test(core): characterize config precedence, reload scope, and partial activation

[jrschumacher]

Release classification

Parent: #3996 — Roadmap to OpenTDF v1.

Proposed class: v1-assurance. Score: 6/10 (correctness/reliability 2/3; compatibility 2/3; operability 2/2; explicit v1 dependency 0/2). No explicit release dependency is assumed. Scores describe release necessity, not implementation value; confirmed security/correctness/compatibility failures override the score.

Roadmap reference: ROAD-S05.

Motivation

The platform exposes composable configuration loaders and update hooks, but the effective reload contract is easier to overstate than the code supports. Loader order is precedence order: the first loader assigning a key wins. Reload iterates until the count of assigned keys stabilizes, unmarshals directly into the live Config, and validates afterward. Watch callbacks reload the whole configuration and then invoke hooks sequentially; OnChange returns at the first hook error. This means an invalid reload may already have mutated the in-memory config object, and a later hook failure may occur after earlier hooks have applied changes. There is no desired/applied ledger or transactional rollback in this layer.

Source: WatchWithNamespaces, OnChange, and Reload, the public Loader.Watch signature, and server hook/watch wiring.

Scoped deliverable

Add source-backed documentation and deterministic tests for loader precedence, iterative reload, namespace information passed to watchers, whole-config reload versus namespace-scoped service hooks, validation failure, sequential hook failure, and overlapping notifications. Publish a representative startup-only versus reloadable settings table. Tests should describe current behavior—even undesirable behavior—without blessing it as the permanent contract.

Non-goals: no admin API, typed configuration governance, desired/applied state store, transactional hook protocol, or behavior change. Do not silently “fix” partial activation in this characterization issue.

Acceptance tests

• Earlier loaders win the same key; later loaders can supply missing keys.
• Dependency iteration and its count-based convergence rule are captured without a hanging test.
• A watcher receives registered namespace/service information and triggers whole-config reload before hooks.
• Validation/unmarshal failure records whether the live config was mutated and confirms hooks do not run after reload returns an error.
• When hook N fails, tests record that hooks before N ran and hooks after N did not.
• Concurrent/overlapping notifications demonstrate the reload mutex boundary and separately document hook serialization (or lack of it).
• Docs distinguish startup options from keys with active service hooks.

Rollout and impact

Actual controls: reloads are serialized by reloadMux; loader Watch implementations decide notification behavior; hooks stop on first error. No rollback ledger or partial-activation switch exists.
Proposed controls: tests/docs only, with no runtime change. A later governance issue can propose last-known-good or staged activation based on this evidence.

Downstream changes: none for this issue, but public custom loaders make compatibility highly coupled. Complexity medium for reliable concurrency tests. Dependencies: none.

v1 rationale

If live reload is supported in v1, operators need an honest failure/activation contract. If owners declare reload best-effort or unsupported, this can be assurance rather than a gate—but the boundary should still be explicit.

Delivery flags

Required downstream adaptation	Existing feature gate	High complexity	High coupling
NO incremental change; baseline loader upgrades remain separate	N/A: characterization only	CONDITIONAL: deterministic concurrent-hook tests	YES: public loaders and service activation