[tgoodyear]

This adds a restore mode to usnm-ingest publish-staged (--restore) and restore and ready steps to scripts/onprem/release.sh, so the disaster recovery runbook can copy the version current.json already names back to Azure and check the app before the Worker is pointed at it. Part of #1061; the issue stays open for the rehearsal, which is an operator exercise this PR documents and does not run.

As built, publish-staged refuses unless the lake's current.json names the release's previous_version. After Phase 4 it names the release's own version, so the runbook's release.sh --step publish stops before copying anything. With --restore the copy is refused unless current.json names this release's version and the same indexes as the release's pending pointer; if it names the rollback, the error says to publish without --restore, and any other version is refused. Past that guard it is the same copy with the same checks, skips, retries and create-only writes. It writes neither the pending pointer nor current.json, and --dry-run writes nothing to Azure. Without --restore nothing changes: the checks, the writes, the log lines and the serialized Published result are as before (the result gains no field, and a test pins its keys).

release.sh --step restore is publish plus the flag, run in the directory that built the version; it stops first if that directory has no reference/index-storage-pending.json. release.sh --run-dir DIR --step ready --app-url URL --index-version V polls /readyz and /v1/meta with bounded tries (USNM_READY_TRIES, USNM_READY_WAIT_SECS) and exits 1 if the app never serves V. It needs only curl and jq, and the URL comes from the flag, so no hostname is committed. The optional arguments of the publish call are now expanded so an empty list is safe on bash before 4.4; the arguments usnm-ingest receives are the same.

While Azure still serves, before Phase 4, the apps and the index account hold the serving version, so a restore copies nothing. It still runs every check against the run directory, and a test against local copies of the lake and the account shows that restoring what they already hold changes nothing. A real copy into an empty account is a separate exercise, on dev or at Phase 4, with the maintainer's approval for anything provisioned. The docs state the limits: the indexes must all sit under one --local-index-root, which will matter once releases have a delta mode; Phase 5's restore point must cover reference too; and the restore needs the release's run directory (state.json, reference/index-storage-pending.json, qw/), which must be kept on tank. They also correct the runbook to provision the index account again, since Phase 4 deletes it.

ADR-0027 was accepted on 2026-10-24, so these four edits, all in their own commit, are for the maintainer's review. ADR-0021's "Amended by" line and ADR-0027 decision 13 said publish-staged retires after Phase 4. ADR-0021's line now says the index account, its endpoint and the Azure serving apps are deleted at Phase 4 and provisioned again in a disaster recovery, where publish-staged --restore copies the index back. Decision 13 says the same and adds that publish-staged and its restore mode stay in the code while the runbook restores serving to Azure. ADR-0027's Phase 4 rollback row names release.sh --step restore instead of release.sh publish, which could not restore. Decision 10's Phase 4 gate list gains "a rehearsed restore of the serving version to Azure (#1061)". Expect a small textual conflict with the open #1042 in release.sh, at the usage sed range line (the two ranges combine to 2,56); the header comments merge cleanly. Whichever PR merges second resolves it.

Checks run locally: cargo fmt --all --check; cargo clippy -p usnm-ingest -p usnm-boinc -p usnm-ocr-worker -p usnm-ocr-dash -p usnm-ocr-adapter --all-targets --locked -- -D warnings; cargo test -p usnm-ingest --locked; bash -n and shellcheck -x -S warning on scripts/onprem/release.sh; release.sh --help; python3 -m unittest scripts/onprem/test_release.py (11 tests, on macOS bash 3.2); prettier 3.9.10 --check on the four changed Markdown files; scripts/ci/no-private-addresses.sh, no-shared-keys.sh and no-positional-args.py. A mutation check that ignored s.restore in the guard and always wrote the pending pointer failed the five new restore tests, as intended, and the source was restored afterwards. After merging origin/main into the branch I reran cargo fmt --all --check; cargo clippy -p usnm-ingest -p usnm-boinc -p usnm-ocr-worker --all-targets --locked -- -D warnings; cargo test -p usnm-ingest --locked; shellcheck on release.sh; test_release.py; no-private-addresses.sh, no-shared-keys.sh, no-positional-args.py and check-names.py --base origin/main, and all passed (prettier and the clippy runs for usnm-ocr-dash and usnm-ocr-adapter were not repeated). scripts/ci/run-unit-tests.py ran 1259 tests in 65 files; four files fail (scripts/boinc/test_acceptance.py, scripts/boinc/test_amd_device_map.py, scripts/boinc/test_boinc_scripts.py and scripts/ci/test_changes.py), and the same four fail the same way on a clean origin/main (2e9c646). I did not run cargo test --workspace, because only usnm-ingest uses publish_staged, onprem::Staged and Published, and I did not run bicep lint or the web npm checks, because no .bicep or web/ file changed.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The readiness check can report success despite a failed metadata transfer, allowing an incorrect cutover decision.

1 open issue

• Require metadata pipeline success before declaring Worker readiness · New

What changed in this PR

Adds a disaster-recovery path to restore the currently serving on-prem version to Azure without changing the lake’s pointers, addressing part of #1061.

Changes:

• Adds publish-staged --restore with serving-version and index checks while preserving normal publish behavior.
• Adds release-script restore and bounded readiness checks, with Rust and Python regression coverage.
• Updates recovery runbooks and architecture decisions; the operator rehearsal remains outstanding.

File	Description
scripts/​onprem/​test_release.py	Tests publish arguments, restore prerequisites, and readiness polling.
scripts/​onprem/​release.sh	Adds restore and readiness steps.
docs/​operations/​rebuild-and-index-storage.md	Documents restore commands, prerequisites, limits, and rehearsal.
docs/​design/​adr/​0027-onprem-serving-behind-cloudflare.md	Retains restore tooling and adds a recovery rehearsal gate.
docs/​design/​adr/​0021-separate-index-storage.md	Clarifies index-account reprovisioning for recovery.
docs/​design/​09-operations-security-cost.md	Updates recovery sequencing and readiness verification.
crates/​usnm-ingest/​src/​onprem.rs	Implements restore guards, pointer preservation, and regression tests.
crates/​usnm-ingest/​src/​cli/​onprem.rs	Exposes and tests the restore flag.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The production recovery procedure and accepted architecture decision amendments need maintainer sign-off, with the empty-account Azure restore rehearsal still outstanding.

No open issues 🎉

1 resolved since last review

• Require metadata pipeline success before declaring Worker readiness

_{Review effort: Balanced}