You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Phase 0 of #955: the scripts that will serve usnewsmap.com from an on-prem container behind a Cloudflare Tunnel. Nothing runs them until Phase 2, and nothing here changes how Azure serves. Everything is in the new scripts/onprem/serving/, with a README.
What's in it
Units. The Quickwit searcher reads a file:// metastore on local disk, read-only. Two API colors, usnm-api@blue and @green, run the same image as Azure, by digest, under podman. cloudflared runs the usnm-serving tunnel. Caddy sits on loopback in front of the colors. Everything listens on loopback only.
gen-env.sh. Writes the API's and the searcher's settings from the sources Bicep reads. --check-app compares the API's settings with the Azure app and fails on any difference it doesn't list. The search-log and OCR-request URLs stay unset until Phase 3.
update-api.sh and its timer. The timer runs it 2 minutes after the last run ends. It reads the registry's usnewsmap-api:main manifest with the container's Arc identity and asks GitHub nothing while that image serves. A new image deploys once its build provenance attestation verifies (ci.yml on main; CI attests after the Azure roll, in a Phase 0 API PR). The script then starts the idle color and switches Caddy once /readyz passes. An image whose color doesn't get ready is rejected, and the old color keeps serving.
apply-tunnel-ingress.sh and tunnel-ingress.json. The tunnel's ingress, kept in the repository, applied with the Cloudflare API token read from Key Vault.
install.sh. Puts it all in place inside the container. It also installs journald retention (30 days), a response cache directory for each color with a 14-day cleanup, and run-cloudflared.sh, which cuts query strings from cloudflared's log lines. Caddy's error log drops the request, so neither logs a search or a visitor's address.
The container, its VLAN and its firewall are in home-infra (tgoodyear/home-infra#59), so no host names or addresses land here.
Testing
test_serving.py: 30 tests that run every script against stub tools, including real deploy, rollback, attestation-wait and install runs in a temporary root. CI's run-unit-tests.py picks it up. They pass on macOS (the Linux-only ones skip) and on Debian 12.
shellcheck is clean at every severity. systemd-analyze verify passes on the units in Debian 12.
Caddy 2.6.2 (Debian 12's version) validates the Caddyfile, a live color switch with caddy reload works, and an upstream failure logs no query string or address. The cloudflared wrapper was run in Debian 12 against sample error lines.
gen-env.sh --dry-run --check-app against prod (read-only) matches ca-usnm-prod apart from the listed differences.
Open items for Phase 2
How a new index version reaches the running container. The searcher reads one release's file:// metastore, so today a new version means a new config and a searcher restart.
api.env sets USNM_TRUST_CF_CONNECTING_IP=true and USNM_READY_REQUIRES_PUBLISHED_INDEXES=true, which come with the API image's attestation from the Phase 0 API PRs. An image from before those PRs ignores both settings and keys rate limits on the loopback peer (USNM_TRUSTED_PROXY_HOPS=0). Until the attestation exists, update-api.sh deploys nothing.
Temporary file names in usnm-store that are unique across processes, so both colors could share one cache directory.
Searcher upgrades, and monitoring for the container.
Merge only after #283 says the run g publish is done.
Units, Caddy and cloudflared config, env-file generation and the update
timer for serving from an on-prem container behind a Cloudflare Tunnel,
in scripts/onprem/serving/. Written and tested only; nothing runs them
until Phase 2, and Azure serving is unchanged.
- usnm-searcher.service: the pinned Quickwit binary on a read-only
file:// metastore, loopback :7280.
- usnm-api@.service: blue (:8081) and green (:8082) colors of the API
image by digest under podman; Caddy on :8080 forwards to one of them.
- cloudflared-usnm-serving.service and tunnel-ingress.json, applied by
apply-tunnel-ingress.sh with the Key Vault-held Cloudflare token.
- gen-env.sh: API and searcher settings from the sources Bicep reads,
with --check-app against the Azure app; search log and OCR requests
unset until Phase 3.
- update-api.sh and its timer: follow CI's prod deployment (public API,
conditional requests), verify the image's build attestation, switch
colors once /readyz passes, reject a commit that doesn't get ready.
- install.sh, journald 30-day retention, 14-day response cache cleanup,
and test_serving.py (stub tools; deploy and install runs on Linux).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndexes
The settings from the Phase 0 API PRs, per their notes: the API takes the
visitor's address from CF-Connecting-IP when the peer is loopback, and
/readyz fails while the local searcher lacks an index current.json names.
Both are on-prem only, so --check-app expects them there alone. An image
from before those PRs ignores them and falls back to the peer address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the Phase 0 scripts for moving serving on-prem behind a Cloudflare Tunnel under #955. The new tooling prepares a local Quickwit searcher and blue/green API deployment without changing current Azure serving.
Changes:
Adds service units, loopback proxying, tunnel ingress, and retention settings.
Adds configuration generation, installation, and attestation-verified API updates.
Documents setup and rollback procedures and adds stub-based tests.
…ut of logs
Review fixes for #1005:
- update-api.sh polls the registry's usnewsmap-api:main with the Arc
identity instead of GitHub's deployments API, whose anonymous
requests count against the 60-an-hour limit even when they come back
304. A new manifest deploys once its attestation (CI's last publish
step, after the Azure roll) verifies; the commit comes from the
attestation and its tag must match. The attestation is asked for each
run for 30 minutes, then every 10, and the manifest is rejected after
3 hours. Rejections are by digest.
- --commit creates the hold file also when that commit serves already.
- Caddy's default log drops the request field, which its error log
fills with the URI and headers when a color can't be reached.
- run-cloudflared.sh cuts query strings from cloudflared's lines: its
error log carries a failed request's URL.
- Each color gets its own cache directory: LocalStore's temporary names
use the process and thread id, and both colors are process 1.
- gen-env.sh resolves a relative --out against the caller's directory.
- Tests: no instrumentation-key-shaped fixture, no URL substring check,
directories skipped in the file scan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ry run or a registry error
Review 2 fixes for #1005:
- --ready-secs defaults to 800: the API's 420 s to load a version plus
its 360 s readiness cap (containerapp.bicep), so a slow but healthy
start isn't rejected. A test ties the default to the Bicep values.
- reject() writes nothing on --dry-run; it reports what it would reject.
- A failed lookup of the commit's tag is tried again next run; only the
registry answering "no such tag" or a different digest rejects.
- The timer, service and README give the deploy time that follows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Persisting transient attestation failures as permanent rejections
scripts/onprem/serving/update-api.sh:254
gh attestation verify can fail for operational reasons, not just an invalid attestation. Even with --bundle, it can retrieve Sigstore trust material over the network unless a custom trusted root is supplied (see the CLI manual). A transient retrieval failure therefore reaches reject() and permanently excludes a valid image from automatic deployment. Preserve the verifier diagnostic and make trust-material retrieval failures retryable, while still requiring successful verification before deployment. Only persist a rejection when invalid attestation data or a policy mismatch is established.
…ack can't be confirmed
Review 3 fixes for #1005:
- With no serving color, a new color that doesn't get ready isn't
rejected: nothing shows the host is well, so it's tried again.
- After a failed switch, the new color stops only once Caddy is shown
to route back to the old one. If that can't be confirmed, both colors
stay up and the hold file stops the timer until someone checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Phase 0 of #955: the scripts that will serve usnewsmap.com from an on-prem container behind a Cloudflare Tunnel. Nothing runs them until Phase 2, and nothing here changes how Azure serves. Everything is in the new
scripts/onprem/serving/, with a README.What's in it
file://metastore on local disk, read-only. Two API colors,usnm-api@blueand@green, run the same image as Azure, by digest, under podman. cloudflared runs theusnm-servingtunnel. Caddy sits on loopback in front of the colors. Everything listens on loopback only.gen-env.sh. Writes the API's and the searcher's settings from the sources Bicep reads.--check-appcompares the API's settings with the Azure app and fails on any difference it doesn't list. The search-log and OCR-request URLs stay unset until Phase 3.update-api.shand its timer. The timer runs it 2 minutes after the last run ends. It reads the registry'susnewsmap-api:mainmanifest with the container's Arc identity and asks GitHub nothing while that image serves. A new image deploys once its build provenance attestation verifies (ci.ymlon main; CI attests after the Azure roll, in a Phase 0 API PR). The script then starts the idle color and switches Caddy once/readyzpasses. An image whose color doesn't get ready is rejected, and the old color keeps serving.apply-tunnel-ingress.shandtunnel-ingress.json. The tunnel's ingress, kept in the repository, applied with the Cloudflare API token read from Key Vault.install.sh. Puts it all in place inside the container. It also installs journald retention (30 days), a response cache directory for each color with a 14-day cleanup, andrun-cloudflared.sh, which cuts query strings from cloudflared's log lines. Caddy's error log drops the request, so neither logs a search or a visitor's address.The container, its VLAN and its firewall are in home-infra (tgoodyear/home-infra#59), so no host names or addresses land here.
Testing
test_serving.py: 30 tests that run every script against stub tools, including real deploy, rollback, attestation-wait and install runs in a temporary root. CI'srun-unit-tests.pypicks it up. They pass on macOS (the Linux-only ones skip) and on Debian 12.systemd-analyze verifypasses on the units in Debian 12.caddy reloadworks, and an upstream failure logs no query string or address. The cloudflared wrapper was run in Debian 12 against sample error lines.gen-env.sh --dry-run --check-appagainst prod (read-only) matchesca-usnm-prodapart from the listed differences.Open items for Phase 2
file://metastore, so today a new version means a new config and a searcher restart.api.envsetsUSNM_TRUST_CF_CONNECTING_IP=trueandUSNM_READY_REQUIRES_PUBLISHED_INDEXES=true, which come with the API image's attestation from the Phase 0 API PRs. An image from before those PRs ignores both settings and keys rate limits on the loopback peer (USNM_TRUSTED_PROXY_HOPS=0). Until the attestation exists,update-api.shdeploys nothing.usnm-storethat are unique across processes, so both colors could share one cache directory.Merge only after #283 says the run g publish is done.
🤖 Generated with Claude Code