Test harness isolation: the fleet installs from itself and reaches nothing outside — and four production fixes it turned up

[MorningLightMountain713]

Overview

The integration fleet reached the public internet on every run — roughly 1.2 million packets left the box per gate, to Canonical, Docker, NodeSource, Syncthing and GitHub. That made the gate depend on five third parties being up and honest, and it meant a node under test was not quite the node we ship. This branch closes that: the fleet now runs with its network sealed, installs its packages from a signed repository the image carries, and records anything that tries to leave.

Most of this is test harness. But it is not only test harness, and the production changes should not be reviewed as if they were — four of the seven commits are production fixes, three of which were found by building the harness rather than by looking for them, and two of the harness commits carry a production change alongside. The split is spelled out below before anything else, so nothing has to be inferred from a commit title.

Gated twice: 83/83 at 5cb0383b7 (60m06s) and 84/84 at 5ec88dfbe (60m35s), the second with the fleet network sealed and tcpdump reporting 0 packets received by filter.

What is production, and what is harness

commit		what it changes
0ac9563d3 endpoints become configuration	production	config/default.js, cloudUIUpdateService, geolocationService, fluxService, imageManager, appSpecHelpers, systemService, scripts/update-cloudui.sh
85fa7b480 a boot sweep is over when it says so	harness + observability	harness wait; fileOperationRecovery gains a fileops:recovered publication (+14 lines)
0c03b6ef9 an apt failure resumes into the same failed task, forever	production	fifoQueue, systemService — no harness files at all
5d5425ddc a legacy node installs its packages from the fleet	harness + production	the harness apt repository, and monitorSystem's package checks become awaited rather than fire-and-forget setImmediate, publishing system:packages-checked
5cb0383b7 one package apt cannot find stops a node installing the others	production	fifoQueue, systemService — no harness files at all
aaf01e5c8 a node installs what a package needs, not what it suggests	production	systemService — no harness files at all
5ec88dfbe the fleet has no way out	harness only	the sinkhole, suite 95

By line count:

	files	changed
production code (ZelBack/, scripts/)	10	+181 / −45
production unit tests (tests/unit/)	6	+343 / −25
harness (test-infra/)	14	+694 / −77

The production surface is four files carrying real behaviour change — fifoQueue.js, systemService.js, cloudUIUpdateService.js and scripts/update-cloudui.sh — plus five files where an endpoint literal became a config lookup, and one additive event publication. fluxEventBus is test-only, so both publications are observability and neither is on a production path.

The apt queue defects

None of these were planned work. All three are invisible while apt succeeds, which is why nothing had found them — and closing the network is precisely what makes apt failure the normal case, so they were fixed one step before they would have started biting.

A failed task resumes into itself, forever

updateAptCache passes retainErrors: false so that a failed apt-get update is dropped rather than left at the head of the queue. queueAptGetCommand built its worker options from retries alone, so retainErrors and retryDelay never arrived and the queue kept the task. monitorAptCache then resumes the queue on every failure, and it is async — it awaits several runCommands first, so its resume() lands after work() has exited and working is back to false. With retries: 0 there is no delay between attempts.

Measured against the real modules: 55,812 worker attempts in two seconds, one core flat out, for as long as the mirror stays unreachable.

The fix is to forward every worker option the caller set.

The queue died the moment that shortcut started working (review round three)

Fixing the payload above switched on a trapdoor the payload bug had been holding
shut. runWorker halted the queue after emitting failed, and an emit is a
synchronous yield: the listener runs right there. monitorAptCache resumes for a
failed update with no await ahead of it, so its resume() landed inside the
emit and the next line overwrote it. The loop broke out with work still queued,
working went false, and nothing restarted it - push() only calls work()
when the queue is not already working. Every apt task behind the update was
stranded, every one pushed later too, and monitorSystem's allSettled never
settled: system:packages-checked was never published and chrony, syncthing and
netcat never installed until FluxOS restarted.

The fix is not to make listeners behave, because a queue cannot police them.
It is to stop depending on when they run: settle state first, then notify, and a
listener may reply instantly, slowly or not at all. Two consequences follow. A
resumed final failure reaches the retry sleep with no retries left, so a
ladder-over break stops it waiting out the full delay - 60s by default - holding
everything behind it. And resume() no longer reassigns finished when called
re-entrantly: work() returns at once while the loop is running, and storing
that resolved promise told clear() the queue was idle mid-flight, on the
apt-is-broken path systemService reaches from these very failures.

An apt command completing is now a published fact, which is what makes this
testable at all. A legacy boot runs a handful of them, so it is an event rather
than a cadence, and which completed - in what order - is the only direct evidence
the queue carried on past one that failed. Suite 94 gains the third regime a real
node boots into, packages missing and apt-get update failing:
createTestEnv({ aptBadSource: true }) adds a source apt cannot reach alongside
the good one, so the update fails as it does behind a dead mirror while the work
behind it stays doable - swapping the good source out would fail that too and
prove only that a broken node stays broken. Proven both ways on a real fleet:
green 8/8 with the fix, and with the halt moved back after the emit all three
tests of that regime fail.

The recovery routine could not see what had failed

runWorker emitted the whole payload as options, but for the {commandOptions, workerOptions} shape the command sits one level down — so options.command was always undefined, and the routine's opening shortcut, which treats an apt-get update failure as not worth panicking over, had never once fired. Every apt failure instead took the full recovery path: up to thirty minutes waiting on the dpkg lock, then fuser -KILL on it, then dpkg --configure -a, then apt-get install --fix-broken — for a node that simply could not reach a mirror.

The fix is to emit commandOptions.

One package apt cannot find stops a node installing any of the others

A retained task went back to the front of the queue, so the next resume handed it straight back to the worker ahead of everything already waiting. Observed on a real node with an empty package index: netcat-openbsd could not be located and worked through its retry ladder, and when the ladder ended a resume granted it a fresh one. chrony and syncthing, queued behind it, were never attempted at all — and there was no ceiling, because each resume started the cycle again for the life of the process.

Retained tasks now go to the back, and a task is handed back at most maxRetainCycles (default 3) times before the queue emits abandoned and gives up. systemService logs that, because the caller already took its error when the first ladder ended and nothing else would ever learn the work had stopped.

Two existing tests encoded two of these behaviours

One asserted that the failed task sits at the head of the queue, with a comment saying so, and five asserted the exact apt argument list. Both were updated deliberately — the expectations were the defect, not the guard.

Every endpoint on the boot and app-lifecycle path is configuration

Endpoint literals scattered through cloudUIUpdateService, geolocationService, fluxService, imageManager, appSpecHelpers, systemService and scripts/update-cloudui.sh move into ZelBack/config/default.js. Configuration lives in the directory fluxbench hashes, so redirecting a node stays detectable tampering — which is why this is configuration and deliberately not an environment variable.

The heading is scoped deliberately, because the sweep is not total. Two endpoints outside that path remain literals and are follow-ups rather than oversights: watchdogService's clone URL for fluxnode-watchdog, and the Azure provider's login.microsoftonline.com authority. Neither runs during boot or an app operation, which is why suite 95 stays green with them in place — that suite proves what the fleet reaches while a gate runs.

Two further literals are deliberately staying, and are not endpoints at all. The Google and Azure registry scopes (.../auth/cloud-platform, containerregistry.azure.net/.default) are permission identifiers those providers define, carried inside an auth request and never fetched — a node with a "configured" scope is a node that fails to authenticate. And paymentService's success_url is handed to Stripe so it can redirect the customer's browser after checkout; FluxOS never opens a connection to it.

The fleet serves its own packages

The dependency closure is resolved at image build into a signed apt repository the image carries. Nothing names a package or a version: whatever apt puts in the archive cache is the closure it resolved, against this image's own package state.

One production change rides with this and is worth a reviewer's attention. monitorSystem's four package checks were fired and forgotten through setImmediate, so nothing could tell a check that ran and found its work done from one that never started. They are now awaited, and the result is published as system:packages-checked with the list of what was actually installed — published whether or not anything was, on the same principle as fileops:recovered, because presence alone cannot distinguish those two states on a seeded node. The checks still run through the same one-at-a-time apt queue; what changes is that monitorSystem now knows when they are finished.

Packages are pre-seeded at image build, so a normal fleet boots into the steady state a production node is already in — monitorSystem() runs for real, finds its prerequisites satisfied, and installs nothing. A per-suite aptSeeded: false flag purges the three packages before FluxOS starts, so the install path is covered too. Purge rather than remove, because a removed package leaves its config behind and reports as deinstall ok config-files, which is neither installed nor absent and which no real node is ever in.

Base packages are served over file:// from the image; only syncthing's source goes over HTTP to the stub, because that is the one source FluxOS actually writes — so the keyring fetch, the source write, apt's HTTP transport and signature verification all run as they do on a node. Signing is load-bearing and proven by mutation: swapping the keyring for an impostor's makes apt refuse both distributions with NO_PUBKEY and the packages become unlocatable.

Syncthing now installs from a .deb rather than a release tarball. Its repository carries only the two most recent releases, so a .deb cannot pin an old version the way a GitHub release can, and the version therefore moves on every rebuild — build-apt-repo.sh records it to syncthing.version and the stub reads that file, so nothing downstream restates it. One side effect is worth naming: the package is built [noupgrade] and the tarball is not, so the upgrades.syncthing.net traffic existed because the harness installed the tarball, and switching closed it by construction.

The fleet has no way out, and says so when something tries

The fleet network is Internal: true — no route off it at all. Nodes resolve through the external stub, which relays each query to the embedded Docker resolver at 127.0.0.11 (the one that knows every container alias on the network) and refuses anything it cannot answer within 300ms, recording the name and the node that asked. Relaying rather than answering from a list means aliases need no second copy in the stub and cannot drift from the ones the runner actually creates.

The redirection goes through /etc/resolv.conf deliberately: FluxOS resolves via c-ares, which does not read /etc/hosts, so a hosts entry would have covered nothing.

Suite 95's third test is the reason the other two mean anything. It deliberately reaches for stats.runonflux.io and asserts the resolver recorded it. On the first run that test failed while both hermeticity assertions passed — the sinkhole was wired to nothing, so "the fleet reached nothing outside" was being recorded against a resolver that recorded nothing at all. Any assertion that something did not happen needs a sibling proving the instrument can see it happen.

systemService hygiene

aptRunner now passes --no-install-recommends. Without it, a node asking for four packages installed twenty-one, including shared-mime-info, dbus, glib and python3-gi — which is where ten legacy nodes sitting in uninterruptible sleep running update-mime-database came from.

Release note. This lands in aptRunner, so it applies to every apt invocation on every existing legacy node, not just the new ones — the change here most likely to surprise someone operationally. The four packages involved are safe without recommends, and an unseeded install was exercised at gate scale. Also operational: all three entry points now delete process.env.NODE_CONFIG at startup, so the node-config environment override no longer reaches FluxOS — nothing in the repo used it, but any external tooling that set it loses its overrides.

addSyncthingRepository() moves to where the install is. It previously ran before any version check and short-circuited only if the source file already existed, so a node whose syncthing was perfectly current still fetched a release key and wrote a keyring and an apt source it would never read.

Measurement

Egress per full gate, captured on the host with tcpdump filtered to the harness subnet:

destination	before	after endpoints	after own packages	after default-deny
total off-box packets	~1,200,000	455,348	2,574	0
Canonical	421,196	425,030	0	0
archive/security.ubuntu.com	1,476	1,752	gone	gone
download.docker.com/deb.nodesource.com	266	296	gone	gone
apt.syncthing.net/syncthing.net	148	140	gone	gone
upgrades.syncthing.net	6	6	gone	gone
github.com	64	6	gone	gone
relays.syncthing.net	6	6	6	gone

The final capture's own tally is 0 packets captured, 0 packets received by filter. Before anything was claimed from that zero, a container on the harness subnet curled archive.ubuntu.com through the same filter and produced 30 packets with the hostname visible — an empty measurement has two explanations, and the boring one is a broken instrument.

Testing

• Full unit suite 5,610 passing, 18 pending, 0 failing.
• Integration gate 87/87 at the current head, 59 minutes, with all six images digest-verified against the tree they were built from.
• Two new integration suites. 94-legacy-package-provisioning covers all three regimes a legacy node can boot into — seeded, unseeded, and unseeded with apt-get update failing — where none were covered before. 95-fleet-reaches-nothing-outside covers both node types, plus the instrument check described above.
• Every fix carries a mutation that kills its own test and nothing else. Two of the queue tests proved nothing on their first pass — one used a zero retry delay, so removing the ladder-over break changed nothing observable, and one asserted on a flag that holds either way — and were rewritten until they bit.
• One rebase note: Restore: acquire before destroying, and stop one dead volume from silencing a node #1779's round hardened the syncthing tarball verification in Dockerfile.fluxos, which this branch then deletes by design — syncthing arrives as a package from the image's signed apt repository, whose signed-by verification supersedes it.

Notes for review

• Based on fix/restart-pacing-crash-only (Reconciler: the only actuator for every operator command, and a ladder that paces faults not operators #1780). That is sequencing, not a code dependency.
• The four production commits could have gone out on their own branch. Keeping them here was a deliberate call: three of them were found by this work, and the fourth is what the harness redirection is built on.
• If you want to review the production change on its own, git diff 6f7016ad3..HEAD -- ZelBack scripts is the whole of it — 10 files, +181/−45.

[codecov]

Codecov Report

❌ Patch coverage is 79.31034% with 12 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (fix/restart-pacing-crash-only@77dce7b). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
ZelBack/src/services/systemService.js	85.29%	5 Missing ⚠️
ZelBack/src/services/utils/appSpecHelpers.js	0.00%	4 Missing ⚠️
ZelBack/src/services/fluxService.js	0.00%	2 Missing ⚠️
ZelBack/src/services/appSecurity/imageManager.js	0.00%	1 Missing ⚠️

Additional details and impacted files

@@                       Coverage Diff                        @@
##             fix/restart-pacing-crash-only    #1781   +/-   ##
================================================================
  Coverage                                 ?   65.25%           
================================================================
  Files                                    ?      176           
  Lines                                    ?    33061           
  Branches                                 ?        0           
================================================================
  Hits                                     ?    21574           
  Misses                                   ?    11487           
  Partials                                 ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Cabecinha84]

PR #1781 — review

Verdict: ACK with one small change requested. The production surface is small, well-reasoned, and the three apt-queue
fixes are real bugs with real tests. I found one defect (harness-only impact, but its guard test is green for the
wrong reason) plus a handful of notes. Nothing here justifies blocking.

Note the base is fix/restart-pacing-crash-only (#1780), so this merges after that one.

What I verified myself

• Ran the six touched unit files with NODE_CONFIG_DIR=tests/unit/globalconfig: 189 passing, 0 failing (systemService
  69, fifoQueue 18).
• npx eslint clean on the four changed production files.
• Config keys stats / pricing / mongodb don't collide with existing top-level keys; all five services that now read
  them already require('config'); zero dangling references to the removed geolocation.statsApiBaseUrl.
• Traced every caller of the apt path — monitorSystem (serviceManager.js:164, not awaited) is the only entry point;
  nothing in routes or other services reaches queueAptGetCommand.
• Wrote a throwaway probe test to check the one claim I doubted (below).

Finding 1 — request a change

system:packages-checked is not published when a check throws, and the test that says it is doesn't test that.

systemService.js:735 awaits Promise.all(Object.values(checks)) inside the try. Any rejection skips straight to catch,
so fluxEventBus.publish never runs.

The guard test — "publishes even when a check throws, so a waiter fails rather than hangs" (systemService.test.js) —
does axios.get.rejects(...), but monitorSyncthingPackage's axios call carries its own .catch() that swallows it.
Nothing ever throws, so the test passes without exercising the path it names.

Probe (stubbing serviceHelper.runCommand to reject, so getPackageVersion genuinely throws):

publish called times = 0
AssertionError: expected +0 to equal 1

Impact is confined to the harness — fluxEventBus.publish is a no-op unless testEventStream is set, so production is
unaffected. But waitForPackagesChecked would burn its full 180s (480s in suite 94) instead of failing fast, which is
precisely the failure mode restartFluxosAndAwaitRecovery was written to eliminate. Worth noting fileOperationRecovery
got this right — its early-return path still publishes, and there's a test proving it. The two are inconsistent.

Fix is one line either way: Promise.allSettled, or move the publish into a finally. And make the test throw somewhere
the code doesn't already catch.

Notes — not blockers

--no-install-recommends changes behaviour on every existing legacy node. It lands in aptRunner, so it applies to all
apt invocations, not just the new ones. The justification is solid (21 packages installed for 4 requested; ten nodes
in D-state running update-mime-database), and the four packages involved are all safe without recommends. The 84/84
gate exercised a real unseeded install. Just flag it in release notes — it's the change most likely to surprise
someone operationally.

addWorker now throws instead of no-op'ing. Behaviour change on a public method of a shared util. Only remaining call
site is a test, so it's contained — but it's a hard failure where there used to be a silent one.

systemService now attaches its worker and failed listener at module-load time, unconditionally, including on Arcane
nodes where monitorSystem early-returns. I confirmed nothing reaches the queue there, so it's inert. Still:
require('systemService') now installs an apt worker as a side effect.

The "config is tamper-evident, environment is not" rationale is stated more strongly than the mechanism supports. It
appears in cloudUIUpdateService.js and again in scripts/update-cloudui.sh's header. node-config honours NODE_CONFIG,
NODE_CONFIG_DIR and custom-environment-variables.js — which is exactly how the harness overrides these same keys. So
moving a literal into config makes those eight endpoints redirectable by anyone who controls the process environment,
which a hardcoded string was not. That's a fine trade (env control ≈ game over anyway, and testability is a genuine
win), but I'd soften the comment so nobody later leans on it as a security property.

Nits: scripts/update-cloudui.sh still documents Usage: npm run update:cloudui and bash scripts/update-cloudui.sh —
both now exit 1. And queueAptGetCommand's JSDoc doesn't mention retryDelay/retainErrors now that they're forwarded,
and still says retries defaults to 3 (the queue default is 5).

Possible flake: the sinkhole resolver refuses at 300 ms. Only names Docker's embedded resolver can't answer locally
reach the stub, so the window is small — but a slow relay would be recorded as an escape attempt and returned NXDOMAIN
to the node. Low risk, worth knowing when a suite-95 failure eventually shows up.

What's genuinely good

The fifoQueue work is the strongest part. I re-read runWorker line by line: the cycle counter on props[2] is sound
(push() builds [payload, resolve], the list getter maps x[0], clear() reads task[1] — the third slot is free), the
retain-to-back plus maxRetainCycles ceiling correctly bounds what was an unbounded retain-and-resume loop, and
emitting commandOptions rather than the wrapper is what makes monitorAptCache's opening shortcut fire for the first
time. The retainErrors: false forwarding is the actual fix for the 55k-attempts spin, and it's the kind of bug that
only bites once the network closes.

The two tests that encoded the old behaviour were re-baselined rather than deleted, with the reasoning written down —
that's the right call and it's visible in the diff.

exec → execFile with an argv array is a strict improvement, and the test deliberately stubs config with a
non-production value so it can't pass by coincidence.

[MorningLightMountain713]

Thanks — the requested change and the notes are all addressed, with one I'd push back on. Branch is now f6c6bc399.

Finding 1 — system:packages-checked. Fixed, and the guard test now throws somewhere the code does not already catch, so it exercises the path it names.

--no-install-recommends — release note added to the body, calling out that it lands in aptRunner and so applies to every apt invocation on every existing legacy node.

addWorker throwing — answered, not changed. The throw fires only when a worker is already installed, and the old behaviour was a silent wrong answer: the caller believes it installed a worker, the queue runs a different one, and nothing observable differs. It also has no production call site at all any more — both remaining callers are its own tests. I've left the method in place; an unused method isn't the same as an unwanted capability.

Module-load side effect — fixed (f84f57abe), but by moving the whole queue rather than just the worker. The existing comment was protecting something real: a queue that arrives without its worker silently accepts work it cannot run, so its behaviour depends on call order. Splitting them would have recreated exactly that. The queue and its worker are now built together on first use, so there is no moment when one exists without the other, and importing the module does nothing. There's a test proving the import is inert that fails if construction is made eager again.

Doc nits — fixed (2cc65b851), plus a third error neither of us had spotted: the JSDoc also said timeout defaults to 60 seconds where aptRunner uses 180. retryDelay and retainErrors are documented, and the numbers are named as the queue's defaults since that is whose they are. update-cloudui.sh now documents the two invocations that actually work.

The tamper-evidence rationale — I don't think this one holds, and I'd rather not soften it. The premise is that node-config honours NODE_CONFIG, NODE_CONFIG_DIR and custom-environment-variables.js. This codebase closes all three, in the first eight lines of app.js:

process.env.NODE_CONFIG_DIR = `${__dirname}/ZelBack/config/`;
// ... NODE_CONFIG is the same door: the config package merges whatever
// JSON it holds over every file, after the directory is settled, so leaving
// it open redirects any endpoint without touching a hashed file - the one
// redirect tamper detection cannot see.
delete process.env.NODE_CONFIG;

The directory is pinned, NODE_CONFIG is deleted, and there is no custom-environment-variables.js in the repo. The comment even names the precise attack — a redirect that touches no hashed file — and shuts it.

The harness also doesn't work the way the note assumes, and for the same reason: it can't override through the environment, which is why entrypoint.sh copies config into the pinned directory instead. Its own comment says so — "app.js hardcodes NODE_CONFIG_DIR to ZelBack/config/ (cannot be overridden from env — fluxbenchd hashes that directory for tamper detection)". Softening the comment would make it less accurate and invite someone to reopen a door that was deliberately closed.

Possible flake on the sinkhole resolver — noted, and worth knowing when a suite-95 failure eventually shows up.

---

Two things worth reporting, because they came out of gating this rather than reviewing it.

The gate caught a real regression in this PR. Moving the runner's overrides out of NODE_CONFIG and into a written local.js made the entrypoint build that file by requiring the per-node config, which spreads shared.js at that moment and freezes the result. Discovery autostart was enabled by a sed against shared.js applied afterwards — so it landed on a file nothing read again, and nodes silently never started discovery. Suites 02 and 04 were both red on it. Fixed in 55545fb5e by having it travel in the same override JSON as everything else, which leaves one path into a node's configuration instead of two with an invisible ordering rule between them.

Separately, this PR now carries an image-provenance check (1103448c9, e8be03dac). A stale stub image cost a full 85-suite gate earlier today: a syncthing-stub four commits behind had no /folder-writes-reset route, so three restore suites died in a before each on an HTML 404 that reads exactly like a restore defect. build-images.sh now stamps every image with a digest of its build context and the runners refuse to start when it doesn't match the tree — an unlabelled image fails too, which is the difference between a guard and a suggestion. Every gate log now opens with images verified against the tree.

Verified: 85 of 85 suites green at f6c6bc399, 62 minutes, MAXN=6.

[Cabecinha84]

PR #1781 — review at current head (1db4be7)

Verdict: request one change. One production regression that your earlier review couldn't have seen — it was introduced
by a change that was in the version you reviewed, but it needs a probe to surface. Everything you flagged on
2026-08-10 is genuinely fixed. Nothing else blocks.

What I verified

┌───────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────┐
│ check │ result │
├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ Full unit suite, PR head │ 5206 passing / 5 pending / 23 failing │
├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ Full unit suite, base │ 5190 passing / 5 pending / 23 failing │
│ (fix/restart-pacing-crash-only) │ │
├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ Failure sets │ byte-identical — all 23 are the pre-existing dockerService tests │
│ │ that need the runonflux/website container. Net +16 tests. │
├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ Six touched test files in isolation │ 159 passing, 0 failing │
├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ npx eslint on the 7 changed production │ clean │
│ files │ │
├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ CI │ build (ubuntu-22.04, 20.x, 7.0) passing │
├───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ Dangling geolocation.statsApiBaseUrl refs │ none │
└───────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────┘

Your Finding 1 is properly closed (101af67): the publish moved into a finally, the checks are Promise.allSettled
with per-check named logging, and the guard test now rejects serviceHelper.runCommand — where nothing catches —
instead of axios, and pins { installed: [] } so it can't pass on a stale payload. I re-read it; it now exercises the
path it names.

The fifoQueue work still holds up on a second read. The props[2] cycle counter is safe (list getter maps x[0], clear()
reads task[1]), abandoned is bounded, and emit('failed', { options: commandOptions }) is what finally lets
monitorAptCache's update shortcut fire. The lazy getQueue() (72511b4) is a better answer to your module-load note
than splitting queue from worker would have been — there's no window where one exists without the other, and import is
now genuinely inert.

Finding — request a change

ZelBack/src/services/systemService.js — moving addSyncthingRepository() after the version check strands a class of
legacy node permanently.

addSyncthingRepository() moved from the top of monitorSyncthingPackage to after the hasNewSources branch. But
updateSyncthingRepository() returns false when /etc/apt/sources.list.d/syncthing.list cannot be read
(systemService.js:452 — "Unable to read syncthing sources"), and that false short-circuits the whole check:

if (!hasNewSources) {
const updated = await updateSyncthingRepository();
if (!updated) return false; // ← returns before addSyncthingRepository()
}
await addSyncthingRepository(); // never reached

So on a node with syncthing installed at < 2.0.0 and no apt source file, the repository is never created and syncthing
is never upgraded. It repeats identically every 24h — the node cannot heal, and the only symptom is one log.warn.

I ran the same probe against both branches (syncthing 1.23.4 installed, fs.stat rejecting, cat returning empty, stats
reporting min 2.0.7):

base: wrote syncthing apt source: true (keyring + sources.list.d/syncthing.list written)
PR: wrote syncthing apt source: false (stops at cat /etc/apt/sources.list.d/syncthing.list)

That state is reachable in production: addSyncthingRepository() returns early without writing whenever addGpgKey fails
(key fetch unreachable, keyring dir not writable) or addAptSource fails. Today those nodes retry daily and recover
once the network comes back; after this PR they never reach the retry. Same for a node whose syncthing came from
Ubuntu universe (1.19.2 on 22.04) rather than the FluxOS installer.

Fix is a one-line move — put await addSyncthingRepository(); immediately after if (upToDate) return false;, before the
hasNewSources branch. That keeps the entire stated benefit (a node already current writes nothing) and restores the
old ordering for every node that actually needs to install. It also keeps the new test green, since that test stubs an
up-to-date node. Worth asking for a unit test on the "1.x installed, no source file" path, since nothing currently
covers it.

Notes — not blockers

1. delete process.env.NODE_CONFIG is an undeclared production change. It landed in 8a0b587 after your review, in
   app.js, apiServer.js and homeServer.js. The reasoning is sound and I agree with him over your original note — the door
   really was open. But the PR body's production/harness table doesn't list it, and the release note only mentions
   --no-install-recommends. Nothing in the repo sets NODE_CONFIG (I grepped), so in-tree it's inert — but any external
   tooling or Arcane deployment that used it loses its overrides silently. It belongs next to --no-install-recommends in
   the release note.

2. The gate evidence doesn't correspond to any commit on the branch. Every SHA cited in the body and in his reply —
   5ec88df, 5cb0383, f6c6bc3, 55545fb, 1103448 — is gone. The whole branch was rebased at 17:02 on 2026-08-11,
   after the 15:15 "85 of 85 green at f6c6bc3" comment, and #1780 underneath it moved at the same time. The content is
   almost certainly the same (author dates all predate the comment), but the 85/85 ran against a different base — and
   #1780's replayed commits touch the reconciler burst ceiling and boot-lock admission, which suites 02/04 exercise.
   Worth asking for a re-gate at the current head before merge.

3. fileOperationRecovery now has the inverse of the bug you found. recoverInterruptedFileOperations publishes
   fileops:recovered after sweepEveryMountedVolume() returns — but executor.reapOrphanedContainers() on that function's
   first line is unguarded. If it throws, no publish, and waitForFileOpsRecovered burns its full 120s. That's exactly the
   failure mode 101af67 fixed in systemService with a finally; the two files are inconsistent again, in the other
   direction. Harness-only impact.

4. Sinkhole resolver, minor. refuse() calls upstream.close() and is also registered as upstream.on('error'), so an
   error on an unbound socket can throw out of the handler and take the stub down. Harness-only, low likelihood. Files
   under your existing 300ms-flake note.

5. Stack depth. 1775 → 1778 → 1777 → 1779 → 1780 → 1781 → 1782. Six PRs deep before development. Nothing to act on,
   just merge-order awareness.