feat(dos): a residential node holds, evacuates, and only then goes to DOS - #1784
Conversation
Cabecinha84
changed the title
MorningLightMountain713
changed the title
f672954 to
ad0faf3
Compare
|
Verdict: don't ACK yet — two things to fix, the rest are notes The design is unusually careful (three-state everywhere, "absence of evidence is not evidence", the
geolocationService.js:387 decides publishedHosting inline, inside setNodeGeolocation, and stores the resulting
So publishedClassification() returns {consulted: false} in the boot pass, publishedHosting is false, and rows 3 and 5 The pass then only re-runs every 3 days (or on an IP change), and every restart resets it. On development the same This is exactly the mistake the PR argues against one function later: getNetworkClassification() refuses to freeze a
appEvacuationSafety.js:201 refuses with ELECTED_PRIMARY and "stand down first" — and nothing implements a stand-down. It compounds with the ordering: queueDelayMs sorts ascending with compareInstanceSeniority, so position 0 (shortest The direction is safe (nothing is deleted), but the node stays in HOLD+EVACUATE forever, never reaches DOS, and logs a Description/claim discrepancies — cheap to fix, but the PR body is the safety argument
Behavioural changes I'd want explicitly signed off (not defects)
What I verified myself
Bottom line: ask for #1 and #2, plus the four prose fixes. #1 is the one I'd insist on — it's a regression that lands |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## fix/same-ip-port-collision #1784 +/- ##
==============================================================
+ Coverage 66.94% 67.77% +0.83%
==============================================================
Files 185 188 +3
Lines 34448 35087 +639
==============================================================
+ Hits 23061 23780 +719
+ Misses 11387 11307 -80 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thanks — this was a careful read, and two of the twelve changed the design rather than the code. Everything below was checked against the source rather than taken or dismissed; where you were right about a mechanism but wrong about its size, I have said so and given the measurement. Tip is 1. Static IP frozen before the table exists — right, and it ended the ruleVerified true in every link. The population is not what you named. Matching every hyperscaler ASN against the fleet gives one slot, an Oracle box. The six "Google" hits are Google Fiber — a consumer ISP. That made the fix worth questioning rather than writing, and the answer went further than deriving-on-read. We measured what "static IP" currently means, by SSH, on a random sample of 500 ArcaneOS slots (442 reachable), reproducing
Of the 228 slots advertising So the rule changed rather than its timing. A node is static only if it holds a public address on an interface — no table, no vendor flag, no operator name confers it. The reasoning is in the body, but briefly: Cost, measured: ~1,550 Arcane slots stop claiming static (95% CI 1,321–1,779), the pool goes 2,800 → ~1,255, and nobody gains — all 102 public-address slots already claim it. 44 apps require The genuine 1:1-NAT case you were pointing at is knowingly refused: ~49 slots fleet-wide, none of which claims static today. Census, method and bounds: 2. An evacuating node could never hand back a
|
| hosts qualifying as data centre | |
|---|---|
development |
1,570 |
| this branch | 1,673 |
105 hosts gain eligibility — ip-api never said "hosting" but the operator sells hosting (38), the reverse-DNS name says vps/cloud/server (37), or a combination. 2 lose it: 212.83.170.245 (Scaleway) and 213.44.137.57 (Bouygues), both hosting addresses whose reverse DNS carries access-network vocabulary, so the classifier reads CONFLICTED rather than guessing. That is the "2 of 1,569" you were reaching for, and it is the whole of the downside.
On the stats fallback you are describing a pre-existing state, not a regression. /fluxlocation returns ten fields — ip, continent, continentCode, country, countryCode, region, regionName, lat, lon, org — and has never carried dataCenter or static. The old path read statsRes.data.data.dataCenter, a field that does not exist, so it was always undefined and always falsy. A node on that fallback has never been eligible for data-centre apps. What this branch changes is that the reason is now explicit instead of accidental. We found it because a commit here fixed the harness stub, which had been synthesising both fields and making the fallback look richer in tests than it is on any real node.
The measurement did surface one genuine defect, which is fixed in 78d5d02b7. proxy was reaching DATACENTER on its own: 8 of the 2,432 hosts had the proxy flag and nothing else — no hosting flag, no hosting operator, nothing in the PTR. That is a VPN artefact, and a home machine behind a VPN exit carries exactly that signature, which is what an owner buying datacenter: true is paying to avoid. It is also the same category error this branch removes from static IP one function away. A DATACENTER verdict now rests on positive hosting evidence; proxy still contradicts a residential reading, so the veto is untouched, it just cannot confer the opposite. New total 1,673; the 27 hosts carrying proxy beside a hosting operator keep it through the operator.
10, 11, 12. Correct
10 — SURPLUS and EVACUATION share one gate, so this is fleet-wide, and a stateful surplus copy can persist while syncthing cannot prove a peer holds the folder. That is the trade against deleting a last remaining copy, and the 12-refusal warning is indeed the only signal. Stated in the body.
11 — Correct, and now stated as a real outage window rather than only as an intended one, including that it is longer where placement is constrained by a geo pin, hardware or enterprise ownership.
12 — Correct, and inert: setPlacementHold and clearPlacementHold have exactly one caller each, so there is no other owner to trample. Left as is.
Your own verification
Re-checked and sound. locationTtlS 7500 is the old 125*60*1000, installingTtlS 900, sigtermExpiryS 420, explorerPollIntervalMs 5000 is the hardcoded value, and installErrorTtlS 3600→86400 matches the 24h the code actually ran while the key was unread. The 23 dockerService failures you saw are the missing docker daemon; the suite is green with it running.
Cabecinha84
left a comment
Cabecinha84
left a comment
There was a problem hiding this comment.
Three things I'd want explicitly signed off — not defects, scope
- An unsigned third-party artifact now drives a destructive path
config.policy.baseUrl is https://raw.githubusercontent.com/RunOnFlux/fluxos-network-policy/main. There is no
signature, no pin, no checksum on the artifact — trust is TLS plus GitHub plus write access to that repo. That was
already true on development, but there a bad orgClasses row costs placement accuracy. Here the same unsigned bytes
drive HOLD → EVACUATE → DOS ≥ 100, and nodeStatusMonitor.js:66 turns isNodeDos() into removeAllAppsLocally(...,
force=true), which is sudo rm -rf on appvolumes.
The design bounds this well, and I want to say so precisely rather than alarm you:
- enforcement needs residential && !arcaneOs, and isArcaneOs() is read from fluxbenchd, so a fully malicious table
reaches at most the 541 non-ArcaneOS slots with readable bench; null bench decides nothing; - DOS is only applied on installed.length === 0, so the destructive step finds nothing;
- the node veto can only ever remove a node from enforcement, never impose one;
- it self-heals — fix the row, next tick clears the hold, the sticky DOS and the settle marker.
But the trust boundary on that repo just moved from "affects placement quality" to "can take non-ArcaneOS nodes out of
service fleet-wide". That's worth a line in the PR body and a conscious sign-off, not a silent inheritance. The PR
body sells the same property as the kill switch ("blank orgClasses and it ends within a day"), which is exactly right
— and the same lever pointed the other way is the risk.
- This is the first removal path in FluxOS that deletes the last copy of an app
appEvacuationSafety.js returns { safe: true, code: 'STATELESS' } before the otherHosts < 1 check, so a stateless app
with instances: 1 and one location — this node — passes the gate, and removeAppLocally(name, null, false, true, true)
deletes the container and the volume (appUninstaller.js:122,226).
SURPLUS has never done this: it requires runningAppList.length > minInstances. The geolocation-redeploy path does, and
the module header cites it as one of the two paths that "have already destroyed customer volumes".
The PR body's justification is "None of the 49 single-instance apps hold synced data — removing one loses a container,
not data." That equates "no syncthing sync mode in containerData" with "holds nothing worth keeping". It's a
defensible product assumption but it is an assumption, and it's the largest residual customer-facing risk in the
change. I'd want it stated in those terms — 49 apps, volume deleted, rebuilt empty from spec elsewhere — rather than
as "a container, not data".
- getMasterIpFromFdm's fdmOk fix changes g: failover fleet-wide
return { ip: null, fdmOk: true } → return { ip: null, fdmOk: answered }. It's the correct fix — if (!fdmOk) continue
in masterSlaveApps was dead code — but making it live means that during a total FDM outage no node elects a primary
for any g: app, where previously a node could take over once checkPeersRunning('all') cleared. A g: app whose primary
dies during an FDM outage now stays down.
That's arguably the right trade (it's the split-brain guard), it's what the evacuation gate needs, and the harness
covers it — but it has nothing to do with residential nodes and it isn't in the PR body's "Behaviour this changes for
every node" list. It should be.
Two small fixes I'd ask for
a. getInterfaceIp reads only the first address on the interface.
for (const addr of iface) {
if (addr.family === 'IPv4' && !addr.internal) return addr.address; // first wins
}
A host with a private primary and a public secondary alias on the default-route interface reads false → DYNAMIC →
loses staticip eligibility. The whole static rule now rests on this one function, so it should scan every address on
the interface and return the first routable one rather than the first non-internal one. One line.
While you're there: the 442-slot census reproduced hasPublicIpOnInterface() over SSH on the host. That's the same
namespace FluxOS runs in on ArcaneOS, so the figure is sound there — but it's worth one sentence confirming there's no
supported deployment where FluxOS reads a container's /proc/net/route, because that would read false for every such
node.
b. getNodeGeolocation() restores staticIp from the database but derives staticIpState separately.
({ geolocation: storedGeolocation, staticIp, dataCenter, ... } = dbData);
staticIpState = dbData.staticIpState ?? STATIC_IP_STATE.UNKNOWN;
On upgrade, a record written by the old proxy || hosting rule restores staticIp: true while staticIpState falls back
to UNKNOWN. isStaticIP() and getStaticIpState() then disagree until the first setNodeGeolocation pass completes — and
stay disagreeing on the 5-minute retry loop if both ip-api and the stats fallback fail. Derive staticIp from the
restored state instead of trusting the stored boolean.
(Related trivia: getStaticIpState is exported and has no consumer outside tests.)
Three prose/code mismatches still in the body
"Instances are counted per host, not per registration." Only otherHostCount is. The BELOW_INSTANCE_COUNT serialisation
gate and reasonToGiveUpApp's SURPLUS test both use raw locations.length. Safe direction for stateful apps — ONLY_HOST
catches the co-located pair — but a stateless app whose "two instances" are two registrations on one residential line
passes the count gate as if it had two hosts.
"a node acts only while it observes the app at full strength, so the first departure makes every other evacuating
holder wait." Not the implemented mechanism. mayEvacuateApp stamps and accrues wholeSince regardless of the app's
strength; nothing checks fullness during the wait. What actually serialises is the safety gate refusing with
BELOW_INSTANCE_COUNT at the moment of action, after which forgetAppObservation resets that app's clock to zero. The
property holds and is more conservative than advertised — every other holder re-serves its full base + position × step
after the app returns to strength — but the described mechanism isn't the one in the code.
Minor, harness-only: applyDos() calls setStickyDosMessage then setStickyDosStateValue, and each publishes. The first
dos:changed therefore carries the new message with dosState: 0. A consumer keying on the message rather than the state
sees an inconsistent first event.
7d2d004 to
44384f6
Compare
|
All eight items serviced: the three sign-offs are made and stated in the body, both fixes are in (one in a different direction, argued below), two of the three mismatches were body defects and are corrected, and the third is deliberate. The three sign-offs1. The unsigned artifact. Signed off, consciously, and the body now carries a paragraph stating the trust boundary rather than inheriting it silently ("The trust boundary this inherits"). Two things worth adding to your framing. First, the root this replaces is weaker than the one it installs: today's classification rests on 2. The last copy. The body paragraph is rewritten, and it now claims more than you asked for rather than less. An owner who deploys 3. The two fixesa. b. The three mismatches"Counted per host." A body defect, corrected — and the invariant is stronger than either of us stated. The spawner refuses to place an instance on an IP that already holds one ("already running on this Flux IP"), and an app's fixed ports make a second copy on one machine moot regardless, so a given app's instances land one per host by construction: the two-registrations-on-one-line scenario is not reachable through placement. The safety gate's per-host dedupe covers the one way a pair can converge afterwards — an IP change — and that gate is the one that guards data. The strength count and SURPLUS counting registrations is, under the placement rule, the same number. "Acts only at full strength." A body defect, corrected to the implemented mechanism: a departure leaves the app short, every other holder's gate refuses to act on a short app and resets that app's queue clock, so each further departure waits for the spawner to restore full strength and then serves its full wait again. As you say, the property holds and is more conservative than the old sentence advertised.
Branch state: rebased onto current |
Cabecinha84
left a comment
Cabecinha84
left a comment
There was a problem hiding this comment.
Findings
- The serialisation between two evacuating holders is temporal, not atomic — this is the one I'd hold the ACK on
The body's termination argument is: "a departure leaves the app short, every other holder's gate refuses to act on a
short app and resets that app's queue clock."
That property only holds if no two holders' tickets mature in the same give-up pass. They can:
- The pass is fired from the block handler, so every node in the fleet runs it within seconds of the same block. A
removal in pass P cannot be visible to another node in pass P — fluxappremoved is broadcast after the volume is
already gone, and both nodes read registryManager.appLocation at the top of the same pass. - The queue step separates first maturity (different position → different duration). It does not separate
re-readiness. mayEvacuateApp returns early on the EVACUATION_INTERVAL_MS (6h) check before it touches wholeSince
(residentialNodeDosService.js:365-371), so a node whose ticket matured while it was blocked by its own departure
interval is ready the instant that interval expires. Two nodes whose 6h intervals expire in the same 22-minute pass,
both mature for the same app, both remove.
The harness test knows this. 55-residential-node-evacuation.js:757 says the tickets fix "a different DURATION per
node, not a different deadline" and rests on "In production both holders start observing together." They don't:
wholeSince is stamped on each node's first give-up pass after its own settling window closes, and those windows open
whenever each node first observed the verdict — days apart across the fleet.
Blast radius today is empty, structurally it isn't. For g: apps the election saves it: the primary gets
STAND_DOWN_REQUIRED and cannot remove in that pass, so the two can't go together — and the body says all 9 stateful
2-instance apps are g:. An r:-only synced app with 2 instances both on residential nodes has no such guard:
canSafelyRemoveApp filters gComponents to empty, skips the election entirely, and each node is the other's
findSyncedPeer at 100%. Both return SYNCED_ELSEWHERE, both delete. Zero such apps in the measured population, nothing
in the code that keeps it that way.
Cheapest fix (~3 lines): in mayEvacuateApp, drop the app's wholeSince entry when the departure-interval gate blocks,
so the position-based ticket is served afresh once the interval clears. That restores position as the separator after
every interval block, which is what the design intends. A per-node deterministic jitter folded into queueDelayMs would
also work.
- wholeSince never observes wholeness
The comment at residentialNodeDosService.js:88 and the body both say a node may act "only once it has seen that app at
full strength for base + position × step." The clock actually starts the first time mayEvacuateApp is asked about the
app, and nothing checks instance count until the ticket matures — the wholeness test lives in canSafelyRemoveApp,
which is only reached once giveUp is already true. An app short for the entire wait still serves the full wait, then
takes one BELOW_INSTANCE_COUNT refusal and resets.
Outcome-equivalent in most traces, but it is a weaker guarantee than the one stated, and it is the same class of
prose/code gap the previous round already corrected four of. Fix the comment or check strength at stamp time.
- SURPLUS + elected primary refuses forever (minor liveness)
STAND_DOWN_REQUIRED is only acted on when decision.reason === 'EVACUATION' (advancedWorkflows.js:3243). The comment
argues a surplus giver-up is never the primary because surplus picks the junior and the primary is senior — but this
branch's own note elsewhere says position is "a correlation, not a guarantee" (a remembered previous primary keeps the
role, a designated leader seeds outside the order). When it isn't, that node refuses that app every pass
indefinitely, surfacing only as a warning every 12 passes, and the surplus is never trimmed. Not destructive. Either
wire the stand-down there too, or state why the correlation is a guarantee for SURPLUS specifically.
- HOLD and EVACUATE are invisible to the operator
The placement hold and the drain produce logs only. getPlacementHold() has exactly one consumer (appSpawner), and
/flux/info carries dos — so an operator sees nothing until the final DOS stage, by which time the node is already
empty. 83 hosts / 218 slots are going to watch apps leave with no readable reason. info.flux.placementHold (plus the
settle/evacuation state) is a few lines and is the difference between self-service and a support queue.
- The static-IP change is wider than the staticip gate — worth an explicit sign-off
isStaticIP() also drives the spawner deferral (appSpawner.js:667, spawnDeferrals.staticIpMs): a node claiming static
defers apps that don't need static, to stay available for apps that do. ~1,550 slots stop deferring on upgrade day.
That is a fleet-wide placement shift the body doesn't cost — arguably a correction, but it isn't only "44 apps, 74
placements."
Second-order: the 10-day window is a new penalty for genuinely static nodes that change address. On development a
hosting-range node recovered static on the very next pass through proxy || hosting; now it reads UNKNOWN for the full
ten days, and checkAppStaticIpRequirements is a hard gate throughout.
- The attrition path deletes volumes and bypasses the new safety gate — needs a number
The body's "nothing evicts those 74 … they drain by attrition" is confirmed, and the shape is worth naming: in
reinstallOldApplications, removeAppLocally(name, null, true, false) — force, volume included — runs before
checkAppRequirements (advancedWorkflows.js:3461 and :3478). So those removals never see appEvacuationSafety.
How many of the 44 staticip apps carry synced components? If the answer is zero, say so and it's closed. If not,
upgrade day starts a slow, unguarded volume-deletion drip — the exact failure mode the rest of this PR was built to
prevent. The mechanism is pre-existing; the trigger is new.
- The node veto is a free evasion, and the "Operator evasion" section doesn't mention it
getNetworkClassification vetoes a published RESIDENTIAL on any evidenceAgainst, and evidenceAgainst includes ip-api
proxy. A residential operator who routes the node through a commercial VPN picks up proxy (usually a hosting-range
address too) and is permanently CONFLICTED → never enforced. It fails open, so it's not a safety bug — but it is the
cheapest evasion available, it costs the operator nothing, and the body currently discusses only config-hashing and
mongo tampering.
One caveat on verification
Every fleet number in the body — the 541/218/83 blast radius, the 442-slot SSH census, the 1,570 → 1,673 datacentre
count — cites fluxModels/investigations/PR1784_*.md, which is not in this repo or this PR. I could verify the code,
the tests, and the mechanisms; I could not verify a single measurement. If those files aren't reachable to whoever
ACKs, the quantitative half of the safety argument is being taken on trust.
Verdict
Don't ACK yet — but it's close, and the gap is small.
The staging is sound, the three-state discipline is real, and the two fixes from the last round
(peer-check-before-election, and static-IP as an observation) genuinely changed the design for the better rather than
papering over the reports. Nothing I found is in the class of "this deletes data today."
What I'd ask for before ACK:
- #1 — reset wholeSince when the departure-interval gate blocks (or add jitter). Small, and it closes the one place
where the body claims a property the code doesn't have on a path that deletes volumes. - #6 — one number: how many of the 44 staticip apps are stateful.
- #4 — expose the placement hold. Cheap, and this feature is otherwise silent to the people it acts on.
Then #2 and #3 as prose-or-code cleanups, and #5 and #7 as things to state in the body rather than change.
|
All seven checked against the branch. Every one was accurate about the code. Four are fixed; three are answered in the body, which is updated and live. 1 and 2 — the ticket (
|
|
Blockers
Changes I'd ask for A. reasonToGiveUpApp, the index === 0 branch bypasses the evacuation gate (advancedWorkflows.js:3090-3100). The if (!runsWriter) return { giveUp: true, reason: 'SURPLUS', detail }; An evacuating residential node that is the newest copy of a surplus app and runs its g: writer stalls that app until B. The ticket collision is narrowed, not closed. 2eb3253 is right, and I confirmed the accounting sits below the C. getDosStaging() doc is wrong for one case. "An EMPTY node reports HOLD with a DOS" is true for a node empty from D. clearPlacementHold() ignores ownership while setPlacementHold(reason) takes one. One caller each today, so inert — E. Two fleet-wide changes, one rollback lever. The residential staging is disarmed by blanking a table. The static-IP Accepted risks I'd want named in the ACK, not silently inherited
|
|
Serviced. Three code changes, one doc, two harness fixes, and one correction to the body. Two of your items I'm not taking, with reasons. Blockers1 — the branch does not merge. Rebased onto You were right that this needed hand-resolution rather than a merge, and for a sharper reason than the conflict count. The That mattered. FDM replying 200 with a non-success body — our own in-band error convention, or a proxy error page — would have counted as "FDM answered, nobody is primary", and the election proceeds from there to self-promotion on evidence that says nothing. 2 — no gate at this head. Agreed, and outstanding. The full harness gate has not run on the post-rebase head and will, with 55 and 96 called out. I'll report the result here. For the record the position was narrower than stated: the last full gate was 75/77 at 3 — the off switch is in the ON position. Confirmed: Not blanking it, and I don't think the ask holds. The staging's first stage is HOLD — no new apps, nothing deleted, for a full 24h window — so the observation period you want is already in the design, and blanking the table would give a worse one, because a blank table shows nothing at all. The switch also doesn't need pre-throwing to work: if release day goes wrong, blanking then stops enforcement within a day, before anything has passed HOLD. And the table is the output of a network scan, so shipping it deliberately empty misrepresents it and costs two extra publications. Taking the other half of your ask instead, which is the part that's load-bearing: a named person sweeping ChangesA — the B — the ticket collision is narrowed, not closed. Agreed on the finding, and the body now says so rather than reading as closed. Not taking the jitter. The collision doesn't come from the step being too small — it's 40 minutes against a 22-minute pass. It comes from the ticket being measured from each node's own observation start, so two holders whose starts differ by about a step mature in the same pass whatever the step is. Jitter separates such a pair by the difference of two fixed offsets, which can be zero, and moves an already-separated pair together as readily as apart. It randomises which pairs collide rather than reducing how often it happens. The structural fix is to stop answering a fleet-wide question with a private measurement: take the turn from the shared clock the pass already runs on, so a node acts only in the pass its rank selects, and leave the observation window doing the one job it is right for — proving this node's view is trustworthy. That replaces the pacing mechanism and its whole test surface to close something nothing can currently reach, so it isn't in this PR. The body now states the residual, names what keeps it empty today, and says the guarantee therefore lives in the election rather than in the evacuation rule. C — the D — The hold was a single slot. Two owners cannot coexist in one: a second So the slot is now a map of owner to reason, the node is held while any owner holds it, and releasing removes only your own entry. The owner is a value from a frozen enum rather than the reason text, because the reason is a message for an operator and identity by message is identity by typo; an owner the enum doesn't carry is refused at the set. E — two fleet-wide changes, one rollback lever. Agreed, and it goes in the release note in its own right. The framing there will be that this corrects static-IP claims rather than tightening them: 55% of current static claims come from nodes behind a UPnP router on a private address, qualifying through a flag about the range rather than anything the node observed. Nothing evicts existing placements — Accepted risks, named
Found while doing thisTwo harness defects, neither from your review. Suite 55 started the FDM outage four statements above the The full gate is still to run. I'll post the result. |
9952b7d to
5258adb
Compare
Cabecinha84
left a comment
Cabecinha84
left a comment
There was a problem hiding this comment.
What I found myself at this head
- The stand-down treats a failed docker stop as a successful one. advancedWorkflows.js, STAND_DOWN_REQUIRED block:
appReconciler.setControllerDesired(identifier, 'stopped', '...');
await appDockerStop(identifier);
standingDown.set(identifier, 0);
The module-level appDockerStop (line ~2218) does not throw on a stop failure — it catches internally and returns {
stopped, running, unavailable, errors }. So the catch whose comment says "a component this node failed to stop is one
it is still writing to, and marking it would make the node unelectable for a component it is running" can only fire on
an unexpected throw, and the mark is set regardless.
Failure scenario: docker refuses the stop or never becomes able to answer → stopped: false, running: [identifier] →
the node marks itself standing down anyway → masterSlaveApps continues past that identifier for up to
STAND_DOWN_PASSES_BEFORE_GIVING_UP (6 passes, ~2h) while it may still be running the g: writer. That is exactly the
state the comment says it is avoiding.
Bounded, and I want to say so plainly: setControllerDesired('stopped') also enqueues a reconcile that will stop it,
the 6-pass cap self-heals, and the next give-up pass refuses with ELECTION_UNKNOWN rather than removing anything. No
data loss. But the guard is decorative and the log line written for this case can never print. Also untested — every
test stubs dockerService.appDockerStop with resolves(). Three lines: check the verdict, continue on !stopped.
- Stale evidence across an address change (minor, arguably pre-existing). getNetworkClassification() looks up the
published table on networkEvidence.ip, which is refreshed on setNodeGeolocation's 3-day cadence, while the enforcement
tick runs every 6h. A node whose address moved between geolocation passes is classified against its previous
address's table row for up to 3 days. It cannot start a drain inside that window (the 24h observed window and the
ticket both restart), but the HOLD is immediate. isDataCenter()/isStaticIP() carried the same latency before this PR.
Worth one sentence, not a change.
Things I checked and cleared
- standingDown's owner derivation (identifier.slice(indexOf('')+1)) and isElectedPrimaryHere's endsWith(''+appName)
are safe: appValidator restricts component names to [a-zA-Z0-9] — no underscores. - installErrorTtlS 3600 → 86400 is not a behaviour change: appConstants had a hardcoded 24h literal and the config key
was dead. Same for locationTtlS, installingTtlS, sigtermExpiryS — all match the literals they replace. - getInterfaceIp moving the non-routable filter into the loop: sole caller is hasPublicIpOnInterface, and the
now-redundant filter there was removed. Strictly better on an interface carrying a private primary plus a public
secondary. - hasPublicIpOnInterface returning null on error: both consumers are in geolocationService and both handle three
states. - fixExplorer deletion: unreferenced, not exported, no route.
- appQueryService.installedApps / listRunningApps called with no args: both handle undefined req/res.
- residentialNodeDosService's null-handling throughout (arcane === null, residential === null, installed === null,
observedMs === null) all fail toward hold placement, do not drain, which is the right direction.
bc5d861 to
6ea39e2
Compare
6dbe7e4 to
e32f375
Compare
b22117e to
c521dcb
Compare
c521dcb to
227d4e1
Compare
…rovokes Suite 60 unmounted an app volume for real and then wrote to the bare directory, expecting EPERM from the immutable mountpoint - on the stated grounds that "the monitor's 3s repair cycle cannot remount in between". The 3s cycle is not what answers. The mount test does: "Volume not mounted. Continuing. Most likely false positive." and then "Volume mounted." 144ms later. The probe then writes to a live volume and gets exit 0, which is a green assertion about the wrong thing when it loses and a red when it wins the other way. Seen on a sequential run today, green on the two gates before it - which is the signature of a race rather than a defect, and the reason it survived. The probe now runs under `unshare -m`, in a mount table of its own. FluxOS never sees the volume go away, so nothing repairs and there is nothing to be faster than. The immutable attribute is on the filesystem and visible from every namespace, so the write is refused for the reason the test is about. The container stop went with it: the app's bind lives in its own namespace and was never what the probe examined. The self-heal half then needs a real unmount of its own, since the namespace probe broke nothing. That half wants the repair to be quick, so it races nobody. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every app seedSyncthingApp builds took the same default port, and forceNonLeader places one app on a node as its subject and the next on that same node as its peer - so two of them met on one node holding one port and the second install was refused, correctly, with "already used with different application". It only surfaced when the first install finished registering before the second started, so suites carrying several of these passed or failed on the gap between two installs. Suite 37 seeds four and failed this way on a gate having passed on the one before it. Counted rather than hashed from the name: a hash collides occasionally and puts two apps back on one port, and a fault that appears sometimes is worse than the one it replaced. Each suite is its own process, so the count is per suite, and it starts clear of 31111 which suites building their own specs still use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit d185e0e)
flux#1784 adds allocateAppPort() in seed-helper.js: the same mechanism where the builders live, covering every builder rather than this one caller. That is the one to keep. It does not exist on this branch yet, so this cannot simply be deleted - buildSeedableSyncthingApp still defaults to a hardcoded port here, and removing this first reopens the fault it fixes. Recorded at the function so whoever rebases across that merge finds the instruction rather than reconciling two counters, or worse, keeping both: 31111+N and 31200+M stay clear only while a suite takes fewer than 89 allocations, and nothing asserts it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit c232318)
…he builder #1795's funnel assigns a port inside buildSeedableApp, where a specification is built - which is the seam with no holes, and the only one that can work at all, because the spec is signed and hashed the moment it is built and a port added afterwards leaves the app carrying a hash of a spec that no longer exists. So allocateAppPort goes rather than merges, as agreed with that session. It was the same class of fix reached one layer too high: a rule in the builders, which four suites route around by hand-writing their compose. Two allocators counting into one range would have stayed clear only while a suite took fewer than 89 allocations from mine, which held today and was asserted nowhere. Suites 55 and 96 declare no port now. 96 lost `ports: [allocateAppPort()]`; 55 lost the parameter and the `ports: [port]` that used it - removing the parameter alone would have left an identifier with no binding and failed the suite at load, which is what checking rather than assuming caught. The history in seed-helper's comment stays, because the funnel's own comment does not carry it: which suites paid for this. Suite 68 hand-picked 31111..31115 to escape it, 55 lost a run, and buildSeedableIndexRefApp pinned its second component to a literal so suite 43 - which builds two - could never place both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven suites on the bc5d861 gate - 36, 45, 46, 47, 48, 51 and 53 - died on `"after all" hook ... Timeout of 30000ms exceeded` with every one of their tests green. The hook is env.teardown(), and 45, 46 and 53 call nothing else in it. Which of its steps took the thirty seconds was not recorded anywhere. Every step is a docker round trip with no timeout of its own - the exec sweep across every node, N sequential container stops, N volume removes each able to fall back to booting a helper container, and a network remove that retries while endpoints detach - and they printed only on THROW. Being slow is not throwing. The per-container dump cannot cover it either: log-on-failure dumps on a TEST failing, and its after-all backstop needs nothing to have passed. So each step reports as it COMPLETES, and the step whose line is missing is the one that ran out. The volume line counts EPERM helper-container fallbacks separately, because that branch boots a container per volume and is the difference between a two-second phase and a long one. Diagnostic only: nothing here changes what teardown does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The veto test timed out after 60s on residential:decided with source node-veto, and the decision it wanted could not be reached rather than having gone the wrong way: node 5 logged 24 consecutive ticks of "no network verdict to act on yet", every published decision carrying classification:null and source:null. getNetworkClassification returns null at three gates - no evidence, no contradiction signals, or a published table that was never consulted. The test waited for the first and its own assertion on evidenceAgainst covers the second. Nothing waited for the third, and the node's log carries no ipLocationStore line at all, so the store never installed or adopted a baseline and publishedClassification was answering consulted:false throughout. The suite already half-knew this - test 1's comment says the verdict "also needs the published table, which arrives later" and works around it by asserting on evidence alone. Waited for on every node now, through the same waitForLocationTable that 64 and 66 use, because the table is the authority the whole suite runs on and bootToReady has already started the ticker by then. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eintroduced This branch introduced its own allocateAppPort and later deleted it in favour of the one in the builder. Stacking onto #1795 replayed both, and resolving the intermediate conflicts kept the call sites alive so the history stayed internally consistent - but the deletion patch only matched the sites its own pre-image named, so ten calls survived a function that no longer exists. Nothing catches that by reading: `node --check` parses an unbound identifier inside a function without complaint, and the suites that call these builders fail at load with a ReferenceError naming neither a port nor this branch. eslint no-undef over test-infra/ finds all ten in one pass, and is worth running before any gate. seed-helper.js is now byte-identical to #1795's: the entire delta was those ten references, so this branch contributes nothing else to that file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ructed reconciler-suite carried nextSeededPort/portForSeededApp with a docstring saying to delete both once the builder-level allocator landed, because that one covers every builder rather than this single caller. #1795's assignPorts is that allocator and it is now underneath this branch, so this is when. Leaving it was not merely redundant. Its counter starts at 31200 - the same ALLOCATION_BASE the allocator hands out from - so the two produce the same numbers, and a port it supplies arrives at assignPorts as a hand-picked port inside the allocator's reserved range: refused outright with "asks for port 31201, which is inside the range the harness hands out". Every suite seeding a syncthing app goes through here. It survived the stacking rebase because the cherry-pick that removed it was skipped as a duplicate of #1795's original, while this branch's own commit adding it replayed. No caller passes a port, so the signature loses the parameter with nothing left referring to it. seedSyncthingApp is now identical to #1795's; the only thing this branch still changes in this file is placeGAppInOrder waiting for each holder to register. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rned Stacking onto #1795 put this branch on a syncthingService whose getConfig takes no request and answers with the config itself. findSyncedPeer still called it as getConfig({}, null) and checked configResponse.status, so it returned early on every invocation and answered null without ever asking a device. Nothing in the rebase objected - both spellings parse, and the call was not in any conflict. Null is the answer that KEEPS an app, so this fails safe; but it means a suite asserting the give-up path can never see it, and the six unit tests that caught it are the only thing between that and a silent behavioural revert. Its sibling checkIfPeersAreSynced twenty lines up already called getConfig the new way, which is what made the pair inconsistent rather than merely old. The tests in "findSyncedPeer excludes this node" stubbed the same two calls with {status, data} envelopes. One of them asserted that findSyncedPeer returns null - which the envelope produced on its own, from the early return - so it passed with the self-exclusion it exists to prove DELETED. Verified by deleting it: 2 failing before the stubs were corrected, 0 after, 95 passing. The dosStaging test stubbed listRunningApps and appsResources with placeholder strings. The projection those feed now reads them, so result.data came back undefined and the failure named the projection rather than the field. Given the fixtures the neighbouring test already defines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… probe's own call Test 2 spent its whole ten-minute budget on a trim that had already been refused, for the right reason, in the one pass it gets. Two faults, and the second is why the first was invisible. The partition from test 1 was never really healed. healPartition drops the iptables rules and says in its own comment that the caller re-runs discovery so the dead cross-group sockets get re-dialled; 68, 70 and 71 all do that and this suite was the only one that did not. Node .12 logged nothing but timed-out handshakes to .10 from the heal onwards, and .10 logged the same back. The precondition then waited for the wrong fact. It polled the two nodes' PEER LISTS until each could see the other, but the trim reads peerComponentState, which GETs /apps/heldcomponents on the newest copy. Those are different facts: the peer map still carried the entry from before the partition while nothing could actually talk, so the wait passed immediately and the single probe found the newest silent. It now makes that exact call, from the node that will make it, and requires the writer in the answer. The single pass is what makes a merely-usually-true precondition a coin toss here: the give-up pass runs on `blockHeight % (removeFluxAppsPeriod * speedMultiplier) === 0` - 44 blocks - and this suite sets no override, so the 53 blocks driveUntil spends buy one attempt and no second. The rule under test never misbehaved: WRITER_UNCONFIRMED on an unreachable peer is the fail-towards-no-trim this design is built to do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…its budget went The per-step timing this branch added named the step on its first run. Across the seven suites that lost the previous gate to an `after all` timeout with every one of their tests green: suite 36 stops 6806ms of 7167ms suite 45 stops 13865ms of 14427ms suite 46 stops 9436ms of 10998ms suite 47 stops 3607ms of 3814ms suite 48 stops 5641ms of 5950ms suite 51 stops 7966ms of 8327ms suite 53 stops 3710ms of 3894ms Everything else is milliseconds: the immutable sweep 59-558ms, the database 0-1ms, the volumes 35-900ms, the network 72-125ms. The EPERM helper-container fallback fired once across all seven. resetSyncState does not appear at all - the totals are the sum of the steps, so it was never the cost. Sixteen containers stopped one at a time at ~200-870ms each is work that scales with the fleet, under an `after` hook budget that is a flat 30s in 123 of 137 suites. 14.4s against 30s on a box running nothing else is a factor of two, which is why they pass on a quiet box and why they did not on a loaded one. The serial loop bought no politeness either: the docker daemon serialises stops across every concurrent suite regardless. The one ordering that means anything is kept - every node goes down before any infra does, so a node is never left running against a stopped mongo writing errors into the dump that read as product faults. infraContainers is the discriminator and carries all six (mongo, daemon, syncthing, external, fdm, registry); within each group nothing depends on order. NOT YET VALIDATED: the gate running as this lands is on the previous code, and a teardown change touches every suite. It needs its own gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… a constant Suite 55's test 16 drove its whole 198-second budget and never saw STAND_DOWN_REQUIRED. Not a wrong answer - no answer. The node was the elected primary, was enforcing correctly (119 ticks of residential=true arcaneOs=false enforce=true), and evaluated primaryapp twenty times: 10 DEPARTURE_INTERVAL and 10 AWAITING_TURN. It never reached the stand-down check because it never got past the pacing gates that precede it. It was holding SEVEN apps by this point - clockapp, firstout, residentapp, secondout, sharedapp, worldapp and primaryapp - collected from the fifteen tests ahead of it. Evacuation gives up one app per departure interval, and a node inside that interval records nothing against any other app's queue ticket, so each departure restarts the turn of everything still held. It shed six inside the window and ran out one departure short of the seventh. DEPARTURE_WAIT_MS is two cycles. That was right when this test was written and stopped being right as tests were added in front of it, with nothing to say so: the budget is a constant and the cost scales with what the node accumulated. Now derived from the node's own localAppCount at the moment the drive starts, which is the same rule the rest of this suite already follows - every wait that must contain a departure is derived, not typed. Reading it from the node rather than counting by hand here means adding a sixteenth test ahead of this one cannot quietly shorten its budget again. One cycle of headroom so the final verdict has a pass to land in. The other seven DEPARTURE_WAIT_MS sites are unchanged: they run earlier, where the node holds few enough apps for two cycles to cover it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every app this suite seeds asks for five instances on a five-node fleet, so each lands on every node, and nothing removes them between tests. By the end of the first describe node 1 holds seven apps. Evacuation gives up ONE app per departure interval, and a node inside that interval records nothing against any other app's queue ticket, so every departure restarts the turn of everything still held. The stand-down tests were therefore queued behind six unrelated apps at roughly a hundred seconds each. On the 2026-08-31 gate the node shed six inside the budget and ran out one departure short of the seventh: primaryapp drew ten DEPARTURE_INTERVAL and ten AWAITING_TURN verdicts and never reached the stand-down check at all - no answer rather than a wrong one - and tests 17 and 18 never ran. dc94eb4 budgeted for that queue by deriving the wait from the node's app count. That was the wrong fix and this supersedes it: it made one test wait thirteen minutes, and it left the next person to add a test to rediscover the same thing. These tests need one node that is the elected primary of one app and is draining. A second fleet costs one boot and gives them exactly that, and the budget goes back to two cycles because two cycles is now true. The fleet setup becomes bootResidentialFleet() rather than being duplicated, and the four node constants move to module scope with it. The moved tests already seed what they use - primaryapp and lockedapp - and return their nodes to service first, so nothing carried over from the describe above them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… wiring #1781 creates the harness network Internal, so docker gives node containers no default route. FluxOS decides whether a node holds a fixed public address by looking for one - hasPublicIpOnInterface reads /proc/net/route and returns false when there is no default route at all - so on the stacked tree EVERY node reads DYNAMIC. Verified across both gates: pre-stack every node logged "public IP on interface: true", and on this gate all 65 dumped nodes logged false, none true. That took suite 21's two static_ip deferrals with it. The node can no longer be STATIC, so the deferral the suite waits for cannot fire, and it is skipped for `datacenter` instead - six of ten tests failing on that one fact. Neither side was wrong. The isolation is the point of #1781, and a node with no route out is not a static endpoint. What was wrong is that the property was a side effect of the wiring rather than something a suite declares. So it is declared, the way geolocation and dataCenter already are, and provisioned as the REAL thing rather than a flag the product honours: the entrypoint installs a default route via the network's own gateway before FluxOS starts. It restores the fact without restoring connectivity - an internal network's gateway forwards nothing outward - and its absence is how a suite asks for a NAT'd node, which is what every node is today. Adding it for everyone would have been the same mistake pointing the other way: every node STATIC, no way to be DYNAMIC, and the topology still deciding. seedMongo's staticIp is driven by the same declaration. It is not dead - it is what a node answers during boot, before its first lookup completes and setNodeGeolocation recomputes - but it was seeded true unconditionally while the recompute said DYNAMIC, so it looked like a control and was silently overwritten. db-client's seedGeolocation and dropAndReseed are deleted: dropAndReseed had no callers anywhere in test-infra, and it was seedGeolocation's only one. The route failure is loud rather than `|| true`. A swallowed failure here reads exactly like a node that was never asked for a route, and the suite then fails somewhere else on a classification it cannot explain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…acing its data A restore stops the component to swap what is underneath it, and appendRestoreTask refuses when it cannot - "could not be stopped, so its data cannot be replaced safely". The suite issued the restore without ever establishing that the app was settled, and on the 2026-08-31 gate the syncthing leader election named a leader and requested the start 11ms before the restore arrived. The stop met a start already in flight, the refusal came back as a 200 whose body was still "Pausing syncthing folder", and the assertion for /Finalizing/ failed on it. Test one waits for the two syncthing DAEMONS to find each other. That is a different fact from this app's own election having finished, and it is the second one the restore depends on. The race was always there. It could not surface while findSyncedPeer returned null without asking a single device - the transition never reached the leader start this early. Repairing that function is what let the election run, which is the second suite this week where fixing a silent always-null unmasked a precondition nobody had needed to state. Waited on the container's STATE rather than a `started` actuation: this suite's own before() installs the app, so an unanchored wait for that event is satisfied by the install rather than by the leader start. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n too Same arithmetic as the stand-down split, one test earlier. Both of these need a departure to actually happen, and a node sheds ONE app per departure interval while a node inside that interval accrues nothing towards any other app's turn - so on the shared fleet they queue behind whatever the thirteen tests above them left on node 1, against a DEPARTURE_WAIT_MS of two cycles. On the 4fc95ac gate that was exactly enough to fail. Node 1 held clockapp and sharedapp - only two, not the seven that killed the stand-down test - and sharedapp drew eight AWAITING_TURN while node 3 drew four DEPARTURE_INTERVAL. Neither node ever departed, so the app never went short, so the BELOW_INSTANCE_COUNT refusal the test waits for on the OTHER node could not be published. One departure interval plus one full ticket against two cycles is no headroom, which is why it passed until something in the tree moved. Splitting rather than widening the budget, for the reason the stand-down split was preferred: a budget sized for the queue leaves the queue there for whoever adds the next test. This is the second time that queue has bitten in one day, at two different tests, which is the argument for removing it rather than measuring it. Three fleets now: 13 tests, 2, and 3. The accumulation still exists in the first describe and is still harmless there - nothing in tests 1 to 13 needs a departure to complete inside a fixed budget. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e undone holdOutPendingNode refuses a node that has not started yet with a one-directional INPUT drop on each running node. Its comment claims "one direction suffices because a connection needs both" and nothing checked it. Suite 71 depends on it - it moves a node's boot out of a 63 second acceptance window - but covers it only by going red for its own reasons when it fails. Asserted on both sides. A drop on INPUT stops the held node's SYN arriving; a running node dialling out still sends one and it is the reply that is dropped. So neither direction can complete, and testing only the outbound half would pass on a pair that connected the other way round - the mistake suite 96 made. The control is the part that makes the rest evidence. An empty peer map satisfies "the held node is absent" exactly as well as a working drop does, so the suite waits until the running nodes have peered WITH EACH OTHER before asserting the held one is missing: absence measured in a fleet where presence is demonstrable, at the same moment. Then the drop is removed and the same node joins, which is what attributes the absence to the drop rather than to a node that was never going to peer. releasePendingNode is the counterpart the primitive did not have - a suite could hold a node out and had no supported way to let it back in. Best-effort per rule, like healPartition, which is the same shape one function down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… when it repeats spawner:noCandidates was published on every pass that placed nothing - roughly every 240ms per node, and true continuously for any fleet whose apps are all at their instance count. fluxEventBus states the rule it broke at the top of the file: FACTS AS EVENTS, CADENCE AS A COUNTER. Subscribing to it shrank the Last-Event-ID replay window from most of a suite to about four minutes, so the client stopped subscribing - leaving product code publishing an event nothing could receive, and a waitForNoCandidates helper no suite called. An event nobody can afford to listen to is not a telemetry surface. So the cadence is now a counter, and the fact it was standing in for is an event: spawner:candidacy, published only when this node's verdict on an app CHANGES - carrying the app and the filter that removed it, or 'candidate'. The survivor breakdown stays in the log line, which is where it was useful today. This is what suite 78's last test needed. It asserted that a node which stood down is one of the two holding the app afterwards - but that is a draw among every eligible node, and the design promises nothing about who wins. Measured on an idle box with the claim rows dumped: only THREE of five non-holders stood down, the other two never claimed at all because the app already read as covered when they looked, and after the withdrawal all five are candidates. A stood-down node wins three times in five, and the suite has been passing on that. Eligibility is a fact about one node, and afterAlreadyHeldOrTried is the filter that would hold a stood-down node out - so the test now waits for any node that stood down to report that verdict flipping back to candidate, and is indifferent to which one wins. The instrument that found this also killed the theory it was written for: every claim list is empty after the withdrawal and identical before it, so nothing was stale and nothing double counted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he withdrawal Every node in stoodDown was a candidate for this app before it claimed - that is how it came to stand down - so an unanchored waitForEvent is answered from the buffer by that earlier verdict and passes before the rival has withdrawn anything. Proven, not reasoned: with a mutant that permanently excludes any node which ever withdrew, the unanchored form still passed 6/6. Anchored, the same mutant fails test 6 and leaves 1-5 green. Two attempts were needed to get that red, and the first failure was informative in its own right. Setting trySpawningGlobalAppCache on withdrawal did not exclude anything, because the decline path deletes that entry on the next pass - the spawner fix two commits down was repairing the damage as fast as the mutation made it. A mutant that stays green means either the test is blind or the mutation is not real, and those need opposite responses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s still up Suite 02 lost the f8fe3db gate with all eleven of its tests green: the `after all` hook died on its flat 30s budget with `db 32831ms`, against the other 145 teardowns in that same gate at 0-2ms. Perfectly bimodal, so it is a hang and not slow work. mongodb 7.5.0's `_close()` ends its pooled sessions with `endSessions`, a w:0 write that needs a connection but never a reply, and it skips that call ONLY when server selection over its CACHED topology description comes back empty. Teardown stopped mongo and then closed the client, which makes that cache a coin flip: usually the monitor has already marked the server Unknown and the call is skipped, but when it has not, close() commits to an operation whose server goes Unknown mid-flight and waits out the full 30s serverSelectionTimeoutMS. `squashError` swallows the error, so nothing is logged and the suite just loses its budget. Instrumented on chud, six suite-02 runs at a time so the stops land together. The in-flight snapshot shows the sequence: `Standalone` on entry to close(), an implicit session appearing as `endSessions` commits, `Unknown` two seconds later, then 29 seconds of nothing. closing after the stop 27 of 30 runs slow, ~33s each closing before the stop 0 of 30 runs slow, slowest 3ms Closing while mongo is still up removes the precondition rather than budgeting for it. The nodes are already stopped by that point, so nothing is writing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ubshell `trap 'kill $CAP1 $CAP2 $CAP3'` reaches three subshells and nothing inside them. For CAP1 the children are `sudo` and the `dmesg` it execs, which are root-owned, so they survive an unprivileged kill of their parent, reparent to init, and leave `grep` blocked forever on a pipe nobody will close. Three processes per gate. Because they stay in the ssh session's cgroup the scope goes `abandoned` and logind never reaps the session, which is why `uptime` reported 54 users on an idle chud while `who` reported none. Documented as leaking since 2026-08-07 and hand-cleared at least three times without the cause being fixed. Found this session: 66 runs' worth on chud and 31 on cindy, the oldest dating from that box's boot. `docker events` was orphaned alongside them, so CAP3 leaks too and the fix cannot stop at CAP1. CAP1 now records the PID of its root half so the trap can signal it directly - kill `dmesg` and `grep` sees EOF and the subshell follows it out - and the trap takes each sidecar's children before the sidecar itself, since killing the parent first loses the ppid link that finds them. Proven both ways on cindy with the sidecars launched exactly as the gate launches them: the current form leaves 3 survivors, this one leaves 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…st the ring 9fcabdc taught bootAndPeer that a deferred node is a hole in the ring exactly like a stub, and stopped one function short. derivePeerThresholds is called with the stub count alone, so a fleet with a deferred node derives thresholds for more dialers than exist - asking each node for a peer that cannot answer until startNode creates it. No suite moves, which is why it went unnoticed. The arc is a step function and every current fleet with a deferred node sits away from a step: 10/1, 11/1 and 12/2 stay at 4, 6/1 stays at 2, 3/1 stays at 1. Checked by enumeration across all five rather than assumed. 6/2 and 7/1 are the nearest shapes that change, so this was a trap set for the next suite written rather than a fault in an existing one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…is cleared
checkLoggedUser had two outcomes sharing one envelope. Not authorised answered
createErrorMessage(PRIVILEGE_RESPONSE.NONE), so data.message was 'none'. The
catch answered createErrorMessage(error.message, ...), so data.message was
whatever was thrown.
data.message is not free text on this endpoint. PRIVILEGE_RESPONSE says so at
the top of the file: the frontend branches on those four strings, in a
separately deployed repo, reading them out of the body whether the status says
success or error. fluxos-frontend's router guard does exactly that, and clears
localStorage.zelidauth on exactly 'none':
const privilege = response?.data?.data?.message || 'none'
fluxStore.setPrivilege(privilege)
if (privilege === 'none') { localStorage.removeItem('zelidauth') ... }
So a thrown error set the privilege to a string like 'No user Flux ID
specificed', which is not 'none', and the stale session stayed in localStorage
instead of being cleared. Access still failed closed - no route's privilege list
contains an error string, so every guarded route bounced to '/' - but the user
was never logged out, and each later navigation repeated the same failure
against the same bad credentials.
The three input throws are not the reachable trigger: the guard only calls with
all three fields present. What reaches the catch is verifyPrivilege itself
failing, which is exactly when the session should end.
message now carries the contract on both paths. The error is not lost - it is
still logged, and name/code still separate a failure from a plain refusal, which
sends neither. The status stays 200, per the rule stated in messageHelper: a
request that reached a handler is answered in the body.
Every other consumer of /id/checkprivilege was checked and none is affected:
cloud_admin reads message only when status is success and validates it against
its enum, fiat-orchestration-backend and FluxDrive test only status, the
palworld and minecraft sites define a caller nothing calls, and flux-mcp only
lists the path. The frontend guard is the only reader of message on the error
path, and it is the one this fixes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ller setNodeGeolocation reschedules itself three ways - 10s while no IP is detected, 3 days on success, 5 minutes on failure - and kept no handle for any of them. It has two callers: serviceManager at boot, and fluxNetworkHelper on EVERY IP change. So a second caller did not resume the loop, it started another one, and nothing could stop either. A node that changed IP three times therefore ran four independent geolocation loops for the life of the process, each calling the geolocation API and writing the same record on its own schedule. On a node that never detects its IP, each of them also logs an error every ten seconds, so the log noise multiplied with every IP change. Nothing short of restarting FluxOS ended any of it. Ownership is the fix, not a guard at the call sites: the loop is a singleton the module owns. One handle, cancelled before it is re-armed, so a second caller resumes the one chain. One pass at a time, because an IP change arriving while the previous pass is still awaiting the geolocation API would otherwise let two passes interleave their writes and settle lastIpChangeDate on whichever finished last rather than on the order the addresses changed - the late caller is held and run afterwards, never dropped. And stopNodeGeolocation is exported, because a timer nobody can stop is a leak wherever it runs. Proven red first, all three: two callers woke two passes three days later, the loop could not be stopped at all, and two callers ran concurrently. 61 passing in this file with the fix, 6,130 in the unit suite with 0 failing. The test file faked setTimeout but not clearTimeout, which left the real clearTimeout unable to cancel a fake id - a cancellation would have silently done nothing and every cancelled timer still fired. Faked together now. Date is still deliberately not faked; the service reads it for the IP-change window. Diagnosed in fluxModels TIMERS_NOBODY_OWNS.md, which measured this from the other side: the same unowned timers are what made the unit suite fail at random, because a timer armed in one test fires inside a later one and writes to a logger three files' worth of exact log.error assertions are counting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eads is quiet Suite 96 test 2 was the single red on the b22117e gate, and the product was right: the second-newest copy reported SURPLUS/WRITER_UNCONFIRMED and trimmed nothing, because peerComponentState's 10s probe found the newest copy silent. The rule is "never trim while you cannot confirm the newest holds the writer", and it held. The test lost because it gets one attempt. The give-up pass runs every removeFluxAppsPeriod * 4 = 44 blocks and the drive spends about 53, so there is exactly one. The precondition was checked up to three minutes before that pass landed - the newest answered the test's identical curl and had stopped answering by the time the product asked. A precondition that is merely usually true decides nothing here, as the file's own comment already warned. driveUntil evaluates its condition before every block, so the condition is where the guarantee belongs: it now waits for the newest copy to be answering before another block is driven. The chain cannot advance into a give-up pass through a window in which that pass would have to refuse. No budget is widened and the pass rate is untouched - driving longer would have reached block 88 and bought a second coin toss rather than removing the coin. Measured on cindy, three concurrent copies of the suite: 3 of 3 green. The same rig failed 2 of 3 on the attempt described below. The attempt not taken, recorded because it looks obviously right and is not. Compressing removeFluxAppsPeriod to 2 the way suite 55 does gives the drive many passes and halves the runtime - and it breaks the fixture. A pass then fires within ~7s of the stub announcing the fourth copy and trims the newest real one before the setup's wait can observe four holders, because the stub announces runningSince - 24h and is therefore the OLDEST, so the surplus rule correctly picks a real node. stopTicker does not save it: blocks already in flight are still processed. THIS SUITE ONLY SURVIVES ITS OWN SETUP BECAUSE THE GIVE-UP PASS IS RARE - the knob that would make it fast is the knob protecting its fixture, and making it fast means the suite owning the clock from before the surplus exists, which is a redesign of the setup rather than a config change. The fixture's failure now names which holder is missing rather than how many there are. Three can mean the stub was never counted or that a real copy was trimmed early, they need opposite fixes, and the count alone cannot tell them apart - it cost most of a session to learn that once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… having called it
Found by Cabecinha84 in review. The STAND_DOWN_REQUIRED block awaited
appDockerStop and then marked the component regardless of what it answered.
The catch beside it says why that is wrong - "a component this node failed to
stop is one it is still writing to, and marking it would make the node
unelectable for a component it is running" - but it can never fire for that
reason: appDockerStop REPORTS a refusal rather than throwing one. It catches
internally and returns { stopped, running, unavailable, errors }, where
`stopped` is read back from docker rather than taken from the stop call.
So on a node where docker simply refuses, or never becomes able to answer, the
result is stopped:false with no throw, and the component was marked anyway. It
is then excluded from the primary election - standingDown.has() is checked in
masterSlaveApps - for up to STAND_DOWN_PASSES_BEFORE_GIVING_UP passes while it
may still be running the g: writer. Exactly the state the comment describes.
Bounded, and worth stating: setControllerDesired('stopped') also enqueues a
reconcile that will stop it, the cap self-heals, and the next give-up pass
refuses with ELECTION_UNKNOWN rather than removing anything. No data loss. The
guard was decorative and its log line could not print.
The verdict is read now. Marking happens only on stopped:true; anything else
logs what docker actually said - which components are still running, whether it
was unavailable, and the errors - and leaves the component unmarked so the next
pass retries.
Proven red first, in the shape a live node sees. The existing test for this
rejects the LOW-LEVEL dockerService.appDockerStop, which the wrapper catches, so
it was passing on an unexpected throw from dockerActual rather than on the
mechanism it names. The new test lets the stop return and has docker answer
cleanly that the container is still up: no throw, stopped:false. It failed
before this change and passes after, and the two cap tests either side of it
still pass, so the success path is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cba4798 to
9a12057
Compare
…vent is left to the harness that reads it
…the reason is a value and not a broadcast
2 participants
A node on a residential connection is only fit to serve the network when it runs ArcaneOS. This puts such a node out of service in three stages, so that the destructive step is only ever reached on a node holding nothing.
DOS >= 100 is not a mark:
nodeStatusMonitorcallsremoveAllAppsLocallyon it andappStartupManagerdoes the same on every boot, so a node in that state has every app directory and volume on it deleted. Reaching it only on an empty node is the whole point of the staging.There is no in-FluxOS off switch, and the one that exists is better. Enforcement acts only on a
RESIDENTIALverdict from the published table, so blankingorgClassesinfluxos-network-policyends it fleet-wide within a day — no release, no node upgrades, nothing to roll back. A config flag would have needed all three.The trust boundary this inherits, stated rather than inherited silently. The table is fetched unsigned from
fluxos-network-policy; trust is TLS, GitHub, and write access to that repo, and this PR moves what those bytes can do from "placement quality" to "stage a non-ArcaneOS residential node out of service". That is a conscious trade. It is bounded — at most the 541 non-Arcane slots with readable bench, one app per 6 hours behind the full safety gate, DOS only on an empty node, self-healing on the corrected row — and it replaces a weaker root, not a stronger one: today's classification rests on plaintexthttp://ip-api.com, forgeable per node by any on-path party with no history to audit, while every byte the policy repo has ever served is in its git history behind branch protection (no force-push, admins bound, validated on publish — though merges do not yet require a second reviewer). Signed manifests (the fluxhashes scheme, in progress) are the successor; this artifact adopts that signing when it lands.What it acts on
Measured against
stats.runonflux.io/fluxinfo(6,115 slots on 2,513 machines, 5,822 slots with readable bench) andapi.runonflux.io/apps/locations(6,340 placements, 1,085 apps), classified by reading the published artifact through this branch's ownipLocationStoreandnetworkClassifier.Enforcement acts only on a
RESIDENTIALverdict from the published table. Of the 541 slots on 316 hosts whose bench confirms they are not ArcaneOS:A legacy node is very nearly as likely to be in a data centre as on a home line, so "not ArcaneOS" is not a usable proxy for "residential" and is not used as one. A further 293 slots, on 83 hosts, have no readable bench at all;
isArcaneOs()returnsnullfor those and anulltick decides nothing.No node in this population vetoes its published verdict: the node veto declines a published
RESIDENTIALon local hosting evidence, and the one fleet host it currently covers —213.44.137.57— runs ArcaneOS, so it is outside the enforceable set to begin with.Why it is staged rather than immediate
63 apps have every copy on a node that qualifies for enforcement:
None of the 49 single-instance apps hold synced data. An owner who deploys
instances: 1chose a service with no continuity guarantee, and the platform's standing semantics for any departing node — expiry, powercut, surplus trim, geolocation redeploy — already remove the instance and its volume; nothing anywhere preserves an unsynced volume, so the rebuild starting empty is not a property of this path. Evacuation is in fact the gentlest departure the platform has: the removal is broadcast immediately and the spawner rebuilds elsewhere at once, where a node that simply dies leaves its stale location record holding the app at apparent full strength for up to 125 minutes (the running-location TTL) before anyone acts. It is still an outage window, and a longer one where placement is constrained by a geo pin, hardware, or enterprise ownership.All 9 stateful ones are
g:apps at exactly 2 instances: seven Palworld and two Minecraft world saves. The ordering is what makes them safe — the junior leaves, the senior copy still holds the world, the replacement syncs from it, and only then does the senior go.218 slots sit on 83 machines. Several FluxNode registrations routinely share one physical connection — a residential line hosting 8 nodes loses all 8 at once. The spawner already refuses to place a second instance of an app on an occupied IP, so one app's instances land one per host by construction; the safety gate's other-host count dedupes to machines as well, covering the pair that converges after placement through an IP change.
Why the evacuation terminates
The departure creates the deficit that drives it. Only 4 apps in the entire fleet carry any surplus — the trim path removes above N and the spawner fills below N, so the network sits at exactly-N and a drain conditioned on staying above the requirement never starts.
So the node leaves first: the app is one short, the spawner is already asking every 120s who is short, a node that is not held volunteers, and the count returning to N is what releases the next holder. Serialisation needs no coordination and no wire change — a departure leaves the app short, every other holder's gate refuses to act on a short app and resets that app's queue clock, so each further departure waits for the spawner to restore full strength and then serves its full wait again.
The queue ticket is an uninterrupted observation, not elapsed time, and that is what keeps position binding on every departure rather than only the first. A node inside its own departure interval cannot act on anything, so it records nothing against its tickets; the block then reads afterwards as one long gap and every ticket starts again. Without that the tickets matured untouched through the six hours, the node came out of the block instantly ready for everything it held, and two holders whose blocks expired on the same pass — the pass is keyed on block height, so the fleet evaluates together — could hand back the same app in the same pass, each reading it at full strength because the removal is broadcast after the volume is already gone. The strength test sits at the same point: an app short of its instance count accrues nothing towards its turn, so the wait means what it says. Both clocks are monotonic, so a step of the system clock cannot mature a ticket against time the node never spent watching.
Position in the shared instance order sets a delay, never a veto. "Only the most junior may leave" deadlocks, because the replacement is itself the most junior and does not want to leave.
The junior copy leaves first, which is the same order
reasonToGiveUpAppalready ranks SURPLUS by and for the same reason: the newest copy stands aside and the senior one goes on holding the data. It also puts the one case that cannot simply leave at the back of the queue rather than the front — the elected primary is the senior instance, and it has to stand down and hand the app back before it can go.The pacing is config-tunable: 6h between evaluations, a 24h settling window before anything moves customer data, 6h between departures, and a queue ticket of 30min + position × 40min. The step has to outlast the pass that reads it —
mayEvacuateAppis reachable only from the give-up pass, which runs everyremoveFluxAppsPeriod(11) ×speedMultiplier(4 post-PON) = 44 blocks = 22 minutes, so maturity is quantised to a 22-minute grid and a shorter step cannot separate adjacent positions on it. That inequality is asserted against production's own config in the unit tests. Two intervals are hardcoded rather than tunable:RETRY_INTERVAL_MS(5min, the retry after a tick that could not decide — it doubles up to and is capped by the check interval) andstaticIpStabilityDays(10).What the grid does not separate. The step separates positions on the 22-minute grid; it does not separate maturities that began at different moments. A ticket is measured from each node's own observation start, so two holders whose starts differ by about a step mature inside the same pass whatever the step is — and the pass is block-keyed, so the fleet evaluates together. What keeps that empty is not the pacing: every stateful 2-instance app on the network is
g:, and the election refuses to let the writer leave, so one of any two colliding holders declines regardless of its turn. That is a property of today's app population, not a guarantee this code makes. Closing it structurally means taking the turn from the shared clock the pass already runs on — a node acts only in the pass its rank selects — rather than from elapsed observation, which is a private measurement answering a fleet-wide question. Per-node jitter does not close it: it separates a colliding pair by the difference of two fixed offsets, which can be zero, and moves an already-separated pair together as readily as apart.The settling window counts time the node observed the verdict, not time that elapsed. Each confirming tick credits the gap since the last, capped at one check interval, and credits nothing once the gap is long enough that the node plainly stopped watching. It is persisted in the
nodeStartupTrackercollection, so restarting FluxOS costs nothing and cannot postpone the drain — and a node that answers once, says nothing for a day, then answers once more has accrued two ticks, not a day.An empty residential node is DOSed on the first decisive tick, with no settling window at all. The window exists to protect customer data and an empty node has none; the DOS deletes nothing and is released the moment the node is attested again. It does mean a mistaken policy row reaches empty nodes in minutes rather than a day.
The classification
A node is
RESIDENTIALonly when something positively says so and nothing contradicts it:The published location table is the authority, and where it holds no verdict nobody does. It is built in
fluxos-network-policyfrom registry evidence a node cannot gather for itself: six thousand nodes cannot each query the RIRs. There is no fallback to the node's own reading — that rule is the published rule with its strongest signal removed, its error rate has never been measured, and it would have decided for the 13% of the enforceable population — 70 of the 541 non-ArcaneOS slots — whose organisation carries no published verdict, on the one path that deletes customer data. Tuning belongs in the policy repo, where a verdict is evidence-backed, auditable, and correctable by hand throughdata/orgclass-overrides.jsonwithout a FluxOS release.What the node's own evidence still does is decline a published
RESIDENTIALabout its own address. An organisation is decided by 80% of its hosts agreeing, so a minority tail of the other kind is guaranteed by construction, and the veto is how one of them steps out until someone adjudicates the range. It can only ever remove a node from enforcement.The evidence a verdict is read against can lag an address change by up to three days.
getNetworkClassification()looks the published table up onnetworkEvidence.ip, which is refreshed onsetNodeGeolocation's three-day cadence, while the enforcement tick runs every six hours — so a node whose address moved between geolocation passes is classified against its previous address's row until the next one. It cannot begin a drain inside that window: both the 24h observed window and the queue ticket restart on the change. The HOLD is immediate, which is the conservative direction.isDataCenter()andisStaticIP()carried the same latency before this PR.Measured at the point of enforcement: of the 1,569 fleet hosts ip-api positively calls hosting, across all 29 hosting ASNs the fleet uses, exactly one carries a published residential verdict —
213.44.137.57, in Bouygues' consumer space — and the node veto covers it. One ledger-level error in 1,569, zero enforced.That is the only accuracy figure worth quoting for this design. A rate measured on the classifier alone cannot be one:
hostingis itself a contradiction, soRESIDENTIALis unreachable on that population whatever the other signals say.Signals are read from the operator (
isp/as), never fromorg, which is the block registrant and frequently a reseller:46.250.240.89carries isp "Contabo Asia Private Limited" and org "Yorkshire Tech Limited". Link asymmetry corroborates and never decides — a bench figure is a speed test's result, not a property of the link, so it can support a verdict the real signals already reached and can never reach one.A verdict also requires that the contradicting signals were consulted at all. When ip-api answers 200 with an unusable body,
geolocationServicefalls back tostats.runonflux.io, which carries none ofhosting/proxy/mobile/isp/as—fluxstatsnever requests them and its/fluxlocationhandler projects them away. On that path an emptyevidenceAgainstmeans nobody looked rather than nothing found, and the node is left unclassified rather than residential.No table means no verdict — not a fallback. A booting node fetches a 4.6 MB artifact and ingests two million rows while a single ip-api call answers in milliseconds, so the table is reliably absent at the moment a booting node would otherwise decide, and nothing enforces on
null.The verdict is not stored.
setNodeGeolocationgathers evidence on its three-day cadence;getNetworkClassification()reaches the verdict when asked, from that evidence and whatever table the node currently holds — which is whatplacementFeasibility,hwRequirementsandappSpawneralready do with the same table.What
staticipis for, and why the rule changedstaticipis two promises, and the apps that ask for it need both. The live consumers are VPN endpoints (cumulusvpn*), Kaspa nodes, andquipbootnode— peers and clients hold anip:portpair written down, so what they need is an endpoint that stays reachable at a fixed address:Both are properties of the host, so only the host can attest to them. The whole rule is now four rows of observation, and no table, vendor flag or operator name appears in it:
Only
STATICsatisfies the requirement.UNKNOWNandDYNAMICboth fail it and are kept apart, so a node that could not answer does not read as one that answered "behind NAT".What the fleet actually looks like
Random sample of 500 ArcaneOS slots, 442 reachable, reproducing
hasPublicIpOnInterface()on each node by SSH. Scaled ×12.3 to the 5,438 reachable Arcane slots.staticIptoday95% CI on the public-address share: 19.1–27.0%.
Of the 228 slots advertising
staticIp: true, 126 — 55% — are behind a UPnP router on an RFC1918 address. Every one of them qualifies through a third-party flag about the address RANGE, never through anything the node observed about itself:statichosting: trueproxy: trueThat is
storedGeolocation.static = proxy || hostingondevelopment. So "static IP" has meant "ip-api says this address sits in a hosting or proxy range" — a statement about the block, not about the node.Why a range flag answers neither promise
DATACENTERverdict, different endpoint.The 12 qualifying on
proxyare the clearest failure: that flag is a VPN artefact, andgeolocationService's own comment said so while the code it commented on granted static on exactly that basis.What it costs
staticipNothing evicts those 74.
checkAppStaticIpRequirementsis reachable only fromcheckAppRequirements, which runs on install and redeploy; there is no periodic compliance sweep in ZelBack. They drain by attrition as owners push spec updates, becausereinstallOldApplicationsremoves before it checks.How much of that attrition touches data, measured: of the apps requiring
staticip, 3 carry synced components — all of themr:, all atinstances: 3, 9 placements between them. Two of those 9 sit on slots that lose static. Every other app requiringstaticipkeeps no synced state at all, so an attrition removal costs them a container the spawner rebuilds from the specification.So the bound is one of three copies, on two apps, re-synced from the two survivors; losing an app outright would need all three of its copies on losing slots and an owner update, which no app in the population is in a position for. The removal is still a removal that the evacuation safety gate never sees, and that is the honest residual here.
The ordering behind it is not this branch's to fix, and is being fixed.
reinstallOldApplicationsdeletes the app and its volume and only then asks whether the node may still run it, so any requirement that fails — disk, geolocation, static IP — fails after the data is gone. It fires today, ondevelopment, for every one of those reasons; this branch widens the set of nodes one of them says no to. The fix is a make-before-break drain: a node that can no longer satisfy a spec keeps serving the old one and keeps its data while being excluded from the counts that decide whether the network needs another instance, and only tears down once a replacement is ready. That is designed and lands in a subsequent PR. Reordering the check alone is not the fix — it converts the deletion into a node stuck on a stale spec forever, which is why it is not done here.The operator's remedy is to put a routed public address on the interface rather than NAT, which is the behaviour worth steering toward.
What else moves with it
isStaticIP()is not read only by thestaticipgate. It also drives a spawner deferral: a node claiming static defers apps that do not require static, so as to stay available for apps that do. So on upgrade day roughly 1,550 slots stop deferring and become immediately available for ordinary placements. That is a fleet-wide placement shift and it is not priced in the table above. It is a correction rather than a regression — those slots were deferring on a claim they could not support — and the static pool that remains, ~1,255 slots, still exceeds the 74 placements that ask for it by more than an order of magnitude.Second-order, and ongoing rather than one-off: a genuinely static node that changes its address now reads
UNKNOWNfor the full ten-day stability window, where ondevelopmenta hosting-range node recoveredstaticon its very next pass throughproxy || hosting.checkAppStaticIpRequirementsis a hard gate throughout that window. This is the same pre-existing ordering described above, reached by a different route, and it closes with the same change.The case this knowingly refuses
Genuine 1:1 NAT, where a fixed address is mapped to a host that never sees it locally. The sample holds 4 such slots (~49 fleet-wide) and none claims a static address today; the fleet's entire hyperscaler population is one Oracle slot. Real, and priced at nil.
The sample was drawn at random and each slot was read directly, so the shares above carry the stated confidence interval rather than being a count of whoever answered.
Behaviour this changes for every node
Surplus removal refuses when no connected peer demonstrably holds a stateful app's data. SURPLUS and EVACUATION share one safety gate, so this applies fleet-wide, not only to residential nodes. A surplus copy may now persist while syncthing cannot prove a peer holds the folder — the trade against deleting a last remaining copy, and the only signal is a warning every 12 refusals. The gate also refuses to hand back a
g:component this node is the elected primary for, and refuses when no FDM can name the primary at all: an election that has not run is not an election that said no.A surplus copy that is running the
g:writer is left alone, and the next copy trims instead. "The newest copy stands aside" is a stand-in for "the least valuable copy stands aside", and when the newest copy is the one writing it is backwards — that is the most valuable copy on the network. The election is allowed to seat the writer anywhere in the instance order: it will not start a node whose data has not finished syncing, so it works down the list until it finds one that is ready, and the designated-leader branch leaves the order outright. The second-newest steps in only on a positive confirmation that the newest is running the writer, asked through the same peer probe the election uses. Silence, a timeout, a refusal and "not running" all mean it does nothing — every node ranks the same shared order, but "who is writing" is each node's own reading and FDM's registration lags it by ~110s, so the rule can only ever fail towards no trim, never towards two. What it does not do is stop the writer to make the ordering come true: an app that is over-served is not down, and an app whose writer was stopped to tidy up a count is./flux/inforeports which stage of being staged out a node is in —dosStaging, one ofnull,HOLDorEVACUATE, beside thedosit leads to. Always present, so it can be counted across the fleet. The first stage was otherwise invisible: a held node takes no new apps and is in every other respect indistinguishable from one that has simply not been given work, and that stage lasts a whole settling window — the only part of this where an operator can still put the node right and lose nothing. It is derived from the settling window rather than from the evacuation flag, because that flag is a per-tick permission for the give-up pass that any undecidable tick turns off, and reporting it as a stage would show a node moving backwards through a staging it never moved backwards through.During a total FDM outage, no node elects a g: primary.
getMasterIpFromFdmnow reports whether any region actually answered, which makes theif (!fdmOk)guard inmasterSlaveAppslive: a node that gets no answer from all three regions skips primary selection for that cycle instead of reading silence as "no primary set" and promoting itself. A g: app whose primary dies while every FDM is unreachable stays down until one answers. That is the trade, and it is the right one: the node the old behaviour promoted was one that could not reach three services on three continents — at least as likely to be network-isolated itself as witnessing a global outage, which is exactly the split-brain writer the election exists to prevent.One instance of an app per host, and the safety gate counts hosts. The spawner refuses to place an instance on an IP that already holds one, and an app's fixed ports make a second copy on one machine moot regardless — so a given app's instances land one per host by construction. The safety gate's "does anyone else hold this?" dedupes locations to distinct machines anyway, covering the one way a pair can converge after placement: an IP change. The strength count and the SURPLUS test count registrations, which under the placement rule is the same number.
An app is no longer written off for twelve hours on a count that included a claim.
trySpawningGlobalApplicationcounts instances twice — once while choosing an app, and again after it has selected and cached it — and the second count returned without clearing the entry the selection had just set. That entry is the candidate filter rather than a re-decision skip, so it removes the app from consideration entirely, and it lives twelve hours. A claim is provisional by design: the fault-domain share is checked after the claim goes out, so a claim is withdrawn seconds later once its node finds the share filled. A node that re-counted inside that window recorded "the network has this covered" for half a day. The entry is now kept when running copies alone meet the count and cleared only when the claims were needed to reach it. Clearing it unconditionally is the other way to be wrong: an app genuinely covered by running copies would re-enter the candidate pool every pass, be declined again, and never be cached — because this node never installs it, and installing is the only other thing that sets the entry. No deferral is added;appsToBeCheckedLatercarries conditions lasting 30–57 minutes and this one lasts seconds.A placement hold is released by its owner, and only by its owner. The hold was a single slot, so a second
setoverwrote the first reason outright and either caller's release cleared the other's. It is now a map of owner to reason, the owner drawn from a frozen enum, an unknown owner refused at theset, andgetPlacementHold()reporting every reason held. An ownership check on the single slot would have deadlocked rather than fixed it: two owners cannot coexist in one, so the first would have been left unable to release and the node held indefinitely.One removal per pass, rather than several spaced 300s apart inside one pass. The sleep held the whole pass open — every app behind the removed one waited on it, and every decision after it was made against an installed-app list read minutes earlier — and the pass is fired unawaited from the block handler, so one long enough to outlive its own interval could overlap the next. Surplus removal is correspondingly slower: one app per pass rather than several, on work that is not urgent because a surplus app is over-served rather than down.
isStaticIP()is observed, not inferred, and is three-state. The table above is the whole rule.publishedHostingand thestaticIpOrgslist of nine hoster names are deleted; the decision collapses from six rows to four, and nothing outside the node's own interface can confer a static address.isStaticIP()is a hard gate inhwRequirements, so apps requiring a stable endpoint are no longer placed on nodes that only look stable from the outside.isDataCenter()rests on positive hosting evidence. It used to be ip-api'shostingflag OR the block registrant's name matching one of nine hosters. It is now aDATACENTERverdict from the classifier: a hosting PTR, the hosting flag, or an operator that sells hosting.proxydeliberately does not qualify — it is a VPN artefact, and a home machine behind a VPN exit carries that signature and nothing else, which is exactly what an owner buyingdatacenter: trueis paying to avoid. Measured over 2,432 fleet hosts with an ip-api record:developmentThe two that lose it are
212.83.170.245(Scaleway) and213.44.137.57(Bouygues), both hosting addresses whose reverse DNS carries access-network vocabulary, so the classifier reads them as CONFLICTED rather than guessing.Also in this branch
config.fluxapps.explorerPollIntervalMsis configurable, defaulting to the current 5000. It is not how often the chain is asked:pollForNewBlocksreads a height cached bydaemonServiceMiscRpcsand refreshed on its owndaemonInfoIntervalMstimer, so the poll is the rate at which a node works through blocks once it knows it is behind. Its share of that refresh window — 5000/30000, 16.7% — is what decides whether a block is still the tip when processed, and everything downstream of block processing inherits it. Production is unchanged; the harness runs 833ms against a 5s refresh, which is the same 16.7%.isSyncedis documented as what it is: true when the block was still the chain tip when fetched, which is not the same as "the node is caught up". The tip comes from a cache refreshed everydaemonInfoIntervalMs, andprocessBlockchains straight into the next block when behind, so a burst arriving between two refreshes is processed back to back and only the last block qualifies — silently skipping expiring app records, trimming surplus instances and reinstalling outdated apps for the rest. Post-PON that is a 30s block against a 30s refresh. It is named rather than worked around: v9 removes the race by pushinghashblockheighttochainTipSource.ipLocationSync.startSync()is split. The restore half — one marker read against rows already in mongo — runs with the database schema prep inserviceManager; the fetch half, a 4.2 MB download and a two-million-row ingest, stays behindwaitForDbReady. The residential verdict was otherwise waiting minutes for data it could have in milliseconds. Splitting rather than moving wholesale is deliberate: on a node with no cache, moving the fetch early would run that ingest concurrently with the app-database rebuild, which is the busiest the database ever is.config.fluxapps.sigtermExpirySreplaces a hardcoded 420s constant, defaulting to 420 so production is unchanged. It must stay belowlocationTtlS:appStartupManagerexpires locations on(cleanShutdown && downtime > sigterm) || downtime > running, so a sigterm window above the running expiry is unreachable and a clean shutdown gets no grace at all.fluxNetworkHelperpublishesdos:changedfrom the sticky DOS setters, carrying the valueisNodeDos()reads.residentialNodeDosServicepublishesresidential:decidedon every terminal path of a tick, withenforcethree-state so a tick that could not decide is distinguishable from one that decided this node is fit to serve. Both are inert on a real node —fluxEventBus.publishreturns immediately unlessconfig.testEventStreamis set.Harness
Three
shared.jskeys —locationTtlS,installingTtlS,installErrorTtlS— were wired to collection-level TTL indexes dropped when expiry moved per-document, so the harness ran on production durations while the file claimed 25×, 15× and 12× compression. They are live, andHARNESS_TUNABLES.mdrecords which are compressible and why the other two are not.explorerPollIntervalMsis 833ms, holding production's 16.7% share of the tip-refresh window rather than the 250ms that was three times more forgiving than production.test-infra/runner/framework/coupled-knobs.jsderives the knobs that only mean something relative to another knob, andtest-env.jsasserts them on the effective config of every node of every fleet before boot. A ratio at or above production's passes; below throws, because that is where the property inverts rather than merely slows. Two pairs are checked:residentialQueueStepMsagainst the give-up pass, andsigtermExpirySagainst bothlocationTtlSand the measured cost of one node boot.The
external-http-stubno longer synthesisesstaticanddataCenterfor/fluxlocation. The real service has never carried either — it batch-queries ip-api for location andorgonly — so the stub made the degraded fallback look richer in the harness than it is on any real node, and put the one path where the classifier loses its contradiction signals beyond anything a suite could observe.Operator evasion
ZelBack/config/default.jscannot be edited to disable this: fluxbench'sGetFluxCheckSum()hashes every file under the backend folder againsthashes.runonflux.io, so a changed config fails the checksum,signfluxnodetransactionrefuses, and the node expires at 640 blocks.The node veto is the cheapest evasion available, and it fails open by design.
getNetworkClassificationdeclines a publishedRESIDENTIALon any evidence against it, andproxyis such evidence — so an operator who routes a residential node through a commercial VPN reads asCONFLICTED, or more often asDATACENTERoutright once the exit address is in a hosting range, and is never enforced against. It is not a safety bug: everything here is built to enforce only on a node that is provably residential, and every unavailable or contradictory input already means "do nothing". It is stated rather than closed because closing it would mean enforcing on contested evidence, which is the trade this design refuses everywhere else. It is not free to the operator either — a commercial VPN exit generally cannot offer the inbound reachability a FluxNode needs, and the added path shows up in the benchmark figures that set the node's tier.Mongo is not hashed, and
geolocationServicerestoresdataCenterfrom the local collection when memory is empty — which is the state at boot. That is pre-existing and is not closable while an unverified machine judges itself. The settling clock is persisted precisely because restarting a service on a cron is near-accidental and worth closing; forging a classification in the database is deliberate tampering and is not.Depends on
This PR is the top of a stack and targets #1795, not
development. Its diff shows everything below it until those land:It sits on #1795 because it needs the harness port allocator that lives in
buildSeedableApp— a specification is signed and hashed the moment it is built, so a port added afterwards leaves the app carrying the hash of a specification that no longer exists — and because suite 98 covers aappSpawnerrefusal that lives on that branch. This branch previously carried six of those commits as cherry-picks; they are gone, and the originals are underneath.fluxos-network-policy#17, #18 and #19, all merged — the artifact carrying the organisation classes, the hand-adjudication filedata/orgclass-overrides.jsonfor the populations a supermajority vote cannot reach, and the rename of class code 2 fromhostingtodatacenter(a ledger-text change; the artifact carries codes, so there is no wire change). It is inert until this ships: the ipLocation stack is not onmaster, so no released node readsorgClasses.Verified end to end: the artifact fetched from the live
raw.githubusercontent.com/.../mainURL nodes read, parsed with this branch'sipLocationStore— 2,009,526 rows, and all 1,599 ledger ranges resolve to exactly the class the ledger claims. Every fleet figure above was produced through that same reader.Per-host lists — every host counted residential, and every host with no evidence either way — can be reproduced from the published artifact with the reader in this branch, and are available to reviewers on request.
The spawner reports its verdict when the verdict changes
spawner:noCandidateswas published on every pass that placed nothing — roughly every240ms per node, and continuously true for any fleet whose apps all sit at their instance
count.
fluxEventBusstates the rule at the top of its own file: facts as events, cadenceas a counter. Publishing the cadence pushed the
Last-Event-IDreplay window down frommost of a run to about four minutes, so nothing could afford to subscribe — leaving product
code publishing an event no listener could receive.
The cadence is a counter now, and the fact it was standing in for is an event:
spawner:candidacy, published only when this node's verdict on an app changes, andcarrying the app and the filter that removed it, or
candidate. The survivor breakdownstays in the log line, which is where it was already the useful thing.
Two unowned-lifetime fixes that came along
Neither is residential DOS. Both were found while gating this branch and both are small, so they
are here rather than trailing behind it.
checkLoggedUseranswered a failure where a privilege belongs.PRIVILEGE_RESPONSEis apublished contract — the frontend branches on those four strings and reads them out of the body
whether the status says success or error. The not-authorised path put
'none'indata.message;the catch put the thrown error's text there. fluxos-frontend's router guard clears
localStorage.zelidauthon exactly'none', so a genuine backend failure set the privilege to astring like
'No user Flux ID specificed'and left the stale session in place, repeating on everynavigation. Access still failed closed — no route's privilege list contains an error string — but
the user was never logged out.
messagenow carries the contract on both paths, with the errorkept in
name/codeand in the log. Every other consumer of/id/checkprivilegewas checked andnone reads
messageon the error path.The geolocation refresh loop was one chain per caller.
setNodeGeolocationreschedules itselfthree ways and kept no handle, and it has two callers:
serviceManagerat boot andfluxNetworkHelperon every IP change. So each IP change started another self-perpetuating loopand none could be stopped — a node that changed IP three times ran four of them for the life of
the process, each hitting the geolocation API and writing the same record, and on a node that never
detects its IP each one logged an error every ten seconds. It is now a singleton the module owns:
one handle cancelled before it is re-armed, one pass at a time so an IP change arriving mid-pass
cannot interleave its writes, and an exported
stopNodeGeolocation. This branch readsisStaticIP,isDataCenterandgetNetworkClassificationfrom that service, which is how itsurfaced.
Testing
55-residential-node-evacuation), 18 tests across three live 5-node fleets:The suite runs on three fleets rather than one because every test it contains seeds an
app asking for five instances on a five-node fleet, and nothing is cleaned up between
tests. Holdings accumulate, and evacuation sheds one app per departure interval, so a
test late in the file starts against a node holding seven apps and runs out of budget
before it reaches the assertion — with the node enforcing correctly the whole time. The
stand-down tests and the one-holder-at-a-time tests each get a fleet of their own, so
what they assert is decided by the rule rather than by how many apps the node happened
to have collected. Sizing a timeout to the accumulation instead would have left the
accumulation in place for the next test written.
96-surplus-trim-and-the-writer), on its own fleet with a stub peerholding a fourth copy of a three-instance
g:app, so the surplus is built rather thanwaited for:
The first holds the second-newest copy where it cannot reach the newest, while the writer
goes on running: a rule that trimmed on silence would remove a copy with the writer up and
healthy. It waits for the pass to report the surplus it declined, so a pass that never
ran cannot read as a pass that declined.
Harness suite 89 waits for the app to be stoppable before replacing its data, and suite
78 covers a spawner claim being withdrawn and the node's candidacy for that app coming
back.
A node's static IP is now declared by the fleet rather than inherited from the
wiring.
hasPublicIpOnInterface()reads/proc/net/routefor a default route, and theharness network is
Internal, so docker gives node containers no route out and everynode read DYNAMIC — making
static_ipdeferral unreachable and silently swapping whichreason a test was waiting for. A suite that wants a static node now gets a real default
route installed via the network's own gateway before FluxOS starts, and its absence is
how a suite asks for a NAT'd node. The route failure is loud, because a swallowed one
reads exactly like a node that was never asked for a route.
Full harness gate, 94 suites,
MAXN=6, at7523568a3: 94 passed, 0 failed.Run after Hosting an app is not owning it: the node operator loses every app-scoped verb, a specification decrypts for its owner alone, and the check that decides who is asking is rebuilt underneath #1791 merged into
developmentand this branch was rippled onto Two Flux nodes on one public address can install onto the same port, and nothing refuses it #1795 at18a705431.157 commits above
development(37 + 120); the ripple verified by--numstatand per-commitpatch-id rather than by the rebase's own report.
The two suites that were red on an earlier gate are green. Suite 02 lost its
after allhook to aharness teardown defect — the mongo client was closed after the mongo container had stopped, and
mongodb 7.5.0 then waits out a full 30s server-selection timeout that
squashErrorswallows,against a hook budget that is a flat 30s. Closing the client while the server is still up removes
the precondition: 27 of 30 runs slow before, 0 of 30 after, and the
dbteardown step has notexceeded 11ms on any gate since, against one outlier of 32,831ms.
Suite 96 asserts that a surplus copy which is also the writer is left alone and the next copy
trimmed instead. The trim is permitted only when a 10s peer probe confirms the newest copy holds
the writer, and the give-up pass runs every 44 blocks — so the drive gets one attempt, and an
earlier gate lost it with the product declining for entirely the right reason.
driveUntilnowwaits for that peer to be answering before it drives another block, so the chain cannot advance
into a pass through a window in which the pass would have to refuse. The budget is unchanged:
driving longer would have bought a second coin toss rather than removing the coin.
The gate is re-run per revision and this section carries the latest result at the head it names.