Publish the tree hash into fluxhashes without wiping the directory - #1785
Publish the tree hash into fluxhashes without wiping the directory#1785MorningLightMountain713 wants to merge 3 commits into
Conversation
…ectory The push used an action that deletes the destination directory before copying, and the source directory held only hashes.js -- so every build removed anything else kept in fluxhashes' src/hashes/. Nothing else lived there yet, so it has cost nothing so far, but it silently forbids ever putting anything beside the list. That matters now: a signed copy of the list is being published there, and its sequence is read from the previously published document. Deleting it would restart the sequence at 1, which every consumer that had accepted a higher one would refuse -- fleet-wide and silent, from a file placement. Clones and edits in place instead. Each attempt re-syncs to origin/master before deciding, so it is idempotent and survives two builds publishing at once: the loser of a push race re-reads and either adds its hash to the new tip or exits because the other build already added it. Previously a race simply lost a hash. Also refuses to publish a hashes.js that does not load, or that does not contain the hash just added. The list is served by requiring it, so a malformed edit would take the endpoint down rather than merely publishing something odd. The other use of that action, mirroring ZelBack/src/services into fluxjsdocs, is left alone: there the destination holds only what is pushed, so replacing the directory is what it should do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
node -e "..." puts the script inside a double-quoted shell string, so the
backticks in the log line were command substitution and ${hashes.length} was
the shell's expansion, not node's. The step aborted rather than validating,
which under bash -e failed the whole publish -- so it would have blocked every
hash rather than passing a bad one, but it would have blocked it on the first
build after merge.
Reads the hash from the environment and single-quotes the script instead.
Found by running the step against a local origin under Linux rather than
reading it. Four paths verified there: a new hash publishes and leaves the rest
of the directory intact, a repeat is a no-op, a malformed hash is refused with
nothing published, and two builds racing both land -- 180 entries from two
concurrent publishes where the previous implementation would have dropped one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The list is opaque md5s: nothing ties an entry to the commit, branch or release that produced it, and the fluxhashes commit message only records a sha -- which stops being resolvable when a branch is force-pushed or deleted after merge. The 2026-08-14 cull had to keep a bucket of entries it could not attribute for exactly this reason. Each publish now writes a row into src/hashes/provenance.json in the same commit that edits hashes.js: date, commit, branch, tag. Rows are created, never rewritten; a tag push finds its hash already listed and fills in the tag on the existing row. The record sits outside the signed payload, so nothing that consumes the list changes. The already-published early exit moves behind the edits: the decision is now "did anything change" rather than "is the hash present", which is what lets a tag annotate a row and lets a hash that predates the record gain one on its next republish. Verified against a local origin under Linux: a new hash lands with its row, a repeat is a no-op, a tag push annotates exactly once, a corrupt record is a red run with nothing pushed, and a rejected push recovers on the retry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cabecinha84
left a comment
There was a problem hiding this comment.
Verdict
Safe to ACK. The change is correct, fail-closed, and has already run in production on the branch itself. There is one
small hardening item I'd raise as a comment (non-blocking), plus two nits.
What it actually does
One file, .github/workflows/nodejs.yml, replacing the cpina/github-action-push-to-another-repository step for the hash
publish with an inline clone-and-edit script. The other use of that action (fluxjsdocs) is untouched, correctly —
there the destination is only the source.
The premise is proven, not just argued
The PR claims the old action wipes src/hashes/ on every build. That is verifiable in the live repo:
- ef5069ae (2026-08-17 08:02, author runonfluxbot — the new code, run from this branch with real secrets) added
src/hashes/provenance.json. - 76c77262 (2026-08-17 08:20, author RunOnFlux — the old cpina action, from another branch) shows removed
src/hashes/provenance.json +0/-10.
18 minutes. That is exactly the failure mode fluxhashes#2's signed document and sequence anchor would hit, fleet-wide
and silently. The dependency ordering in the PR description is real: this must merge before fluxhashes#2.
Also worth noting: the new step has already executed successfully against production fluxhashes (run 32008434407, step
Publish the tree hash to fluxhashes = success, whole job green, GitGuardian pass). And it exercised the non-trivial
path — the hash was already listed, so it pushed only the provenance row, which is the "a hash that predates the
record gains one on its next republish" branch.
Correctness review
I re-ran the sed + both node -e blocks locally against the real hashes.js: 199 → 200 entries, exactly one ]; anchor
in the file (so dropping the g flag is harmless), provenance written in the expected shape.
Traced every path:
┌────────────────────────────┬─────────────────────────────────────────────────────────────────┬──────────────────┐
│ path │ behaviour │ correct │
├────────────────────────────┼─────────────────────────────────────────────────────────────────┼──────────────────┤
│ new hash │ sed inserts, row created, commit, push │ ✅ │
├────────────────────────────┼─────────────────────────────────────────────────────────────────┼──────────────────┤
│ hash already listed, row │ staged diff empty → exit 0 │ ✅ │
│ exists │ │ │
├────────────────────────────┼─────────────────────────────────────────────────────────────────┼──────────────────┤
│ hash listed, no row │ row created, provenance-only commit │ ✅ (observed │
│ │ │ live) │
├────────────────────────────┼─────────────────────────────────────────────────────────────────┼──────────────────┤
│ tag push, row exists, tag: │ fills tag only, original date/commit/branch kept │ ✅ │
│ null │ │ │
├────────────────────────────┼─────────────────────────────────────────────────────────────────┼──────────────────┤
│ tag push, tag already set │ no-op │ ✅ │
├────────────────────────────┼─────────────────────────────────────────────────────────────────┼──────────────────┤
│ push race │ reset --hard origin/master discards the local commit, re-reads, │ ✅ │
│ │ re-decides │ │
├────────────────────────────┼─────────────────────────────────────────────────────────────────┼──────────────────┤
│ anchor pattern gone / sed │ validator throws "new hash is not in the patched file" → red, │ ✅ fail-closed │
│ no-op │ nothing pushed │ │
├────────────────────────────┼─────────────────────────────────────────────────────────────────┼──────────────────┤
│ corrupt provenance │ JSON.parse throws under bash -e → red, nothing pushed │ ✅ fail-closed │
└────────────────────────────┴─────────────────────────────────────────────────────────────────┴──────────────────┘
Details I checked rather than assumed:
- git fetch origin master does update refs/remotes/origin/master — clone writes the
+refs/heads/:refs/remotes/origin/ refspec, so the opportunistic update applies. The reset --hard origin/master is
not stale. - require("./src/hashes/hashes") resolves relative to cwd under node -e, and fluxhashes' package.json has no "type":
"module" — the CJS require is valid. - Default step shell is bash -e, so every unguarded failure aborts the step; the if !grep and if git push are in
condition position and correctly exempt. - GITHUB_REF_NAME is consumed via process.env inside a single-quoted node script — no shell interpolation of a branch
name, so no injection surface. This is the bug the second commit fixed, and it's fixed properly.
Integration with the other two PRs
Schema and format match fluxhashes#2 exactly, which matters because both write the same file:
- Both serialise as JSON.stringify(record, null, 2) + "\n" — no reformat churn, no ping-pong commits.
- flux CI touches only record.hashes; the signer only sets record.signed. Each preserves the other's keys through the
parse/stringify round-trip. - The row shape satisfies fluxhashes#2's validate.js: published YYYY-MM-DD ✅, commit 40-hex (GITHUB_SHA) ✅,
branch/tag null-or-string ✅.
No conflict. The pairing is coherent.
What I'd comment on (none blocking)
-
The PAT lands on disk in the workspace. git clone "https://x-access-token:${API_TOKEN_GITHUB}@..." writes the token
into fluxhashes/.git/config, inside $GITHUB_WORKSPACE, and it stays there for the rest of the job — through npm i,
which runs install scripts from the whole transitive dependency tree, and through codecov-action. The old action ran
in a container and didn't leave credentials in the workspace. One-line fix: clone into "$RUNNER_TEMP" instead, or rm
-rf fluxhashes at the end of the step. This is the only thing I'd actually ask for. -
"push raced with another build" is asserted, not diagnosed. Any push failure — expired PAT, revoked write access,
fluxhashes outage — prints that line three times and then a generic exit 1. Logging git's actual stderr on the failing
attempt would save a real debugging session later. -
The validator doesn't check nothing was lost. It confirms every entry is an md5 and that the new hash is present,
but an edit that dropped entries would pass. Given the PR's whole subject is "stop deleting things", a cheap superset
check against git show HEAD:src/hashes/hashes.js would close the loop. Low likelihood, but the check is nearly free.
Nits: the attempt loop variable is unused; there's no backoff between retries (3 attempts, fine in practice).
One pre-existing property worth flagging to the team (not this PR's fault)
The publish step sits before npm i and the test suite, exactly where it did before. So a tree hash is published as
known-good regardless of whether its tests pass, from any branch push (on: [push], unfiltered). That was tolerable
when the list was advisory; with #1788 gating releases on it, "in the list" starts to mean something stronger. Not a
reason to hold this PR — it changes the mechanism, not the policy — but the policy is worth a separate conversation.
Also: there's no if: github.repository == 'RunOnFlux/flux' guard, so a fork with Actions enabled gets three failed
pushes and a red build before its tests run. Roughly parity with the old behaviour, and forks have Actions off by
default, so it's minor.
|
Background. This PR fixed the publish step in place: an inline clone/edit/push replacing the directory-wiping action, plus a provenance row per publish. The review here (thank you @Cabecinha84 — the findings were real and shaped what follows) and a live rehearsal of the fix surfaced a deeper problem than the one it fixes: the published artifact has two writers with no transaction boundary, and the signing workflow in RunOnFlux/fluxhashes#2 signs whatever list it finds in that repository — so any credential able to write to fluxhashes can get a rewrite signed. Patching the publish step would leave that architecture standing. What replaces it. Publication inverts. flux CI stops writing to fluxhashes entirely and instead dispatches "commit X exists", with a token that can trigger that one workflow and nothing else. The fluxhashes signer fetches the commit from this public repository, computes the tree hash itself from the bytes it fetched, and publishes the list, the signed document and the provenance record in one atomic commit. The replacement lands as a revision of RunOnFlux/fluxhashes#2 plus a much smaller change here: the publish body deleted, a ~10-line dispatch step in its place. Where the review findings went:
Closing unmerged; the provenance-row schema from this PR lives on inside the signer. |
Background
fluxbench validates a FluxOS installation by checking its
ZelBacktree hash against a list of known-good hashes, which this repository's CI publishes into RunOnFlux/fluxhashes on every push. That list is becoming verifiable: fluxhashes#2 signs it with Ed25519, and #1788 gates releases on the signed copy. This PR is the publish side both of those depend on, and it merges first — the current publish mechanism would delete the files they introduce.The hash push uses
cpina/github-action-push-to-another-repository, which deletes the destination directory before copying:The source directory we push holds only
hashes.js, so every build removes anything else kept in fluxhashes'src/hashes/. Nothing else has lived there so far, but RunOnFlux/fluxhashes#2 adds two files there — the signed copy of the list and the provenance record that anchors its sequence — and it also silently drops a hash whenever two builds publish at once, leaving a red X on a branch build as the only trace.What changes
Clones fluxhashes and edits
hashes.jsin place, leaving the rest of the directory alone.origin/masterbefore deciding, so the step is idempotent: the loser of a race re-reads and either adds its hash to the new tip or exits because nothing is left to change.requireing it, so a bad edit would take the endpoint down. The step refuses to push ahashes.jsthat does not load, is not a list of md5s, or does not contain the hash just added — and a provenance record that does not parse or is missing the new row.wgetof the raw file, so it cannot act on a stale CDN copy.The provenance record
Alongside the hash, each publish writes a row into
src/hashes/provenance.jsonin the same commit: date, commit, branch, tag. The list itself is opaque md5s — nothing ties an entry to what produced it, and a commit referenced only by a commit message stops being resolvable when a branch is force-pushed or deleted after merge. The row is what keeps every entry attributable.Rows are created, never rewritten: a hash republished from another branch keeps its original attribution. The one later edit is a tag push — it finds its hash already listed and fills in the tag on the existing row. The record sits outside the signed payload, so nothing that consumes the list changes.
Tested
Against a local origin under Linux (GNU sed,
bash -e, node 20 — the shapeubuntu-latestgives) and end to end on real GitHub in throwaway private copies of both repositories, driving the fluxhashes#2 signing workflow behind it.src/hashes/left intactEvery hash CI computed in the live runs matched the locally computed value, and the resulting files load via
require/JSON.parse.Pairs with RunOnFlux/fluxhashes#2, which adds the files this protects. Merging this one first avoids the first signed document being wiped by the next build.
Not changed
The other use of that action — mirroring
ZelBack/src/servicesintofluxjsdocs— is left alone. There the destination holds only what is pushed, so replacing the directory is exactly what it should do. The action is not the problem; using it for a destination that holds more than the source is.🤖 Generated with Claude Code