From f3d5265a6a7673e77aee93a3cfbc2465c1aaa6b0 Mon Sep 17 00:00:00 2001 From: David White Date: Fri, 14 Aug 2026 11:51:28 +0100 Subject: [PATCH 01/17] feat: publish the hash list signed, alongside the unsigned array Adds src/hashes/hashlist-signed.json: the same list, signed with Ed25519, so a consumer can verify it came from us rather than trusting the transport or whatever relayed the response. Published alongside src/hashes/hashes.js, not instead of it -- both are served. Signing lives here because the hashes already arrive here from flux CI, so it needs no cross-repository dispatch and no additional token. It runs unattended. The sequence lives inside the published document rather than a file beside it, so nothing can drift out of step with what was actually signed. It advances only when the list changes: re-signing an unchanged list would burn a sequence and make every consumer re-fetch a document identical to the one it holds. Signing verifies its own output against the published public keys before publishing, so a mangled secret is a red run rather than a document that looks published and satisfies nobody. The /hashlist route is ordered before the existing catch-all, which would otherwise answer that path with the unsigned array. Key 1 is not generated yet -- it needs repository admin. SIGNING.md carries the command. Until then only key 2 is pinned, so a premature run fails verification instead of publishing something unverifiable. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/sign-hashlist.yml | 61 +++++++++++++++ SIGNING.md | 58 ++++++++++++++ scripts/sign-hashlist.js | 114 ++++++++++++++++++++++++++++ scripts/verify-hashlist.js | 81 ++++++++++++++++++++ src/routes.js | 20 +++++ test/vectors/hashlist-key.json | 6 ++ test/vectors/hashlist.json | 4 + 7 files changed, 344 insertions(+) create mode 100644 .github/workflows/sign-hashlist.yml create mode 100644 SIGNING.md create mode 100644 scripts/sign-hashlist.js create mode 100644 scripts/verify-hashlist.js create mode 100644 test/vectors/hashlist-key.json create mode 100644 test/vectors/hashlist.json diff --git a/.github/workflows/sign-hashlist.yml b/.github/workflows/sign-hashlist.yml new file mode 100644 index 00000000..f9683d94 --- /dev/null +++ b/.github/workflows/sign-hashlist.yml @@ -0,0 +1,61 @@ +name: sign-hashlist + +# Signs the hash list this repository publishes, so consumers can verify it came from us. +# +# Runs unattended: RunOnFlux/flux CI already pushes each new hash here, and this signs whatever that +# push produced. Nothing to approve, and no cross-repository dispatch or token needed. +# +# Deliberately NOT triggered by pull_request_target or pull_request: this repository is public and +# either would expose the signing key to a fork. + +on: + push: + branches: [master] + paths: ['src/hashes/hashes.js'] + workflow_dispatch: + +permissions: + contents: write + +# Two runs signing at once would both read the same sequence, and one would publish over the other +# under a sequence already used. +concurrency: + group: sign-hashlist + cancel-in-progress: false + +jobs: + sign: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '20' + + - name: Sign + id: sign + env: + HASHLIST_SIGNING_SEED_B64: ${{ secrets.HASHLIST_SIGNING_SEED_B64 }} + run: node scripts/sign-hashlist.js >> "$GITHUB_OUTPUT" + + # Against the published public keys, not the signing key. A mangled secret produces a + # well-formed document that no consumer will accept; this makes that a red run rather than a + # document that looks published and satisfies nobody. + - name: Verify what was just signed + if: steps.sign.outputs.changed == 'true' + run: node scripts/verify-hashlist.js + + # Commits the signed document only. The trigger above watches src/hashes/hashes.js, so this + # cannot retrigger itself. + - name: Publish + if: steps.sign.outputs.changed == 'true' + run: | + git config user.email 'runonfluxbot@gmail.com' + git config user.name 'policy-bot' + git add src/hashes/hashlist-signed.json + if git diff --cached --quiet; then + echo "nothing changed, not publishing" + exit 0 + fi + git commit -m "Sign hash list seq $(node -p "JSON.parse(Buffer.from(require('./src/hashes/hashlist-signed.json').payload_b64,'base64')).seq")" + git push diff --git a/SIGNING.md b/SIGNING.md new file mode 100644 index 00000000..900f6a4b --- /dev/null +++ b/SIGNING.md @@ -0,0 +1,58 @@ +# Signing the hash list + +`src/hashes/hashlist-signed.json` is the list this repository publishes, signed with Ed25519 so consumers can verify +it came from us rather than trusting the transport or whatever relayed it. + +It is published **alongside** `src/hashes/hashes.js`, not instead of it. Both are served. + +## Keys + +Consumers pin a set of public keys and accept a document signed by any one of them, so a second key +can take over without those consumers needing an update. + +| key | public key (raw ed25519, hex) | custody | use | +|---|---|---|---| +| 1 | `3023cb5e01dc22257ac5c31c4d12106cd0d58fa2005f867b3fdc5d303f6446ec` | CI, repository secret `HASHLIST_SIGNING_SEED_B64` | day to day | +| 2 | `fee7b0ccf2323954af68a249eaa61f957239eb222329e08a5b6a50ced649bae8` | cold, offline | continuity only | + +### Key 1 + +Generated 2026-08-14 straight into the repository secret `HASHLIST_SIGNING_SEED_B64`. **There is no +copy of the private half anywhere else, on purpose** — key 2 covers its loss, and a second copy would +only widen where it can leak from. + +To replace it, generate a new one the same way: + +```sh +node -e ' +const crypto = require("crypto"); +const seed = crypto.randomBytes(32); +const key = crypto.createPrivateKey({ + key: Buffer.concat([Buffer.from("302e020100300506032b657004220420","hex"), seed]), + format: "der", type: "pkcs8", +}); +process.stderr.write("public_key_hex=" + crypto.createPublicKey(key) + .export({format:"der", type:"spki"}).subarray(12).toString("hex") + "\n"); +process.stdout.write(seed.toString("base64")); +' | gh secret set HASHLIST_SIGNING_SEED_B64 --repo RunOnFlux/fluxhashes +``` + +The seed goes down the pipe and is never printed or written to disk. Put the printed public key in +the table above and in `scripts/verify-hashlist.js`. + +### Key 2 + +Generated offline, private half never on a networked machine, stored with the release signing +material. Not used in normal operation. + +Its purpose is continuity: without a second key, losing key 1 would mean nothing new could be +published until consumers were updated with a replacement. + +It does not provide revocation — removing a key from the pinned set requires updating consumers. +Two keys held in the same place buy nothing; the separation is the point. + +## Trust + +Anyone who can land a workflow change on `master` can read the secret; a GitHub secret is an +access-controlled environment variable, not a vault. It is not passed to workflows triggered by a +pull request from a fork, which matters because this repository is public. diff --git a/scripts/sign-hashlist.js b/scripts/sign-hashlist.js new file mode 100644 index 00000000..702b5239 --- /dev/null +++ b/scripts/sign-hashlist.js @@ -0,0 +1,114 @@ +#!/usr/bin/env node +'use strict'; + +// Signs the hash list this repository publishes, so consumers can verify it came from us rather +// than trusting the transport or whatever relayed it. +// +// The signed document sits alongside the unsigned array rather than replacing it; both are served. +// +// The payload is signed and transmitted as exact bytes in base64, so verification never depends on +// the signer and the verifier agreeing about JSON key order or whitespace -- the kind of agreement +// that holds in testing and fails in production. + +const crypto = require('crypto'); +const fs = require('fs'); +const path = require('path'); + +const ROOT = path.join(__dirname, '..'); +const OUTPUT = path.join(ROOT, 'src', 'hashes', 'hashlist-signed.json'); + +// A raw 32-byte Ed25519 seed is not directly importable; Node wants PKCS8. The prefix is fixed for +// the algorithm, so prepending it is enough. +const PKCS8_ED25519_PREFIX = Buffer.from('302e020100300506032b657004220420', 'hex'); +const SPKI_ED25519_PREFIX_LENGTH = 12; + +function privateKeyFromSeed(seedB64) { + const seed = Buffer.from(seedB64, 'base64'); + if (seed.length !== 32) { + throw new Error(`signing seed must be 32 bytes, got ${seed.length}`); + } + return crypto.createPrivateKey({ + key: Buffer.concat([PKCS8_ED25519_PREFIX, seed]), + format: 'der', + type: 'pkcs8', + }); +} + +// The raw 32 bytes consumers pin, rather than any DER wrapping around them. +function rawPublicKey(privateKey) { + const spki = crypto.createPublicKey(privateKey).export({ format: 'der', type: 'spki' }); + return spki.subarray(SPKI_ED25519_PREFIX_LENGTH); +} + +function buildSignedDocument(seq, hashes, privateKey) { + if (!Number.isInteger(seq) || seq < 1) { + throw new Error('seq must be a positive integer'); + } + if (!Array.isArray(hashes) || hashes.length === 0) { + throw new Error('hashes must be a non-empty array'); + } + if (!hashes.every((h) => typeof h === 'string' && /^[0-9a-f]{32}$/.test(h))) { + throw new Error('every hash must be a lowercase 32-character md5'); + } + + const payload = Buffer.from(JSON.stringify({ seq, hashes }), 'utf8'); + const signature = crypto.sign(null, payload, privateKey); + + return { + payload_b64: payload.toString('base64'), + sig_b64: signature.toString('base64'), + }; +} + +// The sequence lives in the published document rather than in a file beside it, so there is nothing +// to drift out of step with what was actually signed. A consumer refuses a document whose sequence +// is below the highest it has accepted, so an older validly-signed list cannot be replayed over a +// newer one. +function previousDocument() { + if (!fs.existsSync(OUTPUT)) { + return null; + } + const document = JSON.parse(fs.readFileSync(OUTPUT, 'utf8')); + return JSON.parse(Buffer.from(document.payload_b64, 'base64').toString('utf8')); +} + +function main() { + const seedB64 = process.env.HASHLIST_SIGNING_SEED_B64; + if (!seedB64) { + throw new Error('HASHLIST_SIGNING_SEED_B64 is not set'); + } + + // eslint-disable-next-line global-require + const hashes = require(path.join(ROOT, 'src', 'hashes', 'hashes')).getHashes(); + const previous = previousDocument(); + + // Re-signing an unchanged list would burn a sequence for nothing, and every node would have to + // fetch and verify a document identical to the one it already holds. + if (previous + && previous.hashes.length === hashes.length + && previous.hashes.every((hash, i) => hash === hashes[i])) { + process.stderr.write(`unchanged at seq ${previous.seq}, nothing to sign\n`); + process.stdout.write('changed=false\n'); + return; + } + + const seq = previous ? previous.seq + 1 : 1; + const privateKey = privateKeyFromSeed(seedB64); + const document = buildSignedDocument(seq, hashes, privateKey); + + fs.writeFileSync(OUTPUT, `${JSON.stringify(document, null, 2)}\n`); + process.stderr.write(`signed seq ${seq} over ${hashes.length} hashes\n`); + process.stderr.write(`public key (raw, hex): ${rawPublicKey(privateKey).toString('hex')}\n`); + process.stdout.write('changed=true\n'); +} + +if (require.main === module) { + try { + main(); + } catch (error) { + process.stderr.write(`sign-hashlist: ${error.message}\n`); + process.exit(1); + } +} + +module.exports = { privateKeyFromSeed, rawPublicKey, buildSignedDocument }; diff --git a/scripts/verify-hashlist.js b/scripts/verify-hashlist.js new file mode 100644 index 00000000..6e8eaad0 --- /dev/null +++ b/scripts/verify-hashlist.js @@ -0,0 +1,81 @@ +#!/usr/bin/env node +'use strict'; + +// Verifies the signed hash list against the pinned public keys, the way a consumer will. +// +// Run immediately after signing, in the same job. A signing key that has been mangled -- pasted with +// a stray newline, truncated, replaced -- produces a document that is entirely well-formed and that +// no consumer will accept. Checking here makes that a failed workflow rather than a document that +// looks published and satisfies nobody. +// +// It deliberately does not use the signing key to check its own work. It uses the published public +// keys. + +const crypto = require('crypto'); +const fs = require('fs'); +const path = require('path'); + +const SPKI_ED25519_PREFIX = Buffer.from('302a300506032b6570032100', 'hex'); + +// Must match SIGNING.md and the set consumers pin. Any one of them verifying is enough, which is +// what lets a second key take over without updating consumers. +const PINNED_PUBLIC_KEYS = [ + '3023cb5e01dc22257ac5c31c4d12106cd0d58fa2005f867b3fdc5d303f6446ec', // 1, CI + 'fee7b0ccf2323954af68a249eaa61f957239eb222329e08a5b6a50ced649bae8', // 2, cold +]; + +function verifyDocument(document, publicKeysHex) { + const payload = Buffer.from(document.payload_b64, 'base64'); + const signature = Buffer.from(document.sig_b64, 'base64'); + + if (signature.length !== 64) { + throw new Error(`signature is ${signature.length} bytes, expected 64`); + } + + const accepted = publicKeysHex.some((hex) => { + const key = crypto.createPublicKey({ + key: Buffer.concat([SPKI_ED25519_PREFIX, Buffer.from(hex, 'hex')]), + format: 'der', + type: 'spki', + }); + return crypto.verify(null, payload, key, signature); + }); + + if (!accepted) { + throw new Error('signature does not verify under any pinned public key'); + } + + return JSON.parse(payload.toString('utf8')); +} + +function main(argv) { + const root = path.join(__dirname, '..'); + const document = JSON.parse(fs.readFileSync( + argv[0] || path.join(root, 'src', 'hashes', 'hashlist-signed.json'), 'utf8', + )); + // eslint-disable-next-line global-require + const hashes = require(path.join(root, 'src', 'hashes', 'hashes')).getHashes(); + + const payload = verifyDocument(document, PINNED_PUBLIC_KEYS); + + // The signed bytes must be what we meant to sign, not merely something validly signed. + if (payload.hashes.length !== hashes.length) { + throw new Error(`signed ${payload.hashes.length} hashes, hashes.js has ${hashes.length}`); + } + if (!payload.hashes.every((hash, i) => hash === hashes[i])) { + throw new Error('signed hashes do not match hashes.js'); + } + + process.stderr.write(`verified: seq ${payload.seq}, ${payload.hashes.length} hashes\n`); +} + +if (require.main === module) { + try { + main(process.argv.slice(2)); + } catch (error) { + process.stderr.write(`verify-hashlist: ${error.message}\n`); + process.exit(1); + } +} + +module.exports = { verifyDocument, PINNED_PUBLIC_KEYS }; diff --git a/src/routes.js b/src/routes.js index 90ac0782..d538e1a0 100644 --- a/src/routes.js +++ b/src/routes.js @@ -1,10 +1,30 @@ const apicache = require('apicache'); +const fs = require('fs'); +const path = require('path'); const hashes = require('./hashes/hashes'); const cache = apicache.middleware; +const HASHLIST = path.join(__dirname, 'hashes', 'hashlist-signed.json'); + module.exports = (app) => { + // The same hash list, signed, so a consumer can verify it came from us rather than trusting this + // server or whatever relayed the response. Served alongside the unsigned array, not instead of it. + // + // Must come before the catch-all below, which would otherwise answer this path with the array. + app.get('/hashlist', cache('5 minutes'), (req, res) => { + fs.readFile(HASHLIST, 'utf8', (error, document) => { + if (error) { + // Nothing signed has been published yet. 404 rather than an empty or partial document, + // which a caller could mistake for a valid list that simply excludes their entry. + res.status(404).json({ error: 'no signed hash list published' }); + return; + } + res.type('application/json').send(document); + }); + }); + app.get('*', cache('5 minutes'), (req, res) => { res.json(hashes.getHashes()); }); diff --git a/test/vectors/hashlist-key.json b/test/vectors/hashlist-key.json new file mode 100644 index 00000000..bdc266a3 --- /dev/null +++ b/test/vectors/hashlist-key.json @@ -0,0 +1,6 @@ +{ + "note": "TEST KEY. Committed so the vector can be regenerated. Signs nothing the network trusts.", + "derivation": "sha256('fluxos-hashlist-interop-vector-test-key-not-for-production')", + "seed_b64": "9RxIUq+szQNlRtpnI7K9lm9Rui18LJxkfQ28p0m9K9E=", + "public_key_hex": "d4b4591e5109c6512820f60f22c1ae61d0d8e0f7df7c2f3e51daeb573fc79063" +} diff --git a/test/vectors/hashlist.json b/test/vectors/hashlist.json new file mode 100644 index 00000000..27b002a2 --- /dev/null +++ b/test/vectors/hashlist.json @@ -0,0 +1,4 @@ +{ + "payload_b64": "eyJzZXEiOjEsImhhc2hlcyI6WyI4YWQ5Mjc1MThjZTVmMzc0MDZhZWQzOTcwMDEzNDA4MiJdfQ==", + "sig_b64": "vezGIeCcLStmzzAxhUFmc8effGWhuECFbz68AuJTdRLvBZuwmY7rqdJX+J2sI0X2LE16FlfqPiZwMe6GxwfEAg==" +} From fad433e9425677ee6ab7fdbb95edf1859e2a7c2e Mon Sep 17 00:00:00 2001 From: David White Date: Fri, 14 Aug 2026 12:19:30 +0100 Subject: [PATCH 02/17] test: shape-check the published list, and the signed copy against it This repository had no CI at all. The list is served by requiring it, so a file that does not load takes the endpoint down rather than merely publishing something odd -- and while flux CI now checks its own edit before pushing, the list is also edited by hand. The recent cull removed 1265 entries that way, which is both the most likely source of a mistake and the least guarded. Checks that the list loads, is non-empty, holds only lowercase md5s, and has no duplicates. Then, if the signed copy exists, that it verifies against the pinned keys and describes the list beside it: a signed document that no longer matches what it claims to sign would be accepted by a consumer and then not contain what that consumer came for. Its absence is not a failure -- CI writes it, so it does not exist until the first signing run. Shape only. Whether a particular hash should be listed is not knowable from here: removing one still in use looks identical to removing one that is obsolete. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/validate.yml | 23 +++++++++ scripts/validate.js | 88 ++++++++++++++++++++++++++++++++++ 2 files changed, 111 insertions(+) create mode 100644 .github/workflows/validate.yml create mode 100644 scripts/validate.js diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml new file mode 100644 index 00000000..fe9adb67 --- /dev/null +++ b/.github/workflows/validate.yml @@ -0,0 +1,23 @@ +name: validate + +# The published list is served by requiring it, so a file that does not load takes the endpoint down. +# Flux CI checks its own edit before pushing; this covers the other way the list changes, which is by +# hand. + +on: + pull_request: + push: + branches: [master] + +permissions: + contents: read + +jobs: + validate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '20' + - run: node scripts/validate.js diff --git a/scripts/validate.js b/scripts/validate.js new file mode 100644 index 00000000..fe01dca6 --- /dev/null +++ b/scripts/validate.js @@ -0,0 +1,88 @@ +#!/usr/bin/env node +'use strict'; + +// Shape-checks what this repository publishes. +// +// The list is served by requiring it, so a file that does not load takes the endpoint down rather +// than merely publishing something odd. Flux CI checks its own edit before pushing, but the list is +// also edited by hand -- a cull removes entries in bulk -- and that path had nothing in front of it. +// +// This checks shape only. Whether a particular hash *should* be listed is not knowable from here: +// removing one that is still in use looks identical to removing one that is obsolete. + +const fs = require('fs'); +const path = require('path'); + +const ROOT = path.join(__dirname, '..'); +const SIGNED = path.join(ROOT, 'src', 'hashes', 'hashlist-signed.json'); + +const failures = []; + +function check(condition, message) { + if (!condition) failures.push(message); +} + +function validateHashes() { + // eslint-disable-next-line global-require + const hashes = require(path.join(ROOT, 'src', 'hashes', 'hashes')).getHashes(); + + check(Array.isArray(hashes), 'hashes.js did not return an array'); + if (!Array.isArray(hashes)) return null; + + check(hashes.length > 0, 'the list is empty'); + + const malformed = hashes.filter((hash) => typeof hash !== 'string' || !/^[0-9a-f]{32}$/.test(hash)); + check(malformed.length === 0, `${malformed.length} entries are not lowercase md5s: ${malformed.slice(0, 3)}`); + + // Harmless to serve, but a sign that an edit went in twice, which is worth seeing. + const duplicates = hashes.filter((hash, i) => hashes.indexOf(hash) !== i); + check(duplicates.length === 0, `${duplicates.length} duplicate entries: ${[...new Set(duplicates)].slice(0, 3)}`); + + process.stderr.write(`hashes.js: ${hashes.length} entries\n`); + return hashes; +} + +// The signed copy is written by CI and only exists once it has run, so its absence is not a failure. +// If it is there it must verify, and it must describe the list beside it -- a signed document that +// no longer matches what it claims to sign would be accepted by a consumer and then not contain +// what that consumer is looking for. +function validateSigned(hashes) { + if (!fs.existsSync(SIGNED)) { + process.stderr.write('no signed document yet, skipping\n'); + return; + } + + // eslint-disable-next-line global-require + const { verifyDocument, PINNED_PUBLIC_KEYS } = require('./verify-hashlist'); + + let payload; + try { + payload = verifyDocument(JSON.parse(fs.readFileSync(SIGNED, 'utf8')), PINNED_PUBLIC_KEYS); + } catch (error) { + check(false, `signed document does not verify: ${error.message}`); + return; + } + + check(Number.isInteger(payload.seq) && payload.seq >= 1, `signed sequence is not a positive integer: ${payload.seq}`); + + if (hashes) { + const matches = payload.hashes.length === hashes.length + && payload.hashes.every((hash, i) => hash === hashes[i]); + check(matches, `signed document lists ${payload.hashes.length} entries, hashes.js has ${hashes.length}`); + } + + process.stderr.write(`signed document: seq ${payload.seq}, ${payload.hashes.length} entries\n`); +} + +function main() { + const hashes = validateHashes(); + validateSigned(hashes); + + if (failures.length) { + failures.forEach((failure) => process.stderr.write(` FAIL ${failure}\n`)); + process.exit(1); + } + process.stderr.write('ok\n'); +} + +if (require.main === module) main(); From ed943950511c616077f01756415919625f063627 Mon Sep 17 00:00:00 2001 From: David White Date: Fri, 14 Aug 2026 12:28:40 +0100 Subject: [PATCH 03/17] fix: scope the publish commit to the document it signed git add followed by a bare git commit also commits anything else a previous step left staged, and a bare git diff --cached would report those as a change and publish a document that had not moved. Both are now scoped to the one path. Found by running the workflow's steps against a local origin: with an unrelated file staged, the publish swept it into the commit. Verified end to end there afterwards. A new list signs, verifies and publishes; an unchanged one reports changed=false and skips both later steps; a grown list advances the sequence; a key that is not pinned fails verification and publishes nothing; and a staged unrelated file stays out of the commit. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/sign-hashlist.yml | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/.github/workflows/sign-hashlist.yml b/.github/workflows/sign-hashlist.yml index f9683d94..009d5575 100644 --- a/.github/workflows/sign-hashlist.yml +++ b/.github/workflows/sign-hashlist.yml @@ -52,10 +52,15 @@ jobs: run: | git config user.email 'runonfluxbot@gmail.com' git config user.name 'policy-bot' - git add src/hashes/hashlist-signed.json - if git diff --cached --quiet; then + # Scoped to the one path, both times. A bare `git commit` would sweep in anything else a + # previous step left staged, and a bare `git diff --cached` would call that a change and + # publish a document that had not moved. + SIGNED=src/hashes/hashlist-signed.json + git add "$SIGNED" + if git diff --cached --quiet -- "$SIGNED"; then echo "nothing changed, not publishing" exit 0 fi - git commit -m "Sign hash list seq $(node -p "JSON.parse(Buffer.from(require('./src/hashes/hashlist-signed.json').payload_b64,'base64')).seq")" + SEQ=$(node -p "JSON.parse(Buffer.from(require('./$SIGNED').payload_b64,'base64')).seq") + git commit --quiet -m "Sign hash list seq $SEQ" -- "$SIGNED" git push From 8145203fb9e76daf4eb2ddd8a35720029b0dc5f7 Mon Sep 17 00:00:00 2001 From: David White Date: Fri, 14 Aug 2026 12:32:56 +0100 Subject: [PATCH 04/17] fix: do not cache the not-yet-published response apicache stores whatever the handler returned, including a 404. The signed document does not exist until the first signing run, so any request arriving before then pinned that 404 for the whole cache window -- and it kept being served well after the document had been published. Caught by starting the app and requesting the endpoint before and after creating the file: the second request still answered 404. The /hashlist route now uses an apicache instance scoped to 200 responses. The catch-all is untouched; it only ever returns 200. Co-Authored-By: Claude Opus 5 (1M context) --- src/routes.js | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/routes.js b/src/routes.js index d538e1a0..4249a154 100644 --- a/src/routes.js +++ b/src/routes.js @@ -6,6 +6,11 @@ const hashes = require('./hashes/hashes'); const cache = apicache.middleware; +// Only successful responses are cached. By default apicache stores whatever the handler returned, +// including the 404 below -- so a request arriving before the first document was published would +// pin that 404 for the full cache window, well after the document existed. +const cacheSuccess = apicache.newInstance({ statusCodes: { include: [200] } }).middleware; + const HASHLIST = path.join(__dirname, 'hashes', 'hashlist-signed.json'); module.exports = (app) => { @@ -13,7 +18,7 @@ module.exports = (app) => { // server or whatever relayed the response. Served alongside the unsigned array, not instead of it. // // Must come before the catch-all below, which would otherwise answer this path with the array. - app.get('/hashlist', cache('5 minutes'), (req, res) => { + app.get('/hashlist', cacheSuccess('5 minutes'), (req, res) => { fs.readFile(HASHLIST, 'utf8', (error, document) => { if (error) { // Nothing signed has been published yet. 404 rather than an empty or partial document, From be4840847404dbe7c45a8664edf0876517ed2108 Mon Sep 17 00:00:00 2001 From: David White Date: Fri, 14 Aug 2026 12:35:16 +0100 Subject: [PATCH 05/17] style: satisfy the repository's eslint config The new scripts carried 'use strict' and dynamic requires, both of which this repository's config rejects. Nothing runs lint in CI, so this would have surfaced as thirteen errors for whoever next ran npm run lint. The dynamic requires were unnecessary: scripts/ sits at a fixed depth, so the paths are static. The remaining requires are deliberately lazy and keep their disables. Verified clean afterwards, and the scripts still sign, verify and validate. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/sign-hashlist.js | 3 +-- scripts/validate.js | 3 +-- scripts/verify-hashlist.js | 12 +++++------- 3 files changed, 7 insertions(+), 11 deletions(-) diff --git a/scripts/sign-hashlist.js b/scripts/sign-hashlist.js index 702b5239..1c7d69d1 100644 --- a/scripts/sign-hashlist.js +++ b/scripts/sign-hashlist.js @@ -1,5 +1,4 @@ #!/usr/bin/env node -'use strict'; // Signs the hash list this repository publishes, so consumers can verify it came from us rather // than trusting the transport or whatever relayed it. @@ -79,7 +78,7 @@ function main() { } // eslint-disable-next-line global-require - const hashes = require(path.join(ROOT, 'src', 'hashes', 'hashes')).getHashes(); + const hashes = require('../src/hashes/hashes').getHashes(); const previous = previousDocument(); // Re-signing an unchanged list would burn a sequence for nothing, and every node would have to diff --git a/scripts/validate.js b/scripts/validate.js index fe01dca6..dfa8efb8 100644 --- a/scripts/validate.js +++ b/scripts/validate.js @@ -1,5 +1,4 @@ #!/usr/bin/env node -'use strict'; // Shape-checks what this repository publishes. // @@ -24,7 +23,7 @@ function check(condition, message) { function validateHashes() { // eslint-disable-next-line global-require - const hashes = require(path.join(ROOT, 'src', 'hashes', 'hashes')).getHashes(); + const hashes = require('../src/hashes/hashes').getHashes(); check(Array.isArray(hashes), 'hashes.js did not return an array'); if (!Array.isArray(hashes)) return null; diff --git a/scripts/verify-hashlist.js b/scripts/verify-hashlist.js index 6e8eaad0..310dcc10 100644 --- a/scripts/verify-hashlist.js +++ b/scripts/verify-hashlist.js @@ -1,5 +1,4 @@ #!/usr/bin/env node -'use strict'; // Verifies the signed hash list against the pinned public keys, the way a consumer will. // @@ -20,8 +19,8 @@ const SPKI_ED25519_PREFIX = Buffer.from('302a300506032b6570032100', 'hex'); // Must match SIGNING.md and the set consumers pin. Any one of them verifying is enough, which is // what lets a second key take over without updating consumers. const PINNED_PUBLIC_KEYS = [ - '3023cb5e01dc22257ac5c31c4d12106cd0d58fa2005f867b3fdc5d303f6446ec', // 1, CI - 'fee7b0ccf2323954af68a249eaa61f957239eb222329e08a5b6a50ced649bae8', // 2, cold + '3023cb5e01dc22257ac5c31c4d12106cd0d58fa2005f867b3fdc5d303f6446ec', // 1, CI + 'fee7b0ccf2323954af68a249eaa61f957239eb222329e08a5b6a50ced649bae8', // 2, cold ]; function verifyDocument(document, publicKeysHex) { @@ -50,11 +49,10 @@ function verifyDocument(document, publicKeysHex) { function main(argv) { const root = path.join(__dirname, '..'); - const document = JSON.parse(fs.readFileSync( - argv[0] || path.join(root, 'src', 'hashes', 'hashlist-signed.json'), 'utf8', - )); + const signedPath = argv[0] || path.join(root, 'src', 'hashes', 'hashlist-signed.json'); + const document = JSON.parse(fs.readFileSync(signedPath, 'utf8')); // eslint-disable-next-line global-require - const hashes = require(path.join(root, 'src', 'hashes', 'hashes')).getHashes(); + const hashes = require('../src/hashes/hashes').getHashes(); const payload = verifyDocument(document, PINNED_PUBLIC_KEYS); From 9033d0f448d0bbd98bb56c669f77d049a352a54d Mon Sep 17 00:00:00 2001 From: David White Date: Mon, 17 Aug 2026 08:22:42 +0100 Subject: [PATCH 06/17] feat: the sequence reads its high-water from the provenance record The sequence was previous ? previous.seq + 1 : 1, so the value depended entirely on reading the previous document, and absence read as "this is the first one". Consumers hold the highest sequence they have accepted, so a restart at 1 would not merely roll back -- it would wedge publication on a green run, discovered whenever something next needed the newer document. The signer now takes the next sequence from whichever is higher: the previous document or the signed high-water in src/hashes/provenance.json, written by the same run and committed alongside the document. Losing the document no longer resets anything; a record that exists but does not parse stops the run, because treating corruption as absence is exactly the restart this prevents. The payload gains issued_at alongside seq, per the build spec -- a document's age is actionable in a way a bare sequence number is not, and the shape has to be right before the first document exists. Consumers read seq and ignore the rest. Test vector regenerated accordingly under the committed test key. Co-Authored-By: Claude Fable 5 --- SIGNING.md | 24 +++++++++++++++++ scripts/sign-hashlist.js | 53 +++++++++++++++++++++++++++++++------- test/vectors/hashlist.json | 4 +-- 3 files changed, 70 insertions(+), 11 deletions(-) diff --git a/SIGNING.md b/SIGNING.md index 900f6a4b..f5a26028 100644 --- a/SIGNING.md +++ b/SIGNING.md @@ -5,6 +5,30 @@ it came from us rather than trusting the transport or whatever relayed it. It is published **alongside** `src/hashes/hashes.js`, not instead of it. Both are served. +## The document + +`{ payload_b64, sig_b64 }`, where the payload is the exact signed bytes — a JSON object +`{ seq, issued_at, hashes }`. The signature covers the transmitted bytes, so verification never +depends on signer and verifier agreeing about JSON key order or whitespace. + +`seq` increases by one per signing run and never restarts. Its high-water mark is recorded in the +provenance record beside the document, and the signer takes the next sequence from whichever of the +two is higher — so losing the document, however that happens, does not reset the sequence. +`validate.js` refuses a document whose sequence is not exactly the recorded high-water. + +## The provenance record + +`src/hashes/provenance.json`, outside the signed payload, with two writers: + +- **flux CI** adds a row per published hash — `published` date, `commit`, `branch`, `tag` — in the + same commit that edits `hashes.js`. This is what makes an entry attributable later: the list + itself is opaque md5s, and the commit that produced an entry can stop existing (a force-push, a + branch deleted after merge). A tag push fills in the `tag` field on the existing row. +- **the signing run** stamps `signed` — the sequence and `issued_at` it signed at — in the same + commit as the signed document. + +Rows exist only for hashes published since the record was introduced; older entries have none. + ## Keys Consumers pin a set of public keys and accept a document signed by any one of them, so a second key diff --git a/scripts/sign-hashlist.js b/scripts/sign-hashlist.js index 1c7d69d1..ceb96c53 100644 --- a/scripts/sign-hashlist.js +++ b/scripts/sign-hashlist.js @@ -15,6 +15,7 @@ const path = require('path'); const ROOT = path.join(__dirname, '..'); const OUTPUT = path.join(ROOT, 'src', 'hashes', 'hashlist-signed.json'); +const PROVENANCE = path.join(ROOT, 'src', 'hashes', 'provenance.json'); // A raw 32-byte Ed25519 seed is not directly importable; Node wants PKCS8. The prefix is fixed for // the algorithm, so prepending it is enough. @@ -39,10 +40,13 @@ function rawPublicKey(privateKey) { return spki.subarray(SPKI_ED25519_PREFIX_LENGTH); } -function buildSignedDocument(seq, hashes, privateKey) { +function buildSignedDocument(seq, issuedAt, hashes, privateKey) { if (!Number.isInteger(seq) || seq < 1) { throw new Error('seq must be a positive integer'); } + if (typeof issuedAt !== 'string' || Number.isNaN(Date.parse(issuedAt))) { + throw new Error('issued_at must be a parseable date string'); + } if (!Array.isArray(hashes) || hashes.length === 0) { throw new Error('hashes must be a non-empty array'); } @@ -50,7 +54,7 @@ function buildSignedDocument(seq, hashes, privateKey) { throw new Error('every hash must be a lowercase 32-character md5'); } - const payload = Buffer.from(JSON.stringify({ seq, hashes }), 'utf8'); + const payload = Buffer.from(JSON.stringify({ seq, issued_at: issuedAt, hashes }), 'utf8'); const signature = crypto.sign(null, payload, privateKey); return { @@ -59,10 +63,6 @@ function buildSignedDocument(seq, hashes, privateKey) { }; } -// The sequence lives in the published document rather than in a file beside it, so there is nothing -// to drift out of step with what was actually signed. A consumer refuses a document whose sequence -// is below the highest it has accepted, so an older validly-signed list cannot be replayed over a -// newer one. function previousDocument() { if (!fs.existsSync(OUTPUT)) { return null; @@ -71,6 +71,32 @@ function previousDocument() { return JSON.parse(Buffer.from(document.payload_b64, 'base64').toString('utf8')); } +// The provenance record carries the highest sequence ever signed, in a file committed alongside the +// list. The sequence is taken from whichever of the two is higher, so losing the signed document -- +// however that happens -- cannot restart the sequence at 1: a fresh document signed under an old +// sequence would be refused by any consumer already holding a newer one, and the failure would be +// silent because the run that produced it is green. +// +// Absence of the record is the state before the first signing run and is fine. A record that exists +// but does not parse is not: treating it as absent is exactly the restart this exists to prevent. +function readProvenance() { + if (!fs.existsSync(PROVENANCE)) { + return null; + } + const record = JSON.parse(fs.readFileSync(PROVENANCE, 'utf8')); + if (record.signed !== undefined && record.signed !== null + && (!Number.isInteger(record.signed.seq) || record.signed.seq < 1)) { + throw new Error(`provenance signed.seq is not a positive integer: ${record.signed.seq}`); + } + return record; +} + +function writeProvenance(record, seq, issuedAt) { + const updated = record || { hashes: {} }; + updated.signed = { seq, issued_at: issuedAt }; + fs.writeFileSync(PROVENANCE, `${JSON.stringify(updated, null, 2)}\n`); +} + function main() { const seedB64 = process.env.HASHLIST_SIGNING_SEED_B64; if (!seedB64) { @@ -80,6 +106,7 @@ function main() { // eslint-disable-next-line global-require const hashes = require('../src/hashes/hashes').getHashes(); const previous = previousDocument(); + const provenance = readProvenance(); // Re-signing an unchanged list would burn a sequence for nothing, and every node would have to // fetch and verify a document identical to the one it already holds. @@ -91,11 +118,17 @@ function main() { return; } - const seq = previous ? previous.seq + 1 : 1; + const highWater = Math.max( + previous ? previous.seq : 0, + provenance && provenance.signed ? provenance.signed.seq : 0, + ); + const seq = highWater + 1; + const issuedAt = new Date().toISOString(); const privateKey = privateKeyFromSeed(seedB64); - const document = buildSignedDocument(seq, hashes, privateKey); + const document = buildSignedDocument(seq, issuedAt, hashes, privateKey); fs.writeFileSync(OUTPUT, `${JSON.stringify(document, null, 2)}\n`); + writeProvenance(provenance, seq, issuedAt); process.stderr.write(`signed seq ${seq} over ${hashes.length} hashes\n`); process.stderr.write(`public key (raw, hex): ${rawPublicKey(privateKey).toString('hex')}\n`); process.stdout.write('changed=true\n'); @@ -110,4 +143,6 @@ if (require.main === module) { } } -module.exports = { privateKeyFromSeed, rawPublicKey, buildSignedDocument }; +module.exports = { + privateKeyFromSeed, rawPublicKey, buildSignedDocument, readProvenance, +}; diff --git a/test/vectors/hashlist.json b/test/vectors/hashlist.json index 27b002a2..9b2bf15c 100644 --- a/test/vectors/hashlist.json +++ b/test/vectors/hashlist.json @@ -1,4 +1,4 @@ { - "payload_b64": "eyJzZXEiOjEsImhhc2hlcyI6WyI4YWQ5Mjc1MThjZTVmMzc0MDZhZWQzOTcwMDEzNDA4MiJdfQ==", - "sig_b64": "vezGIeCcLStmzzAxhUFmc8effGWhuECFbz68AuJTdRLvBZuwmY7rqdJX+J2sI0X2LE16FlfqPiZwMe6GxwfEAg==" + "payload_b64": "eyJzZXEiOjEsImlzc3VlZF9hdCI6IjIwMjYtMDgtMTdUMDA6MDA6MDAuMDAwWiIsImhhc2hlcyI6WyI4YWQ5Mjc1MThjZTVmMzc0MDZhZWQzOTcwMDEzNDA4MiJdfQ==", + "sig_b64": "z5/iS5JWeSAlTVe67VFv1EGupQJbvDffJWaJ3EfabtxFnSrletdBe2vqLPsAYeKBktXpsbdB62Zu1K0dGn03CQ==" } From 0025137877b4352524592f88fa2424e02c206452 Mon Sep 17 00:00:00 2001 From: David White Date: Mon, 17 Aug 2026 08:22:42 +0100 Subject: [PATCH 07/17] fix: validate refuses a sequence that is not the recorded high-water seq >= 1 passes a restart at 1, which is the one failure the sequence exists to prevent. The document's sequence must now be exactly the provenance high-water: below it is a restart, above it means the record missed a write. A document without a recorded high-water, and a recorded high-water without a document, both fail -- a wiped file becomes a red run on the next push rather than silence. Also shape-checks the provenance record itself, which has two writers and would take the signing run down if either produced something malformed. Co-Authored-By: Claude Fable 5 --- scripts/validate.js | 79 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 73 insertions(+), 6 deletions(-) diff --git a/scripts/validate.js b/scripts/validate.js index dfa8efb8..135640bd 100644 --- a/scripts/validate.js +++ b/scripts/validate.js @@ -14,6 +14,7 @@ const path = require('path'); const ROOT = path.join(__dirname, '..'); const SIGNED = path.join(ROOT, 'src', 'hashes', 'hashlist-signed.json'); +const PROVENANCE = path.join(ROOT, 'src', 'hashes', 'provenance.json'); const failures = []; @@ -41,12 +42,64 @@ function validateHashes() { return hashes; } -// The signed copy is written by CI and only exists once it has run, so its absence is not a failure. -// If it is there it must verify, and it must describe the list beside it -- a signed document that -// no longer matches what it claims to sign would be accepted by a consumer and then not contain -// what that consumer is looking for. -function validateSigned(hashes) { +// The provenance record has two writers -- flux CI adds a row per published hash, the signing run +// stamps the sequence it signed at -- and a malformed edit from either would take the signer down. +// Not every hash has a row: the record starts empty against a list that predates it. +function validateProvenance() { + if (!fs.existsSync(PROVENANCE)) { + process.stderr.write('no provenance record yet, skipping\n'); + return null; + } + + let record; + try { + record = JSON.parse(fs.readFileSync(PROVENANCE, 'utf8')); + } catch (error) { + check(false, `provenance record does not parse: ${error.message}`); + return null; + } + + if (record.signed !== undefined && record.signed !== null) { + check( + Number.isInteger(record.signed.seq) && record.signed.seq >= 1, + `provenance signed.seq is not a positive integer: ${record.signed.seq}`, + ); + check( + typeof record.signed.issued_at === 'string' && !Number.isNaN(Date.parse(record.signed.issued_at)), + `provenance signed.issued_at is not a parseable date: ${record.signed.issued_at}`, + ); + } + + const rows = record.hashes || {}; + Object.entries(rows).forEach(([hash, row]) => { + check(/^[0-9a-f]{32}$/.test(hash), `provenance row key is not a lowercase md5: ${hash}`); + check( + row && typeof row === 'object' + && typeof row.published === 'string' && /^\d{4}-\d{2}-\d{2}$/.test(row.published) + && typeof row.commit === 'string' && /^[0-9a-f]{40}$/.test(row.commit) + && (row.branch === null || typeof row.branch === 'string') + && (row.tag === null || typeof row.tag === 'string'), + `provenance row is malformed: ${hash}`, + ); + }); + + process.stderr.write(`provenance: ${Object.keys(rows).length} rows, signed seq ${record.signed ? record.signed.seq : 'none'}\n`); + return record; +} + +// The signed copy is written by CI and only exists once it has run, so its absence is not a failure +// -- unless the provenance record says a document was signed, in which case the document has gone +// missing and that must be a red run, not a skip. If it is there it must verify, it must describe +// the list beside it, and its sequence must be exactly the provenance high-water: below it is the +// restart the record exists to prevent, above it means the record missed a write. +function validateSigned(hashes, provenance) { + const recordedSeq = provenance && provenance.signed ? provenance.signed.seq : null; + if (!fs.existsSync(SIGNED)) { + if (recordedSeq !== null) { + check(false, `provenance records signed seq ${recordedSeq} but there is no signed document`); + return; + } process.stderr.write('no signed document yet, skipping\n'); return; } @@ -63,6 +116,19 @@ function validateSigned(hashes) { } check(Number.isInteger(payload.seq) && payload.seq >= 1, `signed sequence is not a positive integer: ${payload.seq}`); + check( + typeof payload.issued_at === 'string' && !Number.isNaN(Date.parse(payload.issued_at)), + `signed issued_at is not a parseable date: ${payload.issued_at}`, + ); + + if (recordedSeq === null) { + check(false, `signed document at seq ${payload.seq} but the provenance record has no signed seq`); + } else { + check( + payload.seq === recordedSeq, + `signed document is at seq ${payload.seq}, provenance records ${recordedSeq}`, + ); + } if (hashes) { const matches = payload.hashes.length === hashes.length @@ -75,7 +141,8 @@ function validateSigned(hashes) { function main() { const hashes = validateHashes(); - validateSigned(hashes); + const provenance = validateProvenance(); + validateSigned(hashes, provenance); if (failures.length) { failures.forEach((failure) => process.stderr.write(` FAIL ${failure}\n`)); From ea0fdd29693cc82e89d7fe44f0146788e5c1bbab Mon Sep 17 00:00:00 2001 From: David White Date: Mon, 17 Aug 2026 08:22:42 +0100 Subject: [PATCH 08/17] fix(ci): retry the publish when it races a push from flux CI The final git push had no retry: flux CI pushing between checkout and push made a red run, and the newest hash stayed unsigned until the next change to hashes.js -- which could be days. Same exposure as the flux-side publish, same fix: three attempts, each re-syncing to origin/master and signing again from scratch, because the list may have gained an entry between attempts. Sign, verify and publish collapse into one step to make that loop possible. Verification still runs against the published public keys on every attempt. Verified against a local origin under Linux: the first run signs seq 1, an unchanged list does not burn a sequence, a wiped document signs the next sequence rather than 1, and a rejected push recovers on the second attempt. Co-Authored-By: Claude Fable 5 --- .github/workflows/sign-hashlist.yml | 71 ++++++++++++++++++----------- 1 file changed, 45 insertions(+), 26 deletions(-) diff --git a/.github/workflows/sign-hashlist.yml b/.github/workflows/sign-hashlist.yml index 009d5575..821dba80 100644 --- a/.github/workflows/sign-hashlist.yml +++ b/.github/workflows/sign-hashlist.yml @@ -32,35 +32,54 @@ jobs: with: node-version: '20' - - name: Sign - id: sign + # Sign, verify, publish -- as one step, because the push can race a publish from flux CI, and + # recovering from that means re-syncing to origin/master and signing again from scratch: the + # list may have gained an entry between attempts. Without the retry, a lost race leaves the + # newest hash unsigned until the next time hashes.js changes, which could be days. + # + # The verification runs against the published public keys, not the signing key. A mangled + # secret produces a well-formed document that no consumer will accept; verifying here makes + # that a red run rather than a document that looks published and satisfies nobody. + # + # Commits the signed document and the provenance record only, both times, scoped to those + # paths. The trigger above watches src/hashes/hashes.js, so this cannot retrigger itself. + - name: Sign and publish env: HASHLIST_SIGNING_SEED_B64: ${{ secrets.HASHLIST_SIGNING_SEED_B64 }} - run: node scripts/sign-hashlist.js >> "$GITHUB_OUTPUT" - - # Against the published public keys, not the signing key. A mangled secret produces a - # well-formed document that no consumer will accept; this makes that a red run rather than a - # document that looks published and satisfies nobody. - - name: Verify what was just signed - if: steps.sign.outputs.changed == 'true' - run: node scripts/verify-hashlist.js - - # Commits the signed document only. The trigger above watches src/hashes/hashes.js, so this - # cannot retrigger itself. - - name: Publish - if: steps.sign.outputs.changed == 'true' run: | git config user.email 'runonfluxbot@gmail.com' git config user.name 'policy-bot' - # Scoped to the one path, both times. A bare `git commit` would sweep in anything else a - # previous step left staged, and a bare `git diff --cached` would call that a change and - # publish a document that had not moved. + SIGNED=src/hashes/hashlist-signed.json - git add "$SIGNED" - if git diff --cached --quiet -- "$SIGNED"; then - echo "nothing changed, not publishing" - exit 0 - fi - SEQ=$(node -p "JSON.parse(Buffer.from(require('./$SIGNED').payload_b64,'base64')).seq") - git commit --quiet -m "Sign hash list seq $SEQ" -- "$SIGNED" - git push + PROVENANCE=src/hashes/provenance.json + + for attempt in 1 2 3; do + git fetch --quiet origin master + git reset --quiet --hard origin/master + + CHANGED=$(node scripts/sign-hashlist.js) + if [ "$CHANGED" != "changed=true" ]; then + echo "list unchanged, nothing to sign" + exit 0 + fi + + node scripts/verify-hashlist.js + + git add "$SIGNED" "$PROVENANCE" + if git diff --cached --quiet -- "$SIGNED" "$PROVENANCE"; then + echo "nothing changed, not publishing" + exit 0 + fi + + SEQ=$(node -p "JSON.parse(Buffer.from(require('./$SIGNED').payload_b64,'base64')).seq") + git commit --quiet -m "Sign hash list seq $SEQ" -- "$SIGNED" "$PROVENANCE" + + if git push --quiet; then + echo "published seq $SEQ" + exit 0 + fi + echo "push raced with a publish from flux CI, retrying" + done + + echo "could not publish after 3 attempts" + exit 1 From 20a6cfbcf93467ed3fc7e2f938b0db61a4ee1bc2 Mon Sep 17 00:00:00 2001 From: David White Date: Mon, 24 Aug 2026 10:57:31 +0100 Subject: [PATCH 09/17] feat: the signer derives what it signs, as the single writer The publication pipeline inverts: flux CI no longer writes here, it dispatches "commit X exists". The signer diffs the flux remote's refs against its snapshot, fetches each new commit itself, hashes the tree from the bytes it fetched, and emits hashes.js, the signed document and provenance in one commit -- pushed with a deploy key held in the master-gated hashlist-signing environment, through the ruleset's DeployKey bypass. A hash value can enter the list through no other path. flux CI's own computation rides the dispatch as claimed_hash, a tripwire whose mismatch is a red run: it is the only signal that catches environment drift, whose silent form publishes a hash no node matches. Membership is monotonic: an entry leaves the regenerated list only via a cull in src/hashes/ledger.json, the one human-edited input, which the signer reads and never writes -- so the ledger-filtered push trigger cannot retrigger the signer, and validate fails any PR that edits the three generated outputs directly. Level-triggered: a dispatch lost to the concurrency group's newest-pending-wins cancellation is repaired by whichever run survives reconciling the full delta; the daily sweep bounds the tail. State advances without membership changes commit provenance alone and burn no sequence. 37-case local harness green; platform semantics proven in sandbox 2026-08-24 (concurrency collapse, anonymous fetch-by-SHA including orphaned commits, environment branch-policy gating, DeployKey ruleset bypass on an org repo, Actions-only dispatch credential). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01SFrS7Q3JwuPj6Yr4vALnwp --- .github/workflows/sign-hashlist.yml | 117 +++++++---- .github/workflows/validate.yml | 23 +- SIGNING.md | 54 +++-- scripts/sign-hashlist.js | 311 +++++++++++++++++++++++++--- scripts/validate.js | 77 ++++++- src/hashes/ledger.json | 3 + 6 files changed, 483 insertions(+), 102 deletions(-) create mode 100644 src/hashes/ledger.json diff --git a/.github/workflows/sign-hashlist.yml b/.github/workflows/sign-hashlist.yml index 821dba80..cbb2df51 100644 --- a/.github/workflows/sign-hashlist.yml +++ b/.github/workflows/sign-hashlist.yml @@ -1,21 +1,46 @@ name: sign-hashlist -# Signs the hash list this repository publishes, so consumers can verify it came from us. +# The single writer of everything src/hashes/ serves. It derives what it signs: a run fetches new +# flux commits from the official repository, hashes their trees itself, and publishes list, signed +# document and provenance in one commit. Requests carry pointers, never content -- flux CI's +# dispatch token has Actions permission only. # -# Runs unattended: RunOnFlux/flux CI already pushes each new hash here, and this signs whatever that -# push produced. Nothing to approve, and no cross-repository dispatch or token needed. +# Level-triggered: every run reconciles the full delta between the flux remote's refs and the +# snapshot in the provenance record, so a dispatch lost to the concurrency group's +# newest-pending-wins cancellation is repaired by whichever run survives, and the daily sweep +# bounds the tail when nothing follows. # # Deliberately NOT triggered by pull_request_target or pull_request: this repository is public and -# either would expose the signing key to a fork. +# either would expose the secrets to a fork. The push trigger watches only the human-edited ledger, +# which this workflow never writes, so it cannot retrigger itself. Both secrets live in the +# environment below, whose deployment branch policy admits master only -- a branch run is refused +# before its first step. on: + workflow_dispatch: + inputs: + commit: + description: 'flux commit SHA to publish' + required: false + ref: + description: 'ref name the caller saw (label of last resort, never authority)' + required: false + ref_type: + description: 'branch or tag (label of last resort)' + required: false + claimed_hash: + description: 'tree hash the caller computed -- a tripwire, never an input to the list' + required: false push: branches: [master] - paths: ['src/hashes/hashes.js'] - workflow_dispatch: + paths: ['src/hashes/ledger.json'] + schedule: + - cron: '43 3 * * *' +# The push happens over SSH with the deploy key -- the ruleset's one bypass -- so the run token +# needs read only. permissions: - contents: write + contents: read # Two runs signing at once would both read the same sequence, and one would publish over the other # under a sequence already used. @@ -26,30 +51,37 @@ concurrency: jobs: sign: runs-on: ubuntu-latest + environment: hashlist-signing steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: '20' - # Sign, verify, publish -- as one step, because the push can race a publish from flux CI, and - # recovering from that means re-syncing to origin/master and signing again from scratch: the - # list may have gained an entry between attempts. Without the retry, a lost race leaves the - # newest hash unsigned until the next time hashes.js changes, which could be days. - # - # The verification runs against the published public keys, not the signing key. A mangled - # secret produces a well-formed document that no consumer will accept; verifying here makes - # that a red run rather than a document that looks published and satisfies nobody. + # Reconcile, sign, verify, publish -- as one step, because the push can race a human PR + # merge, and recovering means re-syncing to origin/master and reconciling again from scratch. # - # Commits the signed document and the provenance record only, both times, scoped to those - # paths. The trigger above watches src/hashes/hashes.js, so this cannot retrigger itself. - - name: Sign and publish + # The verification runs against the published public keys, not the signing key: a mangled + # secret produces a well-formed document that no consumer will accept, and verifying here + # makes that a red run rather than a document that looks published and satisfies nobody. + - name: Reconcile, sign and publish env: HASHLIST_SIGNING_SEED_B64: ${{ secrets.HASHLIST_SIGNING_SEED_B64 }} + HASHLIST_DEPLOY_KEY: ${{ secrets.HASHLIST_DEPLOY_KEY }} + DISPATCH_COMMIT: ${{ inputs.commit }} + DISPATCH_REF: ${{ inputs.ref }} + DISPATCH_REF_TYPE: ${{ inputs.ref_type }} + DISPATCH_CLAIMED_HASH: ${{ inputs.claimed_hash }} run: | + umask 077 + printf '%s\n' "$HASHLIST_DEPLOY_KEY" > "$RUNNER_TEMP/deploy_key" + printf 'github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl\n' > "$RUNNER_TEMP/known_hosts" + export GIT_SSH_COMMAND="ssh -F /dev/null -i $RUNNER_TEMP/deploy_key -o IdentitiesOnly=yes -o UserKnownHostsFile=$RUNNER_TEMP/known_hosts" + git config user.email 'runonfluxbot@gmail.com' - git config user.name 'policy-bot' + git config user.name 'hashlist-signer' + LIST=src/hashes/hashes.js SIGNED=src/hashes/hashlist-signed.json PROVENANCE=src/hashes/provenance.json @@ -57,29 +89,36 @@ jobs: git fetch --quiet origin master git reset --quiet --hard origin/master - CHANGED=$(node scripts/sign-hashlist.js) - if [ "$CHANGED" != "changed=true" ]; then - echo "list unchanged, nothing to sign" - exit 0 - fi - - node scripts/verify-hashlist.js - - git add "$SIGNED" "$PROVENANCE" - if git diff --cached --quiet -- "$SIGNED" "$PROVENANCE"; then - echo "nothing changed, not publishing" - exit 0 - fi - - SEQ=$(node -p "JSON.parse(Buffer.from(require('./$SIGNED').payload_b64,'base64')).seq") - git commit --quiet -m "Sign hash list seq $SEQ" -- "$SIGNED" "$PROVENANCE" + RESULT=$(node scripts/sign-hashlist.js) + case "$RESULT" in + changed=false) + echo 'nothing to publish' + exit 0 + ;; + changed=state) + node scripts/validate.js + git add "$PROVENANCE" + git commit --quiet -m 'Reconcile flux refs' -- "$PROVENANCE" + ;; + changed=signed) + node scripts/verify-hashlist.js + node scripts/validate.js + git add "$LIST" "$SIGNED" "$PROVENANCE" + SEQ=$(node -p "JSON.parse(Buffer.from(require('./$SIGNED').payload_b64,'base64')).seq") + git commit --quiet -m "Sign hash list seq $SEQ" -- "$LIST" "$SIGNED" "$PROVENANCE" + ;; + *) + echo "unexpected reconciler output: $RESULT" + exit 1 + ;; + esac - if git push --quiet; then - echo "published seq $SEQ" + if git push --quiet "git@github.com:${GITHUB_REPOSITORY}.git" HEAD:master; then + echo "published ($RESULT)" exit 0 fi - echo "push raced with a publish from flux CI, retrying" + echo 'push raced a human merge, retrying' done - echo "could not publish after 3 attempts" + echo 'could not publish after 3 attempts' exit 1 diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index fe9adb67..7d639916 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -1,8 +1,9 @@ name: validate -# The published list is served by requiring it, so a file that does not load takes the endpoint down. -# Flux CI checks its own edit before pushing; this covers the other way the list changes, which is by -# hand. +# The published list is served by requiring it, so a file that does not load takes the endpoint +# down. The signer runs validate.js itself before pushing; this run covers human PRs and stands as +# the live tripwire on master -- the signer's deploy-key pushes trigger it, so every signing commit +# gets a green check and a red one always means something real. on: pull_request: @@ -20,4 +21,20 @@ jobs: - uses: actions/setup-node@v4 with: node-version: '20' + + # The three outputs are generated by the signer; a human edit to any of them is either a + # mistake or an attempt to put a hash value into the list without deriving it. Either way: + # edit the ledger instead. + - name: Outputs are generated, not edited + if: github.event_name == 'pull_request' + run: | + git fetch --quiet --depth=1 origin "${{ github.event.pull_request.base.sha }}" + CHANGED=$(git diff --name-only "${{ github.event.pull_request.base.sha }}" HEAD -- \ + src/hashes/hashes.js src/hashes/hashlist-signed.json src/hashes/provenance.json) + if [ -n "$CHANGED" ]; then + echo 'these files are generated by the signing workflow; edit src/hashes/ledger.json instead:' + echo "$CHANGED" + exit 1 + fi + - run: node scripts/validate.js diff --git a/SIGNING.md b/SIGNING.md index f5a26028..8429625b 100644 --- a/SIGNING.md +++ b/SIGNING.md @@ -18,16 +18,24 @@ two is higher — so losing the document, however that happens, does not reset t ## The provenance record -`src/hashes/provenance.json`, outside the signed payload, with two writers: - -- **flux CI** adds a row per published hash — `published` date, `commit`, `branch`, `tag` — in the - same commit that edits `hashes.js`. This is what makes an entry attributable later: the list - itself is opaque md5s, and the commit that produced an entry can stop existing (a force-push, a - branch deleted after merge). A tag push fills in the `tag` field on the existing row. -- **the signing run** stamps `signed` — the sequence and `issued_at` it signed at — in the same - commit as the signed document. - -Rows exist only for hashes published since the record was introduced; older entries have none. +`src/hashes/provenance.json`, outside the signed payload, with **one writer — the signing +workflow**, which derives every post-cutover entry itself from commits it fetches from +`RunOnFlux/flux`: + +- a **row per listed hash** — `published` date, `commit`, `branch`, `tag`, `derived`. This is what + makes an entry attributable later: the list itself is opaque md5s, and the commit that produced + an entry can stop existing (a force-push, a branch deleted after merge). First attribution wins; + a new tag on a known commit annotates the existing row. Rows with `derived: false` predate + derivation (grandfathered at cutover) and have no commit to point at. +- a **commits map** (`sha → hash`) so nothing is fetched or hashed twice, and a **refs snapshot** + of the flux remote, which is what the reconciler diffs against — a publication request that gets + lost is repaired by the next run reconciling the full delta. +- **`signed`** — the sequence high-water and `issued_at`, stamped in the same commit as the signed + document. + +`src/hashes/ledger.json` is the one human-edited input: cull marks, reviewed through PRs. The +signer reads it and never writes it. The three output files are generated — `validate` fails any +PR that edits them directly. ## Keys @@ -36,14 +44,16 @@ can take over without those consumers needing an update. | key | public key (raw ed25519, hex) | custody | use | |---|---|---|---| -| 1 | `3023cb5e01dc22257ac5c31c4d12106cd0d58fa2005f867b3fdc5d303f6446ec` | CI, repository secret `HASHLIST_SIGNING_SEED_B64` | day to day | +| 1 | `3023cb5e01dc22257ac5c31c4d12106cd0d58fa2005f867b3fdc5d303f6446ec` | CI, secret `HASHLIST_SIGNING_SEED_B64` in the `hashlist-signing` environment | day to day | | 2 | `fee7b0ccf2323954af68a249eaa61f957239eb222329e08a5b6a50ced649bae8` | cold, offline | continuity only | ### Key 1 -Generated 2026-08-14 straight into the repository secret `HASHLIST_SIGNING_SEED_B64`. **There is no -copy of the private half anywhere else, on purpose** — key 2 covers its loss, and a second copy would -only widen where it can leak from. +Generated 2026-08-14 straight into the secret `HASHLIST_SIGNING_SEED_B64`, which lives in the +`hashlist-signing` GitHub Environment whose deployment branch policy admits `master` only — a +workflow run on any other ref is refused before its first step, so a branch push cannot read it. +**There is no copy of the private half anywhere else, on purpose** — key 2 covers its loss, and a +second copy would only widen where it can leak from. To replace it, generate a new one the same way: @@ -75,8 +85,18 @@ published until consumers were updated with a replacement. It does not provide revocation — removing a key from the pinned set requires updating consumers. Two keys held in the same place buy nothing; the separation is the point. +### The deploy key + +The signer pushes to `master` over SSH with a write deploy key, `HASHLIST_DEPLOY_KEY` in the same +environment — the one bypass on the master ruleset, which otherwise admits only reviewed PRs with +`validate` green (repository admins included). The run's own `GITHUB_TOKEN` stays read-only. To +rotate: generate a fresh keypair, replace the repository deploy key and the environment secret; +the ruleset's `DeployKey` bypass covers whatever write keys the repository holds, so it needs no +change — which is also why the repository must hold exactly this one write deploy key. + ## Trust -Anyone who can land a workflow change on `master` can read the secret; a GitHub secret is an -access-controlled environment variable, not a vault. It is not passed to workflows triggered by a -pull request from a fork, which matters because this repository is public. +Anyone who can land a workflow change on `master` can read the secrets; a GitHub secret is an +access-controlled environment variable, not a vault. Environment scoping means landing that change +requires a reviewed merge — a branch push is no longer enough. Secrets are not passed to workflows +triggered by a pull request from a fork, which matters because this repository is public. diff --git a/scripts/sign-hashlist.js b/scripts/sign-hashlist.js index ceb96c53..32b1ee04 100644 --- a/scripts/sign-hashlist.js +++ b/scripts/sign-hashlist.js @@ -1,21 +1,38 @@ #!/usr/bin/env node -// Signs the hash list this repository publishes, so consumers can verify it came from us rather -// than trusting the transport or whatever relayed it. +// The single writer of everything src/hashes/ serves. // -// The signed document sits alongside the unsigned array rather than replacing it; both are served. +// Reconciles the published list against RunOnFlux/flux itself: diffs the remote's refs against the +// snapshot in the provenance record, fetches each new commit, computes its ZelBack tree hash from +// the bytes it fetched, and emits hashes.js, the signed document and the provenance record together. +// A hash value can enter the list through no other path. A dispatch carries pointers, never +// content, so the credential that sends one needs no write authority -- and the signature means +// "this commit's tree hashes to this value", not "this was in the repository when I ran". // // The payload is signed and transmitted as exact bytes in base64, so verification never depends on -// the signer and the verifier agreeing about JSON key order or whitespace -- the kind of agreement -// that holds in testing and fails in production. +// the signer and the verifier agreeing about JSON key order or whitespace. +// +// stdout carries exactly one line -- changed=false, changed=state or changed=signed -- read by the +// workflow to decide what to commit. Everything human goes to stderr. +const { execFileSync } = require('child_process'); const crypto = require('crypto'); const fs = require('fs'); +const os = require('os'); const path = require('path'); const ROOT = path.join(__dirname, '..'); +const LIST = path.join(ROOT, 'src', 'hashes', 'hashes.js'); const OUTPUT = path.join(ROOT, 'src', 'hashes', 'hashlist-signed.json'); const PROVENANCE = path.join(ROOT, 'src', 'hashes', 'provenance.json'); +const LEDGER = path.join(ROOT, 'src', 'hashes', 'ledger.json'); + +const FLUX_REMOTE = process.env.FLUX_REMOTE || 'https://github.com/RunOnFlux/flux'; + +// fluxbench's pipeline, byte for byte -- flux CI's Check Hash step runs the same one. The awk +// strips filenames before the sort, so the hash depends only on the multiset of file contents; +// LC_ALL=C pins the sort. +const TREE_HASH_PIPELINE = "find ./ZelBack -type f -exec md5sum {} + | awk '{print $1}' | LC_ALL=C sort | md5sum | awk '{printf $1}'"; // A raw 32-byte Ed25519 seed is not directly importable; Node wants PKCS8. The prefix is fixed for // the algorithm, so prepending it is enough. @@ -71,14 +88,10 @@ function previousDocument() { return JSON.parse(Buffer.from(document.payload_b64, 'base64').toString('utf8')); } -// The provenance record carries the highest sequence ever signed, in a file committed alongside the -// list. The sequence is taken from whichever of the two is higher, so losing the signed document -- -// however that happens -- cannot restart the sequence at 1: a fresh document signed under an old -// sequence would be refused by any consumer already holding a newer one, and the failure would be -// silent because the run that produced it is green. -// -// Absence of the record is the state before the first signing run and is fine. A record that exists -// but does not parse is not: treating it as absent is exactly the restart this exists to prevent. +// The provenance record is this writer's own state: attribution rows, the commit-to-hash map, the +// refs snapshot the reconciler diffs against, and the highest sequence ever signed. Absence is the +// state before the first run and starts the cutover bootstrap. A record that exists but does not +// parse is a red run: treating corruption as absence is exactly the restart this exists to prevent. function readProvenance() { if (!fs.existsSync(PROVENANCE)) { return null; @@ -91,10 +104,95 @@ function readProvenance() { return record; } -function writeProvenance(record, seq, issuedAt) { - const updated = record || { hashes: {} }; - updated.signed = { seq, issued_at: issuedAt }; - fs.writeFileSync(PROVENANCE, `${JSON.stringify(updated, null, 2)}\n`); +// The ledger is the human-edited input: cull marks that take entries out of the list through a +// reviewed PR. The signer reads it and never writes it, which is what lets the workflow trigger on +// pushes to it without ever retriggering itself. Malformed is a red run, not a skip. +function readLedger() { + if (!fs.existsSync(LEDGER)) { + return { culls: [] }; + } + const ledger = JSON.parse(fs.readFileSync(LEDGER, 'utf8')); + if (!Array.isArray(ledger.culls)) { + throw new Error('ledger.culls is not an array'); + } + ledger.culls.forEach((cull) => { + if (!cull || typeof cull.hash !== 'string' || !/^[0-9a-f]{32}$/.test(cull.hash)) { + throw new Error(`ledger cull without a valid hash: ${JSON.stringify(cull)}`); + } + }); + return ledger; +} + +function git(args, options = {}) { + return execFileSync('git', args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], ...options }); +} + +// refs/heads/* as listed; refs/tags/* resolved to the commit they point at (the ^{} peel when the +// tag is annotated). One request regardless of ref count. +function lsRemoteRefs(remote) { + const refs = {}; + const peeled = {}; + git(['ls-remote', '--heads', '--tags', remote]).split('\n').filter(Boolean).forEach((line) => { + const [sha, ref] = line.split('\t'); + if (ref.endsWith('^{}')) { + peeled[ref.slice(0, -3)] = sha; + } else { + refs[ref] = sha; + } + }); + Object.entries(peeled).forEach(([ref, sha]) => { refs[ref] = sha; }); + return refs; +} + +function ensureCacheRepo() { + const dir = process.env.FLUX_CACHE_DIR || path.join(os.tmpdir(), 'flux-hash-cache'); + if (!fs.existsSync(path.join(dir, '.git'))) { + fs.mkdirSync(dir, { recursive: true }); + git(['init', '--quiet'], { cwd: dir }); + } + return dir; +} + +// Fetch the named commit from the official remote and hash the pristine tree in a throwaway +// worktree. Nothing from the fetched tree is ever executed; the pipeline only reads bytes. +function deriveTreeHash(cache, sha) { + git(['fetch', '--quiet', '--depth', '1', FLUX_REMOTE, sha], { cwd: cache }); + const parent = fs.mkdtempSync(path.join(os.tmpdir(), 'flux-tree-')); + const worktree = path.join(parent, 'wt'); + try { + git(['worktree', 'add', '--quiet', '--detach', worktree, sha], { cwd: cache }); + const hash = execFileSync('bash', ['-c', TREE_HASH_PIPELINE], { cwd: worktree, encoding: 'utf8' }).trim(); + if (!/^[0-9a-f]{32}$/.test(hash)) { + throw new Error(`tree hash pipeline produced "${hash}"`); + } + return hash; + } finally { + try { + git(['worktree', 'remove', '--force', worktree], { cwd: cache }); + } catch (error) { /* the worktree was never created */ } + fs.rmSync(parent, { recursive: true, force: true }); + } +} + +function writeList(hashes) { + const lines = ['function getHashes() {', ' return [']; + hashes.forEach((hash) => lines.push(` '${hash}',`)); + lines.push(' ];', '}', '', 'module.exports = {', ' getHashes,', '};', ''); + fs.writeFileSync(LIST, lines.join('\n')); +} + +function refLabel(ref) { + return ref.startsWith('refs/heads/') + ? { branch: ref.slice('refs/heads/'.length), tag: null } + : { branch: null, tag: ref.slice('refs/tags/'.length) }; +} + +function dispatchLabel() { + const ref = process.env.DISPATCH_REF || null; + if (!ref) { + return { branch: null, tag: null }; + } + return process.env.DISPATCH_REF_TYPE === 'tag' ? { branch: null, tag: ref } : { branch: ref, tag: null }; } function main() { @@ -104,34 +202,181 @@ function main() { } // eslint-disable-next-line global-require - const hashes = require('../src/hashes/hashes').getHashes(); + const currentList = require('../src/hashes/hashes').getHashes(); const previous = previousDocument(); - const provenance = readProvenance(); - - // Re-signing an unchanged list would burn a sequence for nothing, and every node would have to - // fetch and verify a document identical to the one it already holds. - if (previous - && previous.hashes.length === hashes.length - && previous.hashes.every((hash, i) => hash === hashes[i])) { - process.stderr.write(`unchanged at seq ${previous.seq}, nothing to sign\n`); - process.stdout.write('changed=false\n'); + const provenance = readProvenance() || {}; + const ledger = readLedger(); + const before = JSON.stringify(provenance); + + const today = new Date().toISOString().slice(0, 10); + const rows = provenance.hashes || {}; + const commits = provenance.commits || {}; + const snapshot = provenance.refs || null; + + const current = lsRemoteRefs(FLUX_REMOTE); + process.stderr.write(`flux remote: ${Object.keys(current).length} refs\n`); + + // Cutover bootstrap: no snapshot means nothing has been derived yet. Grandfather everything + // already listed and snapshot the remote as it stands, so only movement from here on is derived. + if (!snapshot) { + currentList.forEach((hash) => { + if (!rows[hash]) { + rows[hash] = { + published: today, commit: null, branch: null, tag: null, derived: false, + }; + } else if (rows[hash].derived === undefined) { + rows[hash].derived = false; + } + }); + process.stderr.write(`bootstrap: grandfathered ${currentList.length} entries, snapshotting ${Object.keys(current).length} refs\n`); + } + + // The work set: every ref that moved since the snapshot, keyed by commit. A tag riding a commit + // that is also a branch tip annotates rather than duplicates. + const work = new Map(); + if (snapshot) { + Object.entries(current).forEach(([ref, sha]) => { + if (snapshot[ref] === sha) return; + // A tag that MOVED keeps its original attribution; only genuinely new tags join the set. + if (ref.startsWith('refs/tags/') && snapshot[ref] !== undefined) return; + const label = refLabel(ref); + if (!work.has(sha)) { + work.set(sha, label); + } else if (label.tag && !work.get(sha).tag) { + work.get(sha).tag = label.tag; + } + }); + } + + const dispatchCommit = (process.env.DISPATCH_COMMIT || '').toLowerCase() || null; + const claimed = (process.env.DISPATCH_CLAIMED_HASH || '').toLowerCase() || null; + if (dispatchCommit && !/^[0-9a-f]{40}$/.test(dispatchCommit)) { + throw new Error(`dispatched commit is not a 40-character sha: ${dispatchCommit}`); + } + if (dispatchCommit && !work.has(dispatchCommit)) { + // Label from our own view of the remote first; the dispatch's ref fields are a fallback label + // for a ref that moved past the commit before we looked, never authority. + const ownRef = Object.keys(current).find((ref) => current[ref] === dispatchCommit); + work.set(dispatchCommit, ownRef ? refLabel(ownRef) : dispatchLabel()); + } + if (dispatchCommit && claimed && commits[dispatchCommit] && commits[dispatchCommit] !== claimed) { + throw new Error(`claimed hash ${claimed} does not match ${commits[dispatchCommit]} already derived for ${dispatchCommit}`); + } + + // Derive. A failed fetch of a dispatched commit is a red run -- the caller named a commit the + // official repository will not serve. A failed fetch from the ref sweep keeps that ref's old + // snapshot entry, so the next run retries it. + const failedRefs = new Set(); + const cache = ensureCacheRepo(); + work.forEach((label, sha) => { + if (commits[sha]) { + const row = rows[commits[sha]]; + if (row && label.tag && !row.tag) { + row.tag = label.tag; + } + return; + } + let hash; + try { + hash = deriveTreeHash(cache, sha); + } catch (error) { + if (sha === dispatchCommit) { + throw new Error(`dispatched commit ${sha}: ${error.message}`); + } + process.stderr.write(`skipping ${sha}: ${error.message}\n`); + Object.entries(current).forEach(([ref, s]) => { if (s === sha) failedRefs.add(ref); }); + return; + } + if (sha === dispatchCommit && claimed && claimed !== hash) { + throw new Error(`claimed hash ${claimed} does not match derived ${hash} for ${sha} -- environment drift or a hostile dispatch, either must be loud`); + } + commits[sha] = hash; + if (!rows[hash]) { + // First attribution wins: a hash republished from another branch keeps its original row. + rows[hash] = { + published: today, commit: sha, branch: label.branch, tag: label.tag, derived: true, + }; + process.stderr.write(`derived ${hash} from ${sha.slice(0, 9)} (${label.tag || label.branch || 'unlabelled'})\n`); + } else if (label.tag && !rows[hash].tag) { + rows[hash].tag = label.tag; + } + }); + + const nextRefs = {}; + Object.entries(current).forEach(([ref, sha]) => { + if (failedRefs.has(ref)) { + if (snapshot && snapshot[ref] !== undefined) { + nextRefs[ref] = snapshot[ref]; + } + return; + } + nextRefs[ref] = sha; + }); + + // Membership: what is listed, minus culls, plus everything derived that is not yet listed. + const culled = new Set(ledger.culls.map((cull) => cull.hash)); + const listed = new Set(currentList); + const retained = currentList.filter((hash) => !culled.has(hash)); + const additions = Object.keys(rows).filter( + (hash) => rows[hash].derived === true && !listed.has(hash) && !culled.has(hash), + ); + const newList = retained.concat(additions); + + // Monotonicity, asserted independently of the construction above: an entry leaves the list only + // when the ledger says so. A regeneration bug must be a red run, never a signed loss. + const surviving = new Set(newList); + currentList.forEach((hash) => { + if (!surviving.has(hash) && !culled.has(hash)) { + throw new Error(`entry ${hash} would vanish without a ledger cull -- refusing to publish`); + } + }); + if (newList.length === 0) { + throw new Error('refusing to publish an empty list'); + } + if (surviving.size !== newList.length) { + throw new Error('the regenerated list contains duplicates -- refusing to publish'); + } + + const listChanged = newList.length !== currentList.length + || newList.some((hash, i) => hash !== currentList[i]); + // No document has ever been published: the bootstrap run signs even an unchanged list. This is + // also the first proof that the stored seed matches the pinned key. + const mustSign = !previous; + + provenance.hashes = rows; + provenance.commits = commits; + provenance.refs = nextRefs; + + if (!listChanged && !mustSign) { + if (JSON.stringify(provenance) === before) { + process.stderr.write(`nothing changed at seq ${previous.seq}\n`); + process.stdout.write('changed=false\n'); + return; + } + // Attribution, tag annotations or the snapshot advanced with the membership intact: worth a + // commit, not worth a sequence. + fs.writeFileSync(PROVENANCE, `${JSON.stringify(provenance, null, 2)}\n`); + process.stderr.write(`state advanced, membership unchanged at seq ${previous.seq}\n`); + process.stdout.write('changed=state\n'); return; } const highWater = Math.max( previous ? previous.seq : 0, - provenance && provenance.signed ? provenance.signed.seq : 0, + provenance.signed ? provenance.signed.seq : 0, ); const seq = highWater + 1; const issuedAt = new Date().toISOString(); const privateKey = privateKeyFromSeed(seedB64); - const document = buildSignedDocument(seq, issuedAt, hashes, privateKey); + const document = buildSignedDocument(seq, issuedAt, newList, privateKey); + provenance.signed = { seq, issued_at: issuedAt }; + writeList(newList); fs.writeFileSync(OUTPUT, `${JSON.stringify(document, null, 2)}\n`); - writeProvenance(provenance, seq, issuedAt); - process.stderr.write(`signed seq ${seq} over ${hashes.length} hashes\n`); + fs.writeFileSync(PROVENANCE, `${JSON.stringify(provenance, null, 2)}\n`); + process.stderr.write(`signed seq ${seq} over ${newList.length} hashes (${additions.length} added, ${currentList.length - retained.length} culled)\n`); process.stderr.write(`public key (raw, hex): ${rawPublicKey(privateKey).toString('hex')}\n`); - process.stdout.write('changed=true\n'); + process.stdout.write('changed=signed\n'); } if (require.main === module) { @@ -144,5 +389,5 @@ if (require.main === module) { } module.exports = { - privateKeyFromSeed, rawPublicKey, buildSignedDocument, readProvenance, + privateKeyFromSeed, rawPublicKey, buildSignedDocument, readProvenance, readLedger, }; diff --git a/scripts/validate.js b/scripts/validate.js index 135640bd..3d309a40 100644 --- a/scripts/validate.js +++ b/scripts/validate.js @@ -3,8 +3,10 @@ // Shape-checks what this repository publishes. // // The list is served by requiring it, so a file that does not load takes the endpoint down rather -// than merely publishing something odd. Flux CI checks its own edit before pushing, but the list is -// also edited by hand -- a cull removes entries in bulk -- and that path had nothing in front of it. +// than merely publishing something odd. The three outputs have one writer -- the signer, which +// runs this as a self-check before pushing -- so no legitimate commit can fail here: a red +// validate on master always means the generator or the repository rules are broken, which is the +// point of running it everywhere. // // This checks shape only. Whether a particular hash *should* be listed is not knowable from here: // removing one that is still in use looks identical to removing one that is obsolete. @@ -15,6 +17,7 @@ const path = require('path'); const ROOT = path.join(__dirname, '..'); const SIGNED = path.join(ROOT, 'src', 'hashes', 'hashlist-signed.json'); const PROVENANCE = path.join(ROOT, 'src', 'hashes', 'provenance.json'); +const LEDGER = path.join(ROOT, 'src', 'hashes', 'ledger.json'); const failures = []; @@ -42,10 +45,11 @@ function validateHashes() { return hashes; } -// The provenance record has two writers -- flux CI adds a row per published hash, the signing run -// stamps the sequence it signed at -- and a malformed edit from either would take the signer down. -// Not every hash has a row: the record starts empty against a list that predates it. -function validateProvenance() { +// The provenance record has one writer -- the signer, which owns the attribution rows, the +// commit-to-hash map, the refs snapshot its reconciler diffs against, and the sequence high-water. +// A malformed record takes the signer down, so shape is enforced here and on every PR. A +// grandfathered row (derived false) predates derivation and has no commit to point at. +function validateProvenance(hashes) { if (!fs.existsSync(PROVENANCE)) { process.stderr.write('no provenance record yet, skipping\n'); return null; @@ -73,20 +77,72 @@ function validateProvenance() { const rows = record.hashes || {}; Object.entries(rows).forEach(([hash, row]) => { check(/^[0-9a-f]{32}$/.test(hash), `provenance row key is not a lowercase md5: ${hash}`); + const commitOk = row && ( + (typeof row.commit === 'string' && /^[0-9a-f]{40}$/.test(row.commit)) + || (row.commit === null && row.derived !== true) + ); check( row && typeof row === 'object' && typeof row.published === 'string' && /^\d{4}-\d{2}-\d{2}$/.test(row.published) - && typeof row.commit === 'string' && /^[0-9a-f]{40}$/.test(row.commit) + && commitOk && (row.branch === null || typeof row.branch === 'string') - && (row.tag === null || typeof row.tag === 'string'), + && (row.tag === null || typeof row.tag === 'string') + && (row.derived === undefined || typeof row.derived === 'boolean'), `provenance row is malformed: ${hash}`, ); }); - process.stderr.write(`provenance: ${Object.keys(rows).length} rows, signed seq ${record.signed ? record.signed.seq : 'none'}\n`); + Object.entries(record.commits || {}).forEach(([sha, hash]) => { + check(/^[0-9a-f]{40}$/.test(sha), `commits map key is not a 40-character sha: ${sha}`); + check(typeof hash === 'string' && /^[0-9a-f]{32}$/.test(hash), `commits map value is not a lowercase md5: ${hash}`); + }); + Object.entries(record.refs || {}).forEach(([ref, sha]) => { + check(ref.startsWith('refs/'), `refs snapshot key is not a ref: ${ref}`); + check(typeof sha === 'string' && /^[0-9a-f]{40}$/.test(sha), `refs snapshot value is not a 40-character sha: ${ref}`); + }); + + // Post-cutover (the snapshot exists), membership is generated from the rows: a listed hash + // without a row means the generator and its record have diverged. + if (record.refs && hashes) { + const unattributed = hashes.filter((hash) => !rows[hash]); + check(unattributed.length === 0, `${unattributed.length} listed hashes have no provenance row: ${unattributed.slice(0, 3)}`); + } + + process.stderr.write(`provenance: ${Object.keys(rows).length} rows, ${Object.keys(record.commits || {}).length} commits, ${Object.keys(record.refs || {}).length} refs, signed seq ${record.signed ? record.signed.seq : 'none'}\n`); return record; } +// The ledger is the human-edited input: cull marks reviewed through PRs. A cull may still be +// listed here -- the signer applies it on its next run -- so consistency with the list is not +// checkable; shape is. +function validateLedger() { + if (!fs.existsSync(LEDGER)) { + process.stderr.write('no ledger yet, skipping\n'); + return; + } + + let ledger; + try { + ledger = JSON.parse(fs.readFileSync(LEDGER, 'utf8')); + } catch (error) { + check(false, `ledger does not parse: ${error.message}`); + return; + } + + check(Array.isArray(ledger.culls), 'ledger.culls is not an array'); + (Array.isArray(ledger.culls) ? ledger.culls : []).forEach((cull, i) => { + check( + cull && typeof cull === 'object' + && typeof cull.hash === 'string' && /^[0-9a-f]{32}$/.test(cull.hash) + && typeof cull.reason === 'string' && cull.reason.length > 0 + && typeof cull.date === 'string' && /^\d{4}-\d{2}-\d{2}$/.test(cull.date), + `ledger cull ${i} is malformed: ${JSON.stringify(cull)}`, + ); + }); + + process.stderr.write(`ledger: ${Array.isArray(ledger.culls) ? ledger.culls.length : 0} culls\n`); +} + // The signed copy is written by CI and only exists once it has run, so its absence is not a failure // -- unless the provenance record says a document was signed, in which case the document has gone // missing and that must be a red run, not a skip. If it is there it must verify, it must describe @@ -141,7 +197,8 @@ function validateSigned(hashes, provenance) { function main() { const hashes = validateHashes(); - const provenance = validateProvenance(); + const provenance = validateProvenance(hashes); + validateLedger(); validateSigned(hashes, provenance); if (failures.length) { diff --git a/src/hashes/ledger.json b/src/hashes/ledger.json new file mode 100644 index 00000000..a1d1499a --- /dev/null +++ b/src/hashes/ledger.json @@ -0,0 +1,3 @@ +{ + "culls": [] +} From e677ce8678bcd9f1d2c9894db103d203c3d35107 Mon Sep 17 00:00:00 2001 From: David White Date: Mon, 24 Aug 2026 11:41:59 +0100 Subject: [PATCH 10/17] polish from independent review: keep both labels on a dual-ref commit, honest comments Three notes from the independent implementation review, none above low severity: a commit arriving as branch tip and tag in one run now keeps both labels regardless of ref iteration order; the outputs guard's comment stops claiming to be a security control (pull_request runs the PR head's workflow copy -- required review is the control, the guard catches mistakes); the monotonicity guard's comment says what it is, defense-in-depth for a future regeneration refactor, mutation-tested. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01SFrS7Q3JwuPj6Yr4vALnwp --- .github/workflows/validate.yml | 8 +++++--- scripts/sign-hashlist.js | 14 ++++++++++---- 2 files changed, 15 insertions(+), 7 deletions(-) diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 7d639916..1f3fc3eb 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -22,9 +22,11 @@ jobs: with: node-version: '20' - # The three outputs are generated by the signer; a human edit to any of them is either a - # mistake or an attempt to put a hash value into the list without deriving it. Either way: - # edit the ledger instead. + # The three outputs are generated by the signer; edit the ledger instead. This catches + # honest mistakes, not attacks: pull_request runs the PR head's copy of this workflow, so a + # hostile PR could weaken the check it is judged by. The control against a hostile PR is + # required review on the ruleset -- this step just makes the mistake loud before a human + # looks. - name: Outputs are generated, not edited if: github.event_name == 'pull_request' run: | diff --git a/scripts/sign-hashlist.js b/scripts/sign-hashlist.js index 32b1ee04..af81d9be 100644 --- a/scripts/sign-hashlist.js +++ b/scripts/sign-hashlist.js @@ -242,8 +242,12 @@ function main() { const label = refLabel(ref); if (!work.has(sha)) { work.set(sha, label); - } else if (label.tag && !work.get(sha).tag) { - work.get(sha).tag = label.tag; + } else { + // A commit arriving as branch tip and tag in the same run keeps both labels, whichever + // ref was seen first. + const existing = work.get(sha); + if (label.tag && !existing.tag) existing.tag = label.tag; + if (label.branch && !existing.branch) existing.branch = label.branch; } }); } @@ -322,8 +326,10 @@ function main() { ); const newList = retained.concat(additions); - // Monotonicity, asserted independently of the construction above: an entry leaves the list only - // when the ledger says so. A regeneration bug must be a red run, never a signed loss. + // Monotonicity: an entry leaves the list only when the ledger says so. The append-with-cull + // construction above cannot trip this today -- it exists so that a future change to true + // regeneration-from-rows turns a dropped entry into a red run, never a signed loss + // (mutation-tested: it catches exactly that). const surviving = new Set(newList); currentList.forEach((hash) => { if (!surviving.has(hash) && !culled.has(hash)) { From 288226795123c9ad28487fb07c79344a35043f29 Mon Sep 17 00:00:00 2001 From: David White Date: Mon, 24 Aug 2026 11:52:58 +0100 Subject: [PATCH 11/17] fix: anchor the outputs guard on the merge commit's first parent The event payload's base.sha is the base branch at the last synchronize event -- stale on any PR whose base has moved since, and the signer moves master daily. Diffing against it flagged every hash published since the PR opened as if the PR had edited it, going red on this very PR. HEAD^1 of the merge-ref checkout is the base as it stands now, so the guard sees exactly what merging would change. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01SFrS7Q3JwuPj6Yr4vALnwp --- .github/workflows/validate.yml | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 1f3fc3eb..2dbc855d 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -18,6 +18,9 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 + with: + # The outputs guard below diffs the merge commit against its first parent. + fetch-depth: 2 - uses: actions/setup-node@v4 with: node-version: '20' @@ -27,11 +30,15 @@ jobs: # hostile PR could weaken the check it is judged by. The control against a hostile PR is # required review on the ruleset -- this step just makes the mistake loud before a human # looks. + # + # HEAD is the PR merge commit and HEAD^1 is the base branch as it stands NOW, so this sees + # exactly what merging the PR would change. (The event payload's base.sha is the base at the + # last synchronize -- stale on any PR whose base has since moved, and the signer moves master + # daily, so diffing against it flags every hash published since the PR was opened.) - name: Outputs are generated, not edited if: github.event_name == 'pull_request' run: | - git fetch --quiet --depth=1 origin "${{ github.event.pull_request.base.sha }}" - CHANGED=$(git diff --name-only "${{ github.event.pull_request.base.sha }}" HEAD -- \ + CHANGED=$(git diff --name-only HEAD^1 HEAD -- \ src/hashes/hashes.js src/hashes/hashlist-signed.json src/hashes/provenance.json) if [ -n "$CHANGED" ]; then echo 'these files are generated by the signing workflow; edit src/hashes/ledger.json instead:' From 9fda0cd76bfe55c6d653f6ec08e0a2680fb9d1ba Mon Sep 17 00:00:00 2001 From: David White Date: Mon, 24 Aug 2026 12:25:07 +0100 Subject: [PATCH 12/17] chore: pin the regenerated key 1 Key 1 was regenerated 2026-08-24 straight into the hashlist-signing environment's secret: a secret's value cannot be moved from the repository level into an environment, the environment scoping is the custody hardening, and nothing had ever consumed the old key -- no production signing run has happened and fluxbench has not shipped -- so regeneration was free. Key 2 (cold) is unchanged. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01SFrS7Q3JwuPj6Yr4vALnwp --- SIGNING.md | 7 +++++-- scripts/verify-hashlist.js | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/SIGNING.md b/SIGNING.md index 8429625b..99d0b6e1 100644 --- a/SIGNING.md +++ b/SIGNING.md @@ -44,14 +44,17 @@ can take over without those consumers needing an update. | key | public key (raw ed25519, hex) | custody | use | |---|---|---|---| -| 1 | `3023cb5e01dc22257ac5c31c4d12106cd0d58fa2005f867b3fdc5d303f6446ec` | CI, secret `HASHLIST_SIGNING_SEED_B64` in the `hashlist-signing` environment | day to day | +| 1 | `14837066068b258bfbd0749702056f7065361af44aed48761834744391cbbaaa` | CI, secret `HASHLIST_SIGNING_SEED_B64` in the `hashlist-signing` environment | day to day | | 2 | `fee7b0ccf2323954af68a249eaa61f957239eb222329e08a5b6a50ced649bae8` | cold, offline | continuity only | ### Key 1 -Generated 2026-08-14 straight into the secret `HASHLIST_SIGNING_SEED_B64`, which lives in the +Generated 2026-08-24 straight into the secret `HASHLIST_SIGNING_SEED_B64`, which lives in the `hashlist-signing` GitHub Environment whose deployment branch policy admits `master` only — a workflow run on any other ref is refused before its first step, so a branch push cannot read it. +(This replaced the 2026-08-14 key, which had only ever lived as a repository-level secret: a +secret's value cannot be moved into an environment, and nothing had ever consumed the old key, so +regenerating was free.) **There is no copy of the private half anywhere else, on purpose** — key 2 covers its loss, and a second copy would only widen where it can leak from. diff --git a/scripts/verify-hashlist.js b/scripts/verify-hashlist.js index 310dcc10..1459b38b 100644 --- a/scripts/verify-hashlist.js +++ b/scripts/verify-hashlist.js @@ -19,7 +19,7 @@ const SPKI_ED25519_PREFIX = Buffer.from('302a300506032b6570032100', 'hex'); // Must match SIGNING.md and the set consumers pin. Any one of them verifying is enough, which is // what lets a second key take over without updating consumers. const PINNED_PUBLIC_KEYS = [ - '3023cb5e01dc22257ac5c31c4d12106cd0d58fa2005f867b3fdc5d303f6446ec', // 1, CI + '14837066068b258bfbd0749702056f7065361af44aed48761834744391cbbaaa', // 1, CI 'fee7b0ccf2323954af68a249eaa61f957239eb222329e08a5b6a50ced649bae8', // 2, cold ]; From d8238412595d8e90aadfa7a0aa9a078b28396213 Mon Sep 17 00:00:00 2001 From: David White Date: Tue, 25 Aug 2026 11:56:55 +0100 Subject: [PATCH 13/17] fix: refuse the empty-tree hash instead of signing it The derivation pipeline yields d41d8cd98f00b204e9800998ecf8427e -- the md5 of an empty stream -- whenever nothing was hashed, and that value is a well-formed 32-hex string, so the existing shape guard passed it straight through to the signature. It means "a node whose ZelBack holds no regular files is genuine FluxOS", and membership is monotonic, so one accident becomes a listed entry only a cull PR removes. Two distinct ways in, and only one of them is a pipeline failure: ZelBack absent find errors, but the pipeline's status is awk's, so without pipefail the caller sees success ZelBack present, empty find exits 0 and emits nothing -- pipefail cannot see this Mutation-tested rather than assumed: with the empty-hash guard removed, the second case lists the empty hash; with pipefail removed, the guard still catches both. So the guard is what makes this correct and pipefail is defence in depth -- it makes the absent case fail on find's own error rather than laundering it into a hash comparison. pipefail is passed as a bash argument rather than folded into TREE_HASH_PIPELINE so the constant stays byte-identical to the pipeline flux CI runs; that identity is what makes claimed_hash comparable at all. Failure routing needed no new code: throwing here already means a red run when the commit was dispatched, and a logged skip that retains the ref's old snapshot entry when it was found by the sweep. A permanently non-code branch therefore cannot wedge publication. Reported by Cabecinha84 on #2. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01HQKpZbuxYqrWeErU7vKgoA --- scripts/sign-hashlist.js | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/scripts/sign-hashlist.js b/scripts/sign-hashlist.js index af81d9be..b2f07c0e 100644 --- a/scripts/sign-hashlist.js +++ b/scripts/sign-hashlist.js @@ -34,6 +34,14 @@ const FLUX_REMOTE = process.env.FLUX_REMOTE || 'https://github.com/RunOnFlux/flu // LC_ALL=C pins the sort. const TREE_HASH_PIPELINE = "find ./ZelBack -type f -exec md5sum {} + | awk '{print $1}' | LC_ALL=C sort | md5sum | awk '{printf $1}'"; +// md5 of an empty stream. The pipeline yields it whenever nothing was hashed -- ZelBack absent +// (find errors; without pipefail the status is awk's, so the failure is otherwise invisible) or +// present but holding no regular files (find succeeds and emits nothing, which pipefail cannot +// see). Both must be refused: the value is a well-formed 32-hex hash that means "a node whose +// ZelBack contains no files is genuine FluxOS", and membership is monotonic, so listing it once +// costs a cull PR to undo. +const EMPTY_TREE_HASH = 'd41d8cd98f00b204e9800998ecf8427e'; + // A raw 32-byte Ed25519 seed is not directly importable; Node wants PKCS8. The prefix is fixed for // the algorithm, so prepending it is enough. const PKCS8_ED25519_PREFIX = Buffer.from('302e020100300506032b657004220420', 'hex'); @@ -161,10 +169,15 @@ function deriveTreeHash(cache, sha) { const worktree = path.join(parent, 'wt'); try { git(['worktree', 'add', '--quiet', '--detach', worktree, sha], { cwd: cache }); - const hash = execFileSync('bash', ['-c', TREE_HASH_PIPELINE], { cwd: worktree, encoding: 'utf8' }).trim(); + // -o pipefail rather than folding it into the constant, so TREE_HASH_PIPELINE stays byte-identical + // to the one flux CI runs -- that identity is what makes claimed_hash comparable at all. + const hash = execFileSync('bash', ['-o', 'pipefail', '-c', TREE_HASH_PIPELINE], { cwd: worktree, encoding: 'utf8' }).trim(); if (!/^[0-9a-f]{32}$/.test(hash)) { throw new Error(`tree hash pipeline produced "${hash}"`); } + if (hash === EMPTY_TREE_HASH) { + throw new Error('nothing was hashed -- this commit has no ZelBack files. Refusing to list the empty-tree hash'); + } return hash; } finally { try { From 636fca10b8f216f62a8055e25cc8e451a543bcf9 Mon Sep 17 00:00:00 2001 From: David White Date: Tue, 25 Aug 2026 12:21:34 +0100 Subject: [PATCH 14/17] docs: state what a dispatch is actually trusted to name The header claimed "a hash value can enter the list through no other path". That is false, and it was load-bearing: it is the reasoning behind accepting looser pins elsewhere, so it needed correcting whether or not the gap gets closed. A fork network shares one object store, so every commit ever pushed to any public fork of flux stays anonymously fetchable by SHA from the official remote -- and a dispatched commit matching no ref is derived anyway, on purpose, because a branch can move past a commit before the signer looks. "Fetchable from RunOnFlux/flux" is therefore a much larger set than "a tree RunOnFlux authored", and a dispatch credential can name anything in it. What is true is narrower: a dispatch carries pointers rather than hash values, so the credential needs no write authority. That bounds the credential; it does not vouch for the tree. And the credential lives in flux's repository secrets, reachable by anyone who can land a workflow change on any branch there -- who can already get a hash listed by pushing one. Same trust boundary, not a defence against it. Wording only. The ancestry check that would close the gap is not taken: it would refuse commits on no ref, which are legitimate after a force-push or a deleted branch and are supported deliberately. Reported by Cabecinha84 on #2. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01HQKpZbuxYqrWeErU7vKgoA --- .github/workflows/sign-hashlist.yml | 6 ++++-- scripts/sign-hashlist.js | 14 +++++++++++--- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/.github/workflows/sign-hashlist.yml b/.github/workflows/sign-hashlist.yml index cbb2df51..178e2601 100644 --- a/.github/workflows/sign-hashlist.yml +++ b/.github/workflows/sign-hashlist.yml @@ -2,8 +2,10 @@ name: sign-hashlist # The single writer of everything src/hashes/ serves. It derives what it signs: a run fetches new # flux commits from the official repository, hashes their trees itself, and publishes list, signed -# document and provenance in one commit. Requests carry pointers, never content -- flux CI's -# dispatch token has Actions permission only. +# document and provenance in one commit. Requests carry pointers, never hash values -- flux CI's +# dispatch token has Actions permission only. That bounds the credential rather than vouching for +# the tree: a fork network shares one object store, so a dispatched commit may be any commit ever +# pushed to flux or to a public fork of it. scripts/sign-hashlist.js states the bound in full. # # Level-triggered: every run reconciles the full delta between the flux remote's refs and the # snapshot in the provenance record, so a dispatch lost to the concurrency group's diff --git a/scripts/sign-hashlist.js b/scripts/sign-hashlist.js index b2f07c0e..71d101e9 100644 --- a/scripts/sign-hashlist.js +++ b/scripts/sign-hashlist.js @@ -5,9 +5,17 @@ // Reconciles the published list against RunOnFlux/flux itself: diffs the remote's refs against the // snapshot in the provenance record, fetches each new commit, computes its ZelBack tree hash from // the bytes it fetched, and emits hashes.js, the signed document and the provenance record together. -// A hash value can enter the list through no other path. A dispatch carries pointers, never -// content, so the credential that sends one needs no write authority -- and the signature means -// "this commit's tree hashes to this value", not "this was in the repository when I ran". +// A dispatch carries pointers, never hash values, so the credential that sends one needs no write +// authority -- and the signature means "this commit's tree hashes to this value", not "this was in +// the repository when I ran". +// +// That bounds the credential; it does not vouch for the tree. GitHub shares one object store across +// a fork network, so every commit ever pushed to any public fork of flux stays anonymously fetchable +// by SHA from the official remote -- and a dispatched commit matching no ref is still derived, on +// purpose, because a branch can move past a commit before we look. So a dispatch is trusted to name +// a commit reachable in flux's object store, forks included, which is a much larger set than "a tree +// RunOnFlux authored". The credential is held only by principals who can already land a branch on +// flux and get a hash listed that way; this is the same trust boundary, not a defence against it. // // The payload is signed and transmitted as exact bytes in base64, so verification never depends on // the signer and the verifier agreeing about JSON key order or whitespace. From f2df1298bf3b7dbfde22883a8f9002f840968175 Mon Sep 17 00:00:00 2001 From: David White Date: Tue, 25 Aug 2026 13:21:22 +0100 Subject: [PATCH 15/17] chore(ci): move the actions to the current majors actions/checkout v4 -> v7, actions/setup-node v4 -> v7. checkout v7 refuses to fetch fork pull request code under pull_request_target and workflow_run, the configuration behind pwn request attacks. Neither workflow here uses those triggers -- validate runs on plain pull_request, which GitHub already denies secrets and write to -- so this changes nothing today. It is worth taking anyway: it makes the dangerous configuration fail closed if one is ever added, rather than depending on the next author knowing why not to. setup-node's v5 automatic caching does not engage here: it keys off a packageManager field in package.json, which this repository does not set, and validate installs nothing regardless -- setup-node only provides the runtime. Prompted by Cabecinha84's note on #2 about fork pull requests executing the list file. That remains true and remains negligible: fork runs get no secrets, a read-only token and an ephemeral runner. Its permissions: {} suggestion is not taken -- the workflow already declares contents: read, which on a public repository grants only what is already public. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01HQKpZbuxYqrWeErU7vKgoA --- .github/workflows/sign-hashlist.yml | 4 ++-- .github/workflows/validate.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/sign-hashlist.yml b/.github/workflows/sign-hashlist.yml index 178e2601..3d6a16f5 100644 --- a/.github/workflows/sign-hashlist.yml +++ b/.github/workflows/sign-hashlist.yml @@ -55,8 +55,8 @@ jobs: runs-on: ubuntu-latest environment: hashlist-signing steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@v7 + - uses: actions/setup-node@v7 with: node-version: '20' diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 2dbc855d..139524ec 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -17,11 +17,11 @@ jobs: validate: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v7 with: # The outputs guard below diffs the merge commit against its first parent. fetch-depth: 2 - - uses: actions/setup-node@v4 + - uses: actions/setup-node@v7 with: node-version: '20' From 1892bdf1db10f241a59eec218b0ab2b5a718a8c5 Mon Sep 17 00:00:00 2001 From: David White Date: Tue, 25 Aug 2026 13:47:19 +0100 Subject: [PATCH 16/17] test: land the reconciler harness and run it in CI The 43 assertions that justified this PR lived in a session scratchpad and were never committed, so npm test printed "No tests have been implemented" and exited 1. The evidence was real but unreproducible by anyone else, and nothing guarded against regressions. The harness drives scripts/sign-hashlist.js against a git remote it builds itself: bootstrap grandfathering, derivation and labelling, no-op and state-only runs, an orphaned dispatched commit, claimed-hash mismatch, culls and their retained audit rows, duplicates, tag annotation, corrupt provenance, corrupt ledger, a wiped document, and v1 record compatibility. No network, no secrets, nothing outside a temp dir -- so it runs on fork pull requests like the rest of validate. Restructured to live in the repository: the working copy, fake remote and all scratch output moved into mktemp with a trap, the source is taken from the repository root rather than a sibling checkout, and .git and node_modules are excluded from the copy. Verified it leaves the working tree clean. Two cases are new, covering the defect this PR fixed one commit earlier. They are split because the two ways in have different guards, and each asserts the message it should die on -- asserting only "the run went red" passes for any reason and tests nothing. Case 17 removes ZelBack and expects the pipeline's own failure; case 18 gives ZelBack only symlinks, so find matches nothing and exits 0, which pipefail cannot see and only the value check catches. Git cannot store an empty directory, so a symlink is what "present but nothing to hash" actually looks like in a commit. Mutation-tested, because coverage proves execution and not that an assertion would notice. Deleting the empty-hash guard fails case 18 with "EMPTY HASH REACHED THE LIST"; deleting pipefail fails case 17, which then observes the guard firing instead. An earlier draft of both cases passed under both mutations -- it asserted redness without asserting cause, and its "empty directory" fixture had silently collapsed into the absent-directory case. Raised by Cabecinha84 on #2 as his main process ask. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01HQKpZbuxYqrWeErU7vKgoA --- .github/workflows/validate.yml | 7 + package.json | 2 +- test/reconciler/run-tests.sh | 251 +++++++++++++++++++++++++++++++++ 3 files changed, 259 insertions(+), 1 deletion(-) create mode 100755 test/reconciler/run-tests.sh diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 139524ec..7fee1301 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -47,3 +47,10 @@ jobs: fi - run: node scripts/validate.js + + # The reconciler's own logic, driven against a local fake flux remote: bootstrap, derivation, + # claimed-hash mismatch, culls, duplicates, corruption, a wiped document, v1 compatibility, + # and the empty-tree hash. No network and no secrets -- it builds its own git remote in a + # temp dir, so it runs on fork pull requests like everything else here. + - name: Reconciler tests + run: npm test diff --git a/package.json b/package.json index 3b2499fa..b7e94925 100644 --- a/package.json +++ b/package.json @@ -5,7 +5,7 @@ "main": "index.js", "scripts": { "start": "nodemon index.js", - "test": "echo 'No tests have been implemented' && exit 1", + "test": "bash test/reconciler/run-tests.sh", "lint": "eslint ./ --fix" }, "author": "Tadeas Kmenta", diff --git a/test/reconciler/run-tests.sh b/test/reconciler/run-tests.sh new file mode 100755 index 00000000..c4683717 --- /dev/null +++ b/test/reconciler/run-tests.sh @@ -0,0 +1,251 @@ +#!/usr/bin/env bash +# Local logic tests for the reconciler. GitHub platform semantics are proven separately (the +# 2026-08-24 sandbox proofs); this drives scripts/sign-hashlist.js against a local fake flux +# remote through every state transition the design names. +set -u + +# Everything scratch lives in a temp dir: the repository itself is never written to, so this is +# safe to run from a clean checkout and leaves nothing behind. +HERE="$(cd "$(dirname "$0")" && pwd)" +SRC="$(cd "$HERE/../.." && pwd)" +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT +WORK="$TMP/work" +FLUX="$TMP/fake-flux" +export HASHLIST_SIGNING_SEED_B64="MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY=" # test only +PASS=0; FAIL=0 + +say() { printf '\n=== %s\n' "$*"; } +ok() { PASS=$((PASS+1)); printf 'PASS %s\n' "$*"; } +bad() { FAIL=$((FAIL+1)); printf 'FAIL %s\n' "$*"; } + +run_signer() { # -> stdout line in $RESULT, exit code in $RC + cd "$WORK" + RESULT=$(node scripts/sign-hashlist.js 2>"$TMP/last-stderr.txt"); RC=$? + cd "$HERE" +} + +jsonq() { node -e "const j=JSON.parse(require('fs').readFileSync('$1','utf8')); console.log($2);"; } +listq() { node -e "console.log(require('$WORK/src/hashes/hashes.js').getHashes()$1);"; } +tree_hash() { (cd "$1" && find ./ZelBack -type f -exec md5sum {} + | awk '{print $1}' | LC_ALL=C sort | md5sum | awk '{printf $1}'); } + +# ---------- setup: fake flux remote with two files, master + dev ---------- +mkdir -p "$FLUX/ZelBack/src" +git -C "$FLUX" init -q -b master +git -C "$FLUX" config user.email t@t; git -C "$FLUX" config user.name t +git -C "$FLUX" config uploadpack.allowAnySHA1InWant true +echo 'alpha' > "$FLUX/ZelBack/src/a.js"; echo 'beta' > "$FLUX/ZelBack/src/b.js"; echo 'root' > "$FLUX/readme.md" +git -C "$FLUX" add -A; git -C "$FLUX" commit -qm c1 +HASH1=$(tree_hash "$FLUX") + +mkdir -p "$WORK" +tar -c -C "$SRC" --exclude=.git --exclude=node_modules . | tar -x -C "$WORK" +rm -f "$WORK/src/hashes/hashlist-signed.json" "$WORK/src/hashes/provenance.json" +# Pin the test key in the work copy, the same substitution the sandbox rehearsals make. +node -e " +const fs=require('fs'); +const {privateKeyFromSeed, rawPublicKey}=require('$WORK/scripts/sign-hashlist.js'); +const pub=rawPublicKey(privateKeyFromSeed(process.env.HASHLIST_SIGNING_SEED_B64)).toString('hex'); +const p='$WORK/scripts/verify-hashlist.js'; +const src=fs.readFileSync(p,'utf8').replace(/const PINNED_PUBLIC_KEYS = \[[^\]]*\];/, \"const PINNED_PUBLIC_KEYS = [\n '\"+pub+\"',\n];\"); +fs.writeFileSync(p,src); +" +printf 'function getHashes() {\n return [\n '\''%s'\'',\n '\''ffffffffffffffffffffffffffffffff'\'',\n ];\n}\n\nmodule.exports = {\n getHashes,\n};\n' "$HASH1" > "$WORK/src/hashes/hashes.js" +export FLUX_REMOTE="$FLUX" +export FLUX_CACHE_DIR="$TMP/cache" + +# ---------- 1. bootstrap: signs the grandfathered list, derives nothing ---------- +say "1 bootstrap" +run_signer +[ "$RC" = 0 ] && [ "$RESULT" = "changed=signed" ] && ok "bootstrap signs" || bad "bootstrap: rc=$RC result=$RESULT" +[ "$(jsonq "$WORK/src/hashes/provenance.json" 'j.signed.seq')" = 1 ] && ok "seq 1" || bad "seq" +[ "$(jsonq "$WORK/src/hashes/provenance.json" "j.hashes['$HASH1'].derived")" = "false" ] && ok "grandfathered underived" || bad "grandfather" +[ "$(jsonq "$WORK/src/hashes/provenance.json" "j.hashes['$HASH1'].commit")" = "null" ] && ok "grandfathered commit null" || bad "commit null" +[ "$(jsonq "$WORK/src/hashes/provenance.json" 'Object.keys(j.refs).length')" = 1 ] && ok "refs snapshotted" || bad "refs" +[ "$(jsonq "$WORK/src/hashes/provenance.json" 'Object.keys(j.commits).length')" = 0 ] && ok "nothing derived at bootstrap" || bad "derived at bootstrap" +node -e " +const {verifyDocument} = require('$WORK/scripts/verify-hashlist.js'); +const {privateKeyFromSeed, rawPublicKey} = require('$WORK/scripts/sign-hashlist.js'); +const pub = rawPublicKey(privateKeyFromSeed(process.env.HASHLIST_SIGNING_SEED_B64)).toString('hex'); +const doc = JSON.parse(require('fs').readFileSync('$WORK/src/hashes/hashlist-signed.json','utf8')); +const p = verifyDocument(doc, [pub]); +if (p.seq !== 1 || p.hashes.length !== 2) throw new Error('payload wrong'); +" && ok "document verifies under the test key" || bad "verify" + +# ---------- 2. unchanged rerun ---------- +say "2 unchanged rerun" +run_signer +[ "$RC" = 0 ] && [ "$RESULT" = "changed=false" ] && ok "no-op" || bad "rerun: rc=$RC result=$RESULT" + +# ---------- 3. new ZelBack commit on master: derived, appended, seq 2 ---------- +say "3 new tree commit" +echo 'gamma' >> "$FLUX/ZelBack/src/a.js"; git -C "$FLUX" commit -qam c2 +HASH2=$(tree_hash "$FLUX"); SHA2=$(git -C "$FLUX" rev-parse HEAD) +run_signer +[ "$RC" = 0 ] && [ "$RESULT" = "changed=signed" ] && ok "signed" || bad "rc=$RC result=$RESULT" +[ "$(jsonq "$WORK/src/hashes/provenance.json" 'j.signed.seq')" = 2 ] && ok "seq 2" || bad "seq" +[ "$(listq ".includes('$HASH2')")" = "true" ] && ok "new hash listed" || bad "not listed" +[ "$(listq '.length')" = 3 ] && ok "appended, nothing lost" || bad "length" +[ "$(jsonq "$WORK/src/hashes/provenance.json" "j.hashes['$HASH2'].branch")" = "master" ] && ok "own-view branch label" || bad "label" +[ "$(jsonq "$WORK/src/hashes/provenance.json" "j.hashes['$HASH2'].derived")" = "true" ] && ok "derived" || bad "derived flag" +[ "$(jsonq "$WORK/src/hashes/provenance.json" "j.commits['$SHA2']")" = "$HASH2" ] && ok "commits map" || bad "commits map" + +# ---------- 4. non-ZelBack commit: same tree hash, state-only, no seq burn ---------- +say "4 non-tree commit" +echo 'docs' >> "$FLUX/readme.md"; git -C "$FLUX" commit -qam c3 +run_signer +[ "$RC" = 0 ] && [ "$RESULT" = "changed=state" ] && ok "state only" || bad "rc=$RC result=$RESULT" +[ "$(jsonq "$WORK/src/hashes/provenance.json" 'j.signed.seq')" = 2 ] && ok "no sequence burned" || bad "seq burned" + +# ---------- 5. dispatch hint for an orphaned commit ---------- +say "5 orphaned dispatch" +git -C "$FLUX" checkout -qb doomed +echo 'delta' > "$FLUX/ZelBack/src/d.js"; git -C "$FLUX" add -A; git -C "$FLUX" commit -qm c4 +HASH4=$(tree_hash "$FLUX"); SHA4=$(git -C "$FLUX" rev-parse HEAD) +git -C "$FLUX" checkout -q master; git -C "$FLUX" branch -qD doomed +DISPATCH_COMMIT=$SHA4 DISPATCH_REF=doomed DISPATCH_REF_TYPE=branch bash -c 'cd '"$WORK"' && node scripts/sign-hashlist.js' >"$TMP/out5.txt" 2>"$TMP/last-stderr.txt"; RC=$? +[ "$RC" = 0 ] && [ "$(cat "$TMP/out5.txt")" = "changed=signed" ] && ok "orphan derived via hint" || bad "rc=$RC $(cat "$TMP/out5.txt")" +[ "$(jsonq "$WORK/src/hashes/provenance.json" "j.hashes['$HASH4'].branch")" = "doomed" ] && ok "dispatch fallback label" || bad "label" + +# ---------- 6. claimed_hash mismatch is a red run that publishes nothing ---------- +say "6 claimed mismatch" +echo 'epsilon' > "$FLUX/ZelBack/src/e.js"; git -C "$FLUX" add -A; git -C "$FLUX" commit -qm c5 +SHA5=$(git -C "$FLUX" rev-parse HEAD) +BEFORE=$(cat "$WORK/src/hashes/provenance.json") +DISPATCH_COMMIT=$SHA5 DISPATCH_CLAIMED_HASH=00000000000000000000000000000000 bash -c 'cd '"$WORK"' && node scripts/sign-hashlist.js' >/dev/null 2>"$TMP/last-stderr.txt"; RC=$? +grep -q 'environment drift or a hostile dispatch' "$TMP/last-stderr.txt" && [ "$RC" != 0 ] && ok "red run" || bad "rc=$RC" +[ "$(cat "$WORK/src/hashes/provenance.json")" = "$BEFORE" ] && ok "published nothing" || bad "state leaked" + +# ---------- 7. correct claimed_hash passes; catches up c5 too ---------- +say "7 claimed match" +HASH5=$(tree_hash "$FLUX") +DISPATCH_COMMIT=$SHA5 DISPATCH_CLAIMED_HASH=$HASH5 bash -c 'cd '"$WORK"' && node scripts/sign-hashlist.js' >"$TMP/out7.txt" 2>"$TMP/last-stderr.txt"; RC=$? +[ "$RC" = 0 ] && [ "$(cat "$TMP/out7.txt")" = "changed=signed" ] && [ "$(listq ".includes('$HASH5')")" = "true" ] && ok "derived with matching claim" || bad "rc=$RC" + +# ---------- 8. cull through the ledger ---------- +say "8 cull" +printf '{\n "culls": [\n { "hash": "ffffffffffffffffffffffffffffffff", "reason": "test cull", "date": "2026-08-24" }\n ]\n}\n' > "$WORK/src/hashes/ledger.json" +run_signer +[ "$RC" = 0 ] && [ "$RESULT" = "changed=signed" ] && ok "resigned" || bad "rc=$RC result=$RESULT" +[ "$(listq ".includes('ffffffffffffffffffffffffffffffff')")" = "false" ] && ok "culled entry gone" || bad "still listed" +[ "$(jsonq "$WORK/src/hashes/provenance.json" "j.hashes['ffffffffffffffffffffffffffffffff'] !== undefined")" = "true" ] && ok "cull keeps its audit row" || bad "row lost" + +# ---------- 9. duplicate entry in the list is refused ---------- +say "9 duplicate refused" +node -e " +const fs=require('fs'); const p='$WORK/src/hashes/hashes.js'; +fs.writeFileSync(p, fs.readFileSync(p,'utf8').replace(\" '$HASH1',\", \" '$HASH1',\n '$HASH1',\")); +" +run_signer +[ "$RC" != 0 ] && grep -q 'duplicates' "$TMP/last-stderr.txt" && ok "red on duplicates" || bad "rc=$RC" +node -e " +const fs=require('fs'); const p='$WORK/src/hashes/hashes.js'; +fs.writeFileSync(p, fs.readFileSync(p,'utf8').replace(\" '$HASH1',\n '$HASH1',\", \" '$HASH1',\")); +" + +# ---------- 10. tag on a known commit annotates without a sequence ---------- +say "10 tag annotation" +git -C "$FLUX" tag v1-test "$SHA2" +run_signer +[ "$RC" = 0 ] && [ "$RESULT" = "changed=state" ] && ok "state only" || bad "rc=$RC result=$RESULT" +[ "$(jsonq "$WORK/src/hashes/provenance.json" "j.hashes['$HASH2'].tag")" = "v1-test" ] && ok "row annotated" || bad "tag" + +# ---------- 11. annotated tag on a NEW commit derives with tag label ---------- +say "11 tagged new commit" +git -C "$FLUX" checkout -q master +echo 'zeta' > "$FLUX/ZelBack/src/z.js"; git -C "$FLUX" add -A; git -C "$FLUX" commit -qm c6 +git -C "$FLUX" tag -a v2-test -m 'release' +HASH6=$(tree_hash "$FLUX") +run_signer +[ "$RC" = 0 ] && [ "$RESULT" = "changed=signed" ] && ok "signed" || bad "rc=$RC result=$RESULT" +TAGLABEL=$(jsonq "$WORK/src/hashes/provenance.json" "j.hashes['$HASH6'].tag || j.hashes['$HASH6'].branch") +[ -n "$TAGLABEL" ] && ok "labelled ($TAGLABEL)" || bad "no label" + +# ---------- 12. corrupt provenance is a red run ---------- +say "12 corrupt provenance" +cp "$WORK/src/hashes/provenance.json" "$TMP/prov-backup.json" +echo 'not json' > "$WORK/src/hashes/provenance.json" +run_signer +[ "$RC" != 0 ] && ok "red on corrupt provenance" || bad "rc=$RC" +cp "$TMP/prov-backup.json" "$WORK/src/hashes/provenance.json" + +# ---------- 13. corrupt ledger is a red run ---------- +say "13 corrupt ledger" +cp "$WORK/src/hashes/ledger.json" "$TMP/ledger-backup.json" +echo '{"culls": [{"hash": "short"}]}' > "$WORK/src/hashes/ledger.json" +run_signer +[ "$RC" != 0 ] && grep -q 'ledger cull' "$TMP/last-stderr.txt" && ok "red on bad cull" || bad "rc=$RC" +cp "$TMP/ledger-backup.json" "$WORK/src/hashes/ledger.json" + +# ---------- 14. validate.js green on the final state ---------- +say "14 validate green" +(cd "$WORK" && node scripts/validate.js 2>"$TMP/validate-out.txt"); RC=$? +[ "$RC" = 0 ] && ok "validate green" || { bad "validate rc=$RC"; cat "$TMP/validate-out.txt"; } + +# ---------- 15. validate red when the signed document is wiped ---------- +say "15 wiped document" +mv "$WORK/src/hashes/hashlist-signed.json" "$HERE/signed-backup.json" +(cd "$WORK" && node scripts/validate.js 2>"$TMP/validate-out.txt"); RC=$? +[ "$RC" != 0 ] && grep -q 'no signed document' "$TMP/validate-out.txt" && ok "wipe is loud" || bad "rc=$RC" +mv "$HERE/signed-backup.json" "$WORK/src/hashes/hashlist-signed.json" + +# ---------- 16. v1 provenance (rows + signed, no refs): bootstrap preserves the sequence ---------- +say "16 v1 compatibility" +node -e " +const fs=require('fs'); const p='$WORK/src/hashes/provenance.json'; +const j=JSON.parse(fs.readFileSync(p,'utf8')); +delete j.refs; delete j.commits; +Object.values(j.hashes).forEach((row)=>delete row.derived); +fs.writeFileSync(p, JSON.stringify(j,null,2)+'\n'); +" +SEQ_BEFORE=$(jsonq "$WORK/src/hashes/provenance.json" 'j.signed.seq') +run_signer +[ "$RC" = 0 ] && ok "v1 record accepted (result=$RESULT)" || bad "rc=$RC" +[ "$(jsonq "$WORK/src/hashes/provenance.json" 'Object.keys(j.refs).length > 0')" = "true" ] && ok "snapshot rebuilt" || bad "no snapshot" +SEQ_AFTER=$(jsonq "$WORK/src/hashes/provenance.json" 'j.signed.seq') +[ "$SEQ_AFTER" -ge "$SEQ_BEFORE" ] && ok "sequence preserved ($SEQ_BEFORE -> $SEQ_AFTER)" || bad "sequence restarted" + +# ---------- 17-18. nothing was hashed: the empty-tree hash must never be listed ---------- +# The pipeline yields d41d8cd98f00b204e9800998ecf8427e -- the md5 of an empty stream -- whenever +# nothing was hashed. It is a well-formed 32-hex value, so nothing downstream tells it apart from +# a real tree hash, and it would mean "a node whose ZelBack holds no regular files is genuine +# FluxOS". Membership is monotonic, so listing it once costs a cull PR. +# +# The two ways in have DIFFERENT guards, so each case asserts the message it should die on -- +# asserting only "the run went red" passes for any reason at all and tests nothing. +EMPTY_HASH=d41d8cd98f00b204e9800998ecf8427e + +say "17 ZelBack absent (find errors; caught by pipefail)" +git -C "$FLUX" checkout -q -b nozelback master +git -C "$FLUX" rm -rq ZelBack +git -C "$FLUX" commit -qm "drop ZelBack entirely" +SHA_NOZB=$(git -C "$FLUX" rev-parse HEAD) +BEFORE=$(cat "$WORK/src/hashes/provenance.json") +DISPATCH_COMMIT=$SHA_NOZB run_signer +grep -q 'Command failed' "$TMP/last-stderr.txt" && [ "$RC" != 0 ] \ + && ok "red on the pipeline's own failure" || bad "rc=$RC -- expected a pipefail death, got: $(tail -1 "$TMP/last-stderr.txt")" +[ "$(listq ".includes('$EMPTY_HASH')")" = "false" ] && ok "empty hash not listed" || bad "EMPTY HASH REACHED THE LIST" +[ "$(cat "$WORK/src/hashes/provenance.json")" = "$BEFORE" ] && ok "published nothing" || bad "state leaked" + +say "18 ZelBack holds no regular files (find exits 0; pipefail cannot see it)" +# Only symlinks and directories: find -type f matches nothing and exits 0, so the pipeline +# succeeds and returns the empty hash. Git cannot store an empty directory, but it stores a +# symlink (mode 120000) -- this is the reachable shape of "present but nothing to hash". +git -C "$FLUX" checkout -q -b emptyzelback master +git -C "$FLUX" rm -rq ZelBack +mkdir -p "$FLUX/ZelBack/sub" +ln -s /dev/null "$FLUX/ZelBack/link.js" +ln -s /dev/null "$FLUX/ZelBack/sub/deep.js" +git -C "$FLUX" add -A +git -C "$FLUX" commit -qm "ZelBack with no regular files" +SHA_EMPTYZB=$(git -C "$FLUX" rev-parse HEAD) +BEFORE=$(cat "$WORK/src/hashes/provenance.json") +DISPATCH_COMMIT=$SHA_EMPTYZB run_signer +grep -q 'nothing was hashed' "$TMP/last-stderr.txt" && [ "$RC" != 0 ] \ + && ok "red on the empty-hash guard" || bad "rc=$RC -- expected the guard to fire, got: $(tail -1 "$TMP/last-stderr.txt")" +[ "$(listq ".includes('$EMPTY_HASH')")" = "false" ] && ok "empty hash not listed" || bad "EMPTY HASH REACHED THE LIST" +[ "$(cat "$WORK/src/hashes/provenance.json")" = "$BEFORE" ] && ok "published nothing" || bad "state leaked" +git -C "$FLUX" checkout -q master + +printf '\n%d passed, %d failed\n' "$PASS" "$FAIL" +[ "$FAIL" = 0 ] From 13732fed258349ed7bfdc8e9755d53fc1ab45417 Mon Sep 17 00:00:00 2001 From: David White Date: Tue, 25 Aug 2026 13:51:00 +0100 Subject: [PATCH 17/17] docs: say what the signature attests, and log the host-key rotation touchpoint Two review notes from Cabecinha84 on #2, both about a reader over-reading what is there. The cutover carries the existing entries across as derived: false, which is correct for a non-destructive cutover but means the signature attests provenance only for what the signer derived afterwards. Said plainly next to the row description, so nobody reads a grandfathered row as "the signer fetched this commit and checked it". The workflow pins GitHub's SSH host key rather than trusting ssh-keyscan at run time. That is the right call and it is also a rotation touchpoint: if GitHub rotates that key the push fails host verification and the run goes red looking exactly like a credential fault. Recorded beside the deploy key rotation steps so it is diagnosed in seconds. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01HQKpZbuxYqrWeErU7vKgoA --- SIGNING.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/SIGNING.md b/SIGNING.md index 99d0b6e1..e205cfe1 100644 --- a/SIGNING.md +++ b/SIGNING.md @@ -26,7 +26,11 @@ workflow**, which derives every post-cutover entry itself from commits it fetche makes an entry attributable later: the list itself is opaque md5s, and the commit that produced an entry can stop existing (a force-push, a branch deleted after merge). First attribution wins; a new tag on a known commit annotates the existing row. Rows with `derived: false` predate - derivation (grandfathered at cutover) and have no commit to point at. + derivation (grandfathered at cutover) and have no commit to point at. **Read the signature accordingly**: + it says these are the hashes this repository published at this sequence, and for `derived: true` + rows it additionally means the signer fetched that commit and computed that hash itself. It + attests no provenance for grandfathered rows — those were carried across at cutover from the + unsigned list, on the authority of whatever published them at the time. - a **commits map** (`sha → hash`) so nothing is fetched or hashed twice, and a **refs snapshot** of the flux remote, which is what the reconciler diffs against — a publication request that gets lost is repaired by the next run reconciling the full delta. @@ -97,6 +101,12 @@ rotate: generate a fresh keypair, replace the repository deploy key and the envi the ruleset's `DeployKey` bypass covers whatever write keys the repository holds, so it needs no change — which is also why the repository must hold exactly this one write deploy key. +The workflow also **pins GitHub's SSH host key** — a single `ssh-ed25519` line written to a +`known_hosts` file for the push, rather than trusting whatever `ssh-keyscan` returns at run time. +It is a rotation touchpoint: if GitHub ever rotates that key the push fails host verification and +the signing run goes red, which looks like a credential fault and is not one. The fix is to update +the pinned line in `.github/workflows/sign-hashlist.yml` against GitHub's published fingerprints. + ## Trust Anyone who can land a workflow change on `master` can read the secrets; a GitHub secret is an