Skip to content

ci: keep the code map's indexes for pin-only and prose-only commits; cancel superseded PR runs - #1116

Merged
cryptskii merged 3 commits into
mainfrom
chore/ci-code-map-cache-and-cancel-superseded-runs
Oct 5, 2026
Merged

cryptskii merged 3 commits into
mainfrom
chore/ci-code-map-cache-and-cancel-superseded-runs

Conversation

@cryptskii

@cryptskii cryptskii commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

The Code map job is the long pole: about 20 minutes on every push, including commits that only re-pin. This PR keeps its expensive half between commits whose code and configuration are the same, and cancels pull-request runs that a newer push has superseded.

Code map: keep the indexes between pin-only and prose-only commits

Where the job's time goes (run 37275560695):

Step Time
Fixture 129 s
Map (three rust-analyzer indexes) 476 s
Pins' self-tests 59 s
Mutation cases 456 s
Everything else seconds

The fixture, the indexes and the mutation cases read only code, configuration and this workflow. They never read a pin or prose. Checked against the sources:

  • The indexes read *.rs/*.kt/*.ts/*.proto/…, the rust-analyzer configs, Cargo.lock/Cargo.toml and the toolchain.
  • The mutation cases read the fixture, ci/*.tsv and the real map.
  • The real map depends on INTENT_MANIFEST.tsv through its root queries, so the manifest stays in the key.

That half is now kept in the Actions cache:

  • Key: every tracked file except *.md and specs/requirements/INTENT_PINS.tsv, plus the runner image (ImageOS, ImageVersion), salted code-map-v1. The key must match exactly: there are no restore-keys, so a near match is a miss.
  • On a hit: the fixture, Index and mutation steps are skipped.
  • Every commit still runs: the map's tables (rebuilt from the indexes and the current manifest), its check, the intent manifest and every pin, and the pins' self-tests. intent_pins.fresh() still refuses a map whose fingerprint is not this tree's.
  • Saved only by a run that verified it: the run must have built the indexes and passed the fixture, map, check and every mutation case. The intent manifest and the pins need not hold, because the commit that re-pins the rows a run left stale is the one that reuses the cache.

Why the key is wider than the map's own fingerprint: the fingerprint covers sources plus a few configs, but not Cargo.toml features, the Makefile, the ci/*.py scripts or the workflow. That never mattered while every run rebuilt. As a cache key it would let a features-only change reuse a map built under different features.

The key was checked locally. Changing INTENT_PINS.tsv or CONFORMANCE_GAPS.md leaves it the same. Changing INTENT_MANIFEST.tsv, Cargo.toml, dsm/src/lib.rs or code-map.yml changes it.

Makefile: requirement-map is now requirement-map-indexes followed by requirement-map-tables. Every recipe line is unchanged, and make requirement-map does what it did (dry-run checked).

Cancel superseded runs

CI, CodeQL and Code map now have a concurrency group per pull request with cancel-in-progress for pull_request events, so a newer push cancels the older run. Pushes to main, schedules and dispatches group by commit SHA and are never cancelled or queued: each merge commit keeps its own result.

Validation

This PR's first Code map run misses the cache by construction. It runs the split targets end to end and saves. Re-running that job on the same commit hits the same key and exercises the cached path. Both results will be posted here.

Not in this PR

A self-hosted runner, which would cut the cold setup (toolchain, NDK, cargo cache).

… or prose; cancel superseded PR runs

Code map: about seventeen of its twenty minutes (the fixture run, the three
rust-analyzer indexes, the mutation cases) read code, configuration and the
workflow, never a pin or prose. That half is now kept in the Actions cache
under a key of every tracked file except *.md and
specs/requirements/INTENT_PINS.tsv, plus the runner image. The key must match
exactly; there are no restore-keys. A commit that re-pins or edits only prose
restores the indexes and fixture run of the commit with the same key, where
they were verified, and skips that half. Everything that reads the changed
files runs on every commit: the map's tables (from the indexes and the
intent manifest's root questions), its check, the intent manifest and every
pin, and the pins' own tests. A map is still refused unless its fingerprint
is this tree's. The cache is saved only by a run that built the indexes and
passed the fixture, the map, its check and every mutation case; the intent
manifest and the pins need not hold, since the commit re-pinning the rows a
run left stale is the one that reuses it.

Makefile: requirement-map is now requirement-map-indexes then
requirement-map-tables. The recipe lines are unchanged and make
requirement-map does what it did.

CI, CodeQL, Code map: a newer push to the same pull request cancels the
older run. Pushes to main, schedules and dispatches are their own groups and
are never cancelled.
The first cached run failed in requirement-map-tables: "dsm_sdk has a build
script but cargo left no output for it under ./target: the index did not
run it". The map reads each shipped crate's build-script `output` (what it
printed to cargo) to refuse one that sets a cfg; the indexes' cargo runs
leave them under target/debug/build and target/<triple>/debug/build, and
rust-cache keeps no workspace crate's. They are kept with the indexes now.
Nothing else under ./target is read: the map's tables name no definition
there.
@cryptskii

Copy link
Copy Markdown
Collaborator Author

Both paths are now validated on ec5145fdf (Code map run 37303915230):

Step Attempt 1 (cache miss: full build, then saves) Attempt 2 (cache hit: same commit)
Fixture 37 s skipped
Index (three rust-analyzer indexes) 252 s skipped
Map (tables from the indexes and the manifest) 29 s 77 s
Check 6 s 7 s
Intent manifest and pins ✓ ✓ 0 failing rows, 0 failing pins, 635 pinned
Pins' self-tests 42 s 59 s
Mutation cases 285 s skipped
Job 13.1 min 4.0 min

On the hit, the log reads Cache restored from key: code-map-v1-…. The Map step takes longer there because it builds requirement_map itself; on the miss path, the fixture step had already built it.

The first cached attempt (run 37295619885, attempt 2) failed in the tables step: dsm_sdk has a build script but cargo left no output for it under ./target. The map reads each shipped crate's build-script output file to refuse one that sets a cfg. Those files are written while indexing, and the CI cargo cache keeps no workspace crate's copy. ec5145fdf keeps them with the indexes. The map reads nothing else from ./target; its tables name no definition there.

For comparison, Code map on recent PRs took 19.9 min (#1115) and 19.7 min (#1114).

@cryptskii
cryptskii merged commit ed1ee84 into main Oct 5, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant