Skip to content

Verify installed plugin provenance - #16

Open
Deicyde wants to merge 30 commits into
facebookresearch:mainfrom
VivienCabannes:split/06-plugin-provenance
Open

Deicyde wants to merge 30 commits into
facebookresearch:mainfrom
VivienCabannes:split/06-plugin-provenance

Conversation

@Deicyde

@Deicyde Deicyde commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Depends on #14 and includes its cut-down head 06b31ac (merge 48a7d74);
GitHub will remove that parent diff after #14 merges. #14 no longer ships
bounded_toml.py, so the module and its depth tests
(tests/test_bounded_toml.py) now live in this PR, where provenance uses them
to parse pyproject.toml and uv.lock.

Review the #16 work in these commits:

  • 3966440 verifies the complete installed plugin tree.
  • 97ddd6a binds scaffolding to verified templates and avoids building Autoform in CI.
  • c7464fe validates Claude host metadata and cleans up cancelled Git process groups.
  • 70825bf isolates checkout, dependency installation, and generated workflow execution.
  • 0db2670 pins the generated runner image.
  • e86468e permits only validated remote uv constraints, for compatibility with Constrain cryptography below 49 on Intel Macs #56.

Summary

  • Resolve an exact source and revision from a checkout or bounded Codex and
    Claude installation records.
  • Fetch that commit through an isolated, deadline-bound partial Git transport;
    compare every tracked file and mode; reject untracked non-derived entries,
    case aliases, executable suffix aliases, symlinks, and runtime configuration.
  • Accept Claude's installer rewrite only when both manifests, the unique
    registry record, cache leaf, full revision, and BUILD_COMMIT form the exact
    deicyde.<sha7> overlay observed in the host cache.
  • Add autoform project provenance and make autoform init consume the remote
    verified template bytes rather than a mutable installed template tree.
  • Generate CI that checks out the exact commit with isolated Git configuration,
    installs only hashed locked wheels into a fresh external environment, never
    builds Autoform or its PEP-517 backend, and launches verified source with
    isolated Python.

Boundaries

  • Secure provenance and local template capture currently require POSIX
    descriptor/no-follow APIs and fail closed on Windows.
  • The result describes file contents observed during the invocation; it cannot
    freeze a mutable installation after a read.
  • Git cannot preflight promised blob sizes. Selected-object transfer has one
    60-second deadline; the 16 MiB file and 64 MiB aggregate limits apply while
    Git decodes fetched objects.

Validation

  • At ce81bef, after merging the cut-down Add bounded Lean project inspection #14: full suite 1,068 passed,
    2 skipped on Python 3.13; ruff check autoform_cli servers tests. The lines
    below were measured at 68feeb2, before that merge.
  • Post-restack project-inspection, provenance, and affected suites: 537 passed,
    1 skipped.
  • Additional unchanged-gate coverage: 143 passed and the pre-existing macOS
    detached-child timing test deselected before stopping the slow real-Lean
    skeleton tail after 41 minutes; CI runs the complete matrix.
  • make lint
  • make check-example
  • Real Codex and Claude caches both verify against recorded commit 1592d6a.
  • Fresh no-cache normal and Pages dependency installs use no Autoform or
    Hatchling distribution and execute the isolated source launcher successfully.
  • Generated YAML parses, checkout shell passes bash -n, and the complete
    generated checkout/sync step succeeds against official pin 45ca2ec.

Reference implementation: #8

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 19, 2026

@Deicyde Deicyde left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review result: changes needed.

  • scaffold.py:197-229 verifies a pin, then rereads mutable templates. A reproduced mutation after verification was emitted into CI while the scaffold recorded the pre-mutation SHA.
  • provenance.py:1252-1288 ignores tracked importable files outside declared roots. A modified top-level sitecustomize.py still verified.
  • The prescribed cachebuster/reinstall workflow changes .codex-plugin/plugin.json, which exact comparison rejects. The current installed Codex cache therefore cannot auto-discover a CI pin.
  • Python local dependencies and in-tree build backends are not rejected, although npm local dependencies are.
  • The bounded-fetch claim is incomplete: the pack is downloaded before blob limits apply, and timeout kills only the direct Git process.
  • Ordinary scaffold unit tests now perform live network provenance fetches; make them offline by default and retain one gated integration test.

The first two are integrity bugs. Capture/stage the verified template bytes, include every executable/importable source in the compared manifest, and add regressions.

@Deicyde Deicyde left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second pass on exact head 651bc566, with 199 focused tests passing. Additional findings:

  • [P1] The second traversal is not an atomic snapshot. At provenance.py:1371-1375, I deterministically changed autoform_cli/__init__.py during pass two, after that root had been read and as servers began. verify_plugin_provenance() returned the recorded SHA while the file was already divergent. Root/directory identity does not observe an in-place write to a previously closed file. The verified snapshot must be staged and consumed by downstream code; another sequential traversal cannot establish the live-tree claim.
  • [P1] The verified lock is not used by generated CI. The workflows at autoform-verify.yml:38 and blueprint-pages.yml:48,55 run uvx --from git+SOURCE@SHA, which resolves dependencies independently of that commit's uv.lock. A fresh-cache run at this head installed fastmcp 3.4.7 and pymdown-extensions 11.0.2; the verified lock pins 3.4.5 and 11.0.1. Existing projects can therefore acquire a different runtime without changing either source or SHA. Fetch the verified commit and run it with uv run --locked --project ..., or provide an equivalent locked installation path.
  • [P2] The advertised resource bounds still permit subprocess amplification. _read_git_blob() starts two git cat-file processes per selected path (provenance.py:726-740, called at 917-934). A valid 20,000-entry tree can reuse one tiny blob, remain below the entry and byte caps, and force roughly 40,000 processes with no aggregate deadline. Use one bounded cat-file --batch session, cache object IDs, and enforce an end-to-end deadline.
  • [P2] Git credential isolation is incomplete. _git_environment() retains HOME, so Git/libcurl still reads ~/.netrc; an exact-flags TLS repro sent the matching Basic credential after a 401 despite the empty credential helper and askpass. This can also make local verification succeed for a private origin while the emitted credential-free CI cannot fetch it. Use a scratch HOME/disabled netrc, or define and test an authenticated-source contract.
  • [P2] Default provenance is unavailable on Windows. _directory_flags() requires POSIX descriptor APIs (provenance.py:358-370). plugin_pin() then converts that platform failure into an unpinned result, so autoform init omits both workflows although the prior checkout pin path was portable and the project declares no Unix-only restriction. Add a safe Windows implementation or make the platform restriction explicit rather than degrading the advertised default.

@Deicyde Deicyde added the review: ready Review complete with no known merge blockers label Oct 2, 2026
@Deicyde Deicyde removed the review: ready Review complete with no known merge blockers label Oct 2, 2026
@Deicyde Deicyde added the review: ready Review complete with no known merge blockers label Oct 2, 2026
@Deicyde Deicyde removed the review: ready Review complete with no known merge blockers label Oct 2, 2026
@Deicyde Deicyde added the review: ready Review complete with no known merge blockers label Oct 2, 2026
@Deicyde Deicyde added awaiting author Review is complete and author action is required and removed review: ready Review complete with no known merge blockers labels Oct 2, 2026
@Deicyde Deicyde added review: ready Review complete with no known merge blockers and removed awaiting author Review is complete and author action is required labels Oct 3, 2026
Compatibility now comes from lean-toolchain plus the Mathlib entry that
lake-manifest.json locks (or package-overrides.json replaces), compared
by commit against the catalog. Files are read with plain size-capped
reads, so inspection works on Windows and through symlinked directories
such as macOS /tmp. Drops the descriptor walk, generation snapshots,
bounded TOML, Lean-name lexer, and material-identity tuple.
A differential run of the previous 190-test suite against the slim
inspector, plus checks against real elan 4.2.0 and Lake 4.32.0, showed
behaviours the cut lost:

- lean-toolchain follows elan: the trimmed first line decides, and a
  blank or malformed first line means elan uses the default toolchain
- a direct Mathlib lock that lakefile.toml no longer requires is unused,
  and lakefile.lean projects stay indeterminate
- a catalog match also needs Mathlib loaded from its repository root
  with lakefile.lean and lake-manifest.json; commits compare in any case
- legacy manifest versions are advisory, null packages mean none, NaN
  is rejected, overrides apply only over a manifest, path Mathlib is
  indeterminate
- lakefile.toml needs a name, a StdVer version, named entries, and
  distinct targets, as Lake requires
- credentials in a Mathlib URL are redacted from reports
- the blueprint vault must be a directory; «mathlib» is mathlib
PR 14 no longer uses bounded_toml.py, so the module stays here with the
provenance code that parses pyproject.toml and uv.lock, and its depth
tests move from the old inspection suite to tests/test_bounded_toml.py.
The project command keeps provenance beside the new inspect and versions
paths.
@Deicyde Deicyde removed the review: ready Review complete with no known merge blockers label Oct 3, 2026
@Deicyde
Deicyde force-pushed the split/06-plugin-provenance branch from 4491ad5 to ce81bef Compare October 3, 2026 03:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant