Skip to content

Skip rendered sites in every Lean capture - #336

Merged
Deicyde merged 2 commits into
mainfrom
fix/skip-rendered-sites-in-lean-capture
Oct 15, 2026
Merged

Deicyde merged 2 commits into
mainfrom
fix/skip-rendered-sites-in-lean-capture

Conversation

@Deicyde

@Deicyde Deicyde commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

autoform render copies every non-Markdown file under the blueprint into its
site, so a .lean file kept in the blueprint lands in the site. mkdocs build
then copies the site, publication.json included, into site/. Only render's
own Lean capture treated a folder holding a complete publication.json as
generated output; it passed that marker through an opaque_markers parameter
no other caller used. check, audit, doctor, work, search, skeleton,
runtime, impact and unused-hypotheses used the default selection (only
.git and the skeleton manifest.json), so they indexed the copies: duplicate
declarations, stale names that still resolved after the source renamed them,
and a different Lean revision from render's for the same tree. Found in the
#307 post-merge review.

This PR treats a folder holding a complete publication.json as generated
output in every Lean capture, exactly like skeleton output.

  • autoform_cli/lean.py: PUBLICATION_MANIFEST, PUBLICATION_SCHEMA,
    PUBLICATION_MANIFEST_MAX_BYTES and _is_complete_publication move here
    from render.py (render already imports lean.py, not the other way round),
    and the marker joins _lean_tree_selection's default markers.
  • opaque_markers is removed from snapshot_project_sources,
    _open_stable_project_sources, open_project_sources and
    _lean_tree_selection; render was its only caller (checked
    autoform_cli, servers, skills, tests).
  • autoform_cli/render.py imports the names from lean.py and still exports
    them (dashboard.py and tests import them from render). It keeps
    exclude_roots=(copy_root,) and its refusal rules. The comment that claimed
    every Lean walk skipped sites is replaced by one at the capture call that is
    now true. _write_publication_manifest replaces publication.json through
    a hidden temporary file in the same folder and os.replace instead of
    rewriting it in place, so a capture never reads a truncated marker. Bytes
    and file mode are unchanged.
  • autoform_cli/README.md: the stop list no longer says "for render"; every
    command skips a folder holding a complete publication.json and stops at an
    unreadable one.

Behavior changes

  • Every command that indexes Lean skips a folder below the Lean root holding a
    complete publication.json (schema v1 or v2, "complete": true, at most
    8 MiB), so render and the other commands index the same files and agree on
    the Lean revision.
  • Every command now stops at an unreadable publication.json below the Lean
    root, as render already did and as all commands do for manifest.json.
  • An incomplete, unrelated or oversized publication.json hides nothing.
  • A capture during a render skips the site being written: render
    writes a complete publication.json into its stage before it copies any
    file, and replaces it, never truncates it, when it adds the file inventory.
    Not changed here: mkdocs build empties site/ and copies files in
    name order, so site/ can hold a copied .lean file before its
    publication.json (checked with mkdocs 1.6.1); a command that indexes during
    the build can read it, and running it again afterwards gives the settled
    result. The README says so.

Tests

  • tests/test_lean_sources.py::test_only_a_complete_publication_manifest_hides_its_folder:
    one tree with render-style site-src/, mkdocs-style site/, a v1 site, and
    incomplete, unrelated and oversized manifests, each over a stale .lean
    copy; asserts exactly which sources snapshot_project_sources captures.
    Fails on main (stale copies under site-src/, site/ and legacy/ are
    captured).
  • tests/test_render.py::test_audit_and_render_skip_a_site_and_its_mkdocs_copy_alike:
    renders into site-src/, copies it to site/ as mkdocs does, renames the
    source declaration, renders again, then audits a lean: that names the old
    declaration. Asserts audit's Lean revision equals render's
    lean_source_revision and that audit reports lean-target-not-found. Both
    assertions fail on main.
  • tests/test_render.py::test_a_capture_during_a_render_skips_the_stage_being_written:
    captures after each .lean copy during a render and asserts only the sources
    are indexed. Fails on main (the capture includes
    .autoform-publication-<hex>/Proofs.lean).
  • tests/test_render.py::test_a_capture_while_render_rewrites_its_manifest_skips_the_stage:
    captures during render's final manifest write (just after an in-place write
    would truncate the file, and on both sides of the replace) and asserts only
    the sources are indexed. Fails when the manifest is rewritten with
    Path.write_bytes (the capture includes
    .autoform-publication-<hex>/Proofs.lean).
  • Touched and dependent files (test_render, test_lean_sources,
    test_windows_source_boundary, test_dashboard, test_audit): 499
    passed, 5 skipped, 1 xfailed on Python 3.13 and on 3.10.
  • Full non-Lean suite on 3.13 (before the manifest-replace commit): 3134 passed, 137 skipped, 1 xfailed, 48 failed.
    All 48 are in tests/test_library_generate.py with "Autoform is not
    installed": the local venv is synced with --no-install-project, and main
    fails the same 48 in that venv.
  • ruff check autoform_cli servers tests is clean.

Overlap with open PRs

MAIN_REF=upstream/main ~/bin/pr_conflicts.sh on the merge of this branch
into upstream/main feec945 (the branch is based on e108ee0 and merges
cleanly into feec945). New overlaps, all with draft PRs that already
conflict with main on graph.py:

No new overlap with any other open PR (#195, #213 to #216, #320 to #335, #337, #338).

render copies every non-Markdown file under the blueprint into its site,
so a .lean file kept in the blueprint lands in the site, and mkdocs then
copies the site, publication.json included, into site/. Only render's
own Lean capture treated a folder holding a complete publication.json as
generated output: it passed the marker through the opaque_markers
parameter, while check, audit, doctor, work, search, skeleton, runtime,
impact and unused-hypotheses used the default selection, whose markers
were only .git and the skeleton manifest. Those commands indexed the
copies, so they saw duplicate declarations, kept resolving names the
source had since renamed, and computed a different Lean revision from
render's for the same tree. The comment on render's marker claimed every
Lean walk skipped sites.

Move the publication manifest's name, schema, byte cap and completeness
check into lean.py and add the marker to the default selection, next to
the skeleton manifest, so every capture skips a site and its mkdocs
copy. render now imports these names from lean.py and still re-exports
them. render was the only caller of opaque_markers, so the parameter and
its plumbing through snapshot_project_sources, open_project_sources and
_lean_tree_selection are gone. render keeps its copy-root exclusion and
refusal rules. The README stop list now says every command skips such a
folder. render writes a complete manifest into its stage before it copies
any file, so a capture during a render skips the stage too (pinned by a
test that captures after each copy). mkdocs build empties site/ and copies
files in name order, so site/ can hold copies before its publication.json;
the README tells users to rerun a command that indexed during the build.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 14, 2026
@Deicyde

Deicyde commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Merge readiness: one small change requested (reviewed at f48f40a)

What holds (independent review plus CI, green on every job including Windows and the real-Lean jobs):

  • check, audit, doctor, render, work and search were run on main and on this branch over trees with and without a rendered site and its mkdocs copy. Two things differ: the skipped site copies (so a stale copy no longer resolves a renamed declaration), and a new stop at an unreadable publication.json.
  • The marker is strict. Only a regular file with schema v1 or v2, "complete": true and at most 8 MiB hides its folder; 26 probe trees covered the near misses. Only render writes a complete manifest, and only into an empty folder or its own site, so it cannot hide real sources.
  • No caller of the removed opaque_markers parameter remains. Draft Scale roadmap inventories and the full DAG explorer #90 is the only open PR that relies on it, and the body names it.
  • Windows render is unchanged. The README's mkdocs ordering was confirmed with mkdocs 1.6.1.

Change requested:

  • The README says render marks the site it is writing before it copies any file, so a command indexing during a render skips it. It does write the marker first. But at the end it rewrites publication.json with Path.write_bytes, which truncates the file before writing. A capture that reads the marker in that gap indexes the stage's .lean copies; a probe reproduced it.
  • Fix: write the final manifest to a temporary file in the stage and os.replace it onto publication.json, as Keep skeleton packets behind their manifest while publishing #258 does for skeleton manifests. I'll push that as a new commit.

Body nits:

  • test_only_a_complete_publication_manifest_hides_its_folder fails on main at the monkeypatch.setattr, not at the captured copies.
  • The body mentions a follow-up commit, an older base and an older overlap run.

It is a draft, so it needs to be marked ready before merging.

render writes a complete publication.json into its stage before it copies
any file, so a Lean capture during the render skips the stage. Once every
page exists it rewrote the manifest with Path.write_bytes, which truncates
the file before writing the new bytes. A capture that read the marker in
that gap found an empty file, did not recognize the folder as a site, and
indexed the stage's .lean copies. The README's claim that a command
indexing during a render skips the site was therefore not quite true
(review comment on #336).

_write_publication_manifest now writes the bytes to a hidden temporary
file in the same folder, opened with O_EXCL and mode 0o666 under the
umask as write_bytes would, and os.replace moves it onto publication.json,
so a reader sees either the previous complete manifest or the new one. The
temporary file is removed if anything fails. Its name is not a marker and
not a .lean file.

test_a_capture_while_render_rewrites_its_manifest_skips_the_stage captures
just after an in-place write would truncate the manifest and on both sides
of the replace, and asserts that only the sources are indexed. It fails on
the parent, where the capture includes the stage's Proofs.lean.

test_only_a_complete_publication_manifest_hides_its_folder now patches the
byte cap with raising=False, so on main it fails at its assertion (stale
copies captured) instead of at the patch.
@Deicyde

Deicyde commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Update: change made, now ready to merge (b007a13)

  • render writes publication.json to a temporary file in the stage and moves it into place with os.replace, so a capture never reads an empty marker. Both manifest writes (into the empty stage, and the final one with the file inventory) go through that one function.
  • test_a_capture_while_render_rewrites_its_manifest_skips_the_stage captures at the moment the manifest is rewritten. It fails on f48f40a, where the stage's .lean copy is captured, and passes now. test_only_a_complete_publication_manifest_hides_its_folder now fails on main at its assertion, as the body says.
  • Independent check:
    • Mode and bytes of publication.json are identical to before under umask 022, 077 and 002.
    • No temporary file is left when the write or the replace fails.
    • The Windows-emulated render tests pass, including a replace onto an existing manifest.
  • CI is green on every job: tests 3.10/3.13, real Lean, the real-Lean skeleton suite, the real REPL, Windows and CLA. It merges cleanly with main. The body is refreshed.

Optional: the new test catches a truncating rewrite through Path.write_bytes. An in-place rewrite by another route would not trip it.

It is a draft, so it needs to be marked ready before merging.

@Deicyde
Deicyde marked this pull request as ready for review October 15, 2026 17:19
@Deicyde
Deicyde merged commit 270a41f into main Oct 15, 2026
7 checks passed
@Deicyde
Deicyde deleted the fix/skip-rendered-sites-in-lean-capture branch October 15, 2026 17:19
Deicyde added a commit that referenced this pull request Oct 15, 2026
Main brings #331, #332, #335, #341, #338, #336, #323, #343 and #337.
lean.py, render.py and tests/test_render.py conflicted, and four
changes were needed outside the conflicts:

- lean.py: snapshot_project_sources and _open_stable_project_sources
  lose the opaque_markers parameter #336 removed and keep this
  branch's names= keyword. The publication constants #336 moved here
  take this branch's autoform-publication/v3, and _REPLACEABLE_SCHEMAS
  moves with _is_complete_publication, since lean.py cannot import
  render.py. Every Lean capture skips a complete v1, v2 or v3 site;
  with main's v2 alone it would index this branch's sites and their
  mkdocs copies.
- render.py: imports the constants and _REPLACEABLE_SCHEMAS from
  lean.py, drops its own _is_complete_publication, and its Lean capture
  passes names= and relies on the marker every capture now has.
- tests/test_render.py: keeps both sides' tests. #336's audit test
  forwards the names= keyword audit passes, and this branch's revision
  test builds TreeSnapshot without root_identity and placeholders,
  which #341 deleted.
- graph.py: _catalog_has_local_ledger takes #337's guards (a link
  urlsplit rejects is skipped, RuntimeError from resolve is caught,
  os.path.isfile replaces Path.is_file), so a catalog's Sources link
  can no longer crash load_graph. A new test in tests/test_graph.py
  covers an invalid URL, an over-long name and a symlink loop.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant