Follow up #171 with statement-end docs, contract tests and doc fixes by Deicyde · Pull Request #208 · facebookresearch/autoform-bot

Deicyde · 2026-10-11T05:10:11Z

Follows up #171, merged as 89dff279, with the findings from our review of its head 516a2f00 that still hold on main. #171 merged before we posted that review, so the fixes land here. There is one commit per group, and each can be reviewed on its own. The branch is on main at 8575a15c.

Commit 1: say where a statement ends when an H1 follows a section

article_parts ends a section at the article's title, its first H1, when the title is written below that section, and the text under the title joins the statement. A later H1 stays in its section. On this branch, article_parts("Intro.\n\n## Sources\n\nCited.\n\n# Title\n\nStatement.\n").statement is "Intro.\n\n\nStatement.", while for "# Title\n\nStatement.\n\n## Sources\n\nCited.\n\n# Appendix\n\nMore.\n" the # Appendix text stays in Sources. test_a_title_below_a_section_ends_that_section pins the first rule, with a ## Depends on section above the title.

The README, the ArticleParts docstring and the missing-statement-text reason said the statement runs to the first ##. They now say what article_parts does. In autoform audit --json, the reason changes from "formalizable article has no statement text before its first H2 section" to "... outside its H2 sections"; the finding code stays the same. Because the README's next sentence now follows one about H1s, "A deeper heading does not end it" becomes "No heading deeper than ## ends it".

test_a_statement_written_below_a_section_is_still_published passed before #171 too, because the old renderer dropped ## Depends on before splitting. Its fixture now puts ## Sources above the title, so it fails on cc7e3a85 (the theorem box is empty) and passes here.

Commit 2: test four checks the statement and search contracts rely on

Each mutant below leaves the named suites passing on main, except the fourth, which hangs the existing test. On this branch, each one fails only the new or extended test:

Check	Mutant	On `main`	On this branch
A fence closes only on a line of its own character	drop `marker[0] == fence[0]`	test_markdown and test_graph: 167 passed	`test_a_fence_closes_only_on_its_own_character` fails (1 failed, 167 passed)
`matched_fields` lists the best field first	list fields in term order	test_search: 61 passed	`test_every_term_must_occur_but_terms_may_sit_in_different_fields` fails (1 failed, 61 passed)
`matched_fields` names each field once	keep duplicates	test_search: 61 passed	the same test fails (1 failed, 61 passed)
An unclosed fence is reread once, not once per fence	never record the unclosed fence	`test_many_fences_that_never_close_do_not_each_reread_the_article` hangs (killed at 90 s)	the same test, now counting fence-pattern reads, fails in about a second
The CLI reference's `autoform search` examples run	`--state proven` in the README	test_search and test_skill_examples: 88 passed (only the verb is checked)	`test_the_reference_search_examples_run` fails; test_skill_examples still passes

In the matched_fields test, one new assertion reverses the term order and one puts both terms in one field. The fence-reread test also asserts at least one counted read per line: if a refactor binds FENCE.match and FENCE_CLOSE.match at import time, the counter sees nothing, and the test now fails (0 >= 40000) instead of passing without counting.

Commit 3: drop `HTML_COMMENT` and a sort in `article_parts`

No behavior change.

article_parts collected the H2 starts and then sorted the title into them. One comprehension now yields the breaks in order. article_parts gives the same output as on main for the 2,156 Markdown files of our test corpus (the bundled example and copies of the Hatcher and Hartshorne blueprints) and for 20,000 random documents.
markdown.HTML_COMMENT has had no consumer since the shared mask replaced it. Nothing in the tree, and no added line in any open PR, refers to it.

Commit 4: correct search and statement-text docs that misdescribe behavior

The README said a title "is matched as the page shows it". Search matches a title without its Markdown markup, as a chapter heading shows it, but the theorem box prints the title as written. With the site's Markdown extensions, # C*-algebras and C*-algebra maps renders as C<em>-algebras and C</em>-algebra maps, the box's bp-thmtitle span keeps both stars, and search folds the title to c-algebras and c-algebra maps, so C*-algebra finds nothing. The README now says only that a title is matched without its Markdown markup, and search._title's docstring now reads "Return a title without its Markdown markup, as a chapter heading shows it." test_a_title_is_matched_as_the_page_shows_it made the same claim in its name and is renamed test_a_title_is_matched_without_its_markup.
Superscript and subscript digits fold to plain digits, but ^ and _ stay: S^1 folds to s^1 and S¹ to s1, H_0 to h_0 and H₀ to h0. So S^1 does not find S¹, and H_0 does not find H₀. Both spellings occur in real blueprints (P^1 and P¹ in Hartshorne, H_0 and H₀ in Hatcher). The README now says to search for each spelling.
An HTML comment that never closes hides every heading and dependency link after it, and a <!-- in inline code or in an indented code block opens one too. We built three articles on this branch, with the comment opened in the statement, opened in the statement by inline code, and opened in ## Sources. In all three the graph loses the Base edge, and the audit reports missing-depends-section and nothing about the comment. Opened in the statement, the comment leaves the statement holding the rest of the article, so the theorem box holds the Sources and Depends on headings and the dependency link. Opened in Sources, it leaves that section holding ## Depends on and the link, and the statement is unchanged. The README said none of this. A short paragraph next to the fence rules now says it, and covers the graph, the audit and the theorem box, not the final HTML page.
autoform search --help named every searched field except ancestors. It now names a containing article's title.
Only a heading written with # marks, alone on its line, gives missing-statement-text. An underlined, quoted or list-item heading gives empty-statement-text, but the README put every heading under the first code. It now names the # form, with ### Setting as the example, and lists the other three under empty-statement-text. test_audit_reports_a_statement_that_shows_a_reader_nothing now also covers Setting\n=======, > ### Setting and - ### Setting. They pass on main as well; the test pins what the README now says.

Not in this PR

Already fixed on main: the statement audit's fence rule. At 516a2f00, a statement after a ``` -- end closer got missing-statement-text, so autoform audit and doctor exited 1 while the theorem box showed the sentence. Since the rewritten "Audit statements that are empty or only a placeholder", _statement_finding consults content_lines only when the published prose has no letter or digit. Our regression test passes on main, and main's test_a_statement_the_site_publishes_from_a_fence_that_never_closes_is_not_missing covers the same path, so this PR adds neither the unclosed_fence_is_text parameter for content() nor the test. A label difference remains: a statement that publishes only a period after such a closer is reported as missing-statement-text rather than empty-statement-text. Both fail the audit.
Deleting test_audit_reports_one_statement_finding_at_a_time, and the audit contract's parenthetical. The test repeats the comment case of test_audit_requires_statement_prose_outside_code_comments_and_headings, and the parenthetical in the README's audit contract repeats the missing-statement-text bullet below it. Report unclosed Markdown fences during audit #202 edits the lines next to both, so each deletion would conflict with it (tests/test_audit.py and autoform_cli/README.md). They can follow once Report unclosed Markdown fences during audit #202 lands.
The shared reverse-edge helper and removing audit._ArticleShape. Both are behavior-neutral golf from our review: search rebuilds render._reverse_edges inline and re-sorts lists that are already sorted, and _ArticleShape copies two facts out of ArticleParts. At the current PR heads, moving the helper to graph.py makes Publish blueprint sites transactionally #165, which merges cleanly with main, conflict in graph.py (one hunk) and render.py (two), and adds the same three hunks to Add verified statement-review bundles #75 and the two render.py hunks to Scale roadmap inventories and the full DAG explorer #90. Removing _ArticleShape adds an audit.py hunk to Add verified statement-review bundles #75 and to Scale roadmap inventories and the full DAG explorer #90, and Keep implementation notes separate from mathematical articles #172's test stub audit_module._ArticleShape(...) then fails with AttributeError. Report unclosed Markdown fences during audit #202 now extends _ArticleShape with the unclosed fence's line (5b383e41), so removing it would conflict with Report unclosed Markdown fences during audit #202 as well. Both can come back, with Add blueprint search and reuse: a description contract, autoform search, and search-before-add rules #143's "builds reverse edges once" counter, after those PRs land.

Open question for @zibo-yang: should an exact title rank first?

Within the title tier, ties break on used_by_count, so a one-word chapter title falls behind longer titles that contain it. These ranks are from main, on read-only copies of the Hatcher and Hartshorne blueprints (with source_units stripped so the current tree loads them):

Query	Blueprint	Rank of the article with that title, now	With the change
`Homology`	Hatcher	42nd of 147	1st
`Schemes`	Hartshorne	71st of 702	1st
`Curves`	Hartshorne	106th of 517 (the chapter)	2nd, behind the leaf also titled `Curves`
`The function field`	Hartshorne	11th of 129	1st

Searching for each of Hatcher's 282 articles by its own title puts that article first 269 times on main (worst: 42nd) and 281 times with the change (worst: 2nd).

The change is one sort-key term after max(ranks), which puts the shorter matching title first among title matches. The annotation also gets the new int, and lists the tuple's first two types in order (main's has them swapped):

@@ -196 +196 @@ def search_blueprint(
-    matches: list[tuple[bool, int, int, SearchHit]] = []
+    matches: list[tuple[int, bool, int, int, SearchHit]] = []
@@ -228 +228,2 @@ def search_blueprint(
-        matches.append((min(ranks), _ranks(words, fields) is None, max(ranks), hit))
+        title_length = len(fields["title"]) if min(ranks) == 0 else 0
+        matches.append((min(ranks), _ranks(words, fields) is None, max(ranks), title_length, hit))
@@ -233 +234 @@ def search_blueprint(
-    matches.sort(key=lambda match: (*match[:3], -match[3].used_by_count, match[3].node_id))
+    matches.sort(key=lambda match: (*match[:4], -match[4].used_by_count, match[4].node_id))
@@ -243 +244 @@ def search_blueprint(
-        hits=tuple(match[3] for match in matches[:limit]),
+        hits=tuple(match[4] for match in matches[:limit]),

It changes the documented order in the README ("Hits sort by ...") and two tests. In each test the first two hits swap: Marker now comes before Holds the marker word, and Convex hull before Convex analysis.

@@ -422,4 +422,4 @@ def test_fields_rank_in_the_documented_order(tmp_path: Path) -> None:
     assert _ids(project, "marker") == [
-        "zeta",
         "zeta/a-title",
+        "zeta",
         "zeta/b-lean",
@@ -483,4 +483,4 @@ def test_filters_and_limit_narrow_the_hits_but_not_the_count(tmp_path: Path) ->
     assert _ids(project, "convex") == [
-        "convexity",
         "convexity/hull",
+        "convexity",
         "convexity/hyperplane",
@@ -496,3 +496,3 @@ def test_filters_and_limit_narrow_the_hits_but_not_the_count(tmp_path: Path) ->
     limited = search_blueprint(project, "convex", limit=1)
-    assert [hit.node_id for hit in limited.hits] == ["convexity"]
+    assert [hit.node_id for hit in limited.hits] == ["convexity/hull"]
     assert limited.total_matches == 4

With those updates, test_search (62) and test_skill_examples (27) pass. Because it changes search's documented ranking, we left it out of this PR. Do you want exact titles first? We can add it here, or you can take it into one of your search PRs.

Overlaps with open PRs

We ran git merge-tree --write-tree --name-only of this branch and of main (8575a15c) against every open PR. Eleven touch at least one of this branch's files: #204 (270ed1f7), #202 (5b383e41), #198 (aa60e95f), #195 (36e85254), #191 (7481b64f), #190 (796da1eb), #183 (788c6f1d), #172 (3e95335f), #165 (de300e07), #90 (2e2633e7) and #75 (efc6b5b6). The branch adds no textual conflict to any open PR:

Search a shared Lean library, generate its index, and teach the skills to reuse it (#203) #204, Report unclosed Markdown fences during audit #202, Add autoform unused-hypotheses: report hypotheses a proof never uses #198, Feedback form #195, Add --write to migrate article-ids #191, Run the CI audit probe without any project code #190, Keep implementation notes separate from mathematical articles #172 and Publish blueprint sites transactionally #165 merge cleanly with both. With Report unclosed Markdown fences during audit #202 merged in, test_audit (72), test_markdown (108) and test_search (62) pass. With Keep implementation notes separate from mathematical articles #172, test_audit (76, including its _ArticleShape stub test) and test_render (83) pass. With Publish blueprint sites transactionally #165, test_render (196) and test_markdown (107) pass.
Allow proof imports with measured impact #183, Scale roadmap inventories and the full DAG explorer #90 and Add verified statement-review bundles #75 already conflict with main, and the branch leaves their conflicted files and hunk counts unchanged (Allow proof imports with measured impact #183: README.md, __main__.py and impact.py; Scale roadmap inventories and the full DAG explorer #90: audit.py, mermaid.py and project/_snapshot.py; Add verified statement-review bundles #75: tests/test_cli.py). Allow proof imports with measured impact #183's __main__.py conflict already spans main's whole search parser, so our --help line falls inside it. With Add verified statement-review bundles #75 merged in, test_markdown passes (110); its audit and render suites import cmarkgfm, which our environment lacks, so we did not run them.
The other ten open PRs touch none of these files and merge cleanly with both.
One semantic conflict, with Search a shared Lean library, generate its index, and teach the skills to reuse it (#203) #204. Search a shared Lean library, generate its index, and teach the skills to reuse it (#203) #204 adds a fourth autoform search example to the CLI reference: autoform search . "separating hyperplane" --lean-root . --library atlas. With both merged, test_the_reference_search_examples_run fails (assert 4 == 3; the rest of test_search passes), and that example exits 2 against the bundled example ("the Lean root has no lake-manifest.json; run lake update first"). Whichever PR lands second needs to adjust the test, for example to run only the examples without --library.
One wording overlap, with Add verified statement-review bundles #75. Add verified statement-review bundles #75 adds review-statement-missing with the reason "article has no mathematical statement before its first H2", computed through article_parts. Commit 1 retires that description for the audit's own reason, so whichever lands second may want to align the two.

Validation at `90c89ed9`

ruff check autoform_cli servers tests: all checks passed.
test_audit 67, test_markdown 107, test_render 82, test_graph 61, test_coverage 54, test_search 62 and test_skill_examples 27 pass, as does tests/test_doctor.py -k bundled_example.

article_parts ends a section at the article's title, its first H1, when the title is written below that section, so the text under the title joins the statement, as test_a_title_below_a_section_ends_that_section pins. A later H1 stays in its section. The README, the ArticleParts docstring and the missing-statement-text reason still said the statement runs to the first `##`; they now say what article_parts does. test_a_statement_written_below_a_section_is_still_published passed before #171 too, because the old renderer dropped `## Depends on` before splitting. Its fixture now puts `## Sources` above the title, which fails on cc7e3a8 (empty theorem box) and passes here.

Four checks that #171's statement and search contracts rely on had no test that fails without them: - A fence closes only on a line of its own character. Dropping `marker[0] == fence[0]` passed test_markdown and test_graph, though that check now decides graph edges, audit sections and the statement span. - The README promises `matched_fields` best first and once each. Listing fields in term order, or keeping duplicates, passed all of test_search. In an existing test, one assertion now reverses the term order and one puts both terms in one field. - test_many_fences_that_never_close_do_not_each_reread_the_article compared output only, so a rewind that never records the unclosed fence hung it (still running after 90 s). It now counts fence-pattern reads and fails such a mutant in seconds. It also asserts at least one read per line, so a refactor that bypasses the counter fails instead of passing unchecked. - The three `autoform search` examples in the CLI reference were checked only for their verb, so `--state proven` passed. They now run against the bundled example.

No behavior change. - article_parts collected the H2 starts and then sorted the title into them; one comprehension now yields the breaks in order. - markdown.HTML_COMMENT has had no consumer since the shared mask replaced it.

- A leaf's title is matched after Markdown rendering, but the theorem box prints it raw, so "matched as the page shows it" did not hold: `C*-algebra` finds nothing in "C*-algebras and C*-algebra maps". The README now says only that a title is matched without its markup, search._title's docstring says a chapter heading shows it that way, and the test named for the old claim is renamed to match. - Superscript and subscript digits fold to digits while `^` and `_` stay, so `S^1` does not find `S¹` nor `H_0` `H₀`. Both spellings occur in real blueprints; the README says to search for each. - An HTML comment that never closes, even one opened by a `<!--` in inline code or an indented code block, hides every later heading and dependency link. The graph loses those edges, and the part of the article the comment opens in keeps the rest, so one opened in the statement puts the dependency list in the theorem box. Nothing reports it, and the README did not say so; it now does, next to the fence rules. - `search --help` listed every searched field but `ancestors`. - Only a heading written with `#` marks, alone on its line, gives missing-statement-text. An underlined, quoted or list-item heading gives empty-statement-text, but the README filed every heading under the first. It now names the `#` form, and test_audit_reports_a_statement_that_shows_a_reader_nothing pins the other three.

Merge main at 82cca35, which carries the final #168 (b553a92) and #111 (521a940) plus #172, #175, #186, #188, #198, #208, #209 and the audit PRs #223 to #229. Main's changes are kept as they are, and this branch's changes apply on top of them unchanged. Only tests/test_shared_lean_runtime.py conflicted, in two places. The imports keep both sides: main's servers.lsp.server import, which #168's new live-process LSP timeout test needs, and this branch's servers.repl.core header-command import. The second conflict is in test_partial_repl_startup_transfers_cleanup_without_masking_the_error, where #168 relaxed a timing bound from 0.5 to 1.5 seconds. This branch replaces that test with test_dirty_repl_generation_transfers_to_retirement_before_lease_release because a cold pool no longer starts a REPL while it is built, so the replacement stays and #168's edit leaves with the old test. The other files both sides changed merged without conflict: servers/lean_runtime.py, the CI workflow, tests/test_repl_core_protocol.py, tests/test_skill_examples.py, skills/setup/SKILL.md and the example README.

Brings in main at 82cca35: the Lean client's request phases (#175), the skeleton probe's roots passed as JSON (#209), autoform unused-hypotheses (#198), the shared checkout pin check (#186), the tree snapshot scan closure (#188), the statement contract follow-ups (#208), retained project cleanup and LSP process ownership (#168, #111), implementation notes outside articles (#172), and the audit fixes #223 to #229. Conflicts: - autoform_cli/audit.py: this branch's validate_coverage_roles import beside main's IMPLEMENTATION_NOTES_DIR import. - autoform_cli/graph.py: this branch's ATLAS_SCHEMA and _MAX_ATLAS_BYTES beside main's IMPLEMENTATION_NOTES_DIR. - autoform_cli/README.md: this branch's area, atlas and catalog paragraphs, then main's implementation-notes paragraph. The publication contract keeps this branch's explorer description and takes main's "Hidden files, including .implementation-notes, are omitted". - skills/roadmap/SKILL.md: main's implementation-notes paragraph after the article-ids instruction, then this branch's paragraph break before the claim rule. Main's "Assert formalization or mathlib: true only after exact verification" stays out there, because this branch moved that rule into its own assertion paragraph further down. - tests/test_visualization.py: both new tests, this branch's checked catalog labels and main's structure page without implementation notes. One change had no textual conflict. #172 has every command that loads the graph reject malformed, empty, symlinked or orphaned implementation notes. Render loads the graph from its capture, which leaves hidden files out, so it skipped that check. It now checks the notes in the vault, the same fix #165 makes for its own capture. The notes are never published or hashed, so reading them in place changes no output. A new test renders a vault with an orphaned note and expects render to stop before it writes anything.

#208 deferred two deletions until #202 landed. The test repeats the comment cases that test_audit_requires_statement_prose_outside_code_comments_and_headings and the title-line test already pin, and the audit contract's parenthetical repeats its missing-statement-text bullet.

Deicyde added 4 commits October 10, 2026 23:43

Drop HTML_COMMENT and a sort in article_parts

0c382c0

No behavior change. - article_parts collected the H2 starts and then sorted the title into them; one comprehension now yields the breaks in order. - markdown.HTML_COMMENT has had no consumer since the shared mask replaced it.

meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 11, 2026

Deicyde mentioned this pull request Oct 11, 2026

Blueprint search and reuse: phase 1 items 1-3 and 5 (#143) #171

Merged

Deicyde marked this pull request as ready for review October 13, 2026 02:28

Deicyde merged commit 2825e93 into main Oct 13, 2026
9 checks passed

Deicyde deleted the fix/statement-contract-followups branch October 13, 2026 02:29

Deicyde mentioned this pull request Oct 13, 2026

Retain Lean project ownership through verified cleanup #168

Merged

This was referenced Oct 13, 2026

Scale roadmap inventories and the full DAG explorer #90

Draft

Follow up the reviews of #243, #245 and #249 #297

Merged

Navigation Menu

Repository navigation

Follow up #171 with statement-end docs, contract tests and doc fixes - #208

Follow up #171 with statement-end docs, contract tests and doc fixes#208
Deicyde merged 4 commits into
mainfrom
fix/statement-contract-followups

Deicyde commented Oct 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Deicyde commented Oct 11, 2026

Commit 1: say where a statement ends when an H1 follows a section

Commit 2: test four checks the statement and search contracts rely on

Commit 3: drop HTML_COMMENT and a sort in article_parts

Commit 4: correct search and statement-text docs that misdescribe behavior

Not in this PR

Open question for @zibo-yang: should an exact title rank first?

Overlaps with open PRs

Validation at 90c89ed9

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Commit 3: drop `HTML_COMMENT` and a sort in `article_parts`

Validation at `90c89ed9`