Rewrite angle-bracket and query links whole, and link only captured notes

[Deicyde]

Closes #303.

Problem

Render's link rewriting published three kinds of dead link, and mkdocs build --strict let each one through:

• _MARKDOWN_LINK had no <...> alternative, so [Smith](<../sources/Smith 2019.pdf>) was cut at the space and published as [Smith](<../sources/Smith). Its bare alternative ended at the first ), so with repository coordinates [S](../sources/Smith(2019).pdf) got a URL ending in Smith%282019 that 404s.
• moved_target split off only the fragment, so a query was resolved as part of the file name. With repository coordinates, ../sources/paper.md?plain=1 became a URL ending in paper.md%3Fplain%3D1.
• With coordinates, every path under sources/ went to the repository once the directory was in the commit, so a link to a note that does not exist got a URL that 404s. check reads only ## Sources, so such a link in an article body or on another page reached the deployed site.

Change

All in autoform_cli/render.py:

• _MARKDOWN_LINK takes the <[^>]+> alternative that graph's _LINK and render's _LINK_DEFINITION already have, and its bare alternative takes one level of balanced parentheses, which Python-Markdown and CommonMark accept, so [S](../sources/Smith(2019).pdf) is read whole. A title may hold balanced parentheses too, and a quoted title an unclosed (, so [S](../sources/paper.md "Smith (2019)") no longer leaves ") on the page, and neither title carries the match on to a later ) and drops the text before it. A backslash-escaped parenthesis counts in neither, since Python-Markdown reads it as text, so (see [x](../sources/a\(b.md) and [y](../sources/c.md)) keeps everything after [x]; the escape is not decoded, so with coordinates [x] is left as written, where the strict build reports it. This also stops an external <https://...> destination with a space from being rewritten as a relative path. The book's page order reads links with the same pattern, so it now follows an angle-bracket link with a space, or a bare one with balanced parentheses, to a book page too.
• An inline link whose destination does not move is left as written, title included. Math or code shaped like a link is then left alone too: read as a link to f(p), $[X,Y](f(p))$ would gain angle brackets, and $[X,Y](f \circ g)$ keeps the \circ g that main drops as a title.
• A rewritten inline destination that holds a space or a parenthesis, from a decoded path or from a query or fragment kept from angle brackets, is written back in angle brackets. Python-Markdown 3.10.3, which the Pages build uses, reads [x](<a(b.md>) and [x](<a b.md>) as one link each. Bare, it reads no link to a(b.md and ends a)b.md at the ), and CommonMark allows no space in a bare destination.
• In a reference definition, whitespace is percent-encoded instead. Python-Markdown reads a definition's destination only up to whitespace, in angle brackets or not, so [paper]: <../sources/paper note.md>, kept on the site, was published as text, and every [Paper][paper] with it, whether render dropped the brackets, as it did before, or kept them.
• moved_target splits the query off the path before decoding and resolving it, and writes it back after the moved path and before the fragment; on a node link, before the anchor. This is the split draft Treat every published page as untrusted #215 makes, with the same comment. The book's page order splits the query off too, so a landing-page link such as roadmap/README.md?plain=1 no longer drops its chapter from the book, as it does on main.
• A sources/ target gets the repository URL only when render's private copy of the captured blueprint holds it. Every captured file is in the verified commit, so that URL resolves. A captured directory Git does not hold, one that is empty or holds only ignored hidden files, still gets a repository URL that 404s. Any other target keeps its site-relative link. The site does not publish sources/ in that mode, so the strict build fails on a link to a missing file and names it. One form still gets past it: a target MkDocs reads as having no file extension, which it only logs, as when the name has none or a decoded ? or # ends the path early. The directory link from Publish blueprint sites transactionally #165 is unchanged.

This changes three outcomes. Capture skips hidden files, so a committed hidden note such as sources/.private/paper.md got a repository link that resolved, and now fails the strict build; #304 proposes that check reject such targets in ## Sources, and #321 does. An unbalanced bare target that Python-Markdown reads as text, as [x] in See [x](../sources/a(b.md) and [y](../sources/c.md)., is now left as written, so with coordinates it stays text too, where it used to get a repository URL; the link after it is still rewritten. And a bare target with an unclosed ( and a quoted title, as [x](../sources/a(b.md "t"), which Python-Markdown reads as a link to a(b.md, is left as written too, so with coordinates the strict build fails on it where it used to get a repository URL that resolved.

The existence check reads the private copy, so on a case-insensitive filesystem a link whose spelling differs from the file only in case still gets a repository URL, as before. The generated Pages workflow renders on Linux, where it does not.

Tests

New render tests, rendered from a committed project with repository coordinates unless noted, cover: an angle-bracket destination with a space; a query with a fragment on a source note link; a missing note; a bare link with balanced parentheses to a note that exists; See [x](../sources/a(b.md) and [y](../sources/c.md)., which must keep [x] as written and lose no text; (see [x](../sources/a\(b.md) and [y](../sources/c.md)), which must keep the text after [x]; titles with parentheses, balanced and unclosed, in either quote and with a space before the ); a link to the sources/ directory; math shaped like a link, which must come out as written; and, without coordinates, a site-relative link whose decoded path holds a space or a (, a query on a node link and on a page link from the coverage page, and a query on the landing page's chapter link, which must keep the chapter in the book. The existing angle-bracket reference-definition test now also runs without coordinates.

Called directly, each test fails when the rule it covers is reverted: the angle-bracket alternative; the balanced bare alternative; the title rules, whether the title part goes back to main's or loses its quoted alternatives, its single-quoted one or the space after a quoted title; the escape handling; the query in moved_target's split, its site-relative return or its node branch, or in the page order; the existence check, or os.path.isfile in its place; either half of bracketed; the encoding in definitions; and leaving unmoved links as written.

The test suite was not run locally; CI runs it. Called directly from Python at this head, the twelve new tests pass, as do both cases of the reference-definition test and fifteen existing render tests that rewrite links, order the book or place statements at their authored slots, among them #318's test for links that leave the blueprint.

Over 29 inputs, among them each form above, links inside an aside, an external URL with parentheses, titles in parentheses, a node link with a title, and math and code spans shaped like links, Python-Markdown 3.10.3 shows the same text before and after the rewrite, with coordinates and without, but in two places: a quoted title holding a ) on a link that moves, as [S](../sources/paper.md "a) b") does with coordinates, still leaves b") on the page, as on main; and an angle-bracket definition with a space, which Python-Markdown reads as text, becomes a working definition.

mkdocs build --strict with MkDocs 1.6.1 failed on each form render writes for a missing file with a space or a parenthesis (an angle-bracket link, a definition with the space encoded, a definition holding the parenthesis) and passed when the files existed.

ruff check autoform_cli servers tests passes.

Landing

Rebased onto main at 7e34a3a; it merges cleanly into d1ad2fb, which adds #337. None of the PRs merged since the previous base, feec945 (#323, #331, #332, #335, #336, #337, #338, #341, #343), touches _MARKDOWN_LINK or _rewrite_links. An earlier rebase resolved the one conflict with #318, which makes a link to a target outside the blueprint keep only its text: replace ends with #318's if href else match.group("label"), so a target outside the blueprint written in angle brackets with a space keeps only its text too. #300 and #301 change the sources base, not moved_target: under #300 there is one only when the blueprint's own bytes are in the commit, which the existence check relies on, and #301's committed spellings change only the URL the check lets through.

git merge-tree --write-tree of each open PR head against this branch, compared with the same against main:

• Treat every published page as untrusted #215 and the drafts that carry its render changes (Add read-back cards and the testimony validator #216, Show statement review on the site #217, Label statement approvals by the repository's code-owner review policy #219, Add statement-review commands #284) get four new conflicts in render.py: on _MARKDOWN_LINK, beside the _FENCE line Treat every published page as untrusted #215 deletes; on the # split, where Treat every published page as untrusted #215 names the separator hashed; on the node branch of moved_target; and on its sources/ and site-relative returns, which bracketed follows. Their other conflicts with main are unchanged. Keep Treat every published page as untrusted #215's lines, except that _MARKDOWN_LINK keeps this branch's pattern and Treat every published page as untrusted #215's sources/ condition gains and os.path.exists(candidate); keep bracketed, which replace still calls. Treat every published page as untrusted #215 percent-encodes every destination it writes, so the decoded-path test should then expect its links as written. The balanced-parentheses test takes the sources/ branch, where Treat every published page as untrusted #215's _URL_SAFE keeps the %28 and %29 that _source_href writes, so it should pass as written. That resolution was not run at this head.

• Scale roadmap inventories and the full DAG explorer #90 gets two new conflicts in render.py. On _MARKDOWN_LINK, keep its label pattern and take this branch's target and title patterns. On the last line of replace, keep Scale roadmap inventories and the full DAG explorer #90's fallback for a link that loses its target, and have its if href: branch return f"[{label}]({bracketed(href)})".

• Leave malformed link targets as written in render and the runtime #327 wraps the lines just after the split in a try, in moved_target and in the book's page order, so the two conflict in both places. Whichever lands second keeps this branch's split above the try; in the page order that is path = bare.partition("#")[0].partition("?")[0], and in moved_target:

          bare = raw[1:-1] if raw.startswith("<") and raw.endswith(">") else raw
          # The query and fragment are not part of the file's name: they are
          # kept as written, beside the path render moves.
          located, separator, fragment = bare.partition("#")
          path, asked, query = located.partition("?")
          try:
              if not path or urlsplit(path).scheme or path.startswith("/"):
                  return None
              candidate = (source_dir / unquote(path)).resolve()
          except ValueError:
              # An unbalanced IPv6 bracket or a decoded NUL names no file to
              # point at, so the link is published as written, like a URL.
              return None
          node_id = node_sources.get(candidate)

• No other open PR gets a new conflict: not Feedback form #195, Extract Lean skeletons one probe per root module #213, Run skeleton probes in parallel and isolate probe failures #214, Keep source ledgers out of the public book #299, Read code-only labels, reference links and nested brackets as links #320, Make check reject Sources links to hidden paths #321, Stop worker groups that Lean's watchdog leaves running at LSP close #322, Close the gaps optional claims left in runs without claims #324, Make check reject a container article that sets a declaration #328, Say how unused-hypotheses differs from Batteries' unusedArguments linter #329, Link folders under the blueprint by the spelling Git committed, and fix render --ref's help #333, Restore SIGCHLD's default so the runner sees real exit statuses #334, Guard #289's linear render and drop caches lazy links made redundant #339 or Pin the comment stripper's events, and end raw strings at their first closer #340. Keep source ledgers out of the public book #299 drops sources/ from the MkDocs build; a site rendered with coordinates has no sources/ to drop, so a link to a missing note still fails the strict build. Guard #289's linear render and drop caches lazy links made redundant #339 swaps the node branch's _anchored_links(...) call for its page-wide links mapping, which the query splice after it reads unchanged.

#320 replaces graph's _LINK, cited above, with markdown.link_targets. #327's try blocks also cover a form this branch starts reading: [x](<https://[::1/a b>) now raises in urlsplit, in moved_target and in the book's page order, as [x](https://[::1/a) already does on main.

[Deicyde]

Review at 0f15a8d

Verdict: Request changes. Render now rewrites angle-bracket destinations and queries whole and, with repository coordinates, links a sources/ target to the repository only when the captured blueprint holds it, so a missing note fails mkdocs build --strict instead of deploying a 404 (#303). Called directly on the issue's three cases, _rewrite_links publishes the expected links, each read by Python-Markdown 3.10.3 as one link. One regression should be fixed first: a bare link with balanced parentheses to a note that exists now stops the Pages build, and a small change to the link pattern fixes it while removing a special case.

Fixes before merge

1. A bare link with balanced parentheses to an existing note fails the Pages build (autoform_cli/render.py:64). The bare alternative [^)\s]+ still ends at the first ), so [S](../sources/Smith(2019).pdf) is resolved as Smith(2019, and the new existence check (render.py:2071) finds no such file. The link stays site-relative, but with coordinates the site publishes no sources/, so the strict build reports the target as not found even though it is in blueprint/sources/. Main deployed a 404 to .../Smith%282019. The PR text names the new outcome. Let the bare alternative take one level of balanced parentheses, which Python-Markdown and CommonMark accept:

   _MARKDOWN_LINK = re.compile(
       r"(?<!!)\[(?P<label>[^\]]*)\]\(\s*(?P<target><[^>]+>|(?:[^()\s]|\([^()\s]*\))+)(?:\s+[^)]*)?\)"
   )

   Every match is then a whole destination, so replace drops its bare-( exception (render.py:2087-2091) and ends with return f"[{match.group('label')}]({bracketed(href)})" if href else match.group("label"). With that patch, _rewrite_links links .../sources/Smith%282019%29.pdf with coordinates and publishes [S](<../sources/Smith(2019).pdf>) without them. The issue's three cases and the directory link come out unchanged, and ruff passes. One case moves the other way. An unbalanced bare target such as [x](../sources/a(b.md) is text to Python-Markdown, and it now stays text with coordinates too, where this PR links it. Replace test_a_bare_link_with_parentheses_stays_bare (tests/test_render.py:1440) with a committed-project case that expects {_REPOSITORY}/blob/{commit}/blueprint/sources/Smith%282019%29.pdf, with _render_statement_link also writing Smith(2019).pdf. In the PR text, drop the bare-( bullet and the balanced-parentheses half of "This changes two outcomes".

Optional

• bracketed() in the definition branch (render.py:2105) changes nothing Python-Markdown reads. Its reference parser takes the destination up to whitespace and strips < and >, so once whitespace is encoded the brackets make no difference. Encode only, and drop "a parenthesis still gets angle brackets" from the PR text.
• The query tests cover only the repository and node branches. Removing {asked}{query} at render.py:2074 would pass every test, yet ../sources/paper.md?plain=1#L3 without coordinates would lose ?plain=1. Add that case.
• Give _render_statement_link a committed=False flag. The no-coordinates cases (angle brackets with a space, the query above, balanced parentheses) can then share one parametrized test. The new tests' GITHUB_* delenv loops have no effect, because passing repository_url="" explicitly skips detection (render.py:384).
• _source_link_issues' docstring (autoform_cli/__main__.py:534-536) still lists "a repository URL that 404s, or a path off the site" among render's broken outputs. After Keep the text of a published link that leaves the blueprint #318 and this PR, render produces neither. Let work and search omit what they can't represent, and have check warn #325 rewrites that docstring but keeps the clause, so whichever of the two lands second can fix it.

Landing

• Merges cleanly into main at 09cb009 and does not conflict with Let work and search omit what they can't represent, and have check warn #325. Leave malformed link targets as written in render and the runtime #327 conflicts in moved_target just after the split: keep this branch's split above Leave malformed link targets as written in render and the runtime #327's try, as the PR text shows. Leave malformed link targets as written in render and the runtime #327 also catches the ValueError this PR starts raising for [x](<https://[::1/a b>) (render.py:2053 and render.py:1290), so landing Leave malformed link targets as written in render and the runtime #327 first closes that window.

Decision for the maintainer

• sources/ targets the capture lacks, under Let work and search omit what they can't represent, and have check warn #325. With coordinates, such a link now fails the strict build. For a missing note, main deployed a 404, and failing matches the no-coordinates build and Make check reject Sources links that name no file in the blueprint #287. But capture skips hidden files, so a committed note such as sources/.private/paper.md now stops the Pages build too, where main gave it a working repository link. Let work and search omit what they can't represent, and have check warn #325's rule that tools must not break a codebase argues against that half. Options: (a) land as is; (b) keep main's repository URL for hidden targets, with and (os.path.exists(candidate) or _is_hidden(relative)) at render.py:2071; (c) keep only the link text, as Keep the text of a published link that leaves the blueprint #318 does for targets outside the blueprint, and add a render warning. Recommendation: (a) for missing notes plus (b), so no link that resolved on main stops the build. The Make check reject Sources links to hidden paths #321 review asks the same question from check's side; decide the two together.
• Balanced parentheses in the other link readers. graph's _LINK (autoform_cli/graph.py:25), markdown.LINK (autoform_cli/markdown.py:80) and Read code-only labels, reference links and nested brackets as links #320's _INLINE_TARGET also end a bare target at its first ). So on main, check already rejects a ## Sources link like the one above to a file that exists. Recommendation: fix render here, and open a follow-up for those readers once Read code-only labels, reference links and nested brackets as links #320 lands, because that change alters check's verdicts.

This PR came from the same PR swarm that reviewed it, so this is the author's review, not an independent one. It was a static review: no tests were run locally. CI at this head: all 7 checks pass, including the real-Lean jobs.

Posted by PR swarm: PR Swarm Lead

[Deicyde]

Review changes addressed at f91863d

• Bare link destinations now take one level of balanced parentheses, so [S](../sources/Smith(2019).pdf) reaches the repository whole instead of as Smith%282019.
• Titles may hold balanced parentheses, and quoted titles an unclosed (, so "Smith (2019)" leaves no ") behind and no match runs past its own link's ): See [x](../sources/a(b.md) and [y](../sources/c.md). keeps [x] as the text Python-Markdown reads it as, and still rewrites [y].
• A backslash-escaped parenthesis counts in neither part, so (see [x](../sources/a\(b.md) and [y](../sources/c.md)) keeps everything after [x].
• One outcome changes and the description now lists it: [x](../sources/a(b.md "t") is left as written, so with coordinates the strict build reports it where main wrote a repository URL.
• Render tests cover each case, and the description says which earlier patterns fail them.
• Rebased onto main at feec945, which includes Let work and search omit what they can't represent, and have check warn #325.

Still open for you: whether a hidden note under sources/ should keep its repository link, as on main, or fail the strict build as it does here. #321's review asks the same question from check's side, so the two are best decided together.

CI is running on this head.

Posted by PR swarm: PR Swarm Lead

[Deicyde]

Follow-up review at a2f35a5

Verdict: Ready to merge. This pass checked each fix from the last round against the code, merged the branch with main at d1ad2fb and with #339, and read the whole diff against main again, including math, code spans, hoisted bodies and reference definitions shaped like links. Every should-fix from the last review is resolved and no new one turned up; what is left is the hidden-note decision below.

Changed in this pass

• Rebased onto main at 7e34a3a; it merges cleanly into d1ad2fb, which adds Report malformed Sources and dependency links instead of crashing #337.
• An inline link whose destination does not move is left as written (autoform_cli/render.py:2082), so $[X,Y](f(p))$ and `xs[i](f(x))` come out unchanged and unmoved links keep their titles.
• A reference definition only percent-encodes whitespace (autoform_cli/render.py:2097); angle brackets change nothing Python-Markdown reads there.
• The book's page order splits off a query too (autoform_cli/render.py:1288), so roadmap/README.md?plain=1 no longer drops its chapter from the book.
• The node branch adds the query in two lines (autoform_cli/render.py:2060) and leaves the line Guard #289's linear render and drop caches lazy links made redundant #339 replaces alone.
• New tests: a page link that stays on the site keeps its query; single-quoted titles and a space before ); a decoded space or ( gets angle brackets; math is left alone; a book link with a query keeps its chapter. The no-coordinates definition test is now a parameter of the existing one, _render_statement_link takes committed=, and the dead GITHUB_* loops and the assertions that could not fail are gone.
• Shorter _MARKDOWN_LINK comment, commit message and Tests section. The PR text now names the right base and open-PR conflicts, notes that an empty captured directory still gets a 404 URL, and says which tests run without coordinates. One test docstring no longer says a space ends a bare destination; Python-Markdown reads it whole.

Decision for the maintainer

• Hidden sources/ notes (decide with Make check reject Sources links to hidden paths #321): capture skips hidden files, so a committed sources/.private/paper.md that resolved on main now fails the strict build. Recommendation: keep main's repository URL for hidden targets with os.path.exists(candidate) or _is_hidden(relative) at autoform_cli/render.py:2068, so no link that resolved on main stops the Pages build (Let work and search omit what they can't represent, and have check warn #325). Whichever of this PR and Make check reject Sources links to hidden paths #321 lands second should also drop "a repository URL that 404s, or a path off the site" from _source_link_issues' docstring (autoform_cli/__main__.py:553): render no longer produces either for a missing or escaping target.
• Other link readers: graph's _LINK and markdown.LINK still end a bare target at its first ), so check rejects a ## Sources link to Smith(2019).pdf that render now reads whole. Recommendation: a follow-up once Read code-only labels, reference links and nested brackets as links #320 lands, since it changes check's verdicts.

This PR came from the same PR swarm that reviewed it, so this is the author's review, not an independent one. No tests were run under pytest locally. The 12 new tests, both cases of the reference-definition test and 15 existing link, page-order and slot tests were called directly at this head and pass, and each of 24 mutants of the new rules fails the test meant to catch it. CI at this head: all 7 checks pass, including the real-Lean jobs.

Posted by PR swarm: PR Swarm Lead