Repository navigation
ci: cut the runtime of the nightly link check - #1215
lundybernard wants to merge 4 commits into
Conversation
|
Detailed PR notes 🤖 AI text below 🤖 This is the runtime PR for #1179. Merge order: any. Baseline, measured on 2026-10-27, before #1207 merged and before #1212.
The three fork runs differ from this branch. Verification:
Reviewer notes:
|
| return None | ||
|
|
||
|
|
||
| def rewrite_github_code_link(uri: str) -> str | None: |
There was a problem hiding this comment.
now that we've updated the ignore/link style for the github urls, do we still want to keep this rewrite of github urls? mostly asking naively without looking too closely.
There was a problem hiding this comment.
Yes, I'd keep it. It covers blob and tree links, which #1214 doesn't touch.
A run checks 110 of those, or 108 once the anchors drop.
On main they cause 21 to 37 rate-limit waits a run, none with the rewrite.
In my trial runs it cuts the link check step ~8, to ~5.
TimMonko
left a comment
There was a problem hiding this comment.
Great! Can we get a re-timing on these now that we've cleaned up the links (and the docs CI). I'm wondering how much time we are saving now that we aren't rate limited by Github and such with all the recent PRs, so therefore whether its worth adding a bit more complexity for the nightly time save.
| # A nightly link check skips the release notes before the newest two minor | ||
| # series. Without LINKCHECK_SCOPE=nightly, the link check reads every page. | ||
| if os.environ.get('LINKCHECK_SCOPE') == 'nightly': |
| env: | ||
| PIP_CONSTRAINT: ${{ github.workspace }}/napari/resources/constraints/constraints_py3.12_docs.txt | ||
| LINKCHECK_SCOPE: ${{ steps.scope.outputs.scope }} # on nightly, conf.py skips the old release notes | ||
| GITHUB_TOKEN: ${{ github.token }} # conf.py sends it to api.github.com only |
There was a problem hiding this comment.
why is this here now, but not previously? I see it's used in the makefile now but I'm curious why it's only showing up in this PR.
There was a problem hiding this comment.
The token only helps on GitHub's API, not on the github.com website.
So this PR checks links to code through the API, and this line gives that check the job token. Without the token, the API stops at 60 calls an hour, and a run checks about 110 code links.
The Makefile doesn't name it, it just hands the step env to sphinx-build.
There was a problem hiding this comment.
Thank you for the explanation!
33bdd20 to
655b184
Compare
|
re-timing on current
I dropped the release-note skip. I force-pushed the branch with one commit fewer. |
The link check reads the links in the page sources, not in the notebook outputs. Notebook execution adds build time but does not add links to check. Refs: napari#1179 Assisted-by: claude-code:claude-opus-5-5 [The Engineer]
The docs hold 325 unique commits?author= links, most of them in the release notes. Each check spends GitHub rate-limit budget, and none of them can report a broken link. Assisted-by: claude-code:claude-opus-5-5 [The Engineer]
output.txt lists only the broken, redirected, and timed-out links. output.json holds every checked URI and its status. Assisted-by: claude-code:claude-opus-5-5 [The Engineer]
github.com answers most of its 429s on blob and tree pages, and it ignores a token on HTML pages. With GITHUB_TOKEN set, the linkcheck-process-uri hook rewrites each blob or tree link to its contents URI on api.github.com, which accepts the token. The workflow sets the token. A run without it checks the github.com page, as before. The check drops the page anchor, and a failure reports under the API URI. Assisted-by: claude-code:claude-opus-5-5 [The Engineer]
655b184 to
3982419
Compare
|
Thanks for the updates @lundybernard! I'm a bit confused by your table in #1215 (comment) suggesting no rate limits, but when I look at https://github.com/3M1LY-lb/napari-docs/actions/runs/43984023760/job/133707442017#step:7:5244 it seems like we get rate-limited by Github still? Mostly trying to understand and look for clarification Would it be helpful to add the author and org checks to the rest API? We can do that here or in a follow-up, whatever you prefer. I think this is ready to go once my questions are answered here It seems our slow part now is down to the docs build, rather than stalling during the link checks -- so thank you! I appreciate how you slimmed down the logic to not have special rules, for now. 🍻 |
|
@TimMonko My table column was too narrow. It counts the waits on code links only (blob and tree), because those are the links the REST API rewrite moves. Yes, I think moving the user and org pages to the REST API would help. I'd rather do that in a follow-up PR, so this one can merge as it is. |
Much of the link phase is idle time after github.com answers with 429.
Every run skips notebook execution and the commits?author= links, and
checks GitHub code links through the REST API with the workflow token.
The report artifact gains output.json, with the status of every checked
URI.
Changes:
build: set NB_EXECUTION_MODE off for linkcheck-files
The link check reads the links in the page sources, not in the
notebook outputs. Notebook execution adds build time but does not add
links to check.
perf: add GitHub commits?author= links to linkcheck_ignore
The docs hold 325 unique commits?author= links, most of them in the
release notes. Each check spends GitHub rate-limit budget, and none of
them can report a broken link.
ci: upload output.json with the link check report
output.txt lists only the broken, redirected, and timed-out links.
output.json holds every checked URI and its status.
perf: check GitHub code links through the REST API
github.com answers most of its 429s on blob and tree pages, and it
ignores a token on HTML pages. With GITHUB_TOKEN set, the
linkcheck-process-uri hook rewrites each blob or tree link to its
contents URI on api.github.com, which accepts the token. The workflow
sets the token. A run without it checks the github.com page, as before.
The check drops the page anchor, and a failure reports under the API
URI.
Refs: #1179
Assisted-by: claude-code:claude-opus-5-5 [The Engineer]