Skip to content

Don't let one failed WebKit export check crash the poller - #276

Open
DanielRyanSmith wants to merge 1 commit into
mainfrom
fix-webkit-poller-crash
Open

DanielRyanSmith wants to merge 1 commit into
mainfrom
fix-webkit-poller-crash

Conversation

@DanielRyanSmith

@DanielRyanSmith DanielRyanSmith commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

When the bot polled for bug information from bugs.webkit.org, occasionally the response would be empty, which crashed the whole process. This change makes it so that an occasional empty response doesn't crash things. It may be that large number of requests all at once are getting throttled, but this should allow partial recovery at the very least.

Details

The WebKit export poller handled each pull request in an async forEach callback. The surrounding try/catch doesn't see errors thrown in those callbacks, so any failure became an unhandled rejection, which exits the process on current Node. One bad response from bugs.webkit.org (or GitHub, or Specref) for one PR therefore killed the whole poll, and the instance with it, before any PR was approved.

Since #274 was deployed, this has happened on every full poll, so the WebKit export PRs that are waiting for approval still haven't been approved:

Poll started (UTC) Crashed Error
2026-10-22 17:14:31 17:15:09 invalid json response body at https://bugs.webkit.org/rest/bug/283585/history?...
2026-10-22 17:38:40 17:39:16 invalid json response body at https://bugs.webkit.org/rest/bug/298271?...

The failing bug changes from poll to poll, and the same requests succeed when made on their own, so this looks like bugs.webkit.org sometimes returning empty responses when the poller sends ~100 logged-in requests at once. Before #274, the same pattern crashed the bot on other transient errors, e.g. an ECONNRESET from api.specref.org on 2026-10-21.

Changes

  • Move the poller into lib/webkit-poller.js so it can be tested (index.js starts the server when it's loaded).
  • Handle pull requests one at a time instead of all at once. This limits the load on bugs.webkit.org.
  • Catch and log errors for each pull request (#N: Unable to check WebKit export; will retry on the next poll.) so the rest are still checked. Skipped PRs are retried on the next poll.
  • Log a summary at the end of each poll: Checked N WebKit-exported pull requests; M failed.
  • Otherwise unchanged: the same webkitExport log entry for each PR, a failure to add labels still doesn't stop the PR from being approved, and a failure to list PRs is still caught and logged by index.js.

Testing

  • npm test passes (lint, 84 tests, coverage). The new test/webkit-poller.js covers:
    • non-export PRs are ignored;
    • dry-run is passed through;
    • a failure on one PR doesn't stop the others (regression test);
    • a label failure still leads to commenting;
    • a comment failure is reported and the poll continues;
    • PRs are handled strictly one at a time;
    • an error listing the PRs still rejects.
  • Confirmed that the old forEach(async …) inside try/catch pattern exits Node 24 with code 1 when one callback throws.

Notes

  • Polls will take longer now that PRs are handled one at a time. PRs not reached before an instance shuts down are picked up on the next poll.
  • If bugs.webkit.org still returns empty responses even when requests come one at a time, the affected PRs will be logged and retried rather than crashing the bot.
  • Planned follow-up: fetch Bugzilla data in batches. Both lookups the poller makes accept many bugs per request (/rest/bug?id=A,B,C&include_fields=id,see_also and /rest/bug/A/history?ids=B&ids=C), so a poll could make about 2–4 Bugzilla requests instead of ~100. That also makes polls fast again. Reusing connections in lib/bugs-webkit.js with a capped keep-alive agent is another option. This PR keeps the minimal crash fix separate; the per-PR error handling is needed either way.
  • Unrelated issue found while doing this, not fixed here: both callers run labelModel.post(n, labels, dryRun), but its signature is (issue, labels, log, dryRun), so --dry-run doesn't stop labels from being added.

Some or all of the code in this change was generated by an AI agent.

The WebKit export poller handled each pull request in an async
`forEach` callback. The surrounding `try`/`catch` doesn't see errors
thrown in those callbacks, so any failure became an unhandled
rejection, which exits the process on current Node. One bad
response from bugs.webkit.org (or GitHub, or Specref) for one PR
therefore killed the whole poll before any PR was approved, and the
instance with it.

This happened on both full polls after #274 was deployed. Empty
responses from bugs.webkit.org (for the history of bug 283585, then
for bug 298271) crashed the process ~37s into each poll, so none of
the export PRs that were waiting for approval got approved.

Move the poller into `lib/webkit-poller.js` so it can be tested, and
handle the pull requests one at a time, catching and logging errors
for each so that the rest are still checked. Skipped PRs are retried
on the next poll. Handling them one at a time also stops the poller
from sending ~100 simultaneous requests to bugs.webkit.org. As
before, a failure to add labels is logged and doesn't stop the PR
from being approved.

Some or all of the code in this commit was generated by an AI agent.

Assisted-by: AI agent
@DanielRyanSmith
DanielRyanSmith force-pushed the fix-webkit-poller-crash branch from 063a771 to 3146dd5 Compare October 22, 2026 18:27
@DanielRyanSmith
DanielRyanSmith marked this pull request as ready for review October 22, 2026 18:27
@DanielRyanSmith

Copy link
Copy Markdown
Collaborator Author

I ran this change against the real wpt repository and bugs.webkit.org through a full WebKit export poll, and it seems to work as intended, including approving this PR: web-platform-tests/wpt#64029

@gsnedders

Copy link
Copy Markdown
Member

Planned follow-up: fetch Bugzilla data in batches.

Before you start on that, you should just review #271 which does that!

@gsnedders

Copy link
Copy Markdown
Member

Looking closer, everything this PR does is already done by #271, except for the refactoring away from index.js.

Given that was already awaiting review, it would've been good to just get that landed!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants