Repository navigation
Reduce bugs.webkit.org API calls - #271
Conversation
DanielRyanSmith
left a comment
There was a problem hiding this comment.
There's some outdated text in lib/comment.js that needs to be updated:
Lines 48 to 49 in 3348953
| bugToPRs[bugId].push(pull_request); | ||
| }); | ||
|
|
||
| const everFixedBugs = await webkit.everFixed(Object.keys(bugToPRs).map(Number)); |
There was a problem hiding this comment.
Gemini found a possible regression:
Details
Webhook Failure Recovery (Potential Regression)
- How it worked before: If the webhook for a newly opened WebKit PR failed, the periodic
pullRequestPollerwould eventually pick it up, fetch its metadata, label itwebkit-export, and post the initial explanation comment. - How it works now: The new
pullRequestPolleronly processes PRs whose associated WebKit bugs are alreadyFIXED(viawebkit.everFixed). If a webhook fails for an unfixed WebKit PR, the bot will never label it or leave the initial explanation comment until the bug actually lands in WebKit (i.e. becomesFIXED). - Recommendation: Optimize the filter in
pullRequestPollerto also process PRs that do not yet have thewebkit-exportlabel. The label status can be read directly from the GitHub PR payload (available inpull_requests), which does not require additional Bugzilla API calls.
There was a problem hiding this comment.
This is true — however, the new behaviour is no different to the behaviour of Chromium and Mozilla exports, which don't have a poller as a backstop.
| .filter(bugId => everFixedBugs.has(Number(bugId))) | ||
| .flatMap(bugId => bugToPRs[bugId].map(pull_request => ({pull_request, webkitBug: everFixedBugs.get(Number(bugId))}))); | ||
|
|
||
| everFixedPRs.forEach(async function({pull_request, webkitBug}) { |
There was a problem hiding this comment.
Hey Sam,
Using forEach with an async callback means these promises aren't awaited by pullRequestPoller
pullRequestPoller() will exit its try block immediately and schedule the next 15-minute waitFor() timer while PRs are still running in the background. If get_metadata throws (e.g. on a network timeout or API error), the rejection won't be caught by the outer try/catch and will emit an unhandled promise rejection.
Swapping this to a standard for (const item of everFixedPRs) loop should fix both the unawaited scheduling and error catching.
| return new Map(); | ||
| } | ||
| const params = new URLSearchParams({ | ||
| id: bugIds.join(","), |
There was a problem hiding this comment.
If we have a large number of open WebKit export PRs (e.g., 100+), joining all bug IDs into a single id query parameter:
const params = new URLSearchParams({
id: bugIds.join(","),
...
});will push the GET request URI past 4KB–8KB, which will get rejected by Cloudflare/Apache with 414 Request-URI Too Large or 400 Bad Request.
We should probably chunk bugIds in batches (e.g., max 50 IDs per request) inside everFixed to keep the URI length well within safe limits:
const CHUNK_SIZE = 50;
const result = new Map();
for (let i = 0; i < bugIds.length; i += CHUNK_SIZE) {
const chunk = bugIds.slice(i, i + CHUNK_SIZE);
const params = new URLSearchParams({ id: chunk.join(",") /* ... */ });
const body = await bugsWebkit.get("/rest/bug?" + params);
if (body && Array.isArray(body.bugs)) {
for (const b of body.bugs) result.set(b.id, b);
}
}
return result;There was a problem hiding this comment.
Yeah, this is a good point — now done this, capping it at 50.
| }); | ||
|
|
||
| const everFixedBugs = await webkit.everFixed(Object.keys(bugToPRs).map(Number)); | ||
| const everFixedPRs = Object.keys(bugToPRs) |
There was a problem hiding this comment.
FYI / Heads-up: Since the poller now only queries everFixedBugs, if an export PR is opened before see_also is added on Bugzilla (or if the initial opened webhook fails for any reason), the poller won't pick it up to add the webkit-export label or post the initial comment until the patch actually lands on WebKit.
I think that's fine given the API savings, but just wanted to make sure that trade-off was intentional!
There was a problem hiding this comment.
I was going to, in a follow-up, try adding some delay to the webhook case to try and avoid #164, to increase the odds of see_also existing.
gsnedders
left a comment
There was a problem hiding this comment.
(I wrote these ages ago and never sent them, whoops!)
| bugToPRs[bugId].push(pull_request); | ||
| }); | ||
|
|
||
| const everFixedBugs = await webkit.everFixed(Object.keys(bugToPRs).map(Number)); |
There was a problem hiding this comment.
This is true — however, the new behaviour is no different to the behaviour of Chromium and Mozilla exports, which don't have a poller as a backstop.
| }); | ||
|
|
||
| const everFixedBugs = await webkit.everFixed(Object.keys(bugToPRs).map(Number)); | ||
| const everFixedPRs = Object.keys(bugToPRs) |
There was a problem hiding this comment.
I was going to, in a follow-up, try adding some delay to the webhook case to try and avoid #164, to increase the odds of see_also existing.
970bf90 to
bf74be2
Compare
When almost all WebKit contributions are in GitHub PRs, looking at these flags on Bugzilla is near useless.
The forEach(async ...) callbacks were never awaited, so a rejection from get_metadata was an unhandled rejection (fatal on modern Node) rather than being caught and logged by the surrounding try/catch. Use a for...of loop so the work is awaited and errors reach the catch block.
Instead of fetching bug details individually for each WebKit export PR, group them by bug ID and query for all bugs that have ever been marked FIXED in a single API call. This significantly reduces the number of API calls in the common case.
Serialize all bugs.webkit.org requests through a shared throttle so outgoing calls are spaced at least 1 second apart, regardless of caller. This protects bugs.webkit.org from accidental bursts (e.g. from the WebKit poller fanning out per-bug history fetches).
Rewrite the per-PR work with async/await instead of a .then() chain, and wrap each PR in its own try/catch so a failure for one PR (e.g. in get_metadata) is logged rather than skipping the remaining PRs for the cycle. Labelling and commenting keep separate error handling, so a failure to label still lets the comment step run, as before.
bugs.webkit.org sits behind a gateway which returns an HTML 403 (rather than a Bugzilla JSON error) for e.g. overly long URLs. errFrom() would throw a confusing SyntaxError from JSON.parse in that case; fall back to an error reporting the HTTP status instead. Also fix err.statusCode, which read response.statusCode: node-fetch responses expose .status, so it was always undefined.
bf74be2 to
7570a40
Compare
Instead of polling each and every WebKit bug for every open GitHub PR, this instead uses a single query to fetch all bugs which have ever been marked as fixed. This has a significant impact on the number of requests made to the Bugzilla API, and should make it practical to enforce a rate limit which we previously did not have.
As a precursor to this, we also drop support for r+ and commit-queue+, given these are largely unused nowadays.