Skip to content

Fix WebKit export approval: verify landings against the WebKit repository - #273

Closed
DanielRyanSmith wants to merge 2 commits into
mainfrom
fix-webkit-who-format
Closed

DanielRyanSmith wants to merge 2 commits into
mainfrom
fix-webkit-who-format

Conversation

@DanielRyanSmith

@DanielRyanSmith DanielRyanSmith commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

wpt-pr-bot stopped approving WebKit export PRs. After investigating, it looks like the format of the response from the bugs.webkit.org API changed. This change fixes the review check to allow the PRs to continue to be auto-approved.

More investigation details (generated):

Details ## Summary

wpt-pr-bot stopped approving WebKit-exported PRs on
2026-10-16. There was no deploy; the serving version dates
from 2026-09-08.

WebKit now lands patches without review+ flags, so the bot treats an export
as reviewed when the WebKit bug is resolved FIXED by ews-feeder or
commit-queue. The check compared history_item.who against full email
addresses (ews-feeder@webkit.org). bugs.webkit.org now applies Bugzilla's
email filtering for logged-out API callers, so who contains only the part of
the address before the @ (ews-feeder). The bot calls the API without
logging in (see follow-ups). Because nothing matched, webkit.flags.reviewed
stayed false and no exports were approved.

The fix

The bot now confirms the landing with the WebKit repository itself:

  1. The who check accepts ews-feeder / commit-queue, with or without
    @webkit.org (other domains and different casing are rejected). It is now
    only a cheap pre-filter.
  2. If the history shows a landing bot resolving the bug FIXED, the new
    webkit.landed() reads the bug's comments and finds the landing bot's
    comment in either format:
    • Committed 324168@main (004e01c34435): <https://commits.webkit.org/324168@main>
    • Committed r266816: <https://trac.webkit.org/changeset/266816> (legacy
      SVN format)
  3. For each referenced commit it fetches https://commits.webkit.org/<ref>/json
    (new lib/commits-webkit.js) and requires:
    • branch === "main"
    • the full hash starts with the abbreviated hash in the comment, if the
      comment has one
    • the commit message references this exact bug
      (https://bugs.webkit.org/show_bug.cgi?id=N or https://webkit.org/b/N,
      not followed by another digit)

Only WebKit committers can land on main, so this can't be spoofed by an
arbitrary Bugzilla user. The ordering behaviour of the existing review+ /
commit-queue+ flag logic is unchanged.

Commit references are parsed from comment text, so commits-webkit.get()
accepts only ^\d+@main$ or ^r\d+$ before building a URL. Errors from
commits.webkit.org are logged and treated as "not landed"; the poller retries
every 15 minutes.

Why not just accept ews-feeder?

The part of an email before the @ is not unique. Anyone can register a
Bugzilla account such as ews-feeder@example.com, and it would also show up as
ews-feeder. Accepting the bare name would let anyone get a PR approved
without review:

  1. Register ews-feeder@<anything>.
  2. File a bug.
  3. Add the PR to the bug's see_also.
  4. Resolve the bug FIXED.
  5. Title a WPT PR "WebKit export of ".

Testing

  • test/webkit.js (26 tests):
    • verified landings for all four account forms
    • rejection of other users, lookalike domains, different casing and a
      missing who
    • an unverified "attacker" landing
    • nonexistent commits, commits not on main, hash mismatches, commits for
      other bugs and bug-number prefixes, comments with no landing reference
    • commits.webkit.org errors
    • the review+ path and flag ordering
  • test/commits-webkit.js (10 tests): fetching, 404 → null, errors, and
    rejection of invalid or path-traversal references.
  • test/get-metadata.js: the existing WebKit 201401 test now stubs the comment
    and commit lookups. Its expected output is unchanged.
  • Local nyc per-file coverage: commits-webkit.js 100%, webkit.js 91%. The
    remaining lines are covered by get-metadata.
  • Live check: ran webkit.flags() against real bugs.webkit.org and
    commits.webkit.org for all 49 open "WebKit export of" PRs in
    web-platform-tests/wpt.
    • #63879, #63886 and all PRs previously approved via this path are now
      reviewed: true.
    • Every reviewed: false result is a WebKit bug that is still open, so
      there are no false negatives.

Notes

  • WebKit committers can land "Unreviewed" commits on main. Those were already
    treated as reviewed before this change (the bot trusted any landing-bot
    FIXED), so this PR doesn't change that.
  • The poller now makes one extra Bugzilla request, plus usually one
    commits.webkit.org request, for each open export whose bug was resolved by a
    landing bot.

After deploy

The poller runs every 15 minutes and approval is one-way, so stuck exports
(including #63879 and #63886) should be approved on the next poll.

bugs.webkit.org now reports the 'who' field of bug history as a bare
login name (e.g. 'ews-feeder') rather than a full email address
(e.g. 'ews-feeder@webkit.org'). This caused WebKit-exported PRs to no
longer be detected as reviewed, and therefore never approved.

Accept either form when checking whether a bug was resolved FIXED by
ews-feeder or commit-queue.
bugs.webkit.org now reports only the local part of email addresses
(e.g. 'ews-feeder') to logged-out API callers. The local part is not a
unique identity: anyone can register e.g. 'ews-feeder@example.com',
resolve their own bug as FIXED, and so have a PR titled 'WebKit export
of <bug>' approved without review.

Treat the 'who' check as a pre-filter only, and confirm the landing by
finding the landing bot's 'Committed <ref>' comment and checking with
commits.webkit.org that the referenced commit is on main and its message
references the bug.
@DanielRyanSmith DanielRyanSmith changed the title Fix WebKit export approval after bugs.webkit.org 'who' format change Fix WebKit export approval: verify landings against the WebKit repository Oct 20, 2026
@DanielRyanSmith

Copy link
Copy Markdown
Collaborator Author

The bot is not approving WebKit PRs until we deploy a new version with a change like this

@gsnedders

gsnedders commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Should've split this (a49ca8c) out of #271 a while ago, probably!

Wait, that doesn't help this at all.

@gsnedders

Copy link
Copy Markdown
Member

The check compared history_item.who against full email addresses (ews-feeder@webkit.org). bugs.webkit.org now applies Bugzilla's email filtering for logged-out API callers, so who contains only the part of the address before the @ (ews-feeder).

We could presumably also just fix how we authenticate, and get the actual emails?

This seems to have regressed with WebKit/WebKit@189ce80.

The bot calls the API without logging in (see follow-ups).

There are no follow-ups linked.

This does seem very LLMish, and I will remind you of the Policy for LLM (AI) Use.

@gsnedders gsnedders left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm gonna have a very quick go at fixing this, based on my earlier change in the larger PR, inc. fixing auth.

Live check: ran webkit.flags() against real bugs.webkit.org and commits.webkit.org for all 49 open "WebKit export of" PRs in web-platform-tests/wpt.

We should add/update fixtures in test/fixtures to verify this, versus just updating our unit tests?

Comment thread lib/metadata/webkit.js
logger = require('../logger');

// Accounts that resolve a bug as FIXED when a reviewed patch lands in WebKit.
const WEBKIT_LANDING_BOTS = ["ews-feeder", "commit-queue"];

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's actually store these by their actual usernames, for when we actually fix authentication:

Suggested change
const WEBKIT_LANDING_BOTS = ["ews-feeder", "commit-queue"];
const WEBKIT_LANDING_BOTS = ["ews-feeder@webkit.org", "commit-queue@webkit.org"];

Comment thread lib/metadata/webkit.js
Comment on lines +9 to +12
// Matches the comment left by WebKit's landing bots, e.g.
// "Committed 324168@main (004e01c34435): <https://commits.webkit.org/324168@main>"
// "Committed r266816: <https://trac.webkit.org/changeset/266816>"
const LANDING_COMMENT_PATTERN = /Committed (\d+@main|r\d+)(?: \(([0-9a-f]{7,40})\))?/g;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an interesting inconsistency here, insofar as we allow any branch for SVN but only main for git.

The history of comments is something like:

https://bugs.webkit.org/show_bug.cgi?id=209237, commit-queue@webkit.org:

Patch

Clearing flags on attachment: 393868

Committed r258643: https://trac.webkit.org/changeset/258643

https://bugs.webkit.org/show_bug.cgi?id=208861, ews-feeder@webkit.org:

Committed r258210: https://trac.webkit.org/changeset/258210

All reviewed patches have been landed. Closing bug and clearing flags on attachment 393151 [details].

https://bugs.webkit.org/show_bug.cgi?id=223117, ews-feeder@webkit.org:

Committed r274424: https://commits.webkit.org/r274424

All reviewed patches have been landed. Closing bug and clearing flags on attachment 423037 [details].

Eight bugs, including https://bugs.webkit.org/show_bug.cgi?id=230351, ews-feeder@webkit.org:

Committed r286611 (244935@trunk): https://commits.webkit.org/244935@trunk

All reviewed patches have been landed. Closing bug and clearing flags on attachment 446169 [details].

And https://bugs.webkit.org/show_bug.cgi?id=247043 contains two examples, ews-feeder@webkit.org:

Committed 256627@main (8f70e6bf626c): https://commits.webkit.org/256627@main

Reviewed commits have been landed. Closing PR #6300 and removing active labels.

Plus:

Committed 252432.920@safari-7614-branch (768d89b728fc): https://commits.webkit.org/252432.920@safari-7614-branch

Reviewed commits have been landed. Closing PR #245 and removing active labels.

Though we can have things only on a branch on their own bug.

I might be tempted to use a regex more like:

Suggested change
// Matches the comment left by WebKit's landing bots, e.g.
// "Committed 324168@main (004e01c34435): <https://commits.webkit.org/324168@main>"
// "Committed r266816: <https://trac.webkit.org/changeset/266816>"
const LANDING_COMMENT_PATTERN = /Committed (\d+@main|r\d+)(?: \(([0-9a-f]{7,40})\))?/g;
// Matches the comment left by WebKit's landing bots, e.g.
// "Committed 324168@main (004e01c34435): <https://commits.webkit.org/324168@main>"
// "Committed 252432.920@safari-7614-branch (768d89b728fc): <https://commits.webkit.org/252432.920@safari-7614-branch>"
// "Committed r286611 (244935@trunk): <https://commits.webkit.org/244935@trunk>"
// "Committed r274424: <https://commits.webkit.org/r274424>"
// "Committed r266816: <https://trac.webkit.org/changeset/266816>"
const LANDING_COMMENT_PATTERN = /^Committed ([0-9]+(?:\.[0-9]+)*@[^\000-\040 ~^:?*[\\]+|r[0-9]+)[ :]/gm;

Comment thread lib/metadata/webkit.js
Comment on lines +64 to +65
var bugURLPattern = new RegExp(
`(?:https?://bugs\\.webkit\\.org/show_bug\\.cgi\\?id=|https?://webkit\\.org/b/)${issue}(?!\\d)`);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't change

var titleWebkitIssueMatch = metadata.title.match(/.*http(?:s?):\/\/bugs.webkit.org\/show_bug.cgi\?id=(\d+).*/i);
and thus the https?://webkit\\.org/b/ alternation is pointless.

We also only have a single bug matching that, which suggests we might not need it: https://github.com/web-platform-tests/wpt/pulls?q=is%3Apr+webkit.org%2Fb%2F+in%3Atitle

Comment thread lib/metadata/webkit.js
for (var match of comment.text.matchAll(LANDING_COMMENT_PATTERN)) {
var ref = match[1], abbreviatedHash = match[2];
try {
var commit = await commitsWebkit.get(ref);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to ensure the abbreviatedHash points at something suitable, and not just trust the canonical identifier, then we should pass it in here, rather than the canonical identifier?

Comment thread lib/commits-webkit.js
throw new Error(`Invalid WebKit commit reference: ${ref}`);
}
const response = await exports.fetch(`${BASE_URL}/${ref}/json`, {
headers: { "User-Agent": "wpt-pr-bot" },

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should match everything else:

const response = await fetch(url, {headers: { "User-Agent": "ganesh" }});

"User-Agent": "ganesh",

"User-Agent": "ganesh"

Suggested change
headers: { "User-Agent": "wpt-pr-bot" },
headers: { "User-Agent": "ganesh" },

@gsnedders

Copy link
Copy Markdown
Member

I'm gonna have a very quick go at fixing this, based on my earlier change in the larger PR, inc. fixing auth.

See #274

@DanielRyanSmith

Copy link
Copy Markdown
Collaborator Author

I'm going to go ahead and drop this PR in favor of #274 . Thanks for taking a look at this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants