[baxen]

Why

Operator-listener deliveries (the push path kgoose uses to wake remote agents) reached the listener 0.25–2s after the mention committed. The delivery worker only found new outbox rows on its idle poll, which backs off from 250ms to 2s. A claimed batch also had to finish completely before the worker claimed again, so one slow endpoint (up to the 5s timeout) held up every mention queued behind it.

What changed

• The saving pod sends its own mentions. enqueue_mentions_in_transaction now returns the ids of the outbox rows it inserted, and the event's AdmittedTx holds them. After commit() succeeds, it hands them to this process's delivery worker over a bounded channel. Rollback hands off nothing. The worker claims exactly those rows with claim_deliveries_by_id, then uses the existing send/complete/retry path.
• One claim query. claim_deliveries_by_id is the poll's query plus AND ($6::uuid[] IS NULL OR o.id = ANY($6)). It has the same due/lease conditions and the same FOR UPDATE SKIP LOCKED. So a row is claimed by the fast path, by this pod's poll, or by another pod's poll, never by two of them.
• Bounded. The hand-off channel holds at most 10 ids, and the worker only reads it while one of its 10 in-flight slots is free. Ids that don't fit are dropped, and the poll picks those rows up. Nothing is spawned outside the worker's JoinSet.
• The poll is unchanged and stays as the safety net. It still runs on the same backoff. In normal operation it finds nothing. It covers rows from paths that don't hand off, a failed fast-path claim, a full hand-off, retries with backoff, and expired leases after a pod dies mid-send.
• No batch head-of-line blocking. The worker keeps up to 10 deliveries in flight in a JoinSet and claims again whenever a slot frees, rather than join_all on a batch of 10.

Why not pg_notify

An earlier version of this PR woke workers with pg_notify from inside the event transaction. Jarrod pointed out the cost. Postgres takes one global lock during the commit of every transaction that ran NOTIFY, and holds it through the WAL flush. So every commit that sends a notification goes through a single queue. While a commit waits in that queue, it still holds its pool connection and its row locks, including the thread_metadata counter rows. A burst of agent mentions would back up behind that lock and slow ordinary ingestion too. The hand-off here never touches Postgres beyond the claim the poll already did. It also doesn't need a dedicated LISTEN connection, and the transaction-pooler and half-open-socket caveats are gone.

The tradeoff: a mention only takes the fast path on the pod that saved it. Every relay pod with BUZZ_OPERATOR_LISTENERS runs a worker, so in practice that's every pod that saves one. Saves on paths without a Db handle (raw-pool helpers) fall back to the poll.

Measured

Local relay (debug build) against the same database. Each latency runs from POST /events accepted to the listener receiving the delivery, with 20 mentions spaced 2.5–4s apart so the worker sits at its idle ceiling:

	median	p90	max
main	684 ms	1208 ms	1931 ms
this branch	3.0 ms	5.5 ms	14.0 ms

The earlier pg_notify version measured 1.9 ms median on the same harness.

Tests

• buzz-db Postgres tests:
  • a committed mention hands exactly its outbox row id to the worker, and an event without a listener mention hands off nothing
  • a rolled-back mention hands off nothing
  • 4 pollers and 4 by-id claimers race over 200 rows: every row is claimed exactly once with attempts = 1, and leased rows can't be claimed by id again. Removing SKIP LOCKED makes this fail.
• buzz-relay unit tests drive the production run_delivery_loop:
  • a handed-off row delivers before any idle poll fires, through a by-id claim
  • a row that was never handed off is still delivered by the poll
  • when the by-id claim fails, the poll still delivers the row
  • hand-offs arriving faster than the poll interval don't push the poll back (it runs on a fixed deadline)
  • a stalled delivery does not hold up a later row
  • in-flight deliveries stay at 10, and the loop never claims by id with no free slot
• Each loop test fails under its matching mutation: no hand-off branch, a poll timer that resets every iteration, and no free-slot guard.
• The poll and the by-id claim are separate SQL texts built from one macro. They share every condition, and each gets its own prepared plan.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-18T23:22:56.178022Z	ea9a023	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

🔐 Codex Security Review

Note: This is an automated, security-focused review generated by Codex.
Use it as a supplement to human review; false positives are possible.

Scope

• Exact PR diff: 8f2c7710e958952b53e8e280cde52eb0d51b7312...ea9a023712fa08e610805f6f4a88891aa8c2235e
• Model: gpt-6.1-sol

💡 Click "edited" above to see earlier reviews for this PR.

---

Review Summary

Overall Risk: NONE

No concrete security, correctness, or reliability regressions found in the authorized PR range. Post-commit delivery retains durable outbox claims, bounded concurrency, and existing community and event-access checks.

Findings

No concrete security, correctness, or reliability findings were identified.

Notes

• Review used read-only source and history inspection. Builds and tests were not executed, as requested.

---

_{Generated by Codex Security Review |
Requested by: @baxen |
Workflow run}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ea9a023712

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Filter fast-path claims to listeners routed by this pod

When pods temporarily have different listener route sets, such as during a rolling configuration update, the saving pod claims every committed ID even if another pod is the only one able to route its listener. deliver_one then releases the unroutable claim with next_attempt_at 30 seconds in the future, preventing the correctly configured pod from claiming it and turning the new fast path into a deterministic delay. Restrict committed claims to listener pubkeys routed by this pod, or only hand off locally routable rows.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Bound the returned handoff IDs

A mention can match many registered listeners and targets, but fetch_all materializes every inserted UUID and retains all of them in AdmittedTx until commit even though the handoff channel accepts at most 10 before discarding the remainder. Large fan-outs therefore add unbounded request memory and commit latency, potentially exhausting the relay before the intended bounded queue provides protection. Stream/count the returned rows while retaining only a fixed number of IDs for the fast path.

AGENTS.md reference: AGENTS.md:L240-L248

Useful? React with 👍 / 👎.

[wesbillman]

🤖 Star Lord’s automated source review via Wes’s account, integrating independent Rocket, Mantis, and Groot review lanes.

One P2 latency issue confirmed; no additional actionable defects found. This is a non-blocking COMMENT review.

[P2] Restrict committed claims to listeners this pod can route. I independently confirmed the existing inline finding. During a rolling listener configuration update, one pod can register listener B globally while another saving pod still routes only listener A. The new committed-ID path at crates/buzz-relay/src/operator_listener.rs:89–93 claims B’s row on the saving pod without a route filter. The unchanged no-route branch at lines 325–337 releases it with next_attempt_at = now() + 30s, preventing the correctly configured pod from delivering it until then. The old poll could also win this race; the immediate local handoff makes that delay much more likely for mentions saved on the mismatched pod. Filter the by-ID claim by locally configured listener pubkeys, leaving other rows due for a routable pod. A regression test should show that an unroutable local handoff leaves the row immediately claimable by the routable worker. This is a rollout latency regression; refunded attempts do not remove the existing 15-minute outbox retention limit.

Optional: the returned-ID allocation concern points to a useful fixed-size collector before the channel: fetch_all and the transaction vector retain all inserted IDs even though only ten can be queued. The extra allocation is visible in source; I have not established a realistic exhaustion threshold, so I am not adding a second defect finding on that basis.

The successful-commit boundary, durable poll fallback, shared SQL lease/claim fencing, ten-task delivery limit, and fixed poll deadline otherwise held up under source inspection. The public PR description has no attached screenshots and no actionable disclosure found. Minimalism 9/10, elegance 9/10, correctness 8/10 pending the route filter above.

Reviewed head ea9a023712fa08e610805f6f4a88891aa8c2235e against base tip bbb5183f33df3c6a5f7c587cc41233bc555cd15c; three-dot merge-base 7c1a8967e0d2e12fb8b2ec3ae9783869b4a8be9f (six Rust files). Source only: no tests, builds, application execution, or deployment probes run. One CI snapshot reported 60 successful and 31 skipped check runs; it does not verify runtime behavior. Author latency and mutation-test claims, multi-pod rollout timing, and production performance were not independently reproduced.