Skip to content

batch_operator: crank chkdispute so a resolved dispute can unpause the epoch - #582

Merged
heifner merged 1 commit into
masterfrom
fix/wire-batchop-crank-chkdispute
Aug 21, 2026
Merged

batch_operator: crank chkdispute so a resolved dispute can unpause the epoch#582
heifner merged 1 commit into
masterfrom
fix/wire-batchop-crank-chkdispute

Conversation

@heifner

@heifner heifner commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

The bug

sysio.chalg::opendispute sends sysio.epoch::pause. chkdispute is the ONLY action that tallies the Tier-1 votes, dispatches the winning envelope via resolvedisp, and lifts that pause. Nothing drove it.

plugins/  chkdispute  → 0 files       chkcons  → 2 files (batch_operator_plugin.cpp:351)
          votedispute → 0 files       chkuwchal → 0 files

No plugin in the tree references chalg at all, and batch_operator_plugin knows exactly two action names:

constexpr auto action_deliver = "deliver";
constexpr auto action_chkcons = "chkcons";

The comment at sysio.chalg.cpp:346"batch operators call this on their ~15s cadence" — asserted a cadence that did not exist.

Why the sibling survives and this one cannot

chkuwchal's header states the contract outright:

"Nobody has to run it on a cadence: sysio.uwrit::chklocks (inlined from every sysio.epoch::advance) pokes it … which works precisely because the chain is NOT paused during an underwriter challenge."

An envelope dispute is the inverse: it pauses advance, so the advance-driven poke is structurally unavailable. chkuwchal also has a LAPSED deadline path; chkdispute has none — if (!resolved) return;, "No plurality / tie-break — an unresolved tally just keeps waiting for more votes."

So the sibling has two independent liveness mechanisms and chkdispute had zero. Tier-1 reaches quorum → dispute stays OPENopen_disputes never decrements → epoch paused until someone hand-cranks it.

The change

Crank it from the epoch tick that already runs (--batch-epoch-poll-ms, 15s default), scanning sysio.chalg::disputes for OPEN rows.

Gated on is_active, not is_elected — deliberate, and the one design call worth reviewing. chkcons is elected-only to avoid burning trx slots every tick from every operator. Here the elected operator may be exactly the one offline or the one that delivered the non-canonical envelope — often why the dispute exists. Disputes are rare, and chkdispute asserts the dispute is still OPEN, so a redundant push from a second operator is a cheap no-op logged at dlog.

Full-table scan rather than the byepoch index: the table retains RESOLVED rows as the audit trail, but disputes are rare and poll_own_status already scans a comparably-sized table each tick. The comment names the index as the escape hatch if that ever changes.

Accepted design decision

Pausing until Tier-1 votes is correct, and there is deliberately no timeout escape. This PR moves the liveness dependency from "operators must deliver" to "a majority of Tier-1 must vote", and that is the intended trade — the alternatives (plurality, or abandon-and-resume) either pick a winner on weak evidence or hand back the original deadlock. Recorded here so it is an explicit decision rather than an implicit one.

Contract file

contracts/sysio.chalg/src/sysio.chalg.cpp is comment-only — it corrects the false cadence claim. No .wasm/.abi appears in this diff, and a comment cannot change the compiled artifact, so contracts_unit_test was not re-run for it. Flag it if you want that gate run anyway.

Verification

  • E2E flow in progress.

…e epoch

`sysio.chalg::opendispute` sends `sysio.epoch::pause`, and `chkdispute` is the
only action that tallies the Tier-1 votes, dispatches the winning envelope and
lifts that pause. Nothing drove it: `chkdispute` appears in no plugin in the
tree, and `batch_operator_plugin` knows exactly two action names, `deliver` and
`chkcons`.

The sibling case shows why that is not survivable here. `chkuwchal` needs no
cadence because `sysio.uwrit::chklocks` pokes it from every
`sysio.epoch::advance` — which works precisely because an underwriter challenge
does NOT pause the chain. An envelope dispute halts `advance` itself, so no
inline poke can reach `chkdispute`, and it has no deadline-lapse path either.
The result: Tier-1 reaches quorum, the dispute stays OPEN, `open_disputes` never
decrements, and epoch advancement is paused until someone hand-cranks it.

Crank it from the epoch tick that already runs (`--batch-epoch-poll-ms`, 15s),
scanning `sysio.chalg::disputes` for OPEN rows. Gated on `is_active` rather than
`is_elected`: the elected operator may be the one offline or the one that
delivered the non-canonical envelope — often the reason the dispute exists.
Disputes are rare, and `chkdispute` asserts the dispute is still OPEN, so a
redundant push from a second operator is a cheap no-op.

Also corrects the `chkdispute` comment, which asserted this cadence already
existed. Comment-only: no contract artifact changes, and no .wasm/.abi is in
this diff.

Change-Id: I2495d8d4dadae912336e06d1698168ab34f20366
@heifner

heifner commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

e2e gate: GREEN — and it proves the crank exists in production

Run 32518249867 · Release · 19:23:48 → 20:13:11 UTC (49 min) · conclusion=success

ref pinned
wire-sysio ba321d5bd14f9dfffa2f6e884f4e08f81ca93ad7 (#582)
wire-tools-ts 68b71962304b2707e7960bdb36d6e2a0732cb435 (Wire-Network/wire-tools-ts#81)
everything else manifest default
FLOW_INCLUDE set — running ONLY: flow-batch-operator-slashing flow-batch-operator-termination

## E2E flow results
- ✅ batch-operator-slashing (442s)
- ✅ batch-operator-termination (1011s)
All E2E flows passed.

Why this is the proof, not just a green tick

flow-batch-operator-slashing ran with the harness's own crankChkdispute deleted (#81). Its dispute step now only polls for resolution — nothing in the flow tallies the votes. It resolved anyway, so the only thing that could have called sysio.chalg::chkdispute is batch_operator_plugin::crank_open_disputes from #582.

It also settles the open assumption in #81: the bootstrapped batch operators do keep their daemons and stay opreg-ACTIVE across the paused epoch, and their tick reaches the dispute even though schbatchgps has moved them outside the serving group. That is exactly why the crank is gated on is_active rather than is_elected — had it followed chkcons and gated on is_elected, it would not have fired here.

Durations line up with the known-good baseline

Against run 32184443818 (the last green 5-flow gate): slashing 444s → 442s, termination 1012s → 1011s. No added latency, and batch-operator-termination — the flow #573 broke — is unaffected by touching the same plugin file.

Also verified

pnpm test on the #81 worktree: 1956/1956, 225/225 suites, exit 0 (483s, 8 projects). This supersedes the "1954/1956 with 2 pre-existing failures" noted in the #81 description — the two ClusterBuildDefaultsExternal failures are an intermittent /tmp bind-registry lock race, shown pre-existing (identical with the change stashed) and non-deterministic (absent on a clean re-run). Unrelated to either PR; worth its own ticket if it recurs.

Merge order

#582 first, then Wire-Network/wire-tools-ts#81. Reversed, the flow has no crank and times out.

@heifner
heifner merged commit 7b3e6ef into master Aug 21, 2026
13 checks passed
@heifner
heifner deleted the fix/wire-batchop-crank-chkdispute branch August 21, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants