Skip to content

fix(cost): make paid accounting durable#360

Merged
drewstone merged 5 commits into
mainfrom
fix/run-cost-account
Jul 13, 2026
Merged

fix(cost): make paid accounting durable#360
drewstone merged 5 commits into
mainfrom
fix/run-cost-account

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Problem

Paid campaign work could exceed configured spend, lose in-flight spend across crashes, double-spend across processes, or report incomplete totals as complete.

Solution

  • Make CostLedger.runPaidCall the single admission, reservation, receipt, resume, and summary path for campaign paid calls.
  • Persist a bounded pending reservation before provider dispatch, settle it to the strict provider receipt, and block unresolved work until reconciliation.
  • Serialize file-backed writers with stale-owner recovery, append and fsync events, and fail loudly on every malformed or nonempty partial event.
  • Route built-in proposers, policy/analysis edits, judges, analyst calls, and campaign execution through the same ledger.
  • Compose the reference judge from feat(judges): add reference equivalence scoring #358 and include judge cost in campaign totals without a parallel counter.
  • Resolve Hono 4.12.30 and ws 8.21.0 at the package boundary.

This branch includes the three open #358 commits so the cost change can consume that contract directly. After #358 lands, GitHub will reduce this PR to the cost commit.

Checks

  • pnpm test: 3,010 passed, 2 skipped
  • pnpm typecheck: passed
  • pnpm lint: passed with 4 existing warnings, 0 errors
  • NODE_OPTIONS=--max-old-space-size=8192 pnpm build: passed
  • pnpm verify:package: passed
  • pnpm audit --prod: no known vulnerabilities
  • Focused stale-lock, partial-log, concurrent reservation, crash/resume, bounded spend, paid built-in, and judge-cost regressions: passed

tangletools
tangletools previously approved these changes Jul 13, 2026

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 92c5309d

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-13T15:03:38Z

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Value Audit — better-approach-exists

Verdict better-approach-exists
Concerns 2 (1 medium-concern, 1 low)
Heuristic 0.1s
Duplication 0.0s
Interrogation 257.2s (2 bridge agents)
Total 257.3s

💰 Value — better-approach-exists

A coherent, well-tested redesign that makes campaign cost accounting durable, crash-safe, and ceiling-enforced through a single runPaidCall admission path — but the filesystem durability adapter reinvents an existing in-repo primitive and adds an unnecessary runtime dependency.

  • What it does: Replaces the old scattergun CampaignCostMeter (observe/observeTokens/observeModel) with one method — CostLedger.runPaidCall — that is the single path for every paid LLM/subprocess call. Before provider dispatch it persists a bounded 'pending' reservation; on completion it settles that reservation to a strict provider receipt (tokens + cost); on crash the pending record blocks new paid work until r
  • Goals it achieves: (1) Prevent overspend: reserve the enforced maximum before dispatch, reject if it would breach the ceiling. (2) Survive crashes: persist every reservation and receipt so a killed process leaves a replayable event log, not a silent gap. (3) Prevent double-spend: callId uniqueness + provider idempotency key means a redrive after crash can't charge twice. (4) Never report an incomplete total as compl
  • Assessment: The core design is strong and in the grain of the codebase. CostLedger keeps its place as the substrate primitive (src/cost-ledger.ts, no upward deps); the filesystem adapter lives in the campaign layer (src/campaign/storage.ts), respecting the documented layering rule. The pending→settled state machine, zod-validated event schema, typed PaidCallResult (succeeded/false discriminated union that cal
  • Better / existing approach: The repo already has a dependency-free, tested, cross-process append+fsync+lock primitive in src/campaign/search-ledger-file.ts: appendSearchLedgerLine(path, line) does O_CREAT|O_WRONLY|O_APPEND + full writeAll (handles partial writes) + fsyncSync(fd) + fsyncSync(dir); withSearchLedgerFileLock(ledgerPath, run) does hard-link atomic CAS lock with stale-owner recovery (pid liveness via process.kill(
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 2
  • Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

A single durable paid-call admission path (CostLedger.runPaidCall) replaces the loose observe/observeTokens meter across every LLM-spend surface, with file-backed reservations, ceiling enforcement, crash/resume reconciliation, and consistent helper wiring at all 15+ call sites.

  • Integration: Fully reachable and already exercised. runPaidCall is wired into driver.ts:396, executor.ts:114, judges.ts:318, llm-judge.ts:187, semantic-concept-judge.ts:320, intent-match-judge.ts:193, judge-panel.ts, reviewer.ts:253, analyst/structure-findings.ts:120, every campaign proposer (gepa, skill-opt, llm-policy-edit, analysis-edit, memory, halo, trace-analyst), distillation/run-distillation.ts:180, wi
  • Fit with existing patterns: Matches the codebase grain. It supersedes the prior loose CampaignCostMeter (observe/observeTokens, which had no admission control, durability, or ceiling) with one typed primitive that returns PaidCallResult callers must inspect — aligning with the repo's 'No fallbacks. Fail loud.' doctrine and the external-boundary typed-outcome convention. Durable persistence rides the existing CampaignStora
  • Real-world viability: Robust beyond the happy path. The 41-test cost-ledger suite (passing locally) covers the hard paths the PR claims: concurrent reservation atomicity, crash/resume pending-record blocking, stale filesystem lock recovery (10s stale timer), partial-final-event loud failure, bounded-spend ceiling rejection before dispatch, abort-race late receipt capture, provider-maximum-violation retention, unpriced-
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 1

🔎 Heuristic Signals

🟡 Cruft: magic number added src/cost-ledger.test.ts

  •      inputTokens: 1000,
    

💰 Value Audit

🟠 Filesystem adapter reinvents append+fsync+lock primitive from search-ledger-file.ts and adds proper-lockfile dep [duplication] ``

src/campaign/storage.ts:71-108 implements fsCampaignStorage().append with inline openSync('a')+writeFileSync+fsyncSync(fd)+fsyncSync(dir) and proper-lockfile.lockSync for cross-process serialization. src/campaign/search-ledger-file.ts:22-43 already exports appendSearchLedgerLine (identical durability: O_APPEND + writeAll + fsync file + fsync dir) and withSearchLedgerFileLock (identical stale-owner recovery via hard-link CAS + pid liveness). Both are schema-agnostic and reusable as-is. The new co


What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass What it asks
Heuristic Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication Do added function/class names already exist elsewhere in the repo?
Value Audit What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

value-audit · 20260713T154736Z

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 67b6101d

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-13T16:50:16Z

@drewstone
drewstone merged commit 7fc0bf9 into main Jul 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants