Bound Codex SQLite recovery sweeps

[khaliqgant]

Summary

• bound Codex SQLite recovery to 50 eligible threads per sweep, ordered newest-first, with a persisted resume cursor
• default SQLite-only recovery to a 30-day window with Rust SyncOptions and AI_HIST_CODEX_SQLITE_MAX_AGE_DAYS overrides
• batch projection/catalog lookups, skip complete rollouts, emit CaptureProgress, and run uncatalogued sources before established ones
• document the recovery policy and add cap/resume, cutoff, fingerprint, progress, and source-order coverage

Fixes #508

Validation

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets --all-features -- -D warnings
• RUST_TEST_THREADS=4 cargo test --workspace --all-features
• RUST_TEST_THREADS=4 cargo test -p ai-hist --no-default-features
• cargo build -p ai-hist --no-default-features
• cargo publish --dry-run -p ai-hist --allow-dirty
• native binding contract verification
• env -u XDG_DATA_HOME npm test in sdk-ts (281 passed)
• node --test scripts/*.test.mjs (174 passed)
• node scripts/check-public-api.mjs
• node scripts/smoke-history-packages.mjs
• out-of-tree Rust consumer against this checkout
• node scripts/benchmark-sync.mjs --gate
• rust-oleum quality gate

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Note

Medium Risk
Changes core ingest/sync ordering, catalog ownership during Codex SQLite batches, and fingerprint fast-path behavior; mis-tuned cutoffs or cursor bugs could delay or skip thread recovery, though behavior is heavily tested.

Overview
Codex SQLite recovery is now bounded and resumable instead of walking every incomplete thread in one sync. Each sweep processes at most 50 eligible threads (newest first), persists a codex_sqlite_sweep_cursor_v1 cursor for the backlog, and marks coverage incomplete until the cursor clears or retries finish—so other providers can run between batches and scoped ticks still advance a pending backlog.

Policy knobs: SyncOptions::codex_sqlite_max_age_days (builder + field) and AI_HIST_CODEX_SQLITE_MAX_AGE_DAYS default to a 30-day updated_at window (0 disables). The source fingerprint includes the Codex SQLite cap/cutoff so policy changes cannot be skipped by the fast path.

Supporting behavior: sweep logic moves into providers/codex/sqlite/sweep, batches projection and raw_path catalog lookups via with_raw_path_cache, emits CaptureProgress as codex-sqlite, and runs sources with no catalog rows before established ones in registry order. Docs and changelog describe the recovery policy.

^{Reviewed by Cursor Bugbot for commit 305b68a. Bugbot is set up for automated code reviews on this repo. Configure here.}

---

Agent Relay sessions

• codex session 01a14571 · opened via gh pr create · last active 2026-10-16

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-16T17:21:00.153253Z	5b61acd	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 45 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 30055933-9a98-437f-afe6-d62c617a45e4

📥 Commits

Reviewing files that changed from the base of the PR and between f595d7d and 305b68a.

📒 Files selected for processing: 3

• crates/ai-hist/src/ingest/codex_history.rs
• crates/ai-hist/src/providers/codex/sqlite.rs
• crates/ai-hist/src/providers/codex/sqlite/sweep.rs

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 433f9eb5-4c67-4743-94f4-a3f25fb1fd50

📥 Commits

Reviewing files that changed from the base of the PR and between 5b61acd and f595d7d.

📒 Files selected for processing: 8

• crates/ai-hist/src/ingest.rs
• crates/ai-hist/src/ingest/codex_history.rs
• crates/ai-hist/src/providers/codex/sqlite.rs
• crates/ai-hist/src/providers/codex/sqlite/sweep.rs
• crates/ai-hist/src/providers/codex/sqlite/sweep/cursor.rs
• crates/ai-hist/src/providers/codex/sqlite/tests.rs
• crates/ai-hist/src/providers/mod.rs
• crates/ai-hist/src/providers/sweep.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Codex SQLite recovery now processes eligible threads in bounded, newest-first batches and resumes from a persisted cursor. A configurable age cutoff, sweep-scoped catalog lookups, updated source ordering, and thread progress reporting are also added.

Changes: Codex SQLite recovery

Layer / File(s)	Summary
Recovery policy and API crates/ai-hist/src/session_store/options.rs, crates/ai-hist/src/session_store.rs, crates/ai-hist/public-api.txt, CHANGELOG.md, docs/session-catalog.md, docs/sourcing-sdk.md	SyncOptions adds a Codex SQLite age cutoff. The default is 30 days; zero disables it. SessionStore::sync applies the selected policy to the sweep. The changelog and documentation describe the recovery limits and options.
Bounded selection and capture crates/ai-hist/src/providers/codex/sqlite/*, crates/ai-hist/tests/fixture_corpus.rs	The sweep selects up to 50 eligible threads newest-first and stores a cursor when more remain. It captures selected threads, skips unchanged stamps unless repairs are requested, and reports incomplete work. Tests cover batch resumption, cutoff behavior, capture progress, and retry handling.
Sweep coordination and reporting crates/ai-hist/src/catalog_owner.rs, crates/ai-hist/src/ingest.rs, crates/ai-hist/src/ingest/*, crates/ai-hist/src/providers/sweep.rs, crates/ai-hist/src/providers/mod.rs	Sources with no session rows run before sources with existing rows. Catalog lookups use sweep-scoped cached paths. Codex SQLite reports thread progress, its policy contributes to source fingerprints, and pending backlog marks coverage unread.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant SessionStore
  participant SweepPhases
  participant CodexSQLiteSweep
  participant CodexSQLiteDB
  participant SessionCatalog
  SessionStore->>SweepPhases: run controlled sync
  SweepPhases->>CodexSQLiteSweep: dispatch Codex SQLite source
  CodexSQLiteSweep->>CodexSQLiteDB: select eligible threads by cutoff and cursor
  CodexSQLiteDB-->>CodexSQLiteSweep: return bounded thread batch
  CodexSQLiteSweep->>SessionCatalog: capture and normalize selected threads
  CodexSQLiteSweep->>CodexSQLiteDB: persist stamps and next cursor

Loading

Suggested reviewers: willwashburn

Merge Risk: 🔵 Low · up to f595d

Recovery can proceed and resume across ticks, but a failed thread may miss recovery if it ages beyond the cutoff before the backlog drains. This is a bounded risk to accept or address before merging.

Pre-merge checks: 4 · 1

Failed checks: 1 inconclusive

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 40.28% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 72 functions across 14 files. (1 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly and concisely identifies the main change: bounding Codex SQLite recovery sweeps.
Description check		The description directly explains the recovery limits, resume cursor, age cutoff, progress reporting, source ordering, documentation, tests, and validation for this changeset.
Linked Issues check		Issue #508 is active and defines the coding requirements. The reviewed changes select Codex SQLite threads newest-first by updated_at_ms and ID, cap each sweep at 50, and persist a cursor. They appl…
Out of Scope Changes check		The changed source, tests, options, fingerprints, catalog cache, source ordering, and documentation support the recovery requirements in #508. The changes do not implement the separately excluded desk…

Full details: Linked Issues check

Explanation

Issue #508 is active and defines the coding requirements. The reviewed changes select Codex SQLite threads newest-first by updated_at_ms and ID, cap each sweep at 50, and persist a cursor. They apply the 30-day default cutoff with SyncOptions::codex_sqlite_max_age_days and AI_HIST_CODEX_SQLITE_MAX_AGE_DAYS overrides. They skip complete rollouts, batch projection and catalog lookups, report codex-sqlite progress, and run uncatalogued sources first. The added tests cover cap, ordering, cursor resumption, cutoff, fingerprint changes, progress, scoped backlog continuation, and source ordering. The implementation preserves read-only SQLite access and adds the required changelog entry.

Full details: Out of Scope Changes check

Explanation

The changed source, tests, options, fingerprints, catalog cache, source ordering, and documentation support the recovery requirements in #508. The changes do not implement the separately excluded desktop delivery-phase lock work or lazy parser-generation repair. No unrelated credentials, secrets, caches, network access, or remote backfill changes appear in the reviewed scope.

Full details: Docstring Coverage

Explanation

Docstring coverage is 40.28% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 72 functions across 14 files. (1 skipped: 1 too large.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{A rabbit checks the newest thread,
Then saves the cursor where it led.
Fifty hop through the SQLite night,
The next batch waits, in order right.
“Done!” says the rabbit, paws held high.}

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Confidence Score: 5/5

[High impact] The PR appears safe to merge; no new blocking issue was found.

Summary

Codex SQLite recovery now reads up to 50 eligible threads per sweep, newest-first, and saves its position between sweeps.

• Codex SQLite recovery works through 50-thread newest-first batches.
• Sources without catalog rows get their turn before established sources.
• Codex capture reuses a batch of catalog ownership lookups.

No new actionable issues were found.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Start Codex recovery] --> B{Supported SQLite pair?}
  B -- No --> C[Clear saved cursor]
  B -- Yes --> D[Resume newest-first selection]
  D --> E[Read up to 50 eligible threads]
  E --> F{More threads or retries?}
  F -- Yes --> G[Save cursor and keep coverage incomplete]
  F -- No --> H{Source changed during traversal?}
  H -- Yes --> I[Require another pass from newest]
  H -- No --> J[Allow completed coverage]

Loading

_{Reviews (4) · Last reviewed commit: "Reset unavailable Codex SQLite backlog" · Reviewed by Greptile}

[khaliqgant]

Addressed CodeRabbit’s outside-diff scoped-watch finding in f595d7d: a persisted Codex SQLite cursor now makes every scoped tick continue the next bounded page, even when the event only names history.jsonl. Added an integration regression proving the pending SQLite thread is captured on that scoped tick.

[capy-ai]

Capy found no issues.