Skip to content

Codex session provider #16

Description

@Saber5656

Title

Codex session provider

Summary

Implement the Codex CLI source: date-directory-pruned discovery of rollout
files (with recursive fallback), session_index.jsonl title mapping,
event/task counting, originator excludes, optional archived-sessions scan —
per DESIGN.md §7.4 and the research doc.

Context

~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl (envelope
{timestamp, type, payload}) is unofficial and version-drifting;
docs/research/codex-session-format.md is the normative contract. Date
sharding gives a cheap fast path; automation noise (orchestrators, cron) is
a real problem on the reference machine, handled via
exclude_originators + per-originator visibility in worklog sources (29).

Scope

  • internal/source/codex/provider.go, session.go, index.go, tests,
    synthetic fixtures.

Detailed Requirements

  1. New(cfg config.SourcesCodex) *Provider; ID() = model.SourceCodex.
  2. Discovery fast path: for each calendar day D touching
    [rng.Start − 24h, rng.End + 24h], list
    <sessions_dir>/YYYY/MM/DD/rollout-*.jsonl. Fallback: whenever the fast
    path finds zero files for the whole padded range (and
    <sessions_dir> exists), run a recursive walk capped at depth 4 with an
    mtime prefilter (≥ rng.Start − 48h) — this covers any future layout
    drift, including trees that still look date-sharded but moved the files. Missing sessions dir → codex_dir_missing, zero
    events, nil error. include_archived adds the same scan over
    archived_dir.
  3. Title index: single tolerant pass over cfg.CodexSessionIndex() building
    map[id]thread_name (jsonlutil; missing/unreadable file → warning
    codex_index_unreadable, continue without titles). Cap: 1_000_000 index
    lines (beyond → stop + same warning; degenerate file defense).
  4. Per rollout file (jsonlutil pass):
    • envelope: timestamp, type, payload (via Obj).
    • session_meta.payload: id, cwd, cli_version, originator.
      Missing session_meta (torn file) → per research doc: id/start from
      filename rollout-<ISO-ts>-<uuid>.jsonl (regex
      ^rollout-(\d{4}-\d{2}-\d{2}T\d{2}-\d{2}-\d{2})-([0-9a-f-]{36})\.jsonl$),
      cwd from modal turn_context.payload.cwd, else Project="".
    • counts from event_msg.payload.type, computed over in-range lines
      only
      (midnight-spanning sessions report per-day counts, mirroring
      the Claude Code provider contract): user_message → UserMsgs,
      agent_message → AgentMsgs, task_started → Tasks (count starts;
      ignore task_complete for the count, simpler and stable);
      token_count ignored.
    • span: min/max envelope timestamps of in-range lines; include session
      iff ≥ 1 in-range line.
    • title precedence: index[id] → first in-range user_message text
      (include_prompts only, first line ≤ 120 runes) →
      codex session <first 8 of id>.
  5. exclude_originators: exact string match against
    session_meta.originator → skip the whole file silently (privacy-style
    silent drop, mirroring zsh excludes; per-originator counts surface in
    issue 29 instead).
  6. Dedup live-vs-archived by session id: live wins
    (model.DedupSessions handles final dedup, but avoid double work: track
    seen ids during this Collect).
  7. Event mapping: Kind KindAgentSession, Project = cwd raw,
    Ref = session id, Meta: agent="codex", user_messages,
    agent_messages, tasks, originator, cli_version (omit empties).
  8. Warnings mirror 14: codex_malformed_lines (>20% & ≥10 lines, basename
    only), unreadable file → codex_file_skipped (both are part of the
    canonical DESIGN §13 taxonomy; constants exist from issue 03).

Acceptance Criteria

  • Fixture day-sharded tree: sessions on D−1/D/D+1; range=D returns
    exactly D's sessions plus a midnight-spanning D−1 session (fixture
    line at 00:10 of D).
  • session_meta-less torn fixture: id/start recovered from filename;
    session emitted.
  • Index present → thread_name Title; index missing → warning + fallback
    titles; include_prompts=true path asserted.
  • exclude_originators=["auto-orchestrator"] fixture skipped entirely;
    non-matching originators kept.
  • Archived dedup: same id in live and archived → one Event, live data.
  • Fallback walk: fixture with non-sharded layout still found via mtime
    walk; fixture with a date-sharded-looking tree whose files moved one
    level deeper also found (zero-fast-path-hits trigger asserted).
  • Midnight-spanning fixture: per-day counts differ per the in-range
    rule (two Collect calls, counts hand-computed).
  • Robustness fixtures (oversized line, 30% malformed, unknown types,
    ANSI + fake token in message text) behave per 13/14 patterns (raw
    passthrough asserted).
  • Read-only proof: fixture tree hash identical before/after.

Validation

go test -race -cover ./internal/source/codex/ in PR.

Dependencies

04, 08, 13.

Non-goals

SQLite files in ~/.codex (ADR-003: never opened), ~/.codex/history.jsonl
(v2), token accounting (token_count shape unconfirmed, research doc),
project normalization (18).

Design References

  • docs/research/codex-session-format.md (normative)
  • docs/DESIGN.md §7.4, §13
  • docs/decisions/ADR-003-read-only-sources-and-stateless-runs.md

Source of truth: docs/issues/15-codex-provider.md (PR #1, branch docs/v1-design). If this issue and the repo docs disagree, the docs win. Execution order and dependencies: docs/ISSUE_PLAN.md (this is issue 15 of 33).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions