Read DSH session format v4 written by dsh 0.2.0-rc.2

docs/providers/dsh.md

@@ -23,10 +23,11 @@ sessions/--<slugified-cwd>--/<session-id>/
   session.v1.jsonl[.zstd]    format v1
   session.v2.jsonl[.zstd]    format v2
   session.v3.jsonl[.zstd]    format v3
+  session.v4.jsonl[.zstd]    format v4
 ```
 
 Separate sessions are all counted, including sessions with only legacy v0/v1
-logs alongside sessions using v2/v3. Generation selection applies only within
+logs alongside sessions using v2/v3/v4. Generation selection applies only within
 one session directory; a newer-format session never supersedes another session.
 
 Both compression variants are read. Migrated generations are immutable and may
@@ -44,7 +45,7 @@ older snapshot. The log is append-only JSONL whose first line is the session hea
 
 Every later line is one event `{ type, seq, time, data }`. Formats v0/v1 keep
 stream chunks as top-level events (with delta runs packed into storage rows).
-Formats v2/v3 embed the compact stream in each `assistant/message` or
+Formats v2/v3/v4 embed the compact stream in each `assistant/message` or
 `assistant/attempt`. The parser reads:
 
 | Event | Used for |
@@ -70,22 +71,22 @@ None at the provider level; the log file is the cached source path and the norma
 
 ## Quirks
 
-- **DSH is a developer preview.** The parser explicitly supports released formats v0-v3 and checks both the canonical generation filename and header. A future version is skipped with a notice; **a version bump upstream still requires a semantic reader update, not just relaxing the check.**
+- **DSH is a developer preview.** The parser explicitly supports released formats v0-v4 and checks both the canonical generation filename and header. A future version is skipped with a notice; **a version bump upstream still requires a semantic reader update, not just relaxing the check.** v4 (written by `@deepseek-ai/dsh` 0.2.0-rc.2) was admitted after verifying against DSH's official `sessionFormatCatalog` (`recovery: 'strict'`, `validation: 'current'`, `@deepseek-ai/dsh-session-format-catalog@0.2.0-rc.2`) that the parser's whole consumption surface is unchanged: dense `seq`, usage at `assistant/message`'s `data.usage` (which gains an informational `totalTokens` sum) or its embedded stream, the tagged end-seed inheritance rule, and `llm/retry-started` attempt slots. What v4 changes — tool-result messages lifted to role `tool`, unknown content tags namespaced `plugin:<name>`, surface `surfaceOp` append/replace metadata, new header fields (`agentPreset`, `origin`), and untagged `session/end-seed` markers that unseeded sessions now also write — is ignored by the reader or already handled by it.
 - **The JSONL backend only.** DSH also ships an opt-in SQLite persistence backend (`@deepseek-ai/dsh-session-persistence-sqlite`); it is not the default and is not read.
 - **DSH records tokens, never dollars.** `usage` is `{ inputTokens, outputTokens, cacheReadTokens?, cacheWriteTokens?, reasoningTokens? }` with no cost field, so every call is priced from the shared tables. The buckets are disjoint on input; `reasoningTokens` is informational detail already included in `outputTokens`, as documented in the [DSH TokenUsage contract](https://github.com/deepseek-ai/deepseek-harness/blob/c291e7961a515f6d7af9304e7fd1d257929aef26/docs/subsystems/llm-streaming.md#tokenusage). CodeBurn preserves raw output and applies the shared inclusive-output rule to pricing, cached reads, and display. pi-ai routes do not persist separate reasoning detail. Complete valid usage keeps `costIsEstimated` false; incomplete or inconsistent usage is reported with a notice and marked estimated. An attempt without usage is omitted with a notice rather than represented as an exact zero.
 - **`assistant/message` usage wins over the `assistant/chunk` sample** for the same `(turn, step)` — the two are adjacent reports of one API call, not two calls. A late chunk never overwrites a final report, so the two are never summed.
 - **The model comes from the message, not the request.** `data.message.source.model` is what actually served the step; the current `request/context` model is the fallback, followed by `request/header`. A changed header model clears the previous context fallback. The `provider` field there (`deepseek-official`) is the upstream LLM route, not the tool — the codeburn provider name is always `dsh`.
-- **A forked session's log replays its parent's events.** v0/v1 use header `seedLength` only when `parentSession` is present, preserving the legacy non-fork behavior; v2/v3 use the last `session/end-seed` marker carrying `{ inherited: true }`. CodeBurn excludes the inherited prefix to avoid billing the parent's calls twice.
+- **A forked session's log replays its parent's events.** v0/v1 use header `seedLength` only when `parentSession` is present, preserving the legacy non-fork behavior; v2/v3/v4 use the last `session/end-seed` marker carrying `{ inherited: true }` (v4 unseeded sessions also write untagged markers, which define no cut). CodeBurn excludes the inherited prefix to avoid billing the parent's calls twice.
 - **`user/message` also carries agent-injected context** (runtime snapshots, skill bodies, file-change notices) under `source.kind: 'plugin'`. Only `kind: 'user'` messages become the preview.
 - **Delta chunks are packed.** Runs of streamed deltas are stored as `text-chunks` / `reasoning-chunks` / `tool-call-chunks` storage rows rather than one event per line. They carry no usage and no tool identity the `tool/call` event lacks, so they are ignored — as is any event type the parser does not know.
 - **A torn final zstd frame is ignored.** A crashed writer leaves an incomplete trailing frame; the complete frames before it parse normally. A structurally corrupt file is skipped whole with a notice rather than throwing.
 
 ## When fixing a bug here
 
-`v3-retry.jsonl` also covers a failed `assistant/attempt`, scheduled retry, and successful settlement with exact `totalTokens`. The same official strict restore and reducer yield input 110, output 24, cache read 33, and cache write 7 (174 total tokens).
+`v3-retry.jsonl` also covers a failed `assistant/attempt`, scheduled retry, and successful settlement with exact `totalTokens`. The same official strict restore and reducer yield input 110, output 24, cache read 33, and cache write 7 (174 total tokens); `v4-retry.jsonl` is the v4-stamped equivalent.
 
 1. Reproduce with a minimal session dir: `sessions/--proj--/<id>/session.jsonl` (uncompressed is easiest to hand-write).
 2. `tests/fixtures/dsh/bash-tool-turn.jsonl` is the upstream `examples/acp-agent/tests/snapshots/bash-tool-turn/session.jsonl` snapshot with its template placeholders filled in — refresh it from the DSH repo when the format moves.
 3. Run `tests/providers/dsh.test.ts`.
 4. `.zstd` fixtures must compress **each batch separately**; one `zstdCompressSync` over the whole file is a single-frame layout DSH never writes.
-5. `tests/fixtures/dsh/v0.jsonl` through `v3.jsonl` are minimal synthetic, sanitized format fixtures. Each was restored with DSH's official `sessionFormatCatalog` (`recovery: 'strict', validation: 'current'`) and folded through `tokenUsageProjectionDefinition` at DSH commit `c291e7961a515f6d7af9304e7fd1d257929aef26`. All four yield uncached input 100, full output 20, cache read 30, and cache write 5; the provider tests assert those buckets, including inclusive output and informational reasoning detail. v0/v1 cite their top-level chunk via `sourceEventSeqs`; v2/v3 carry the embedded stream.
+5. `tests/fixtures/dsh/v0.jsonl` through `v3.jsonl` are minimal synthetic, sanitized format fixtures. Each was restored with DSH's official `sessionFormatCatalog` (`recovery: 'strict', validation: 'current'`) and folded through `tokenUsageProjectionDefinition` at DSH commit `c291e7961a515f6d7af9304e7fd1d257929aef26`. All four yield uncached input 100, full output 20, cache read 30, and cache write 5; the provider tests assert those buckets, including inclusive output and informational reasoning detail. v0/v1 cite their top-level chunk via `sourceEventSeqs`; v2/v3 carry the embedded stream. `v4.jsonl` and `v4-retry.jsonl` are the v4 companions, restored with `@deepseek-ai/dsh-session-format-catalog@0.2.0-rc.2` (same strict/current policy) and carrying the same buckets plus the informational `totalTokens` field every v4 usage records.

src/daily-cache.ts

@@ -256,7 +256,10 @@ import type { DateRange, ProjectSummary } from './types.js'
 // miss sessions there and must be re-derived after the default path is fixed.
 // v51: honor redirected Copilot and Cursor editor data roots. Backfill settled
 // days that previously missed usage stored under APPDATA or XDG_CONFIG_HOME.
-export const DAILY_CACHE_VERSION = 51
+// v52: DSH session format v4 (dsh 0.2.0-rc.2) is read; days finalized while
+// those sessions were skipped re-derive. Calls only rise, so no
+// PENDING_REDERIVE_PROVIDER_VERSIONS entry is needed.
+export const DAILY_CACHE_VERSION = 52
 const MIN_SUPPORTED_VERSION = 28
 
 /// Providers whose per-day CALL COUNT means something different at

src/providers/dsh.ts

@@ -34,7 +34,8 @@ const ZSTD_MAGIC = 0xfd2fb528
 // the caller skips the WHOLE file rather than counting the frames it got to.
 const MAX_FRAME_DECODED_BYTES = 64 * 1024 * 1024
 
-const SUPPORTED_SESSION_FORMAT_VERSIONS = new Set([0, 1, 2, 3])
+// v4 (dsh 0.2.0-rc.2) only lifts tool-result roles and adds metadata this parser ignores.
+const SUPPORTED_SESSION_FORMAT_VERSIONS = new Set([0, 1, 2, 3, 4])
 const SESSION_LOG_NAME = /^session(?:\.v(\d+))?\.jsonl(?:\.zstd)?$/u
 
 const MIN_REASONABLE_TIMESTAMP_MS = 1_000_000_000_000

src/session-cache.ts

@@ -465,9 +465,9 @@ export const PROVIDER_PARSE_VERSIONS: Record<string, string> = {
   grok: 'authoritative-usage-v4',
   // Estimated from message text: Grok Bot's local mirror records no tokens.
   grokbot: 'estimated-usage-v1',
-  // v0-v3 generations, embedded attempt streams, retry accounting, and the
+  // v0-v4 generations, embedded attempt streams, retry accounting, and the
   // version-specific inherited-prefix rules all change cached DSH calls.
-  dsh: 'session-formats-v0-v3-attempts-v5',
+  dsh: 'session-formats-v0-v4-attempts-v6',
   // cost-provenance-v3: preserve Hermes included/estimated/actual status and
   // rebuild the provider section alongside the v3 lifetime ledger. The parse
   // bump is required with the ledger bump: seeding a new ledger from a section

tests/fixtures/dsh/v4-retry.jsonl

@@ -0,0 +1,11 @@
+{"type":"session","version":4,"id":"fixture-retry","createdAt":1786707340000,"cwd":"/fixture/project","delegationDepth":0,"isSeeded":false}
+{"type":"turn/start","data":{"turn":1},"seq":0,"time":1786707340000}
+{"type":"step/start","data":{"turn":1,"step":1},"seq":1,"time":1786707340001}
+{"type":"request/header","data":{"header":{"config":{"provider":"deepseek-official","model":"deepseek-v4-flash"}},"reason":"initial"},"seq":2,"time":1786707340002}
+{"type":"request/context","data":{"provider":"deepseek-official","model":"deepseek-v4-flash"},"seq":3,"time":1786707340003}
+{"type":"assistant/attempt","data":{"turn":1,"step":1,"stream":[{"type":"chunk","time":1786707340000,"chunk":{"type":"usage","usage":{"inputTokens":10,"outputTokens":4,"cacheReadTokens":3,"cacheWriteTokens":2,"reasoningTokens":2,"totalTokens":19}}}]},"seq":4,"time":1786707340004}
+{"type":"llm/retry","time":1786707340005,"data":{"retryId":"fixture-retry-1","turn":1,"step":1,"provider":"deepseek-official","mode":"normal","policyKey":"default","retry":1,"maxRetries":1,"delayMs":0,"failure":{"code":"SERVER","message":"Fixture failure"}},"seq":5}
+{"type":"llm/retry-started","data":{"turn":1,"step":1,"retryId":"fixture-retry-1","retry":1},"seq":6,"time":1786707340006}
+{"type":"assistant/message","surfaceOp":"append","data":{"turn":1,"step":1,"message":{"id":"fixture-success","role":"assistant","content":[{"type":"text","text":"Done"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash"}},"usage":{"inputTokens":100,"outputTokens":20,"cacheReadTokens":30,"cacheWriteTokens":5,"reasoningTokens":8,"totalTokens":155},"stream":[{"type":"chunk","time":1786707340000,"chunk":{"type":"usage","usage":{"inputTokens":100,"outputTokens":20,"cacheReadTokens":30,"cacheWriteTokens":5,"reasoningTokens":8,"totalTokens":155}}}]},"seq":7,"time":1786707340007}
+{"type":"step/end","data":{"turn":1,"step":1},"seq":8,"time":1786707340008}
+{"type":"turn/end","data":{"turn":1,"reason":{"kind":"completed"}},"seq":9,"time":1786707340009}

tests/fixtures/dsh/v4.jsonl

@@ -0,0 +1,6 @@
+{"type":"session","version":4,"id":"fixture-session","createdAt":1786707340000,"cwd":"/fixture/project","delegationDepth":0,"isSeeded":false}
+{"type":"turn/start","data":{"turn":1},"seq":0,"time":1786707340000}
+{"type":"step/start","data":{"turn":1,"step":1},"seq":1,"time":1786707340000}
+{"type":"assistant/message","surfaceOp":"append","data":{"turn":1,"step":1,"message":{"id":"fixture-message","role":"assistant","content":[{"type":"text","text":"Done"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash"}},"usage":{"inputTokens":100,"outputTokens":20,"cacheReadTokens":30,"cacheWriteTokens":5,"reasoningTokens":8,"totalTokens":155},"stream":[{"type":"chunk","time":1786707340000,"chunk":{"type":"usage","usage":{"inputTokens":100,"outputTokens":20,"cacheReadTokens":30,"cacheWriteTokens":5,"reasoningTokens":8,"totalTokens":155}}}]},"seq":2,"time":1786707340000}
+{"type":"step/end","data":{"turn":1,"step":1},"seq":3,"time":1786707340000}
+{"type":"turn/end","data":{"turn":1,"reason":{"kind":"completed"}},"seq":4,"time":1786707340000}

tests/providers/dsh.test.ts

@@ -119,7 +119,7 @@ async function writePlainSession(projectDirName: string, sessionDirName: string,
 }
 
 async function writeVersionedSession(
-  version: 1 | 2 | 3,
+  version: 1 | 2 | 3 | 4,
   projectDirName: string,
   sessionDirName: string,
   lines: string[],
@@ -147,7 +147,7 @@ async function writeVersionedSession(
 // packages/session/session-format-v1-to-v2/src/codec.ts (v2), and
 // packages/session/session-format-v2-to-v3/src/codec.ts (v3).
 // Embedded usage and retry settlements follow packages/llm/token-meter/src/usage-projection.ts.
-function versionedHeader(version: 1 | 2 | 3, opts: { id?: string; cwd?: string; isSeeded?: boolean } = {}) {
+function versionedHeader(version: 1 | 2 | 3 | 4, opts: { id?: string; cwd?: string; isSeeded?: boolean } = {}) {
   return JSON.stringify({
     type: 'session',
     version,
@@ -244,7 +244,7 @@ describe('dsh provider - session discovery', () => {
     await writeFile(join(corrupt, 'session.jsonl'), sessionHeader({ id: 'session-corrupt' }) + '\n')
     await writeFile(join(corrupt, 'session.v3.jsonl'), '{broken\n')
     await writeFile(join(unknown, 'session.jsonl'), sessionHeader({ id: 'session-unknown' }) + '\n')
-    await writeFile(join(unknown, 'session.v4.jsonl'), JSON.stringify({ type: 'session', version: 4 }) + '\n')
+    await writeFile(join(unknown, 'session.v5.jsonl'), JSON.stringify({ type: 'session', version: 5 }) + '\n')
 
     expect(await createDshProvider(tmpDir).discoverSessions()).toEqual([])
   })
@@ -377,17 +377,17 @@ describe('dsh provider - session discovery', () => {
 })
 
 describe('dsh provider - parsing', () => {
-  it('counts failed and successful attempts from the official-validated retry fixture', async () => {
-    const content = await readFile(join(import.meta.dirname, '../fixtures/dsh/v3-retry.jsonl'), 'utf8')
-    const path = await writeVersionedSession(3, '--fixture--', 'fixture-retry', content.trim().split('\n'))
+  it.each([3, 4] as const)('counts failed and successful attempts from the official-validated v%s retry fixture', async (version) => {
+    const content = await readFile(join(import.meta.dirname, `../fixtures/dsh/v${version}-retry.jsonl`), 'utf8')
+    const path = await writeVersionedSession(version, '--fixture--', 'fixture-retry', content.trim().split('\n'))
     const calls = await parseAll(createDshProvider(tmpDir), path)
     expect(calls.map(call => [call.inputTokens, call.outputTokens, call.reasoningTokens])).toEqual([[10, 4, 2], [100, 20, 8]])
     expect(calls.every(call => call.model === 'deepseek-v4-flash')).toBe(true)
     expect(calls.reduce((sum, call) => sum + call.inputTokens + billableOutputTokens('dsh', call.outputTokens, call.reasoningTokens)
       + call.cacheReadInputTokens + call.cacheCreationInputTokens, 0)).toBe(174)
   })
 
-  it.each(([0, 1, 2, 3] as const).flatMap(version => [false, true].map(retry => ({ version, retry }))))(
+  it.each(([0, 1, 2, 3, 4] as const).flatMap(version => [false, true].map(retry => ({ version, retry }))))(
     'replaces non-adjacent settlements per step in v$version (retry=$retry)', async ({ version, retry }) => {
     const sample = (step: number, input: number) => version <= 1
       ? chunkUsage(1, step, { inputTokens: input, outputTokens: 10 }, 1786707340000)
@@ -416,7 +416,7 @@ describe('dsh provider - parsing', () => {
     expect((await parseAll(createDshProvider(tmpDir), path)).map(call => call.inputTokens)).toEqual([100, 100, 100, 100])
   })
 
-  it.each([0, 1, 2, 3] as const)('matches official DSH totals for the sanitized v%s fixture', async version => {
+  it.each([0, 1, 2, 3, 4] as const)('matches official DSH totals for the sanitized v%s fixture', async version => {
     const content = await readFile(join(import.meta.dirname, `../fixtures/dsh/v${version}.jsonl`), 'utf8')
     const lines = content.trim().split('\n')
     const path = version === 0
@@ -471,7 +471,7 @@ describe('dsh provider - parsing', () => {
     expect(calls[0]).toMatchObject({ inputTokens: 10, outputTokens: 'outputTokens' in usage ? usage.outputTokens : 0, costIsEstimated: true })
   })
 
-  it.each([1, 2, 3] as const)('uses only the usage layout belonging to format v%s', async (version) => {
+  it.each([1, 2, 3, 4] as const)('uses only the usage layout belonging to format v%s', async (version) => {
     // v1-to-v2 consumes top-level chunks and introduces settlement streams.
     // A row from the other physical layout must not become an extra call.
     const filePath = await writeVersionedSession(version, '--home-u-proj--', 'session-layout', [
@@ -498,7 +498,7 @@ describe('dsh provider - parsing', () => {
     expect(calls[0]).toMatchObject({ model: 'deepseek-v4-pro', inputTokens: 111, outputTokens: 22 })
   })
 
-  it.each([2, 3] as const)('reads v%s embedded streams and prefers message usage', async (version) => {
+  it.each([2, 3, 4] as const)('reads v%s embedded streams and prefers message usage', async (version) => {
     const filePath = await writeVersionedSession(version, '--home-u-proj--', `session-v${version}`, [
       versionedHeader(version, { id: `session-v${version}` }),
       JSON.stringify({
@@ -566,7 +566,7 @@ describe('dsh provider - parsing', () => {
     ])
   })
 
-  it.each([2, 3] as const)('excludes the inherited prefix using the last tagged v%s end-seed marker', async (version) => {
+  it.each([2, 3, 4] as const)('excludes the inherited prefix using the last tagged v%s end-seed marker', async (version) => {
     const filePath = await writeVersionedSession(version, '--home-u-proj--', `session-v${version}-fork`, [
       versionedHeader(version, { id: `session-v${version}-fork`, isSeeded: true }),
       JSON.stringify({