Repository navigation
Index native history reads and incrementally persist JSONL checkpoints - #441
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checkpoints currently reread and rewrite the complete transcript, and chat consumers must decode it again for small lookups. This adds opt-in incremental saves and a bounded, disposable index of row offsets, fingerprints, and small projections. Normal appends advance both the primary and previous-complete backup by their new tails; unchanged transcripts remain untouched. Edits and truncation retain atomic replacement.
A durable append intent makes updated readers recover the complete backup after an interrupted batch, including interruption between valid JSON lines. External replacement, in-place edits, and truncation invalidate indexes. Event association fingerprints avoid retaining full prompts. No event payloads or transcript messages are removed.
Validation in the owned spark-2 DTU: 406 session tests passed, Ruff check/format passed, and the built wheel passed an installed-package read/write smoke. Tests cover partial and valid-line crash boundaries, CLI changes, late invalid input, backup recovery, no-op saves, bounded caching, and association equivalence.
The API is opt-in. Writers retain the session ownership-lock requirement. Consumers enabling it must use intent-aware Foundation readers, including any CLI installation sharing those sessions; older writers remain supported. Full-list callers still serialize/hash their supplied prefix, so this does not remove every history-proportional CPU cost. CI-to-transcript reconstruction is a separate contract change.