Skip to content

fix(copilot): include copilot legacy chat sessions - #1232

Open
JaylyDev wants to merge 6 commits into
getagentseal:mainfrom
JaylyDev:jayly/copilot-legacy-json-sessions
Open

fix(copilot): include copilot legacy chat sessions#1232
JaylyDev wants to merge 6 commits into
getagentseal:mainfrom
JaylyDev:jayly/copilot-legacy-json-sessions

Conversation

@JaylyDev

@JaylyDev JaylyDev commented Sep 2, 2026

Copy link
Copy Markdown

Summary

Adding support for VS Code's GitHub Copilot chat sessions prior to February 2026 to copilot's parser.

How it works:

The parser scan for chatSessions/*.json and parses the legacy chat sessions if there is a requests array in root JSON object. The schema of those JSON files and its definitions like IToolCallRound and ThinkingData can be found in https://github.com/microsoft/vscode-copilot-chat.

This old format reads all of input, output, reasoning messages and tool calling outputs to determine number of tokens used by a model, which is also stored in the request object. It reads exact token counts (promptTokens, completionTokens, usage) when available, otherwise it fallbacks to character-based estimation.

This parser also maps the model used in auto model selection to the actual model used.

Testing

  • I have tested this locally against real data (not just unit tests)
  • npm test passes
  • npm run build succeeds
  • I installed the tool and generated real sessions by using it
  • npm run dev -- today shows correct costs and session counts for this provider
  • npm run dev -- models --provider copilot shows correct model names and pricing
  • Screenshot or terminal output attached below proving it works with real data

Terminal output of running npm run dev -- models --provider copilot -p lifetime

Before:

image

After:

image

@ozymandiashh ozymandiashh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking on the legacy format. I can't merge this as it stands: the tests pass, but the defects are all inside untested branches, and several change numbers for users on the current format too. Everything below was measured on the branch merged with current main (clean merge, full suite green apart from the known #1216 flake).

  1. Reasoning tokens are billed as web-search requests. src/providers/copilot.ts:702-709 passes reasoningTokens as the sixth argument to calculateCost, which is webSearchRequests. A session with an 8,000-char thinking block on claude-sonnet-4-6 comes out at $20.04 where the true cost is about $0.03, with costIsEstimated: false. Every other call site in the file passes 0 there.

  2. The rewrite regresses the current .jsonl path. The old parser read metadata.promptTokens and metadata.outputTokens || completionTokens because the two fields do not live in the same object (see chatSessionSampleRequest in the tests). The new code requires both in one object, so the repo's own real-data shape (metadata.promptTokens: 32543 + root completionTokens: 490) falls through to char estimation: 32,543 input tokens become 2. Rows with no token fields, previously skipped, are now emitted as estimates. That's not a behavior-preserving refactor.

  3. Model ids get mangled instead of resolved. normaliseLegacyModelId turns every dotted version into dashes and the repo has no dash-to-dot normalisation, so anything not in the aliases you added loses or changes pricing: gemini-2.0-flash-001 $0.15/Mtok to $0, gpt-4.5-preview $75 to $30 (matches a different row), gpt-5.3-codex-spark folded into the non-Spark SKU. Resolve through the alias mechanism; don't rewrite ids at the provider.

  4. Unresolvable model now falls back to claude-sonnet-4-5. Main returned unknown and $0 plus the unknown-model warning. The branch invents premium spend ($3.00 on a 1M-token request with no model info) and suppresses the diagnostic.

  5. Double-count with transcripts. hasChatSessionFiles() (line 3358) still matches .jsonl only, so a workspace with only legacy .json sessions no longer suppresses GitHub.copilot-chat/transcripts/. Both sources are discovered, dedup keys are in different namespaces, and docs/providers/copilot.md:366 documents this exact invariant. Extend the gate to .json, and key legacy sessions on session.sessionId || basename so the two formats of one session dedup against each other.

  6. Scope. models.ts display-name changes for copilot-*-auto (affects dash, menubar, JSON), deriveGeminiShortName now overriding the short-name table, eight JetBrains model tokens, the inferJetBrainsProject regex rewrite (which drops C:/repo on Windows because the downstream filter still requires a leading slash), and a win32 early-return added to an unrelated existing test. Please move all of that out; this PR should be the legacy chat sessions and nothing else.

There is no test coverage for the new cost path (grep -c 'renderedUserMessage|toolCallRounds|CacheBreakpoint' tests/ is 0), which is why CI is green. Fixtures exercising 1-5 are required for a re-review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants