Repository navigation
[https://nvbugs/6826000][fix] Size the DeepSeek-V4-Pro AgentX ctx KV pools for the compressed history - #20970
chenfeiz0326 wants to merge 1 commit into
Conversation
…pools for the compressed history The ctx servers of both DeepSeek-V4-Pro DSpark AgentX e2e lanes split their KV cache 0.55 / 0.22 / 0.23 over the DeepSeek-V4 life cycles: the SWA windows, the compressed history with the indexer keys, and the compressor state windows. With per_conversation reuse only the history grows with the retained conversations, so its pools filled and dropped cached prefixes while ~150 GiB per rank of the windowed host pools stayed unused. Attention-DP ranks whose queue wait outlived that short retention re-prefilled every waiting prompt and stayed collapsed; each run collapsed a random 8-10 of 16 ranks (con1156: 6.90-7.70 req/s, CV 4.8 %). Give the history 64 % of the pools. con1156 over four node sets: 29.97-30.41 req/s (CV 0.7 %), prefix reuse 85 % -> 95 %, no collapsed rank. Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Update context KV pool ratios tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-dspark_agentx_con1156_ctx2_dep8_gen1_dep8_eplb0_dspark3_ccb-NIXL.yaml, tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-dspark_agentx_con1456_ctx3_dep8_gen1_dep16_eplb0_dspark5_ccb-NIXL.yaml |
Both configurations change the pool ratios to [0.18, 0.64, 0.18]. Comments identify the pool categories and describe retained-history usage. |
Priority:
Estimated code review effort: 1 (Trivial) | ~5 minutes
Change: Bug fix
Merge Risk: 
303e6
-
The allocation change matches the cache manager’s lifecycle ordering and is ready to merge.
Pre-merge checks: 
-
Check name Status Explanation Title check The title names the change (resizing the DeepSeek-V4-Pro AgentX context KV pools for compressed history) and follows the [NVBugs ID][type] Summary template. It matches the diff, which changes only the… Docstring Coverage No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0… Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Description check The description is complete and relevant. It explains the cause, the pool-ratio change, measured results, test coverage, limitations, and the required host_cache_size follow-up. The checklist is mostl… Full details: Title check
-
Explanation
The title names the change (resizing the DeepSeek-V4-Pro AgentX context KV pools for compressed history) and follows the [NVBugs ID][type] Summary template. It matches the diff, which changes only the pool_ratio in two perf-sanity configs.
Full details: Docstring Coverage
-
Explanation
No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)
Full details: Description check
-
Explanation
The description is complete and relevant. It explains the cause, the pool-ratio change, measured results, test coverage, limitations, and the required host_cache_size follow-up. The checklist is mostly filled out, although the API-change checklist item from the template is omitted.
-
Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
|
Superseded by #20985, which removes the AgentX cases, these two lanes included, from perf-sanity. Closing without merging. |
Description
Fixes https://nvbugs/6826000: the DeepSeek-V4-Pro DSpark AgentX e2e perf-sanity lanes on GB300 are unstable on main
(con1156: 6.49-8.72 req/s, CV ~8 %; con1456: 18.40-23.32 req/s, CV 5.4 %, 27 % spread over 42 post-merge runs).
Their
gen_only_no_context(CV < 1 %) andctx_only(~2 %) components are stable; only the e2e composition swings,and each run's throughput tracks its mean TTFT (r = -0.92 / -0.99).
Cause. Each ctx attention-DP rank settles into one of two states. A good rank reuses ~95 % of the prompt and
answers in seconds. A collapsed rank queues 130-180 requests, waits longer than its KV cache can hold their
prefixes, re-prefills them (whole-prefix miss 34-54 %), and stays collapsed. Every run collapses a random 8-10 of
the 16 ranks (con1156), so its throughput is random.
The retention horizon that permits this comes from the ctx
kv_cache_config.pool_ratio: [0.55, 0.22, 0.23]. ForDeepSeek-V4 the ratio is per KV cache manager v2 life cycle: the SWA windows, the compressed history with the indexer
keys, and the compressor state windows. With
per_conversationreuse only the history grows with the retainedconversations, yet it got 22 % of each pool. Measured per ctx rank: the history pool (18.8 GiB GPU + 39.6 GiB host)
was 100 % full and dropped ~60K cached blocks from host per hour, while the windowed pools used <= 31 % / <= 75 % of
their GPU share and never touched their 140 GiB of host tier.
Change.
pool_ratio: [0.18, 0.64, 0.18]in both lanes: history 54.8 GiB GPU + 115.2 GiB host, each windowedpool 14.4 GiB GPU + 32.4 GiB host. All three pools are now in use.
Test Coverage
GB300 (oci-jhb), CI wheel GenPostMergeBuilds 3191 (main abb94ab), the lanes' CI config with only this change, 3600 s
window and 3 warmup requests per lane as in CI. Each rep on a different node set.
TPOT rises from 11.1 to 15.1 ms (con1156) because the gen server now decodes 4x as many requests; e2e regressions are
gated on throughput only.
Do not merge yet: needs a
host_cache_sizefollow-up commit. Thectx_onlytests of both lanes read the samectx config and run the benchmark client on a ctx node. The old split never used the windowed host pools (158 GiB/node
of the 720 GiB/node
host_cache_sizewas ever touched); with this change the tier fills (500-594 GiB/node at the endof the 1800 s window) and 2 of 4
ctx_onlyruns were killed by the 900 GB/node host OOM cgroup (the e2e ctx nodescarry no client and survived 720 GiB/node in all 7 runs). The follow-up lowers the ctx
host_cache_sizefrom193273528320 (180 GiB) to 103079215104 (96 GiB) in both lanes. Measured with both changes:
That is within 1.1 % of the 180 GiB numbers above; all 6 pytest passed.
ctx_onlywith both changes: no OOM in 12runs (host touched 243-263 GiB/node, capped at 384). With a separate aiperf artifact directory per run (my first
concurrent local runs shared one, which broke two con1156 clients; CI runs one per checkout), all passed:
ctx_onlygets faster but noisier, so these twoctx_onlylanes may start flagging once their baselines settle.The ctx server is not the noisy part: it prefills the same uncached work in every window (con1156 170-174 Mtok,
con1456 177-181 Mtok, CV 1.1 %). The window is. One 8-GPU ctx server carries the lane's full concurrency, so the
1800 s
ctx_onlywindow (set for every AgentX lane in #20163) opens on an empty queue, takes 70-81 % of its requestsin the first 10 min and then saturates, before and after this change alike. With the larger history pool far more
of the opening burst is reused, so each run's count depends on how much reuse that transient leaves (con1156
78-81 %, con1456 55-58 %), and con1156 starts to recover inside the window at a point that varies by run. A
ctx_onlymeasurement that does not hinge on this transient (for example a window that reaches a steady state, orgating on uncached prefill throughput) would be a harness change for every AgentX
ctx_onlylane, so it is not partof this PR.
Also measured:
attention_dp_config.kv_cache_routing_new_conv_placement: least_queuedalone lowers con1156 to6.51 req/s and adds nothing on top of this change (30.25 req/s), so it is not part of it.
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
No new code paths: the change is two perf-sanity configs, exercised by their lanes.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.🤖 Generated with Claude Code
Dev Engineer Review
Both GB300 DeepSeek-V4-Pro DSpark AgentX perf-sanity configurations change the context worker’s
kv_cache_config.pool_ratiofrom[0.55, 0.22, 0.23]to[0.18, 0.64, 0.18]. The comments identify the pool order and explain the larger compressed-history allocation underper_conversationreuse. No source code, APIs, or other settings changed.The PR description reports improved con1156 throughput and prefix reuse, but also higher TPOT. These results are author-reported; con1456 runs were still in progress, and
ctx_onlybaselines were not measured.QA Engineer Review
The changes modify two perf-sanity YAML lane configurations. They add no test functions, test IDs, selectors, or waivers, and they do not change CI or manual-QA test lists. The changed files configure performance lanes, not integration tests. Coverage verdict: needs follow-up because con1456 results were pending and
ctx_onlybaselines were not measured.Per-File QA Perspective
tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-dspark_agentx_con1156_ctx2_dep8_gen1_dep8_eplb0_dspark3_ccb-NIXL.yaml: The context worker now assigns 64% of the KV cache pools to compressed history and 18% to each windowed pool. Verify the lane runs with the new allocation and track the reported TPOT increase; this configuration is not an integration test-list entry.tests/scripts/perf-sanity/disaggregated/gb300_deepseek-v4-pro-dspark_agentx_con1456_ctx3_dep8_gen1_dep16_eplb0_dspark5_ccb-NIXL.yaml: The context worker uses the same revised pool allocation. Verify the con1456 lane completes and assess its performance; this configuration is not an integration test-list entry.