[chenfeiz0326]

Dev Engineer Review

The change removes 20 AgentX cases from GB300 post-merge performance-sanity lists and removes their duration records. It also deletes AgentX-only test-db files and associated pipeline stages. AgentX configs, client, hooks, local-run instructions, and two QA performance copies remain available.

The change adjusts multi-node split counts to match the remaining cases. The GB300 single-node stage retains 17 cases. The author reports that the change removes 16 stages from each post-merge pipeline.

The change narrows getMultiGpuFileChanged by removing one test-list path from its matching logic. Verify that edits to that list no longer need to trigger multi-GPU-related handling.

QA Engineer Review

The changes affect test lists and test-duration data. They remove AgentX test entries and some whole test-db files, adjust stage splits, and remove a deleted test reference from L0_MergeRequest.groovy. The change does not modify test functions or test implementations.

The author reports that a consistency script found no remaining AgentX entries in perf-sanity test-db files and verified multi-node split counts. The author also reports that test-list validation and duplicate-waiver checks pass. Reported unit-test results are 252 passed for test_disagg_e2e_configs.py, 97 passed for test_perf_submit.py, and 13 passed for test_perf_agentx_client.py. test_cache_transceiver_precheck_run.py reportedly had the same 95 passes as on main; the remaining cases require a full build. No GPU runs were performed.

Coverage verdict: needs follow-up. The author reports passing list checks and unit tests, but no GPU runs were performed to validate the remaining post-merge performance-sanity coverage.

Per-File QA Perspective

• jenkins/L0_MergeRequest.groovy: Removes a deleted test-list reference from multi-GPU change detection. Verify that the remaining path rules still trigger the expected handling.
• jenkins/L0_Test.groovy: Updates post-merge shard counts and removes AgentX and other listed stages. Verify the resulting stage membership and split configuration.
• tests/integration/defs/.test_durations: Removes 20 AgentX duration records. Check that each removed record corresponds to a removed test entry.
• tests/integration/test_lists/test-db/l0_gb300_multi_gpus_perf_sanity.yml: Removes GB300 performance-sanity entries, including AgentX cases. Check that the remaining list still matches the intended CI coverage.
• tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx2_node1_gpu2_gen1_node4_gpu16.yml: Deletes an AgentX-only test list. Confirm that the associated stage no longer references it.
• tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx2_node2_gpu8_gen1_node2_gpu8.yml: Deletes an AgentX-only test list. Confirm that the associated stage no longer references it.
• tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx3_node2_gpu8_gen1_node4_gpu16.yml: Deletes an AgentX-only test list. Confirm that the associated stage no longer references it.
• tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_gen1_node2_gpu8.yml: Removes AgentX entries. Check that remaining cases and stage splits are consistent.
• tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_gen1_node4_gpu16.yml: Removes AgentX entries while retaining a non-AgentX DeepSeek case. Verify the remaining case is still included in CI.
• tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_node2_gpu8.yml: Deletes an AgentX-only test list. Confirm that the associated stage no longer references it.
• tests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_node4_gpu16.yml: Deletes an AgentX-only test list. Confirm that the associated stage no longer references it.

Description

AgentX lanes are not a good fit for perf-sanity. An AgentX lane's throughput depends on how its prefill and
decode workers are rate-matched, and that ratio is retuned as the AgentX workload is optimized. A fixed AgentX
lane therefore reports false regressions and spurious instability rather than real ones. NVBug 6826000 is an
example: the DeepSeek-V4-Pro e2e lanes swing with a 5-8 % CV on main, and about 40 % of their post-merge runs
were flagged.

This PR stops running the AgentX cases in CI perf-sanity by removing them from the test lists. The AgentX
configs in tests/scripts/perf-sanity/ stay, so each lane can still be run by hand
(jenkins/scripts/perf/local/README.md).

Config (tests/scripts/perf-sanity/)	Test-list entries removed
aggregated/gb300_minimax_m3_fp4_agentx_grace_blackwell.yaml	3 aggregated (con1, con20, con45)
disaggregated/gb300_deepseek-v41-flash-fp4-dspark_agentx_con1579_*	e2e, ctx_only, gen_only_no_context
disaggregated/gb300_deepseek-v4-pro-dspark_agentx_con{1156,1456}_*	e2e, ctx_only, gen_only_no_context
disaggregated/gb300_kimi-k3-fp4-dspark_agentx_con{138,220,63,88}_*	ctx_only, gen_only_no_context

• Test lists and stages. 20 test-db entries go, all GB300 post-merge.
  • Five test-db files held only AgentX cases. They are deleted together with their stages:
    GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge,
    GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-Node4-GPU16-Post-Merge and the three
    GB300-*-Disagg-PerfSanity-AgentX-*-Post-Merge stages. L0_MergeRequest.groovy drops its reference to
    l0_gb300_multi_nodes_perf_sanity_node2_gpu8.yml.
  • Multi-node perf stages need exactly one case per split, so two stages shrink to their remaining cases:
    GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-GEN1-NODE2-GPU8-Post-Merge goes from 6 splits to 4, and
    GB300-16_GPUs-4_Nodes-PyTorch-PerfSanity-GEN1-NODE4-GPU16-Post-Merge from 6 to 1.
  • Single-node perf-sanity stages now run at most 6 cases per split. GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge
    goes from 7 splits to 4 (2/5/5/5 cases, at most 1.8 h of recorded durations per split), and
    DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge from 3 to 1 (6 cases, 1.1 h). The other single-node
    perf-sanity stages already run one split with at most 6 cases.
  • In total, each post-merge pipeline runs 21 fewer stages.
• Durations. The 20 AgentX keys in tests/integration/defs/.test_durations are removed.
• What stays:
  • The AgentX configs, the AgentX client (agentx_client.py and its hooks in test_perf_sanity.py) and the
    local-run instructions.
  • The ctx_only, gen_only and gen_only_no_context modes, which all have non-AgentX cases.
  • The QA perf suite's two DeepSeek-V4-Pro AgentX copies in tests/scripts/perf/disaggregated/, with their
    QA-list entries.
• Side effects:
  • DWDP context servers no longer run in CI perf-sanity; only the Kimi-K3 AgentX configs used them.
  • num_serve_frontends and ctx_router_config are now exercised only by the QA copies.
  • This supersedes [https://nvbugs/6826000][fix] Size the DeepSeek-V4-Pro AgentX ctx KV pools for the compressed history #20970, a pool_ratio fix for the two V4-Pro lanes.

Test Coverage

No GPU runs: the change only removes test-list entries and adjusts split counts.

• Consistency script. Every perf/test_perf_sanity.py id left in the perf-sanity test-db files resolves to
  an existing config and server config, none of them an AgentX config. Every multi-node perf-sanity stage's
  split count equals its case count. On the pre-change tree the same script flags the 20 AgentX entries, and it
  flags the new split counts when paired with the old test lists.
• Split simulation. Grouping each single-node stage's cases with the copy of pytest-split's least-duration
  packing in jenkins/scripts/perf/submit.py and tests/integration/defs/.test_durations gives the per-split
  counts above. No split holds more than 6 cases, and the longest split stays well under the ~4 h partition
  walltime.
• Test-list checks. scripts/check_test_list.py --validate (2022 entries) and --check-duplicate-waives
  pass.
• Unit tests:
  • tests/unittest/others/test_disagg_e2e_configs.py: 252 passed.
  • tests/unittest/scripts/test_perf_submit.py: 97 passed.
  • tests/unittest/scripts/test_perf_agentx_client.py: 13 passed.
  • tests/unittest/others/test_cache_transceiver_precheck_run.py: the same 95 passes as on main. Its
    remaining cases need a full build.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/TensorRT-LLM
• Branch: main
• File: .coderabbit.yaml
• Commit: 1582528
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: c01d842f-6b91-4508-995e-85e7a7f2c2b0

Commits

Reviewing files that changed from the base of the PR and between 09a16d2 and 4b91866.

Files selected for processing: 1

• jenkins/L0_Test.groovy

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The pull request removes or revises GB300 performance-sanity test entries, configurations, and timing records. It also reduces shard counts for selected Jenkins stages and removes several multi-node stages.

Changes

PerfSanity selection updates

Layer / File(s)	Summary
Update test inventory and timing records tests/integration/test_lists/test-db/*, tests/integration/defs/.test_durations	Several GB300 test entries and multi-node configurations are removed. Timing records are added for selected GB300 runs, while other records are removed.
Adjust Jenkins PerfSanity stages jenkins/L0_Test.groovy, jenkins/L0_MergeRequest.groovy	Shard counts are reduced for selected B200 and GB300 stages. Several GB300 multi-node stages are removed, and one removed test-list path no longer triggers the multi-GPU file-match check.

Priority: Low

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Other

Suggested reviewers: tburt-nv, brnguyen2, juney-nvidia

Merge Risk: Minimal · up to 4b918

No unresolved merge-blocking risk was identified in the PerfSanity updates.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title follows the required template: NVBugs ID, a [test] type, and a summary. It names the main change, which removes the AgentX cases from the perf-sanity test lists. It matches the changeset, wh…
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The pull request includes complete Description and Test Coverage sections. It explains the problem, the implementation, retained functionality, side effects, validation steps, and reported test result…

Full details: Title check

Explanation

The title follows the required template: NVBugs ID, a [test] type, and a summary. It names the main change, which removes the AgentX cases from the perf-sanity test lists. It matches the changeset, which removes 20 AgentX test-db entries and related stages.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

Full details: Description check

Explanation

The pull request includes complete Description and Test Coverage sections. It explains the problem, the implementation, retained functionality, side effects, validation steps, and reported test results. It also includes the PR Checklist with the review confirmation checked. The pull request title cannot be assessed because it was not provided in the available context.

Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

[yiqingy0]

With the 4 AgentX cases removed, could we decrease the number of splits for GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge?

[chenfeiz0326]

@yiqingy0 Done in 4b91866: GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge now uses 4 splits instead of 7, the fewest that keep every split at 6 cases or fewer (2/5/5/5 cases, at most 1.8 h of recorded durations per split). DGX_B200-8_GPUs-PyTorch-PerfSanity-Post-Merge also goes from 3 splits to 1 (6 cases).

[chenfeiz0326]

/bot skip --comment "Only remove old perf tests, no need to run the whole CI pipeline"

[tensorrt-cicd]

PR_Github #80567 [ skip ] triggered by Bot. Commit: 4b91866 Link to invocation

[tensorrt-cicd]

PR_Github #80567 [ skip ] completed with state SUCCESS. Commit: 4b91866
Skipping testing for commit 4b91866

Link to invocation