[yuxianq]

@coderabbitai summary

Description

Enable FP8 E4M3 KV cache for standard non-MLA PrimsTSFmha context, decode, and mixed batches. Fused QKV preprocessing quantizes Q using the shared Q/K/V scale, and the kernels consume FP8 Q/K/V while writing native BF16/FP16 output. Decode reads preprocessing-produced scales on the GPU, including during CUDA graph replay.

Use native BF16/FP16 decode output from the canonical PrimTS kernels, size workspace for FP8 inputs, and validate scalar scale metadata. Context tests pass the current chunk's Q length separately from its cached prefix. Host-scale and device-scale decode plans have distinct persistent compilation keys, so cached kernels retain the correct argument types.

The vendor lock pins the FlashInfer device-scale changes on top of trtllm-prims-ts-dev-20261012 at aa709061230a. The rebase retains the latest TensorRT-LLM compatibility patch and drops BF16-output changes already present in canonical.

Test Coverage

Rebased onto TensorRT-LLM d9c4f3ba3a9e and FlashInfer canonical aa709061230a. Validated on B200 with PyTorch 26.09, CUTLASS DSL 4.8.0, and the current base PR's CI wheel (build 67023):

• 257 TensorRT-LLM tests passed: 223 adapter tests and 34 GPU regression cases. Coverage includes context/decode/mixed batches, cached context, both cache managers, four decode reduction modes, and graph replay with updated device scales.
• 12 FlashInfer FP8 decode GPU cases passed, including BF16 output with both host and device scales for identical geometry. A separate process reloaded a cached separate-reduction case successfully.
• 336 FP8/BF16 planning configurations passed.
• 976 source contract/trace consistency checks passed (2 skipped). The regenerated GPU trace matches the committed JSON.
• Pre-commit, targeted Python compilation, range-diff auditing, and offline/local/remote vendor verification passed.

Model-quality and performance benchmarks have not been run.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

schema_version: 1
vendors:
  flashinfer-prims-ts:
    unpaired_reason: >-
      This FP8 decode delta is based on the TRT-LLM dated canonical FlashInfer
      branch trtllm-prims-ts-dev-20261012 and is reviewed with this integration
      PR. No standalone flashinfer-ai/flashinfer PR is open; the vendor promotion
      workflow will promote the reviewed delta from the temporary source branch
      to that dated canonical branch.

[yuxianq]

@yuxianq — vendor promotion needs author action:

The pinned temporary source commit 539197151dd22e751c8e2845fdcad32ccefe7911 does not contain the latest canonical branch tip ae3851ab5f958ba22f154059df5b38321f6369d2.

Fix: Fetch https://github.com/yuxianq/flashinfer.git / trtllm-prims-ts-dev-20261012 and rebase the temporary source branch prims-tsfmha-fp8-non-mla--kernels onto ae3851ab5f958ba22f154059df5b38321f6369d2. Push it, then use scripts/vendor/manage.py to update the consumer lock and vendored snapshot to the rebased SHA. The consumer lock's old pin alone is not a sufficient base; updating only the PR metadata or temporary branch without its lock pin will not fix this.

The bot will recheck automatically on a later poll.

[yuxianq]

@yuxianq — the PrimTS upstream refresh #20155 has merged. Please update this source-update PR before merging it.

The new canonical source branch is yuxianq/flashinfer:trtllm-prims-ts-dev-20261012, initially pinned at ae312f7cc1723b06fda5168105ade0f328ee47d8.

1. Rebase your temporary FlashInfer branch onto the new canonical branch; retain only changes not already included upstream, resolving any API differences.
2. Rebase this TRT-LLM PR onto latest main, including the refreshed compatibility patch and integration changes.
3. Keep vendors.flashinfer-prims-ts in 3rdparty/vendor_sources.lock.yaml pointing to your temporary source branch, update its immutable commit, and regenerate the selected vendored files and digests. Do not restore the old canonical branch/pin while resolving conflicts.
4. Add the required <!-- vendor-promotion:start --> / <!-- vendor-promotion:end --> block to the PR description, listing upstream PRs for the remaining source changes (or a justified unpaired_reason). The vendor bot provides a copyable template.
5. Verify the vendored snapshot and rerun the relevant tests and CI.

The publishing bot now targets the new dated canonical branch. After this source-update PR merges, its lock-only promotion will target that branch. The old trtllm-prims-ts-dev branch is intentionally unchanged.

[yuxianq]

/bot run

[tensorrt-cicd]

PR_Github #77487 [ run ] triggered by Bot. Commit: a29bcb6 Link to invocation

[tensorrt-cicd]

PR_Github #77487 [ run ] completed with state SUCCESS. Commit: a29bcb6
/LLM/main/L0_MergeRequest_PR pipeline #63875 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

[yuxianq]

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

The pinned temporary source commit 539197151dd22e751c8e2845fdcad32ccefe7911 does not contain the latest canonical branch tip ae3851ab5f958ba22f154059df5b38321f6369d2.

Fix: Fetch https://github.com/yuxianq/flashinfer.git / trtllm-prims-ts-dev-20261012 and rebase the temporary source branch prims-tsfmha-fp8-non-mla--kernels onto its latest tip (observed ae3851ab5f958ba22f154059df5b38321f6369d2). Push it, then use scripts/vendor/manage.py to update the consumer lock and vendored snapshot to the rebased SHA. The consumer lock's old pin alone is not a sufficient base; updating only the PR metadata or temporary branch without its lock pin will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: a29bcb6b922b. This is a snapshot; newer results are posted as new comments.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-tsfmha-fp8-non-mla--kernels, pin 539197151dd22e751c8e2845fdcad32ccefe7911.

Supersedes previous vendor validation.

[yuxianq]

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs maintainer action:

Waiting for the preceding vendor promotion: the base lock still points to repository https://github.com/heyuhhh/flashinfer.git, branch trtllm-prims-ts-dev.

Fix: A maintainer must finish the preceding lock-only promotion back to yuxianq/flashinfer / trtllm-prims-ts-dev-20261012. After it merges, the author should rebase this consumer PR onto the latest base branch and rebase its temporary source branch onto the new canonical pin, then update the lock/vendor snapshot. No metadata edit can unblock the preceding promotion.

The bot will recheck automatically on a later poll.

Checked PR revision: a29bcb6b922b. This is a snapshot; newer results are posted as new comments.

Supersedes previous vendor validation.

[yuxianq]

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

Source update is not a fast-forward.

Fix: Fetch canonical repository https://github.com/yuxianq/flashinfer.git, branch trtllm-prims-ts-dev-20261012, and rebase the temporary source branch prims-tsfmha-fp8-non-mla--kernels onto its latest tip without merge commits, not just the consumer lock's pin 0e0761ef5ef10cb0762cc1b890a5c1a329bb49bf. Then use scripts/vendor/manage.py to update 3rdparty/vendor_sources.lock.yaml and the vendored snapshot to the rebased SHA, and push the source branch and consumer PR. Editing metadata alone will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: a29bcb6b922b. This is a snapshot; newer results are posted as new comments.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-tsfmha-fp8-non-mla--kernels, pin 539197151dd22e751c8e2845fdcad32ccefe7911.

Supersedes previous vendor validation.

[yuxianq]

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs maintainer action:

Waiting for the preceding vendor promotion: the base lock still points to repository https://github.com/yunruis/flashinfer.git, branch trtllm-prims-ts-dev-20261023.

Fix: A maintainer must finish the preceding lock-only promotion back to yuxianq/flashinfer / trtllm-prims-ts-dev-20261012. After it merges, the author should rebase this consumer PR onto the latest base branch and rebase its temporary source branch onto the new canonical pin, then update the lock/vendor snapshot. No metadata edit can unblock the preceding promotion.

The bot will recheck automatically on a later poll.

Checked PR revision: a29bcb6b922b. This is a snapshot; newer results are posted as new comments.

Supersedes previous vendor validation.

[yuxianq]

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion needs author action:

Source update is not a fast-forward.

Fix: Fetch canonical repository https://github.com/yuxianq/flashinfer.git, branch trtllm-prims-ts-dev-20261012, and rebase the temporary source branch prims-tsfmha-fp8-non-mla--kernels onto its latest tip without merge commits, not just the consumer lock's pin aa709061230a58858375c513056e7733fa74294a. Then use scripts/vendor/manage.py to update 3rdparty/vendor_sources.lock.yaml and the vendored snapshot to the rebased SHA, and push the source branch and consumer PR. Editing metadata alone will not fix this.

The bot will recheck automatically on a later poll.

Checked PR revision: a29bcb6b922b. This is a snapshot; newer results are posted as new comments.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-tsfmha-fp8-non-mla--kernels, pin 539197151dd22e751c8e2845fdcad32ccefe7911.

Supersedes previous vendor validation.

[yuxianq]

/bot run

[yuxianq]

Vendor validation: flashinfer-prims-ts

@yuxianq — Vendor promotion metadata and source attribution are valid. This is not a code-review approval. Promotion still requires the source PR to merge.

Checked PR revision: 50de5d1e9ad4. This is a snapshot; newer results are posted as new comments.

Checked source: https://github.com/yuxianq/flashinfer.git / prims-tsfmha-fp8-non-mla--kernels, pin ad4b6bc638237f32496b7f7a379a76e28cccb360.

Supersedes previous vendor validation.

[tensorrt-cicd]

PR_Github #81384 [ run ] triggered by Bot. Commit: 50de5d1 Link to invocation

[tensorrt-cicd]

PR_Github #81384 [ run ] completed with state FAILURE. Commit: 50de5d1
/LLM/main/L0_MergeRequest_PR pipeline #67060 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation