Repository navigation
Conversation
The decode step after the target forward (lm_head, soft-cap, DSpark
verification and the draft block) ran ~420 small kernels on VR200. This
replaces the largest chains in the SM107 tp4/tp8 target with fused ops:
- triton_draft_attention: the draft block's window gather from the paged
context, masking, concatenation, score GEMM, softmax and value GEMM as
one split-K flash-decode kernel plus a merge.
- triton_latent_tail: each draft layer's query rope, latent RMSNorm, rope
and concatenation, and the same for every layer of the context features.
- triton_markov_{input,gate,logits}: the recurrent Markov head's
embedding/concatenation, gate chain and logit add fused with the
greedy argmax.
- symm_logits_gather_softcap: the lm_head shards pushed over NVLink into
symmetric buffers and widened and soft-capped locally, instead of the
NCCL all-gather, the reorder, the upcast and the three soft-cap ops,
for target and draft logits. A step with more than 64 logit rows keeps
the NCCL path.
The draft attention, latent tail and Markov head have no other path in
this target: the draft context is always paged and the lm_head always
gathers the whole vocabulary. Each op keeps its torch reference.
The rope, gate chain, logit add, argmax and logits gather are bitwise
equal to the chains they replace; the RMSNorm reduces in its own order;
the attention merges an online softmax and is within bf16 rounding.
Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
In the Mistral Large 4 SM107 (VR200) TP4/TP8 target, the step after the target's last layer ran about 420 small kernels: about 1.2 ms per step at one sequence and 1.5 ms at four. That step covers the lm_head and soft-cap, DSpark verification and acceptance, the three-layer draft block, the draft lm_head and the recurrent Markov head. The time went to launch latency and to intermediate tensors written to and read back from HBM, not to math. This PR fuses the largest chains into Triton ops under the target's
ops/directory.What changes (all inside
mistral_large_4_nvfp4__sm_107__tp4/; Python and Triton JIT only, no C++/CUDA;tensorrt_llm/_torch/speculative/is unchanged)ops/dspark/draft_ops.py: triton_draft_attention). Each draft layer used to gather its 2048-token context window from the paged cache, mask it, concatenate the block's own keys, then run the score GEMM, softmax and value GEMM. This is now one split-K flash-decode kernel that reads the pages through the page table, plus a merge. It uses 32 splits for up to four sequences and 8 above that.triton_latent_tail). Each draft layer's query rope, latent RMSNorm, rope and concatenation, and the same for every layer of the context features, now run as one kernel that writes straight into the per-layer output.triton_markov_input,triton_markov_gate,triton_markov_logits). The embedding and concatenation, the gate chain, and the logit add fused with the greedy argmax (ties take the first index, astorch.argmaxdoes).ops/comm/symm_logits_gather.py: symm_logits_gather_softcap). This replaces the NCCL all-gather, the rank-major reorder, the fp32 upcast and the three soft-cap ops, for both target and draft logits.DecodeTargetandPrefillTargethad identicaldspark_forwardbodies; both now call one_dspark_block.There are no feature switches. The target's draft context is always paged (
DraftModel._paged_ctx_cache), and its lm_head always gathers the full vocabulary, since the route excludes attention DP. The draft attention, latent tail and Markov head therefore have no other path;apply_markov_chain_logitsasserts the worker passes novocab_slice. Each new op is anOpWrapperthat keeps its torchreference.Numerics
VR200 results
The first three subsections were measured on an earlier revision of this change. It had one env switch per direction, all off, so it ran the same code path as this PR. The PR itself removes the switches and was not re-run on GPU.
Six serving points. The baseline is this PR's base (
416b15a861, same tree as the measured baseline commit). Setup: DSpark K3 with a forced acceptance length of 2.772, 20 min per point,stream_interval=1, one run per point. Step time = 2.772 / (1/ITL p50).Same-build A/B (TP4, one run per arm, all four directions reverted vs all on): per-step median 9.031 → 8.544 ms at 1×16k (−5.4%) and 12.414 → 11.662 ms at 4×16k (−6.1%). Two runs of the same arm differ by 12–79 µs.
Post-target nsys (rank 0, CUDA graph node trace, 30 steps):
Op timings (CUDA graph replay):
Not done yet
test_modeling_v2_*.pyundertests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp4/. The op-level and shadow checks above ran from out-of-tree scripts.Test Coverage
None in the repo yet; this is a draft. The validation above ran out of tree on VR200 (op-level tests against each op's reference, in-model shadow comparisons, same-build A/B and the six-point serving run). Unit tests for the five Triton ops and the multi-GPU logits gather will follow in this PR before it leaves draft.
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.🤖 Generated with Claude Code