Skip to content

[None][perf] Fuse the Mistral Large 4 SM107 DSpark post-target step - #22431

Draft
yt0428 wants to merge 1 commit into
NVIDIA:feat/mistral-large-4from
yt0428:codex/ml4-vr200-post-target
Draft

yt0428 wants to merge 1 commit into
NVIDIA:feat/mistral-large-4from
yt0428:codex/ml4-vr200-post-target

Conversation

@yt0428

@yt0428 yt0428 commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

@coderabbitai summary

Description

In the Mistral Large 4 SM107 (VR200) TP4/TP8 target, the step after the target's last layer ran about 420 small kernels: about 1.2 ms per step at one sequence and 1.5 ms at four. That step covers the lm_head and soft-cap, DSpark verification and acceptance, the three-layer draft block, the draft lm_head and the recurrent Markov head. The time went to launch latency and to intermediate tensors written to and read back from HBM, not to math. This PR fuses the largest chains into Triton ops under the target's ops/ directory.

What changes (all inside mistral_large_4_nvfp4__sm_107__tp4/; Python and Triton JIT only, no C++/CUDA; tensorrt_llm/_torch/speculative/ is unchanged)

  • Draft attention (ops/dspark/draft_ops.py: triton_draft_attention). Each draft layer used to gather its 2048-token context window from the paged cache, mask it, concatenate the block's own keys, then run the score GEMM, softmax and value GEMM. This is now one split-K flash-decode kernel that reads the pages through the page table, plus a merge. It uses 32 splits for up to four sequences and 8 above that.
  • Latent tail (triton_latent_tail). Each draft layer's query rope, latent RMSNorm, rope and concatenation, and the same for every layer of the context features, now run as one kernel that writes straight into the per-layer output.
  • Markov head (triton_markov_input, triton_markov_gate, triton_markov_logits). The embedding and concatenation, the gate chain, and the logit add fused with the greedy argmax (ties take the first index, as torch.argmax does).
  • Logits gather (ops/comm/symm_logits_gather.py: symm_logits_gather_softcap). This replaces the NCCL all-gather, the rank-major reorder, the fp32 upcast and the three soft-cap ops, for both target and draft logits.
    • Each rank pushes its vocabulary shard over NVLink into every peer's symmetric buffer, then a device barrier, then each rank widens and soft-caps locally.
    • The buffer holds 64 rows × 2 slots (target, draft), 50 MiB per rank, built at load.
    • A step with more than 64 logit rows keeps the NCCL path. That only happens with more than 16 generation requests per step.
  • DecodeTarget and PrefillTarget had identical dspark_forward bodies; both now call one _dspark_block.

There are no feature switches. The target's draft context is always paged (DraftModel._paged_ctx_cache), and its lm_head always gathers the full vocabulary, since the route excludes attention DP. The draft attention, latent tail and Markov head therefore have no other path; apply_markov_chain_logits asserts the worker passes no vocab_slice. Each new op is an OpWrapper that keeps its torch reference.

Numerics

  • Op level, random data: the rope, the Markov chain and the argmax are bitwise equal to the chains they replace. The logits gather is bitwise equal for every finite bf16 value, subnormals included, and across 50 CUDA graph replays alternating the two slots. The attention's RMS error against a float64 reference is below the original chain's.
  • In-model shadow check (TP4, eager, every call also runs the original chain on the live tensors):
    • Target and draft logits gathers: 852 and 788 calls, all bitwise equal, including prefill and mixed steps.
    • Markov ops and query rope: bitwise equal in every call.
    • Latent RMSNorm: 3460 of 1.1e9 elements differ by 1 ulp (reduction order).
    • Draft attention: in 2313 of 2364 calls it is closer to float64 than the original chain. Mean relative RMS error is 4.8e-3 vs 5.8e-3.
  • End to end: greedy outputs are not reproducible run to run on this engine even without the change, because of the split radix Top-K's atomics and per-launch autotuning. Draft tokens accepted per request-step were 2.221 and 2.294 on two baseline runs and 2.274 with this change.

VR200 results

The first three subsections were measured on an earlier revision of this change. It had one env switch per direction, all off, so it ran the same code path as this PR. The PR itself removes the switches and was not re-run on GPU.

Six serving points. The baseline is this PR's base (416b15a861, same tree as the measured baseline commit). Setup: DSpark K3 with a forced acceptance length of 2.772, 20 min per point, stream_interval=1, one run per point. Step time = 2.772 / (1/ITL p50).

Point 1/ITL p50 (tok/s/user) Step (ms) Total tok/s/GPU OK / errors
TP4 c=1 289.9 → 306.1 (+5.6%) 9.56 → 9.06 +1.4% 146/1 → 146/1
TP4 c=4 271.4 → 293.7 (+8.2%) 10.21 → 9.44 +4.1% 244/0 → 245/0
TP4 c=8 216.5 → 225.6 (+4.2%) 12.81 → 12.29 +1.1% 456/0 → 460/0
TP8 c=1 312.9 → 330.6 (+5.6%) 8.86 → 8.39 +1.9% 146/1 → 146/1
TP8 c=4 297.0 → 313.2 (+5.5%) 9.33 → 8.85 +1.0% 246/0 → 246/0
TP8 c=8 237.9 → 253.8 (+6.7%) 11.65 → 10.92 +0.8% 467/0 → 476/0
  • The baseline ran on TS4B nodes and this run on TS3 nodes; both are 4×VR200 per node.
  • The 1/ITL gains are well above any rack offset. The ~1% total-throughput deltas may include one; total throughput also counts input tokens and prefill, which this PR does not touch.
  • TTFT p90 is mixed: TP8 c=1 went from 1.88 s to 2.04 s, and the other points are flat or lower.
  • The single c=1 error also occurs on the baseline.

Same-build A/B (TP4, one run per arm, all four directions reverted vs all on): per-step median 9.031 → 8.544 ms at 1×16k (−5.4%) and 12.414 → 11.662 ms at 4×16k (−6.1%). Two runs of the same arm differ by 12–79 µs.

Post-target nsys (rank 0, CUDA graph node trace, 30 steps):

Post-target time Kernels Whole step
1×16k 1195 → 706 µs 423 → 195 8703 → 8174 µs
4×16k 1528 → 759 µs 417 → 202 11998 → 11231 µs
  • Saving by phase (1×16k / 4×16k):
    • Draft preparation and its 3 layers: −353 / −537 µs
    • Markov head: −83 / −94 µs
    • Target gather: −18 / −65 µs
    • Draft gather: −30 / −70 µs
    • Sampling and acceptance (unchanged): −8 / −1 µs
  • Leaving one direction out (ms per step, single runs, about ±0.05 ms noise):
Direction 1×16k 4×16k
Draft attention 0.089 0.316
Latent tail 0.127 0.186
Markov head 0.067 0.101
Logits gather 0.009 0.136

Op timings (CUDA graph replay):

Op New Original chain
Draft attention, per layer (1 / 4 / 8 sequences) 10.1 / 12.7 / 15.8 µs 68.7 / 126 / 172 µs
Latent tail 1.5–1.8 µs 46–106 µs
Markov input / gate / logits+argmax (1 sequence) 1.7 / 1.6 / 2.6 µs 10.4 / 7.6 / 12.9 µs
Logits gather (4 / 16 / 32 rows) 11.0 / 16.6 / 26.4 µs 32.6 / 84.9 / 113.8 µs

Not done yet

  • Unit tests in the repo's layout, like the sibling ops' test_modeling_v2_*.py under tests/unittest/_torch/modeling_v2/models/mistral_large4/mistral_large_4_nvfp4__sm_107__tp4/. The op-level and shadow checks above ran from out-of-tree scripts.
  • A GPU run of this exact revision.
  • Gather timings above 32 rows.

Test Coverage

None in the repo yet; this is a draft. The validation above ran out of tree on VR200 (op-level tests against each op's reference, in-model shadow comparisons, same-build A/B and the six-point serving run). Unit tests for the five Triton ops and the multi-GPU logits gather will follow in this PR before it leaves draft.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

The decode step after the target forward (lm_head, soft-cap, DSpark
verification and the draft block) ran ~420 small kernels on VR200. This
replaces the largest chains in the SM107 tp4/tp8 target with fused ops:

- triton_draft_attention: the draft block's window gather from the paged
  context, masking, concatenation, score GEMM, softmax and value GEMM as
  one split-K flash-decode kernel plus a merge.
- triton_latent_tail: each draft layer's query rope, latent RMSNorm, rope
  and concatenation, and the same for every layer of the context features.
- triton_markov_{input,gate,logits}: the recurrent Markov head's
  embedding/concatenation, gate chain and logit add fused with the
  greedy argmax.
- symm_logits_gather_softcap: the lm_head shards pushed over NVLink into
  symmetric buffers and widened and soft-capped locally, instead of the
  NCCL all-gather, the reorder, the upcast and the three soft-cap ops,
  for target and draft logits. A step with more than 64 logit rows keeps
  the NCCL path.

The draft attention, latent tail and Markov head have no other path in
this target: the draft context is always paged and the lm_head always
gathers the whole vocabulary. Each op keeps its torch reference.

The rope, gate chain, logit add, argmax and logits gather are bitwise
equal to the chains they replace; the RMSNorm reduces in its own order;
the attention merges an online softmax and is within bf16 rounding.

Signed-off-by: yt0428 <51468697+yt0428@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant