Repository navigation
[None][perf] Cut fused QK norm + RoPE kernel time by up to 64% on long prefills by sharing RoPE cos/sin across heads - #22429
Conversation
Problem: fusedQKNormRopeKernel runs one warp per (token, head). Every Q and K head of a token recomputes the same RoPE cos/sin (frequency, YaRN ramp, position load, sincos), and a large prefill launches one warp per head (40960 blocks for 8192 tokens with 32 Q + 8 K heads). In a diagnostic build that replaces the angle math and position loads with constants, the kernel time drops by 51.3% at that size. Solution: Let one warp process several consecutive heads of one token. The per-head work (load, RMSNorm warp reduction, NeoX or interleaved pairing, rotation, store) moves unchanged into a device-inline helper. The kernel computes each thread's cos/sin once per warp while the first head is loading, prefetches the next head, and calls the helper for each of its heads. The launcher picks the heads per warp as the largest divisor of the processed head count that keeps at least two warps per token and at least 16 warps per SM. If no divisor fits (for example 17 heads), it splits the heads as evenly as possible over the fewest warps that keep these limits, and the last warp of a token gets fewer heads. Small batches such as decode keep one head per warp and the same grid. This rule uses only the SM count; it was chosen in development sweeps on GB200 that included the shapes measured below. Same arithmetic per element. No new flag or env. Performance: Standalone kernel screening on one GB200 (sm_100f; shared node, untimed development runs): BF16 QKV, CUPTI kernel duration, fresh QKV after an L2 flush, median of 30 launches per variant in each of three separate processes. Kernel time change, then old -> new kernel time (change) in processes 1, 2 and 3: - Qwen3-VL-8B decoder call (32 Q + 8 K heads, head_dim 128, interleaved mRoPE), 8192 tokens, 40960 -> 2048 blocks: -59.5% to -59.3%; 122.6405 -> 49.9365 us (-72.704 us), 122.6725 -> 49.824 us (-72.8485 us), 122.576 -> 49.696 us (-72.880 us). - Same call, 1158 tokens: -47.2% to -47.1%; 19.184 -> 10.128 us (-9.056 us), 19.168 -> 10.144 us (-9.024 us), 19.200 -> 10.128 us (-9.072 us). - Same call, 8 and 32 tokens (decode, same grid): -4.5% to -4.3%; 8 tokens: 2.816 -> 2.688 us (-0.128 us), 2.816 -> 2.688 us (-0.128 us), 2.816 -> 2.688 us (-0.128 us); 32 tokens: 3.008 -> 2.880 us (-0.128 us), 3.008 -> 2.880 us (-0.128 us), 3.008 -> 2.880 us (-0.128 us). - Synthetic head layouts (arange positions, head_dim 64/128/256, 16 to 64 Q heads) at 8 to 8192 tokens: -62.9% to +2.1%; the largest drop is head_dim 256 with 16 Q + 8 K heads at 8192 tokens (143.968 -> 53.392 us in process 2). Only head_dim 64 with 32 Q + 8 K heads at 32 and 128 tokens is slower (-1.2% to +1.2% and +2.1%; 128 tokens: 3.104 -> 3.168 us (+0.064 us), 3.104 -> 3.168 us (+0.064 us), 3.104 -> 3.168 us (+0.064 us)), inside the old kernel's own run-to-run spread in each process. - Qwen3-VL-235B-A22B at TP4 (16 Q + 1 K heads: 17 heads, no divisor fits, 9 + 8 heads per warp), 8192 tokens, 17408 -> 2048 blocks: -55.1%; 52.640 -> 23.6475 us (-28.9925 us), 52.672 -> 23.648 us (-29.024 us), 52.608 -> 23.616 us (-28.992 us). On an 8192-token Qwen3-VL-8B prefill trace this kernel is 5.0% of decoder GPU time, so the predicted decoder saving is 3.0%. Timing was measured on GB200 only. Full Qwen3-VL-8B prefill of 8192 tokens (BF16 model, TP1, one GB200, eager mode, 1 output token): one untimed development run per build on a shared node (old build A, new build B', and a second old-build run A2), sums of GPU kernel times of the prefill forward from Nsight Systems traces, change against A. Wall time and TTFT were not measured. - This kernel, text: A 4.839521 ms -> B' 2.050050 ms, -2.789471 ms (-57.6%); A2 4.868547 ms (+0.6%) - This kernel, image+text: A 5.042434 ms -> B' 2.066177 ms, -2.976257 ms (-59.0%); A2 4.990019 ms (-1.0%) - Decoder, text: A 97.684574 ms -> B' 96.215550 ms, -1.469024 ms (-1.5%); A2 97.623586 ms (-0.1%) - Decoder, image+text: A 98.423840 ms -> B' 96.149150 ms, -2.274690 ms (-2.3%); A2 98.220387 ms (-0.2%) The kernel saving is 2.9% and 3.0% of A's decoder GPU time. The decoder change is smaller than the kernel saving because the other decoder kernels, which this change does not touch, took 1.4% and 0.8% more time in the B' run (-0.1% and -0.2% in the A2 run). Validation: Outputs bitwise equal to the old kernel (whole output buffer plus an untouched guard region) in 177 cases: Qwen3-VL real positions (text, image, video, decode steps), layers 0 and 35, the FP8 out-of-place path, head_dim 64/128/256, plain RoPE, Gemma-style norm, YaRN, partial rotary, interleaved pairing, no QK norm and several head counts, at 8 to 8192 tokens. Also bitwise equal in 192 more cases with 2 to 72 heads per call (BF16 in place and FP8 out of place, 8 to 8192 tokens); in 51 of them the last warp of a token gets fewer heads. Existing tests on one GB200, before and after this change: test_fused_qk_norm_rope.py, test_fused_qk_norm_rope_gate.py and test_modeling_v2_fused_qk_norm_rope.py give 598 passed, 32 skipped, 0 failed on both, with the same result for every test id. Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
Problem: The fused QK norm + RoPE launcher gives one warp several heads of a token when a call is large, and one head per warp when it is small. Which path a test takes depends on its token count and on the GPU's SM count, and the existing tests compare against a reference with loose tolerances. A bug in the several-heads path could pass on some GPUs. Solution: Add test_fused_qk_norm_rope_large_call_matches_small_calls. It runs the op once on 4096 tokens (each warp gets half of the heads, rounded up, on any GPU with up to 512 SMs) and again on the same tokens in 4-token calls (one head per warp on any GPU with more than 10 SMs), and requires the two outputs to be bitwise equal. Cases: the Qwen3-VL-8B layout with interleaved mRoPE, NeoX and interleaved pairing, YaRN, partial rotary, head_dim 64 and 256, Gemma-style norm, no QK norm, 68 heads and 17 heads (16 Q + 1 KV with mRoPE, so the two warps of a token get 9 and 8 heads), each for the BF16 in-place op and the FP8 out-of-place op (20 cases). Performance: Test only. The 20 cases take 1.01 s in total on one GB200. Validation: One GB200, the three fused QK norm + RoPE test files: the new test passes on this PR's kernel and on the old kernel (618 passed, 32 skipped on both), and fails in 20 of 20 cases on a deliberately broken kernel whose several-heads loop drops the next head's load. Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
|
No actionable comments were generated in the recent review. WalkthroughThe fused QK norm/RoPE kernel now groups consecutive heads per warp and reuses RoPE values across the group. The launcher selects the grouping and passes its dimensions to the kernel. New tests compare large BF16 and FP8 calls with concatenated smaller calls. ChangesFused QK Norm and RoPE
Estimated code review effort: 4 (Complex) | ~45 minutes Change: Refactor Sequence Diagram(s)sequenceDiagram
participant LaunchPath
participant fusedQKNormRopeHeadsPerWarp
participant fusedQKNormRopeKernel
participant PerHeadDeviceHelper
LaunchPath->>fusedQKNormRopeHeadsPerWarp: Select heads per warp
fusedQKNormRopeHeadsPerWarp-->>LaunchPath: Return heads per warp
LaunchPath->>fusedQKNormRopeKernel: Pass heads per warp and warps per token
fusedQKNormRopeKernel->>PerHeadDeviceHelper: Pass each head input and shared RoPE values
Suggested reviewers: Merge Risk:
|
|
@yuxianq, could you help to review this PR as trt-llm-torch-attention-devs? Thanks~ Assignment explanation
Affinity and workload are snapshots at selection, not live counters. Existing reservations on this PR are counted once. |
Dev Engineer Review
fusedQKNormRopeKernelnow processes multiple consecutive heads of one token per warp. The per-head work moves into a device helper, and RoPE cos/sin values are computed once per warp for those heads. The diff touchescpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cuonly (+192/-121 per the AI summary; the 2-file stat is 307 insertions and 121 deletions in total).fusedQKNormRopeHeadsPerWarp(line 916) selects the heads-per-warp value fromnumTokens,totalProcHeads, andsmCount. It starts at one head per warp and raises the value only when the warp-count limits hold. The launcher (lines 978–1011) passesheadsPerWarpandwarpsPerTokeninto all three template instantiations (64, 128, 256).headsPerWarp = 1path, but the selector body was not read in this review.test_fused_qk_norm_rope_gate.pyandtest_modeling_v2_fused_qk_norm_rope.py. Neither file is in the diff, which changes only two files. Those tests were not changed by this PR, or they are not part of it. The reported 618 passed / 32 skipped / 0 failed result is not established by the current head.QA Engineer Review
tests/unittest/_torch/attention/kernels/parallel_hw_agnostic/test_fused_qk_norm_rope.py(+115 lines).test_fused_qk_norm_rope_large_call_matches_small_calls(line 595). It is parameterized overcaseandout_fp8, covering BF16 in-place and FP8 out-of-place modes.test_fused_qk_norm_rope,test_fused_qk_norm_rope_gemma_mrope,test_fused_qk_norm_rope_to_fp8, andtest_fp8_producer_meta_keeps_dynamic_num_tokens) are not changed by this diff, per the raw summary.tests/integration/test_lists/test-db/l0_cpu.ymlat line 72. The new test is covered by that file-level entry. Per-test selectors were not checked.headsPerWarp > 1) for the parameterized shapes.Per-File QA Perspective
cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu: Source change. QA should verify outputs match the previous kernel across head counts, head dimensions, mRoPE, YaRN, and FP8 output. Also verify that small batches (decode) still run with one head per warp, and that the kernel runs on GPUs with different SM counts.tests/unittest/_torch/attention/kernels/parallel_hw_agnostic/test_fused_qk_norm_rope.py: New large-call bitwise test. Listed intests/integration/test_lists/test-db/l0_cpu.ymlthrough the file path. QA should confirm the test runs in the intended CI stage.Description
Why.
fusedQKNormRopeKernel(fused QK RMSNorm + RoPE, used by QK-norm attention layers such as Qwen3 and Qwen3-VL) runs one warp per (token, head), so every Q and K head of a token computes the same RoPE cos/sin again, and an 8K-token prefill launches 40960 blocks. On Qwen3-VL-8B this kernel is 5.0% of the decoder GPU time of an 8K-token prefill. With this change the kernel time drops by 59.3% to 59.5% at that size, outputs stay bit-identical, and decode-size calls do not get slower.Who calls it. On main, this kernel runs by default in the attention of Qwen3, Qwen3-MoE and Qwen3-VL (dense and MoE), Qwen2 with YaRN, GLM-4.5/4.6 with QK norm and MiniMax-M3, in the gpt-oss-120b, MiniMax-M3 and Qwen3.8-Flash-Next modeling_v2 targets, and in the DFlash drafter. Qwen3-Next and Qwen3.5 gated attention layers use a separate fused gate kernel; Gemma3, Gemma4, EXAONE4 and the visual_gen models do not call it.
Change. One warp now handles several consecutive heads of the same token. It computes cos/sin once and reuses it for each of its heads, and it loads the next head while working on the current one. The per-head work (load, RMSNorm, NeoX or interleaved pairing, rotation, store) moves unchanged into a device-inline helper, so the math per element is the same. The launcher picks the number of heads per warp as the largest divisor of the head count that still gives at least two warps per token and at least 16 warps per SM. If no divisor fits (for example 17 heads), it splits the heads as evenly as possible over the fewest warps that keep these limits, so the last warp of a token gets fewer heads. Small batches such as decode keep one head per warp and the same grid as before. This rule uses only the SM count; it was chosen in development sweeps on GB200 that included the Qwen3-VL-8B and synthetic shapes measured below. There is no new option, flag or config.
Performance. Standalone kernel screening on one GB200 (sm_100f) in untimed development runs on a shared node: CUPTI kernel duration, fresh inputs after an L2 flush, median of 30 launches per variant, three separate processes.
Only head_dim 64 with 32 Q + 8 K heads at 32 and 128 tokens is slower (-1.2% to +1.2% and +2.1%), inside the old kernel's own run-to-run spread in each process. At 8,192 tokens the grid goes from 40960 to 2048 blocks. From the 5.0% share, the expected decoder GPU time saving of an 8K-token prefill is 3.0%.
Other callers' head layouts, same method on one GB200 (three processes, kernel time change of the median; head counts and RoPE settings from each model's config.json, arange positions):
With 17 heads per GPU (the TP4 row) no divisor fits, so the launcher splits a token's heads as evenly as possible: 6 + 6 + 5 at 1,024 tokens and 9 + 8 at 8,192 tokens; decode sizes keep one head per warp. No cell in this table is slower. The same runs also timed these layouts at 64, 128, 250, 512, 1,158 and 2,048 tokens: only gpt-oss-120b at 64 tokens is slower (-0.6% to +1.1%), inside the old kernel's run-to-run spread.
Full Qwen3-VL-8B prefill of 8,192 tokens (one GB200, sums of GPU kernel times from Nsight Systems traces of untimed development runs on a shared node, not end-to-end latency): this kernel's GPU time changes by -57.6% for text and -59.0% for image+text, which is 2.9% and 3.0% of decoder GPU time. Decoder GPU time changes by -1.5% and -2.3%; a second run of the old build changes it by -0.1% and -0.2%. The decoder change is smaller than the kernel saving because the other decoder kernels, which this change does not touch, took 1.4% and 0.8% more time in the new-build run (their change in the second old-build run: -0.1% and -0.2%).
Test Coverage
test_fused_qk_norm_rope_large_call_matches_small_callsintest_fused_qk_norm_rope.py: it runs the op once on 4096 tokens (each warp gets half of the heads, rounded up, on any GPU with up to 512 SMs) and again on the same tokens in 4-token calls (one head per warp on any GPU with more than 10 SMs), and requires bitwise equal outputs. 20 cases: the Qwen3-VL-8B layout with interleaved mRoPE, NeoX and interleaved pairing, YaRN, partial rotary, head_dim 64 and 256, Gemma-style norm, no QK norm, 68 heads and 17 heads (16 Q + 1 KV with mRoPE, so the two warps of a token get 9 and 8 heads), each for the BF16 in-place op and the FP8 out-of-place op. On one GB200 it takes 1.01 s, passes with this change and with the old kernel, and fails in 20 of 20 cases with a deliberately broken kernel whose several-heads loop drops the next head's load.pytest tests/unittest/_torch/attention/kernels/parallel_hw_agnostic/test_fused_qk_norm_rope.py tests/unittest/_torch/attention/kernels/test_fused_qk_norm_rope_gate.py tests/unittest/_torch/modeling_v2/attention/test_modeling_v2_fused_qk_norm_rope.pygives 618 passed, 32 skipped, 0 failed on both, with the same result for every test id. By the launcher rule, on GB200 the 256-token cases oftest_fused_qk_norm_rope.pywith 16 to 64 Q heads run with several heads per warp. The 16 FP8 cases with 8 Q + 1 K + 1 V heads at 256 tokens and all cases with 32 or fewer tokens run with one head per warp.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.