Skip to content

[None][perf] Cut fused QK norm + RoPE kernel time by up to 64% on long prefills by sharing RoPE cos/sin across heads - #22429

Open
yechank-nvidia wants to merge 2 commits into
NVIDIA:mainfrom
yechank-nvidia:yechank/qknorm-rope-share-cos-sin
Open

yechank-nvidia wants to merge 2 commits into
NVIDIA:mainfrom
yechank-nvidia:yechank/qknorm-rope-share-cos-sin

Conversation

@yechank-nvidia

@yechank-nvidia yechank-nvidia commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Dev Engineer Review

  • fusedQKNormRopeKernel now processes multiple consecutive heads of one token per warp. The per-head work moves into a device helper, and RoPE cos/sin values are computed once per warp for those heads. The diff touches cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu only (+192/-121 per the AI summary; the 2-file stat is 307 insertions and 121 deletions in total).
  • The new host helper fusedQKNormRopeHeadsPerWarp (line 916) selects the heads-per-warp value from numTokens, totalProcHeads, and smCount. It starts at one head per warp and raises the value only when the warp-count limits hold. The launcher (lines 978–1011) passes headsPerWarp and warpsPerToken into all three template instantiations (64, 128, 256).
  • Selector behavior with small batches: the AI summary says decode keeps one head per warp. This matches the default headsPerWarp = 1 path, but the selector body was not read in this review.
  • No public API or declaration changes were reported.
  • Performance claims (59.3%–59.5% kernel-time reduction on GB200 for the Qwen3-VL-8B 8,192-token case; 1.5% and 2.3% decoder GPU-time reductions) come from the PR objectives. They were not verified in this review. Only GB200 was measured. Wall time and TTFT were not measured.
  • The PR objectives name test_fused_qk_norm_rope_gate.py and test_modeling_v2_fused_qk_norm_rope.py. Neither file is in the diff, which changes only two files. Those tests were not changed by this PR, or they are not part of it. The reported 618 passed / 32 skipped / 0 failed result is not established by the current head.

QA Engineer Review

  • Test file changed: tests/unittest/_torch/attention/kernels/parallel_hw_agnostic/test_fused_qk_norm_rope.py (+115 lines).
  • New test: test_fused_qk_norm_rope_large_call_matches_small_calls (line 595). It is parameterized over case and out_fp8, covering BF16 in-place and FP8 out-of-place modes.
  • Coverage per the AI summary: mRoPE with distinct temporal, height, and width coordinates; YaRN; partial rotary dimensions; varied head dimensions; disabled QK normalization; uneven head counts. For 4096 tokens, a large call is compared bitwise with concatenated four-token calls.
  • Existing tests in the same file (test_fused_qk_norm_rope, test_fused_qk_norm_rope_gemma_mrope, test_fused_qk_norm_rope_to_fp8, and test_fp8_producer_meta_keeps_dynamic_num_tokens) are not changed by this diff, per the raw summary.
  • Test listing: the file is referenced in tests/integration/test_lists/test-db/l0_cpu.yml at line 72. The new test is covered by that file-level entry. Per-test selectors were not checked.
  • Not established: whether the new test passes on the current head. No completed run result is available. The PR objective's pytest result (618 passed, 32 skipped, 0 failed) was reported by the author and was not verified here.
  • Coverage verdict: needs follow-up. The bitwise large-versus-small comparison covers the new multi-head path. Confirm the test runs and passes in CI, and confirm the large-call case hits the multi-head path (headsPerWarp > 1) for the parameterized shapes.

Per-File QA Perspective

  • cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu: Source change. QA should verify outputs match the previous kernel across head counts, head dimensions, mRoPE, YaRN, and FP8 output. Also verify that small batches (decode) still run with one head per warp, and that the kernel runs on GPUs with different SM counts.
  • tests/unittest/_torch/attention/kernels/parallel_hw_agnostic/test_fused_qk_norm_rope.py: New large-call bitwise test. Listed in tests/integration/test_lists/test-db/l0_cpu.yml through the file path. QA should confirm the test runs in the intended CI stage.

Description

Why. fusedQKNormRopeKernel (fused QK RMSNorm + RoPE, used by QK-norm attention layers such as Qwen3 and Qwen3-VL) runs one warp per (token, head), so every Q and K head of a token computes the same RoPE cos/sin again, and an 8K-token prefill launches 40960 blocks. On Qwen3-VL-8B this kernel is 5.0% of the decoder GPU time of an 8K-token prefill. With this change the kernel time drops by 59.3% to 59.5% at that size, outputs stay bit-identical, and decode-size calls do not get slower.

Who calls it. On main, this kernel runs by default in the attention of Qwen3, Qwen3-MoE and Qwen3-VL (dense and MoE), Qwen2 with YaRN, GLM-4.5/4.6 with QK norm and MiniMax-M3, in the gpt-oss-120b, MiniMax-M3 and Qwen3.8-Flash-Next modeling_v2 targets, and in the DFlash drafter. Qwen3-Next and Qwen3.5 gated attention layers use a separate fused gate kernel; Gemma3, Gemma4, EXAONE4 and the visual_gen models do not call it.

Change. One warp now handles several consecutive heads of the same token. It computes cos/sin once and reuses it for each of its heads, and it loads the next head while working on the current one. The per-head work (load, RMSNorm, NeoX or interleaved pairing, rotation, store) moves unchanged into a device-inline helper, so the math per element is the same. The launcher picks the number of heads per warp as the largest divisor of the head count that still gives at least two warps per token and at least 16 warps per SM. If no divisor fits (for example 17 heads), it splits the heads as evenly as possible over the fewest warps that keep these limits, so the last warp of a token gets fewer heads. Small batches such as decode keep one head per warp and the same grid as before. This rule uses only the SM count; it was chosen in development sweeps on GB200 that included the Qwen3-VL-8B and synthetic shapes measured below. There is no new option, flag or config.

Performance. Standalone kernel screening on one GB200 (sm_100f) in untimed development runs on a shared node: CUPTI kernel duration, fresh inputs after an L2 flush, median of 30 launches per variant, three separate processes.

Case Kernel time change
Qwen3-VL-8B decoder call (32 Q + 8 K heads, head_dim 128, mRoPE), 8,192 tokens -59.5% to -59.3%
Same call, 1,158 tokens -47.2% to -47.1%
Same call, 8 and 32 tokens (decode, same grid) -4.5% to -4.3%
Synthetic head layouts (arange positions), head_dim 64/128/256, 16 to 64 Q heads, 8 to 8,192 tokens -62.9% to +2.1%

Only head_dim 64 with 32 Q + 8 K heads at 32 and 128 tokens is slower (-1.2% to +1.2% and +2.1%), inside the old kernel's own run-to-run spread in each process. At 8,192 tokens the grid goes from 40960 to 2048 blocks. From the 5.0% share, the expected decoder GPU time saving of an 8K-token prefill is 3.0%.

Other callers' head layouts, same method on one GB200 (three processes, kernel time change of the median; head counts and RoPE settings from each model's config.json, arange positions):

Layout (heads per GPU) 8 tokens 32 tokens 1,024 tokens 8,192 tokens
Qwen3-VL-235B-A22B, TP1 or attention DP: 64 Q + 4 KV heads, head_dim 128, mRoPE -5.3% to -4.3% -6.1% -46.6% to -46.3% -61.8% to -61.6%
Same model, TP4: 16 Q + 1 KV heads (17 heads) -4.5% to -3.4% -4.8% to -3.2% -32.4% to -31.9% -55.1%
Qwen3-30B-A3B: 32 Q + 4 KV heads, head_dim 128 -9.1% to -6.9% -6.7% to -4.5% -34.6% to -34.1% -53.8%
gpt-oss-120b: 64 Q + 8 KV heads, head_dim 64, YaRN, no QK norm -4.1% to -3.4% -3.8% to -3.2% -41.1% to -40.8% -64.4% to -64.2%
Qwen3.8-Flash-Next: 24 Q + 2 KV heads, head_dim 256, rotary 64, Gemma norm, mRoPE -10.9% to -10.5% -12.1% to -11.7% -36.3% to -36.1% -63.7%
MiniMax-M3, TP8: 8 Q + 1 KV heads, head_dim 128, rotary 64, Gemma norm, FP8 output -13.9% to -11.0% -13.0% to -12.0% -22.9% to -22.5% -46.3%

With 17 heads per GPU (the TP4 row) no divisor fits, so the launcher splits a token's heads as evenly as possible: 6 + 6 + 5 at 1,024 tokens and 9 + 8 at 8,192 tokens; decode sizes keep one head per warp. No cell in this table is slower. The same runs also timed these layouts at 64, 128, 250, 512, 1,158 and 2,048 tokens: only gpt-oss-120b at 64 tokens is slower (-0.6% to +1.1%), inside the old kernel's run-to-run spread.

Full Qwen3-VL-8B prefill of 8,192 tokens (one GB200, sums of GPU kernel times from Nsight Systems traces of untimed development runs on a shared node, not end-to-end latency): this kernel's GPU time changes by -57.6% for text and -59.0% for image+text, which is 2.9% and 3.0% of decoder GPU time. Decoder GPU time changes by -1.5% and -2.3%; a second run of the old build changes it by -0.1% and -0.2%. The decoder change is smaller than the kernel saving because the other decoder kernels, which this change does not touch, took 1.4% and 0.8% more time in the new-build run (their change in the second old-build run: -0.1% and -0.2%).

Test Coverage

  • New test test_fused_qk_norm_rope_large_call_matches_small_calls in test_fused_qk_norm_rope.py: it runs the op once on 4096 tokens (each warp gets half of the heads, rounded up, on any GPU with up to 512 SMs) and again on the same tokens in 4-token calls (one head per warp on any GPU with more than 10 SMs), and requires bitwise equal outputs. 20 cases: the Qwen3-VL-8B layout with interleaved mRoPE, NeoX and interleaved pairing, YaRN, partial rotary, head_dim 64 and 256, Gemma-style norm, no QK norm, 68 heads and 17 heads (16 Q + 1 KV with mRoPE, so the two warps of a token get 9 and 8 heads), each for the BF16 in-place op and the FP8 out-of-place op. On one GB200 it takes 1.01 s, passes with this change and with the old kernel, and fails in 20 of 20 cases with a deliberately broken kernel whose several-heads loop drops the next head's load.
  • Bit checks against the old kernel: the whole output buffer plus an untouched guard region is bitwise equal in 177 cases: Qwen3-VL real positions (text, image, video, decode steps), layers 0 and 35, the FP8 out-of-place path, head_dim 64/128/256, plain RoPE, Gemma-style norm, YaRN, partial rotary, interleaved pairing, no QK norm and several head counts, at 8 to 8,192 tokens. Another 192 cases with 2 to 72 heads per call (BF16 in place and FP8 out of place, 8 to 8,192 tokens) are also bitwise equal; in 51 of them the last warp of a token gets fewer heads.
  • Existing unit tests and the new test, one GB200, old kernel and this change: pytest tests/unittest/_torch/attention/kernels/parallel_hw_agnostic/test_fused_qk_norm_rope.py tests/unittest/_torch/attention/kernels/test_fused_qk_norm_rope_gate.py tests/unittest/_torch/modeling_v2/attention/test_modeling_v2_fused_qk_norm_rope.py gives 618 passed, 32 skipped, 0 failed on both, with the same result for every test id. By the launcher rule, on GB200 the 256-token cases of test_fused_qk_norm_rope.py with 16 to 64 Q heads run with several heads per warp. The 16 FP8 cases with 8 Q + 1 K + 1 V heads at 256 tokens and all cases with 32 or fewer tokens run with one head per warp.
  • Qwen3-VL-8B end-to-end runs above: the greedy output tokens of the old and new builds are the same.
  • Timing was measured on GB200 only.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Problem:
fusedQKNormRopeKernel runs one warp per (token, head). Every Q and K
head of a token recomputes the same RoPE cos/sin (frequency, YaRN ramp,
position load, sincos), and a large prefill launches one warp per head
(40960 blocks for 8192 tokens with 32 Q + 8 K heads). In a diagnostic
build that replaces the angle math and position loads with constants,
the kernel time drops by 51.3% at that size.

Solution:
Let one warp process several consecutive heads of one token. The
per-head work (load, RMSNorm warp reduction, NeoX or interleaved
pairing, rotation, store) moves unchanged into a device-inline helper.
The kernel computes each thread's cos/sin once per warp while the first
head is loading, prefetches the next head, and calls the helper for each
of its heads. The launcher picks the heads per warp as the largest
divisor of the processed head count that keeps at least two warps per
token and at least 16 warps per SM. If no divisor fits (for example 17
heads), it splits the heads as evenly as possible over the fewest warps
that keep these limits, and the last warp of a token gets fewer heads.
Small batches such as decode keep one head per warp and the same grid.
This rule uses only the SM count; it was chosen in development sweeps on
GB200 that included the shapes measured below. Same arithmetic per
element. No new flag or env.

Performance:
Standalone kernel screening on one GB200 (sm_100f; shared node, untimed
development runs): BF16 QKV, CUPTI kernel duration, fresh QKV after an
L2 flush, median of 30 launches per variant in each of three separate
processes. Kernel time change, then old -> new kernel time (change) in
processes 1, 2 and 3:
- Qwen3-VL-8B decoder call (32 Q + 8 K heads, head_dim 128, interleaved
  mRoPE), 8192 tokens, 40960 -> 2048 blocks: -59.5% to -59.3%; 122.6405
  -> 49.9365 us (-72.704 us), 122.6725 -> 49.824 us (-72.8485 us),
  122.576 -> 49.696 us (-72.880 us).
- Same call, 1158 tokens: -47.2% to -47.1%; 19.184 -> 10.128 us (-9.056
  us), 19.168 -> 10.144 us (-9.024 us), 19.200 -> 10.128 us (-9.072 us).
- Same call, 8 and 32 tokens (decode, same grid): -4.5% to -4.3%; 8
  tokens: 2.816 -> 2.688 us (-0.128 us), 2.816 -> 2.688 us (-0.128 us),
  2.816 -> 2.688 us (-0.128 us); 32 tokens: 3.008 -> 2.880 us (-0.128
  us), 3.008 -> 2.880 us (-0.128 us), 3.008 -> 2.880 us (-0.128 us).
- Synthetic head layouts (arange positions, head_dim 64/128/256, 16 to
  64 Q heads) at 8 to 8192 tokens: -62.9% to +2.1%; the largest drop is
  head_dim 256 with 16 Q + 8 K heads at 8192 tokens (143.968 -> 53.392
  us in process 2). Only head_dim 64 with 32 Q + 8 K heads at 32 and 128
  tokens is slower (-1.2% to +1.2% and +2.1%; 128 tokens: 3.104 -> 3.168
  us (+0.064 us), 3.104 -> 3.168 us (+0.064 us), 3.104 -> 3.168 us
  (+0.064 us)), inside the old kernel's own run-to-run spread in each
  process.
- Qwen3-VL-235B-A22B at TP4 (16 Q + 1 K heads: 17 heads, no divisor
  fits, 9 + 8 heads per warp), 8192 tokens, 17408 -> 2048 blocks:
  -55.1%; 52.640 -> 23.6475 us (-28.9925 us), 52.672 -> 23.648 us
  (-29.024 us), 52.608 -> 23.616 us (-28.992 us).
On an 8192-token Qwen3-VL-8B prefill trace this kernel is 5.0% of
decoder GPU time, so the predicted decoder saving is 3.0%. Timing was
measured on GB200 only.
Full Qwen3-VL-8B prefill of 8192 tokens (BF16 model, TP1, one GB200,
eager mode, 1 output token): one untimed development run per build on a
shared node (old build A, new build B', and a second old-build run A2),
sums of GPU kernel times of the prefill forward from Nsight Systems
traces, change against A. Wall time and TTFT were not measured.
- This kernel, text: A 4.839521 ms -> B' 2.050050 ms, -2.789471 ms
  (-57.6%); A2 4.868547 ms (+0.6%)
- This kernel, image+text: A 5.042434 ms -> B' 2.066177 ms, -2.976257 ms
  (-59.0%); A2 4.990019 ms (-1.0%)
- Decoder, text: A 97.684574 ms -> B' 96.215550 ms, -1.469024 ms
  (-1.5%); A2 97.623586 ms (-0.1%)
- Decoder, image+text: A 98.423840 ms -> B' 96.149150 ms, -2.274690 ms
  (-2.3%); A2 98.220387 ms (-0.2%)
The kernel saving is 2.9% and 3.0% of A's decoder GPU time. The decoder
change is smaller than the kernel saving because the other decoder
kernels, which this change does not touch, took 1.4% and 0.8% more time
in the B' run (-0.1% and -0.2% in the A2 run).

Validation:
Outputs bitwise equal to the old kernel (whole output buffer plus an
untouched guard region) in 177 cases: Qwen3-VL real positions (text,
image, video, decode steps), layers 0 and 35, the FP8 out-of-place path,
head_dim 64/128/256, plain RoPE, Gemma-style norm, YaRN, partial rotary,
interleaved pairing, no QK norm and several head counts, at 8 to 8192
tokens. Also bitwise equal in 192 more cases with 2 to 72 heads per call
(BF16 in place and FP8 out of place, 8 to 8192 tokens); in 51 of them
the last warp of a token gets fewer heads.
Existing tests on one GB200, before and after this change:
test_fused_qk_norm_rope.py, test_fused_qk_norm_rope_gate.py and
test_modeling_v2_fused_qk_norm_rope.py give 598 passed, 32 skipped, 0
failed on both, with the same result for every test id.

Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
Problem:
The fused QK norm + RoPE launcher gives one warp several heads of a
token when a call is large, and one head per warp when it is small.
Which path a test takes depends on its token count and on the GPU's SM
count, and the existing tests compare against a reference with loose
tolerances. A bug in the several-heads path could pass on some GPUs.

Solution:
Add test_fused_qk_norm_rope_large_call_matches_small_calls. It runs the
op once on 4096 tokens (each warp gets half of the heads, rounded up, on
any GPU with up to 512 SMs) and again on the same tokens in 4-token
calls (one head per warp on any GPU with more than 10 SMs), and requires
the two outputs to be bitwise equal. Cases: the Qwen3-VL-8B layout with
interleaved mRoPE, NeoX and interleaved pairing, YaRN, partial rotary,
head_dim 64 and 256, Gemma-style norm, no QK norm, 68 heads and 17 heads
(16 Q + 1 KV with mRoPE, so the two warps of a token get 9 and 8 heads),
each for the BF16 in-place op and the FP8 out-of-place op (20 cases).

Performance:
Test only. The 20 cases take 1.01 s in total on one GB200.

Validation:
One GB200, the three fused QK norm + RoPE test files: the new test
passes on this PR's kernel and on the old kernel (618 passed, 32 skipped
on both), and fails in 20 of 20 cases on a deliberately broken kernel
whose several-heads loop drops the next head's load.

Signed-off-by: Yechan Kim <161688079+yechank-nvidia@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The fused QK norm/RoPE kernel now groups consecutive heads per warp and reuses RoPE values across the group. The launcher selects the grouping and passes its dimensions to the kernel. New tests compare large BF16 and FP8 calls with concatenated smaller calls.

Changes

Fused QK Norm and RoPE

Layer / File(s) Summary
Per-head processing and grouped kernel
cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu
A per-head device helper handles head-level normalization, RoPE pairing and application, and output storage. The kernel groups consecutive heads per warp and reuses RoPE values across them.
Head grouping and launch configuration
cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu
A selector chooses heads per warp. The launcher computes warps per token and passes both scheduling values to the kernel.
Large-call equivalence tests
tests/unittest/_torch/attention/kernels/parallel_hw_agnostic/test_fused_qk_norm_rope.py
Parameterized BF16 in-place and FP8 out-of-place tests compare 4096-token results with concatenated four-token results. Cases include mRoPE, YaRN, partial rotary dimensions, and uneven head counts.

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant LaunchPath
  participant fusedQKNormRopeHeadsPerWarp
  participant fusedQKNormRopeKernel
  participant PerHeadDeviceHelper
  LaunchPath->>fusedQKNormRopeHeadsPerWarp: Select heads per warp
  fusedQKNormRopeHeadsPerWarp-->>LaunchPath: Return heads per warp
  LaunchPath->>fusedQKNormRopeKernel: Pass heads per warp and warps per token
  fusedQKNormRopeKernel->>PerHeadDeviceHelper: Pass each head input and shared RoPE values
Loading

Suggested reviewers: schetlur-nv, brb-nv

Merge Risk: Minimal · up to 50b76

The change regroups heads per warp in the fused QK norm and RoPE kernel to cut prefill time, with no API or configuration change. New bitwise-equivalence tests cover both output modes across the listed configurations. No concrete defect was found, so the change looks ready to merge.

Security architecture risk: Minimal · up to 50b76

The optimization preserves the existing operation interfaces and buffer ownership. No introduced or worsened security or failure-containment risk was identified in the inspected change.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: the fused QK norm + RoPE kernel now shares RoPE cos/sin across heads, with a reported kernel-time reduction. The raw summary confirms a warp processes several consecut…
Description check Passed The description covers the why, the change, performance data, and test coverage, so it is substantially complete against the template. It explains the rationale, the launcher head-grouping rule, and t…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: the fused QK norm + RoPE kernel now shares RoPE cos/sin across heads, with a reported kernel-time reduction. The raw summary confirms a warp processes several consecutive heads of one token and reuses cos/sin, so the title matches the changeset. It includes a [None][perf] prefix that follows the repository's title template.

Full details: Description check

Explanation

The description covers the why, the change, performance data, and test coverage, so it is substantially complete against the template. It explains the rationale, the launcher head-grouping rule, and the callers affected. It has a Description section and a Test Coverage section, and it includes a completed PR checklist. It does not follow the template's Description guidance to explain the issue and solution 'in short'; it is long, though it still reads clearly. The template's leading '@coderabbitai summary' line is present in the template but the author's text does not include it, which is non-critical.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 3705333 and 50b7665.

Files selected for processing: 2

  • cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu
  • tests/unittest/_torch/attention/kernels/parallel_hw_agnostic/test_fused_qk_norm_rope.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@yuxianq

yuxianq commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

@yuxianq, could you help to review this PR as trt-llm-torch-attention-devs? Thanks~

Assignment explanation
Componentskernels/auxiliary (85%); integration/attention-tests (15%)
Tagsfeatures:rope (50%); features:qk-norm (35%); features:quantization (15%); phase:prefill (80%); phase:decode (20%); representation:gqa (100%)
PR effort8 points — Reorganizes an existing fused operation for performance without introducing a model or public feature. Review requires deep reasoning about head coverage, warp synchronization, shared RoPE values, aliasing, and occupancy tradeoffs across output types. The omitted launch-selection implementation increases uncertainty.
ImportanceImportant — Sensitive CUDA correctness/performance change: new token/head assignment, sequential head processing, and prefetching must preserve in-place safety, Q/K/V handling, warp reductions, and numerical results.
Affinity (0–1)0.842 component/tag match; 0.882 after PIC preference
Outstanding points at selection66 → 74 (+8)
Points already reserved on this PR0
Effective weight1.3
Completed review points (preceding 168 hours)38
Combined points after assignment112
Demand-derived target81.90 points (soft share)
Additional fair share before assignment0.00 points
Share explanationWhole PR / qualification or coverage constraint exceeds a soft share; assignment is allowed.
Selection ruleaffinity with 0.3-share guard
Assignment scope (yuxianq)all

Affinity and workload are snapshots at selection, not live counters. Existing reservations on this PR are counted once.

@yuxianq
yuxianq requested review from yuxianq and removed request for PerkzZheng and farazkh80 November 17, 2026 03:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants