Repository navigation
Description
Description
A constant-trip-count for n in 0..8 loop is not fully unrolled by cuda-oxide's default middle-end (opt -O2). As a result, a small register array indexed by the loop variable is demoted to .local memory. Nothing is reported unless the build runs with --verbose. This is not specific to tensor-core intrinsics: the same happens with plain f32 arithmetic in the loop body. It matters most for mma.sync accumulator arrays, where the natural loop form silently spills the accumulators.
The cause is LLVM's full-unroll cost threshold at -O2 (150). The same IR fully unrolls, and SROA then removes the alloca, at -O3 or at -O2 -unroll-threshold=250 or higher. #[unroll] on the loop is a working workaround.
Error / actual behavior
PTX per kernel, on current main (348895e), cargo oxide build --arch sm_121:
| kernel | mma.sync / body copies |
.local depot |
st.local / ld.local |
|---|---|---|---|
fma_loop (plain loop, f32 math) |
loop kept, 1 body | 128 B | 36 / 12 |
mma_loop (plain loop, mma_m16n8k16_f32_bf16) |
1 mma.sync in a loop |
128 B | 36 / 12 |
mma_loop_unroll_attr (same loop with #[unroll]) |
8 mma.sync |
0 | 0 / 0 |
| hand-unrolled (constant index per copy) | 8 mma.sync |
0 | 0 / 0 |
The default build prints no diagnostic. With --verbose, the existing local-memory diagnostic finds it:
warning: local `acc` (`[[f32; 4]; 8]`, 128 bytes) in `mma_loop` could not be promoted to registers
= note: indexed by a non-constant value; the allocation survives LLVM scalar replacement and remains stack-backed
= help: use constant indexing or a small match/select when the index range is bounded
The kept *.ll shows a clean counted loop after opt -O2: one induction variable, icmp eq i64 %next, 9, and a body that loads and stores acc through getelementptr [16 x i8], ptr %acc, i64 %n. The only side exits are the slice bounds checks, which go to llvm.trap. Re-running the toolchain's own opt (LLVM 23.1.1, rust 1.99.1) on that IR:
opt invocation |
alloca left in fma_loop / mma_loop |
|---|---|
-O2 -mcpu=sm_121 (what the pipeline runs) |
1 / 1 |
-O2 -mcpu=sm_121 -unroll-threshold=200 |
1 / 1 |
-O2 -mcpu=sm_121 -unroll-threshold=250 |
0 / 0 |
-O2 -mcpu=sm_121 -unroll-threshold=300 (LLVM's -O3 value) |
0 / 0 |
-O3 |
0 / 0 |
Expected behavior
A short constant-trip loop over a small register array compiles to registers, as it does with -O3 (nvcc and clang compile device code at O3 by default). At the very least, the default build should tell the user that a local stayed in memory.
Minimal reproduction example
use cuda_device::wmma::mma_m16n8k16_f32_bf16;
use cuda_device::{DisjointSlice, cuda_module, kernel, thread};
#[cuda_module]
mod kernels {
use super::*;
// Plain loop, plain f32 math: acc stays in .local.
#[kernel]
pub fn fma_loop(a: &[u32], b: &[u32], mut out: DisjointSlice<f32>) {
let lane = thread::threadIdx_x() as usize;
let av = [a[lane * 4], a[lane * 4 + 1], a[lane * 4 + 2], a[lane * 4 + 3]];
let mut acc = [[0f32; 4]; 8];
for n in 0..8 {
let bv = [b[(n * 32 + lane) * 2], b[(n * 32 + lane) * 2 + 1]];
let x = f32::from_bits(av[0] ^ bv[0]);
let y = f32::from_bits(av[1] ^ bv[1]);
acc[n] = [acc[n][0] * x + y, acc[n][1] * y + x, acc[n][2] + x * y, acc[n][3] - x * y];
}
let s = (acc[0][0] + acc[1][1]) + (acc[2][2] + acc[3][3]) + (acc[4][0] + acc[5][1]) + (acc[6][2] + acc[7][3]);
if let Some(o) = out.get_mut(thread::index_1d()) { *o = s; }
}
// The mma.sync version of the same loop: also stays in .local.
#[kernel]
pub fn mma_loop(a: &[u32], b: &[u32], mut out: DisjointSlice<f32>) {
let lane = thread::threadIdx_x() as usize;
let av = [a[lane * 4], a[lane * 4 + 1], a[lane * 4 + 2], a[lane * 4 + 3]];
let mut acc = [[0f32; 4]; 8];
for n in 0..8 { // `#[unroll]` here gives 8 mma.sync and no .local
let bv = [b[(n * 32 + lane) * 2], b[(n * 32 + lane) * 2 + 1]];
acc[n] = unsafe { mma_m16n8k16_f32_bf16(acc[n], av, bv) };
}
let s = (acc[0][0] + acc[1][1]) + (acc[2][2] + acc[3][3]) + (acc[4][0] + acc[5][1]) + (acc[6][2] + acc[7][3]);
if let Some(o) = out.get_mut(thread::index_1d()) { *o = s; }
}
}cargo oxide build --arch sm_121, then look at the two .entry bodies in the PTX (__local_depot, st.local, mma.sync count). Add --verbose to see the diagnostic above.
Possible fixes
Either or both. I'm happy to open a PR for whichever you prefer.
- Use LLVM's
-O3full-unroll threshold in the middle-end. Pass-unroll-threshold=300inoptimization_args(cuda-oxide-codegen/src/ptx.rs), or rundefault<O3>. This makes the natural loop form compile as intended. The cost is somewhat larger unrolled code in general. The#[nounroll]attribute already exists to opt out per loop. - Make the local-memory diagnostic visible by default, at least for locals indexed by a loop induction variable with a constant trip count, and mention
#[unroll]in its help text. Today it runs only with--verbose, and the help suggests constant indexing or match/select, not the attribute that fixes it.
Operating System
Ubuntu 24.04.5 LTS (aarch64)
GPU and GPU Architecture
NVIDIA GB10 (sm_121)
CUDA driver version
580.178.04
rustc version
rustc 1.99.1 (58cf8f9c3 2026-10-13), LLVM 23.1.1. cuda-rust main at 348895e, backend built from that tree by cargo oxide.
Activity
Code4me2 commentedon Nov 14, 2026
Measurements and a question before I open a PR.
Cost of the bug. The loop from the report sits inside a runtime-length K loop. On sm_121, with 8
mma.syncper K step:main-unroll-threshold=300for n in 0..8That is 22× slower. ptxas doesn't recover the array either: the SASS keeps a 128-byte stack frame.
Global threshold (
-unroll-threshold=300, LLVM's-O3value). The smoketest is unchanged at 256/257. 22 of the 991 kernels change: none gains local memory, and spill stores drop from 56 to 0. Total PTX grows by 3.9%. The changed kernels I can run here (tiled_gemm,hashmap_v3) are within noise. It works, but it changes every kernel, and a slightly larger loop body still spills silently.Targeted alternative. AMDGPU's
getUnrollingPreferences(source) raises the threshold only for a loop that indexes a small staticalloca(≤ 960 B) with a loop-varying value. NVPTX has no equivalent. The same rule could go in one of three places:#[unroll]when it has a constant trip count, indexes a small local array with the induction variable, and stays within the existing copy limits. This doesn't depend on LLVM's cost model, covers both the llc and libNVVM routes, and#[nounroll]still overrides it.llvm.loop.unroll.fullmetadata on such loops beforeopt. I haven't checked whether libNVVM honours it.nvptx64target, but takes longest.Separately, showing the local-memory diagnostic without
--verbose, and pointing its help at#[unroll], would catch the cases none of these do.Which would you accept? I'd suggest (1) plus the diagnostic. The threshold change is also ready as a stopgap. It includes an
unroll_smokeregression kernel that fails onmainand passes with the fix.