Skip to content

[BUG]: constant-trip loops stay rolled at opt -O2, silently demoting loop-indexed register arrays (e.g. mma accumulators) to .local #1815

Open

Description

@Code4me2

Description

A constant-trip-count for n in 0..8 loop is not fully unrolled by cuda-oxide's default middle-end (opt -O2). As a result, a small register array indexed by the loop variable is demoted to .local memory. Nothing is reported unless the build runs with --verbose. This is not specific to tensor-core intrinsics: the same happens with plain f32 arithmetic in the loop body. It matters most for mma.sync accumulator arrays, where the natural loop form silently spills the accumulators.

The cause is LLVM's full-unroll cost threshold at -O2 (150). The same IR fully unrolls, and SROA then removes the alloca, at -O3 or at -O2 -unroll-threshold=250 or higher. #[unroll] on the loop is a working workaround.

Error / actual behavior

PTX per kernel, on current main (348895e), cargo oxide build --arch sm_121:

kernel mma.sync / body copies .local depot st.local / ld.local
fma_loop (plain loop, f32 math) loop kept, 1 body 128 B 36 / 12
mma_loop (plain loop, mma_m16n8k16_f32_bf16) 1 mma.sync in a loop 128 B 36 / 12
mma_loop_unroll_attr (same loop with #[unroll]) 8 mma.sync 0 0 / 0
hand-unrolled (constant index per copy) 8 mma.sync 0 0 / 0

The default build prints no diagnostic. With --verbose, the existing local-memory diagnostic finds it:

warning: local `acc` (`[[f32; 4]; 8]`, 128 bytes) in `mma_loop` could not be promoted to registers
  = note: indexed by a non-constant value; the allocation survives LLVM scalar replacement and remains stack-backed
  = help: use constant indexing or a small match/select when the index range is bounded

The kept *.ll shows a clean counted loop after opt -O2: one induction variable, icmp eq i64 %next, 9, and a body that loads and stores acc through getelementptr [16 x i8], ptr %acc, i64 %n. The only side exits are the slice bounds checks, which go to llvm.trap. Re-running the toolchain's own opt (LLVM 23.1.1, rust 1.99.1) on that IR:

opt invocation alloca left in fma_loop / mma_loop
-O2 -mcpu=sm_121 (what the pipeline runs) 1 / 1
-O2 -mcpu=sm_121 -unroll-threshold=200 1 / 1
-O2 -mcpu=sm_121 -unroll-threshold=250 0 / 0
-O2 -mcpu=sm_121 -unroll-threshold=300 (LLVM's -O3 value) 0 / 0
-O3 0 / 0

Expected behavior

A short constant-trip loop over a small register array compiles to registers, as it does with -O3 (nvcc and clang compile device code at O3 by default). At the very least, the default build should tell the user that a local stayed in memory.

Minimal reproduction example

use cuda_device::wmma::mma_m16n8k16_f32_bf16;
use cuda_device::{DisjointSlice, cuda_module, kernel, thread};

#[cuda_module]
mod kernels {
    use super::*;

    // Plain loop, plain f32 math: acc stays in .local.
    #[kernel]
    pub fn fma_loop(a: &[u32], b: &[u32], mut out: DisjointSlice<f32>) {
        let lane = thread::threadIdx_x() as usize;
        let av = [a[lane * 4], a[lane * 4 + 1], a[lane * 4 + 2], a[lane * 4 + 3]];
        let mut acc = [[0f32; 4]; 8];
        for n in 0..8 {
            let bv = [b[(n * 32 + lane) * 2], b[(n * 32 + lane) * 2 + 1]];
            let x = f32::from_bits(av[0] ^ bv[0]);
            let y = f32::from_bits(av[1] ^ bv[1]);
            acc[n] = [acc[n][0] * x + y, acc[n][1] * y + x, acc[n][2] + x * y, acc[n][3] - x * y];
        }
        let s = (acc[0][0] + acc[1][1]) + (acc[2][2] + acc[3][3]) + (acc[4][0] + acc[5][1]) + (acc[6][2] + acc[7][3]);
        if let Some(o) = out.get_mut(thread::index_1d()) { *o = s; }
    }

    // The mma.sync version of the same loop: also stays in .local.
    #[kernel]
    pub fn mma_loop(a: &[u32], b: &[u32], mut out: DisjointSlice<f32>) {
        let lane = thread::threadIdx_x() as usize;
        let av = [a[lane * 4], a[lane * 4 + 1], a[lane * 4 + 2], a[lane * 4 + 3]];
        let mut acc = [[0f32; 4]; 8];
        for n in 0..8 {      // `#[unroll]` here gives 8 mma.sync and no .local
            let bv = [b[(n * 32 + lane) * 2], b[(n * 32 + lane) * 2 + 1]];
            acc[n] = unsafe { mma_m16n8k16_f32_bf16(acc[n], av, bv) };
        }
        let s = (acc[0][0] + acc[1][1]) + (acc[2][2] + acc[3][3]) + (acc[4][0] + acc[5][1]) + (acc[6][2] + acc[7][3]);
        if let Some(o) = out.get_mut(thread::index_1d()) { *o = s; }
    }
}

cargo oxide build --arch sm_121, then look at the two .entry bodies in the PTX (__local_depot, st.local, mma.sync count). Add --verbose to see the diagnostic above.

Possible fixes

Either or both. I'm happy to open a PR for whichever you prefer.

  1. Use LLVM's -O3 full-unroll threshold in the middle-end. Pass -unroll-threshold=300 in optimization_args (cuda-oxide-codegen/src/ptx.rs), or run default<O3>. This makes the natural loop form compile as intended. The cost is somewhat larger unrolled code in general. The #[nounroll] attribute already exists to opt out per loop.
  2. Make the local-memory diagnostic visible by default, at least for locals indexed by a loop induction variable with a constant trip count, and mention #[unroll] in its help text. Today it runs only with --verbose, and the help suggests constant indexing or match/select, not the attribute that fixes it.

Operating System

Ubuntu 24.04.5 LTS (aarch64)

GPU and GPU Architecture

NVIDIA GB10 (sm_121)

CUDA driver version

580.178.04

rustc version

rustc 1.99.1 (58cf8f9c3 2026-10-13), LLVM 23.1.1. cuda-rust main at 348895e, backend built from that tree by cargo oxide.

Activity

  1. Code4me2 commented on Nov 14, 2026

    @Code4me2
    ContributorAuthor

    Measurements and a question before I open a PR.

    Cost of the bug. The loop from the report sits inside a runtime-length K loop. On sm_121, with 8 mma.sync per K step:

    main -unroll-threshold=300
    plain for n in 0..8 240.7 ms 11.05 ms
    hand-unrolled 11.05 ms 11.21 ms

    That is 22× slower. ptxas doesn't recover the array either: the SASS keeps a 128-byte stack frame.

    Global threshold (-unroll-threshold=300, LLVM's -O3 value). The smoketest is unchanged at 256/257. 22 of the 991 kernels change: none gains local memory, and spill stores drop from 56 to 0. Total PTX grows by 3.9%. The changed kernels I can run here (tiled_gemm, hashmap_v3) are within noise. It works, but it changes every kernel, and a slightly larger loop body still spills silently.

    Targeted alternative. AMDGPU's getUnrollingPreferences (source) raises the threshold only for a loop that indexes a small static alloca (≤ 960 B) with a loop-varying value. NVPTX has no equivalent. The same rule could go in one of three places:

    1. cuda-oxide's MIR unroll pass. Treat an unannotated loop as #[unroll] when it has a constant trip count, indexes a small local array with the induction variable, and stays within the existing copy limits. This doesn't depend on LLVM's cost model, covers both the llc and libNVVM routes, and #[nounroll] still overrides it.
    2. llvm.loop.unroll.full metadata on such loops before opt. I haven't checked whether libNVVM honours it.
    3. Upstream: port the AMDGPU rule to NVPTX's TTI. This also fixes clang and rustc's nvptx64 target, but takes longest.

    Separately, showing the local-memory diagnostic without --verbose, and pointing its help at #[unroll], would catch the cases none of these do.

    Which would you accept? I'd suggest (1) plus the diagnostic. The threshold change is also ready as a stopgap. It includes an unroll_smoke regression kernel that fails on main and passes with the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions