Skip to content

perf(mir-lower): carry proved alignment through an array-element address - #777

Merged
nihalpasham merged 2 commits into
NVIDIA:mainfrom
vyncint:perf/array-element-alignment
Aug 10, 2026
Merged

nihalpasham merged 2 commits into
NVIDIA:mainfrom
vyncint:perf/array-element-alignment

Conversation

@vyncint

@vyncint vyncint commented Aug 10, 2026 •

Copy link
Copy Markdown
Contributor

Refs #745 — the array-element half you scoped in the #746 merge review.

#746 taught a field address to record what it proves, so c.x + c.y on an
#[repr(C, align(8))] element fuses into one ld.global.v2.b32. An element
address recorded nothing, so the same two f32 reached through an array index
still exported two align 4 loads.

One correction to the scope note

The note said c.0[0] + c.0[1] still emits two narrow loads. On current main
that exact spelling already fuses — I checked before building anything. The
local copy is what saves it: let c = pts[i] lets SROA load the whole 8-byte
aggregate, and the alignment comes from its type.

The shape that genuinely still failed is reading through the place:

let c = pts[i]; c.0[0] + c.0[1]      // already one wide load
pts[i].0[0] + pts[i].0[1]            // two narrow loads
let l = &pts[i].0; l[0] + l[1]       // two narrow loads   <- the table idiom

So the cost depended on whether the source copied the element or read through
it, which is not a distinction a programmer should have to know. The example
covers the reading-through form, since the copying form was never broken.

The arithmetic

Two facts combine. Whatever the base pointer already proved beats the array
type alone when the array is a field of an over-aligned aggregate —
&table[i].lanes carries the outer struct's alignment, which [f32; 2] does
not know. Element i then sits at byte i * stride:

index provable
constant 0 base
constant i gcd(base, i * stride)
runtime gcd(base, stride) — what every element satisfies

All from rustc's layout. Uncertain cases decline, and the same power-of-two
guard you added to the field stamp applies here.

PTX

kernel before after
lanes_through_ref 2 ld.global.b32 1 ld.global.v2.b32
hot_lanes 4 ld.global.b32 2 ld.global.v2.b32
packed_pair (align-4 control) 2 ld.global.b32 unchanged
hot_packed (align-4 control) 4 ld.global.b32 unchanged

Measured — A10G (sm_86, 595.71.05)

4194304 threads, 30 timed launches after 5 warmup, all bit-correct. Same example
built by both compilers:

kernel stock main this speedup
packed_pair 105.0 104.6 1.00x
aligned_pair (#746's) 104.9 104.8 1.00x
lanes_through_ref 104.9 105.0 1.00x
hot_packed (control) 1253.5 1253.5 1.00x
hot_aligned (#746's) 628.6 633.0 1.00x
hot_lanes 1253.6 629.5 1.99x

Evidence: stock .gpu-evidence/20260810T050351Z.log, this branch
20260810T045728Z.log.

Streaming does not move and should not — bandwidth-bound at ~480 GB/s, where two
consecutive loads already coalesce into the same transactions. hot_packed is
identical to the byte across both builds, so the comparison is controlled and
it guards against claiming alignment an align-4 base never had.

Gates

  • Full smoketest on the A10G: 196 / 196, 0 fail, 1224s
    (.gpu-evidence/20260810T050600Z.log).
  • Four mir-lower unit tests mirroring yours on the field path: element zero
    inherits the base, a nonzero constant index narrows to the gcd, a runtime
    index claims only the stride, and a base with no over-alignment changes
    nothing.
  • verify-code-shape.sh extended to both new kernels, so the fusion stays
    CI-checked without a GPU.
  • Locally: fmt (3 scopes), clippy -D warnings, 11 unit-test suites.

Neighbour

#750 also touches mir-lower/src/convert/ops/aggregate.rs (enum payload
storage) — different region, no overlap with the alignment machinery; whichever
lands first, the other rebases trivially.

Closes #745.

vyncint and others added 2 commits August 10, 2026 12:06
NVIDIA#746 taught a *field* address to record what it proves, so `c.x + c.y` on an
`#[repr(C, align(8))]` element fuses into one `ld.global.v2.b32`. An element
address recorded nothing, so the same two f32 reached through an array index
still exported two `align 4` loads and LoadStoreVectorizer refused to fuse them.

The failing shape is narrower than it first looks, which is worth stating because
it decides what the example has to cover. Copying the element into a local first
already fused -- SROA then loads the whole aggregate and the alignment comes from
its type. Only reading *through* the place keeps the access on the address path:

    let c = pts[i]; c.0[0] + c.0[1]      // already one wide load
    pts[i].0[0] + pts[i].0[1]            // two narrow loads
    let l = &pts[i].0; l[0] + l[1]       // two narrow loads

So the cost depended on whether the source copied the element or read through it,
which is not a distinction the programmer should have to know about.

Two facts combine at an element address. Whatever the base pointer already proved
is stronger than the array type alone when the array is a field of an
over-aligned aggregate -- `&table[i].lanes` carries the outer struct's alignment,
which `[f32; 2]` does not know. Element `i` then sits at byte `i * stride`, so it
proves `gcd(base, i * stride)`, and `base` itself at index zero. A runtime index
can land anywhere, so it claims only `gcd(base, stride)`, which every element
satisfies. All of it comes from rustc's layout; nothing is claimed that the
source did not already guarantee, and every uncertain case declines.

PTX, two adjacent f32 lanes of an `#[repr(C, align(8))]` element:

    kernel               before                 after
    lanes_through_ref    2 ld.global.b32        1 ld.global.v2.b32
    hot_lanes            4 ld.global.b32        2 ld.global.v2.b32
    packed_pair          2 ld.global.b32        2 ld.global.b32     (align-4 control)
    hot_packed           4 ld.global.b32        4 ld.global.b32     (align-4 control)

Measured on an A10G (sm_86, driver 595.71.05), 4194304 threads, 30 timed launches
after 5 warmup, every kernel bit-correct against a host reference:

    kernel               stock      this   speedup
    packed_pair          105.0     104.6     1.00x
    aligned_pair         104.9     104.8     1.00x
    lanes_through_ref    104.9     105.0     1.00x
    hot_packed          1253.5    1253.5     1.00x
    hot_aligned          628.6     633.0     1.00x
    hot_lanes           1253.6     629.5     1.99x

As with NVIDIA#746, streaming access does not move and should not: at ~480 GB/s it is
bandwidth-bound, and two loads from consecutive addresses already coalesce into
the same transactions. The win is where load count is the limit -- a 4096-entry
table with 64 dependent lookups per thread. `hot_packed` is identical to the
byte across both builds, so the comparison is controlled, and it is also the
guard that this claims no alignment an align-4 base never had.

Refs NVIDIA#745

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
The element-address alignment claim multiplied the index by
get_type_size(element_ty), an LLVM-level approximation that answers a
guessed 8 for the MIR dialect element types it does not model:
MirStructType, MirTupleType, MirEnumType, nested MirArrayType, and
also MirFP16Type, the importer's f16 scalar. For [[f32; 3]; 4] under
an align-8 base the real stride is 12, so a runtime element address is
only 4-aligned, but the guess claimed 8; the inner index-0 address
inherits that stamp, the scalar f32 load exports align 8, and
LoadStoreVectorizer then legally fuses two lanes into a v2.b32 that
faults at runtime for odd indexes. An [f16; N] array had the same
over-claim (guessed stride 8, real 2).

Strides now come from mir_element_stride: exact scalar sizes
(including MirFP16Type at 2 bytes), generic-pointer 8, rustc's stored
total_size (padding included) for MIR aggregates via mir_stored_size,
arrays multiplied out with checked_mul on both the aggregate and
scalar recursion paths, and None for anything else so the claim
declines instead of guessing. mir_stored_size learns the MirTupleType
arm it was missing (tuples record total_size like structs do), which
also sharpens the struct-slot-map field advance that already
preferred it.

constant_index_value additionally fails closed on constants wider
than 64 bits, matching integer_constant_u64 in the extract-element
fast path: APInt::to_u64 truncates, and a truncated index would feed
the gcd a wrong byte offset.

The five new lowering tests each fail on the previous stride sourcing
and pass now: nested [f32; 3] elements claim gcd(8, 12) = 4 for
runtime and index-1 addresses (the miscompile chain above), constant
index 2 proves the full 8 again from the exact byte offset, tuple
strides come from recorded layout, f16 elements claim their exact
stride of 2, and an element with no recorded size claims nothing.
@nihalpasham

Copy link
Copy Markdown
Collaborator

The element-address claim and the gcd reasoning are good and the two kernels vectorize as advertised.

I layered a stride-sourcing fix on top:

  • get_type_size guesses 8 for MIR aggregate and f16 element types, which over-claimed alignment on nested arrays;
  • strides now come from rustc layout or the claim declines, with five regression tests.
  • Full sweep 196/196 on sm_120.

Merging,

closes #745.

@nihalpasham

Copy link
Copy Markdown
Collaborator

Thanks @vyncint, the element-address claim and the gcd reasoning are solid, and the two kernels vectorize as advertised. We layered a stride-sourcing fix on top: get_type_size guesses 8 for MIR aggregate and f16 element types, which over-claimed alignment on nested arrays; strides now come from rustc layout or the claim declines, with five regression tests. Full smoketest sweep 196/196 on sm_120. Merging, closes #745.

@nihalpasham nihalpasham added perf Performance of generated code or of the compiler itself IR-lowering Lowering between dialects (dialect-mir to LLVM dialect) labels Aug 10, 2026
@nihalpasham
nihalpasham merged commit 7825c51 into NVIDIA:main Aug 10, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

IR-lowering Lowering between dialects (dialect-mir to LLVM dialect) perf Performance of generated code or of the compiler itself

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reading a lane of an over-aligned aggregate drops its abi_align, costing 2x on cache-resident lookups

2 participants