Repository navigation
perf(mir-lower): carry proved alignment through an array-element address - #777
Merged
Merged
Conversation
NVIDIA#746 taught a *field* address to record what it proves, so `c.x + c.y` on an `#[repr(C, align(8))]` element fuses into one `ld.global.v2.b32`. An element address recorded nothing, so the same two f32 reached through an array index still exported two `align 4` loads and LoadStoreVectorizer refused to fuse them. The failing shape is narrower than it first looks, which is worth stating because it decides what the example has to cover. Copying the element into a local first already fused -- SROA then loads the whole aggregate and the alignment comes from its type. Only reading *through* the place keeps the access on the address path: let c = pts[i]; c.0[0] + c.0[1] // already one wide load pts[i].0[0] + pts[i].0[1] // two narrow loads let l = &pts[i].0; l[0] + l[1] // two narrow loads So the cost depended on whether the source copied the element or read through it, which is not a distinction the programmer should have to know about. Two facts combine at an element address. Whatever the base pointer already proved is stronger than the array type alone when the array is a field of an over-aligned aggregate -- `&table[i].lanes` carries the outer struct's alignment, which `[f32; 2]` does not know. Element `i` then sits at byte `i * stride`, so it proves `gcd(base, i * stride)`, and `base` itself at index zero. A runtime index can land anywhere, so it claims only `gcd(base, stride)`, which every element satisfies. All of it comes from rustc's layout; nothing is claimed that the source did not already guarantee, and every uncertain case declines. PTX, two adjacent f32 lanes of an `#[repr(C, align(8))]` element: kernel before after lanes_through_ref 2 ld.global.b32 1 ld.global.v2.b32 hot_lanes 4 ld.global.b32 2 ld.global.v2.b32 packed_pair 2 ld.global.b32 2 ld.global.b32 (align-4 control) hot_packed 4 ld.global.b32 4 ld.global.b32 (align-4 control) Measured on an A10G (sm_86, driver 595.71.05), 4194304 threads, 30 timed launches after 5 warmup, every kernel bit-correct against a host reference: kernel stock this speedup packed_pair 105.0 104.6 1.00x aligned_pair 104.9 104.8 1.00x lanes_through_ref 104.9 105.0 1.00x hot_packed 1253.5 1253.5 1.00x hot_aligned 628.6 633.0 1.00x hot_lanes 1253.6 629.5 1.99x As with NVIDIA#746, streaming access does not move and should not: at ~480 GB/s it is bandwidth-bound, and two loads from consecutive addresses already coalesce into the same transactions. The win is where load count is the limit -- a 4096-entry table with 64 dependent lookups per thread. `hot_packed` is identical to the byte across both builds, so the comparison is controlled, and it is also the guard that this claims no alignment an align-4 base never had. Refs NVIDIA#745 Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
The element-address alignment claim multiplied the index by get_type_size(element_ty), an LLVM-level approximation that answers a guessed 8 for the MIR dialect element types it does not model: MirStructType, MirTupleType, MirEnumType, nested MirArrayType, and also MirFP16Type, the importer's f16 scalar. For [[f32; 3]; 4] under an align-8 base the real stride is 12, so a runtime element address is only 4-aligned, but the guess claimed 8; the inner index-0 address inherits that stamp, the scalar f32 load exports align 8, and LoadStoreVectorizer then legally fuses two lanes into a v2.b32 that faults at runtime for odd indexes. An [f16; N] array had the same over-claim (guessed stride 8, real 2). Strides now come from mir_element_stride: exact scalar sizes (including MirFP16Type at 2 bytes), generic-pointer 8, rustc's stored total_size (padding included) for MIR aggregates via mir_stored_size, arrays multiplied out with checked_mul on both the aggregate and scalar recursion paths, and None for anything else so the claim declines instead of guessing. mir_stored_size learns the MirTupleType arm it was missing (tuples record total_size like structs do), which also sharpens the struct-slot-map field advance that already preferred it. constant_index_value additionally fails closed on constants wider than 64 bits, matching integer_constant_u64 in the extract-element fast path: APInt::to_u64 truncates, and a truncated index would feed the gcd a wrong byte offset. The five new lowering tests each fail on the previous stride sourcing and pass now: nested [f32; 3] elements claim gcd(8, 12) = 4 for runtime and index-1 addresses (the miscompile chain above), constant index 2 proves the full 8 again from the exact byte offset, tuple strides come from recorded layout, f16 elements claim their exact stride of 2, and an element with no recorded size claims nothing.
Collaborator
|
The element-address claim and the gcd reasoning are good and the two kernels vectorize as advertised. I layered a stride-sourcing fix on top:
Merging, closes #745. |
Collaborator
|
Thanks @vyncint, the element-address claim and the gcd reasoning are solid, and the two kernels vectorize as advertised. We layered a stride-sourcing fix on top: get_type_size guesses 8 for MIR aggregate and f16 element types, which over-claimed alignment on nested arrays; strides now come from rustc layout or the claim declines, with five regression tests. Full smoketest sweep 196/196 on sm_120. Merging, closes #745. |
nihalpasham
approved these changes
Aug 10, 2026
This was referenced Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #745 — the array-element half you scoped in the #746 merge review.
#746 taught a field address to record what it proves, so
c.x + c.yon an#[repr(C, align(8))]element fuses into oneld.global.v2.b32. An elementaddress recorded nothing, so the same two f32 reached through an array index
still exported two
align 4loads.One correction to the scope note
The note said
c.0[0] + c.0[1]still emits two narrow loads. On currentmainthat exact spelling already fuses — I checked before building anything. The
local copy is what saves it:
let c = pts[i]lets SROA load the whole 8-byteaggregate, and the alignment comes from its type.
The shape that genuinely still failed is reading through the place:
So the cost depended on whether the source copied the element or read through
it, which is not a distinction a programmer should have to know. The example
covers the reading-through form, since the copying form was never broken.
The arithmetic
Two facts combine. Whatever the base pointer already proved beats the array
type alone when the array is a field of an over-aligned aggregate —
&table[i].lanescarries the outer struct's alignment, which[f32; 2]doesnot know. Element
ithen sits at bytei * stride:baseigcd(base, i * stride)gcd(base, stride)— what every element satisfiesAll from rustc's layout. Uncertain cases decline, and the same power-of-two
guard you added to the field stamp applies here.
PTX
lanes_through_refld.global.b32ld.global.v2.b32hot_lanesld.global.b32ld.global.v2.b32packed_pair(align-4 control)ld.global.b32hot_packed(align-4 control)ld.global.b32Measured — A10G (sm_86, 595.71.05)
4194304 threads, 30 timed launches after 5 warmup, all bit-correct. Same example
built by both compilers:
mainpacked_pairaligned_pair(#746's)lanes_through_refhot_packed(control)hot_aligned(#746's)hot_lanesEvidence: stock
.gpu-evidence/20260810T050351Z.log, this branch20260810T045728Z.log.Streaming does not move and should not — bandwidth-bound at ~480 GB/s, where two
consecutive loads already coalesce into the same transactions.
hot_packedisidentical to the byte across both builds, so the comparison is controlled and
it guards against claiming alignment an align-4 base never had.
Gates
(
.gpu-evidence/20260810T050600Z.log).mir-lowerunit tests mirroring yours on the field path: element zeroinherits the base, a nonzero constant index narrows to the gcd, a runtime
index claims only the stride, and a base with no over-alignment changes
nothing.
verify-code-shape.shextended to both new kernels, so the fusion staysCI-checked without a GPU.
-D warnings, 11 unit-test suites.Neighbour
#750 also touches
mir-lower/src/convert/ops/aggregate.rs(enum payloadstorage) — different region, no overlap with the alignment machinery; whichever
lands first, the other rebases trivially.
Closes #745.