Skip to content

perf(mir-lower): carry proved alignment through a field store - #795

Merged
nihalpasham merged 1 commit into
NVIDIA:mainfrom
vyncint:perf/store-proved-alignment
Aug 11, 2026
Merged

nihalpasham merged 1 commit into
NVIDIA:mainfrom
vyncint:perf/store-proved-alignment

Conversation

@vyncint

@vyncint vyncint commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

The store-side counterpart of #746 / #777. Those taught the load path to use
the alignment a field address had already proved; the store path never asked.

The defect

convert_load asks its result type for an ABI alignment and, when that answers
nothing, falls back to what the address itself proved when it was computed.
convert_store only ever asked the stored value's type:

if let Some(align) = value_abi_align(ctx, operands_info, val) {

mir_type_abi_align answers only for tuples, structs, enums, unions and arrays
— a scalar records nothing. So an f32 written into a field of a
#[repr(C, align(8))] aggregate exported at LLVM's default align 4, even
though stamp_field_address_alignment had already recorded the real 8 on the
GEP. LoadStoreVectorizer will not widen past an alignment it cannot prove, so an
adjacent pair stayed two st.global.b32.

The fix is to consult the same record the load does, for the same reason. The
value's own type still answers first, so nothing that already had an answer
changes.

PTX

kernel element before after
aligned_store repr(C, align(8)) 2 × st.global.b32 1 × st.global.v2.b32
lanes_store array lanes, align 8 2 × st.global.b32 1 × st.global.v2.b32
hot_aligned same, cache-resident 4 × st.global.b32 2 × st.global.v2.b32
packed_store repr(C), align 4 2 × st.global.b32 unchanged ✓
hot_packed align 4, cache-resident 4 × st.global.b32 unchanged ✓

lanes_store reaches the same two f32 by array index rather than by name, so
this covers the element path from #777 as well as the field path from #746.

Measurement — A10G (sm_86), driver 595.71.05

Two baseline and two patched runs, interleaved ABAB so GPU clock drift
cannot favour whichever ran first. Same host binary, same source; only the
compiler differs.

kernel baseline patched
hot_aligned cache-resident 140.3 / 140.2 µs 89.7 / 90.6 µs 1.56x
hot_packed align-4 control 140.6 / 140.7 µs 140.5 / 141.0 µs 1.00x
aligned_store streaming 3.8 / 3.9 µs 3.9 / 3.9 µs 0.99x
lanes_store streaming 3.9 / 3.9 µs 3.9 / 3.9 µs 1.00x

All five kernels bit-correct in all four runs.

The win is issue, not bandwidth, and I want to be explicit that two of these
rows show nothing. A wide store occupies one LSU slot where two narrow ones
occupy two, so halving the count only pays when stores are the limiting
resource. In the cache-resident kernel each thread rewrites its own two-element
window 64 times, the traffic is absorbed by cache, and what remains is issue —
1.56x. In the streaming kernels the same bytes still have to reach memory and
two adjacent stores already coalesce into the same transactions, so there is
nothing to win, and there isn't any. hot_packed is the control that must not
move, and does not.

The 1.08x on packed_store is a single 4.6 µs outlier in one baseline run
against 3.8–3.9 elsewhere; its PTX is unchanged by this patch, so it cannot be
a real effect.

Blast radius

This touches the store path shared by all device code, so I compiled all 203
examples both ways
and hashed every artifact: 168 of 169 PTX files are
byte-identical.
The only one that changes is the new example. No existing
kernel's codegen moves — which also means the full execute suite cannot regress,
since identical PTX runs identically.

Tests

Four unit tests mirroring the load path's, and they discriminate: three fail
on an unpatched compiler.
The fourth,
declines_non_power_of_two_field_alignment, passes either way by design — it
guards against emitting a non-power-of-two align N that llc rejects, rather
than asserting the new behaviour. Worth keeping for the same reason the load
side keeps its copy: over-claiming on a store is worse than on a load, because
a wide store to an under-aligned address is undefined rather than merely slow.

verify-code-shape.sh is self-validating in the same way — it fails on an
unpatched compiler, which is exactly what the 203-example baseline sweep
reports (aligned_field_stores ... FAIL) and the patched sweep does not.

Local gates: fmt, clippy -D warnings, mir-lower 201 lib + 104 integration
tests, both example-contract guards, and smoketest --compile-only. On the A10G,
smoketest in execute mode over aligned_field_stores, aligned_field_loads
(regression check) and vecadd: 3/3 PASS.

The one pre-existing failure in the full compile sweep is
cutile_inter_kernel (exit 101), which fails identically with and without this
patch.

`convert_load` asks its result type for an ABI alignment and, when that
answers nothing, falls back to what the address itself proved when it was
computed. `convert_store` only ever asked the stored value's type. A
scalar records no alignment, so an `f32` written into a field of a
`#[repr(C, align(8))]` aggregate exported at LLVM's default `align 4`
even though `stamp_field_address_alignment` had already recorded the real
8 on the GEP. LoadStoreVectorizer will not widen past an alignment it
cannot prove, so an adjacent pair stayed two `st.global.b32`.

The store now consults the same record the load does, for the same
reason. That is the whole change; the value's own type still answers
first, so nothing that already had an answer changes.

Measured on an A10G (sm_86, driver 595.71.05), two baseline and two
patched runs interleaved ABAB so clock drift cannot favour either:

| kernel                        | before   | after   |        |
|-------------------------------|----------|---------|--------|
| `hot_aligned` cache-resident  | 140.2 us | 90.2 us | 1.56x  |
| `hot_packed` align-4 control  | 140.6 us | 140.8 us| 1.00x  |
| `aligned_store` streaming     | 3.8 us   | 3.9 us  | 0.99x  |
| `lanes_store` streaming       | 3.9 us   | 3.9 us  | 1.00x  |

The win is issue, not bandwidth. A wide store occupies one LSU slot where
two narrow ones occupy two, so halving the count only helps when stores
are the limit -- which is why the cache-resident kernel moves 1.56x and
the streaming pair does not move at all: there the same bytes still have
to reach memory and two adjacent stores already coalesce into the same
transactions. Both are reported rather than only the flattering one.

Blast radius: compiling all 203 examples both ways leaves 168 of 169 PTX
artifacts byte-identical. The only one that changes is the new example, so
no existing kernel's codegen moves.

The four new unit tests mirror the load path's. Three of them fail
without the change; `declines_non_power_of_two_field_alignment` passes
either way by design, since it guards against over-claiming rather than
asserting the new behaviour. `verify-code-shape.sh` is likewise
self-validating: it fails on an unpatched compiler, which is what the
203-example baseline sweep reports.

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
@nihalpasham nihalpasham added perf Performance of generated code or of the compiler itself IR-lowering Lowering between dialects (dialect-mir to LLVM dialect) codegen Device code-generation pipeline (Rust MIR to IR to PTX) labels Aug 11, 2026
@nihalpasham

Copy link
Copy Markdown
Collaborator

Verified on sm_120:

  • the four unit tests pass here, and the shape guards are self-validating (building your example against an unpatched backend fails with the missing vectorized-store error);
  • llc-21 and llc-22 agree on the store opcode spellings, and spot-checked existing examples are byte-identical to main.

Clean mirror of the load-side mechanism and a nice measured win.

Merging.

@nihalpasham
nihalpasham merged commit ab3f12c into NVIDIA:main Aug 11, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

codegen Device code-generation pipeline (Rust MIR to IR to PTX) IR-lowering Lowering between dialects (dialect-mir to LLVM dialect) perf Performance of generated code or of the compiler itself

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants