Repository navigation
perf(mir-lower): carry proved alignment through a field store - #795
Merged
Merged
Conversation
`convert_load` asks its result type for an ABI alignment and, when that answers nothing, falls back to what the address itself proved when it was computed. `convert_store` only ever asked the stored value's type. A scalar records no alignment, so an `f32` written into a field of a `#[repr(C, align(8))]` aggregate exported at LLVM's default `align 4` even though `stamp_field_address_alignment` had already recorded the real 8 on the GEP. LoadStoreVectorizer will not widen past an alignment it cannot prove, so an adjacent pair stayed two `st.global.b32`. The store now consults the same record the load does, for the same reason. That is the whole change; the value's own type still answers first, so nothing that already had an answer changes. Measured on an A10G (sm_86, driver 595.71.05), two baseline and two patched runs interleaved ABAB so clock drift cannot favour either: | kernel | before | after | | |-------------------------------|----------|---------|--------| | `hot_aligned` cache-resident | 140.2 us | 90.2 us | 1.56x | | `hot_packed` align-4 control | 140.6 us | 140.8 us| 1.00x | | `aligned_store` streaming | 3.8 us | 3.9 us | 0.99x | | `lanes_store` streaming | 3.9 us | 3.9 us | 1.00x | The win is issue, not bandwidth. A wide store occupies one LSU slot where two narrow ones occupy two, so halving the count only helps when stores are the limit -- which is why the cache-resident kernel moves 1.56x and the streaming pair does not move at all: there the same bytes still have to reach memory and two adjacent stores already coalesce into the same transactions. Both are reported rather than only the flattering one. Blast radius: compiling all 203 examples both ways leaves 168 of 169 PTX artifacts byte-identical. The only one that changes is the new example, so no existing kernel's codegen moves. The four new unit tests mirror the load path's. Three of them fail without the change; `declines_non_power_of_two_field_alignment` passes either way by design, since it guards against over-claiming rather than asserting the new behaviour. `verify-code-shape.sh` is likewise self-validating: it fails on an unpatched compiler, which is what the 203-example baseline sweep reports. Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Collaborator
|
Verified on sm_120:
Clean mirror of the load-side mechanism and a nice measured win. Merging. |
nihalpasham
approved these changes
Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The store-side counterpart of #746 / #777. Those taught the load path to use
the alignment a field address had already proved; the store path never asked.
The defect
convert_loadasks its result type for an ABI alignment and, when that answersnothing, falls back to what the address itself proved when it was computed.
convert_storeonly ever asked the stored value's type:mir_type_abi_alignanswers only for tuples, structs, enums, unions and arrays— a scalar records nothing. So an
f32written into a field of a#[repr(C, align(8))]aggregate exported at LLVM's defaultalign 4, eventhough
stamp_field_address_alignmenthad already recorded the real 8 on theGEP. LoadStoreVectorizer will not widen past an alignment it cannot prove, so an
adjacent pair stayed two
st.global.b32.The fix is to consult the same record the load does, for the same reason. The
value's own type still answers first, so nothing that already had an answer
changes.
PTX
aligned_storerepr(C, align(8))st.global.b32st.global.v2.b32lanes_storest.global.b32st.global.v2.b32hot_alignedst.global.b32st.global.v2.b32packed_storerepr(C), align 4st.global.b32hot_packedst.global.b32lanes_storereaches the same two f32 by array index rather than by name, sothis covers the element path from #777 as well as the field path from #746.
Measurement — A10G (sm_86), driver 595.71.05
Two baseline and two patched runs, interleaved ABAB so GPU clock drift
cannot favour whichever ran first. Same host binary, same source; only the
compiler differs.
hot_alignedcache-residenthot_packedalign-4 controlaligned_storestreaminglanes_storestreamingAll five kernels bit-correct in all four runs.
The win is issue, not bandwidth, and I want to be explicit that two of these
rows show nothing. A wide store occupies one LSU slot where two narrow ones
occupy two, so halving the count only pays when stores are the limiting
resource. In the cache-resident kernel each thread rewrites its own two-element
window 64 times, the traffic is absorbed by cache, and what remains is issue —
1.56x. In the streaming kernels the same bytes still have to reach memory and
two adjacent stores already coalesce into the same transactions, so there is
nothing to win, and there isn't any.
hot_packedis the control that must notmove, and does not.
The 1.08x on
packed_storeis a single 4.6 µs outlier in one baseline runagainst 3.8–3.9 elsewhere; its PTX is unchanged by this patch, so it cannot be
a real effect.
Blast radius
This touches the store path shared by all device code, so I compiled all 203
examples both ways and hashed every artifact: 168 of 169 PTX files are
byte-identical. The only one that changes is the new example. No existing
kernel's codegen moves — which also means the full execute suite cannot regress,
since identical PTX runs identically.
Tests
Four unit tests mirroring the load path's, and they discriminate: three fail
on an unpatched compiler. The fourth,
declines_non_power_of_two_field_alignment, passes either way by design — itguards against emitting a non-power-of-two
align Nthat llc rejects, ratherthan asserting the new behaviour. Worth keeping for the same reason the load
side keeps its copy: over-claiming on a store is worse than on a load, because
a wide store to an under-aligned address is undefined rather than merely slow.
verify-code-shape.shis self-validating in the same way — it fails on anunpatched compiler, which is exactly what the 203-example baseline sweep
reports (
aligned_field_stores ... FAIL) and the patched sweep does not.Local gates:
fmt,clippy -D warnings,mir-lower201 lib + 104 integrationtests, both example-contract guards, and
smoketest --compile-only. On the A10G,smoketestin execute mode overaligned_field_stores,aligned_field_loads(regression check) and
vecadd: 3/3 PASS.The one pre-existing failure in the full compile sweep is
cutile_inter_kernel(exit 101), which fails identically with and without thispatch.