Skip to content

fix(runtime): pair dot operands in register order - #364

Open
FeathBow wants to merge 1 commit into
tile-ai:mainfrom
FeathBow:fix/dot-register-order
Open

FeathBow wants to merge 1 commit into
tile-ai:mainfrom
FeathBow:fix/dot-register-order

Conversation

@FeathBow

Copy link
Copy Markdown
Contributor

Why

T.dot pairs its operands by register index. Each unit numbers its local elements from the smallest physical stride up, as register_strides does. The CUDA runtime paired them by CuTe index, which counts the first mode fastest. The two agree only while each local view has at most one mode longer than 1, or lists its modes from the smallest stride up.

A lane holding a (2, 4) block stored by rows beside a (2, 4) parameter met a[l, 1, 0] with b[0, 1].

What

  • dot_impl::contract reads both operands through views whose modes run from the smallest stride up, coalesced. A swizzled operand keeps its swizzle over the reordered affine layout. The load width is taken from the reordered views.
  • A reordered view can now run contiguously where the original did not, so contract tests both base pointers against the load width and reads one element at a time when either is not aligned.
  • The runtime spec states how ops::dot pairs its operands and when it falls back to single elements.
  • Tests on the card contract a (32, 2, 4) operand split by rows against a (2, 4) parameter, with the operand read from global memory, from registers and from swizzled shared memory. A fourth case puts the rows nine elements apart, so most blocks start off a 16 byte boundary.

Contract

  • ops::dot pairs lhs and rhs by register index whatever the order of their modes. An operand with one mode, which includes every kernel parameter, is read as before.
  • An operand of several modes needs static strides, or it does not compile.
  • A wide read happens only when both base pointers are aligned to its width.

Risk

The only other Dot in the repository is dot_tiers. Each of its operands has at most one mode longer than 1, so it pairs as before.

@FeathBow

Copy link
Copy Markdown
Contributor Author
  • On an sm_89 card the four new cases fail on main and pass here. In the first three, every lane of main returns 3436 where the access relation gives 3422. In the fourth, main returns 3456 where it gives 3470.
  • With the reordering but without the base pointer test, the fourth case stops with a misaligned address.
  • The full suite on that card matches main apart from the four new cases, and the installed smoke tests match. racecheck and synccheck report nothing on the new cases.
  • dot_tiers compiled for sm_89 gives the same output as main bit for bit, and both tiers match torch.
  • On an sm_90 card the four new cases also fail on main and pass here, and the full suite matches main apart from them. racecheck and synccheck report nothing on the new cases. dot_tiers gives the same output as main bit for bit, and both tiers match torch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant