Repository navigation
Preserve low-precision rounding in gated residual mixing - #212
yuchenwang3 wants to merge 1 commit into
Conversation
Signed-off-by: Yuchen Wang <93700456+yuchenwang3@users.noreply.github.com>
|
@hjh0119 Could you take a look at the gated-residual rounding fix? The GPU run passes all 16 tests, including BF16/FP16 helper checks and the TELinear unit. The description includes the timing cost of the eager path; does that tradeoff look reasonable? |
|
Do we have additional empirical evidence supporting the necessity of casting to bf16? |
|
The existing measurements establish a rounding difference, but I do not yet have full-model evidence that preserving it improves training stability or actor/rollout log-probability agreement. This patch preserves the intermediate BF16/FP16 stores in the eager gate chain; it does not convert FP32 activations to BF16. In the saved CUDA negative control (four streams, width 2560), the compiled BF16 helper differs from eager by up to 0.0078125 in both output and input gradients. The patched helper matches eager exactly. There is a cost: at 4096 rows, interleaved forward/backward wall medians were 0.264 ms compiled and 0.368 ms eager (40 samples after 10 warmups). The case for the change is exact eager-reference parity. If tolerance-based parity is the intended contract, these measurements alone do not justify disabling fusion. Would you prefer to retain the compiled path unless a full-model mismatch demonstrates a need for this stricter behavior? |
The compiled gated-residual helpers can fuse away the low-precision rounding after division, sigmoid, and multiplication. Their outputs and gradients can then differ from the same gate chain evaluated as separate BF16/FP16 operations.
This keeps those operation boundaries for BF16/FP16 and retains the compiled helpers for FP32. The eager regions are also excluded from an outer
torch.compile. Projections, weights, and the gate formulas are unchanged. The existing GR unit tolerates compilation drift; this makes the low-precision helpers agree with its eager reference exactly, at the cost of fusion in that path.Validation:
_mix_and_reducereproduces the difference: with 17 rows, four streams, and width 64, BF16 output/input-gradient maximum absolute differences from eager are 0.00390625/0.00244140625. FP16 differs too. These are helper results, not full-model error bounds.pre-commit run --all-filespassed.Implementation and tests were generated with OpenAI Codex.