Repository navigation
Conversation
Co-authored-by: OpenAI Codex Signed-off-by: TAJh <taojiaheng1@huawei.com>
Documentation build overview
70 files changed ·
|
There was a problem hiding this comment.
Code Review
This pull request introduces support for Qwen3.8-27B on Ascend NPU, featuring GDN (Gated Delta Network) head tensor parallelism. It adds the --qwen-gdn-tensor-parallel flag to shard GDN heads, projections, and convolutions across the tensor parallel group, along with necessary weight sharding/merging utilities, gradient synchronization for shared norms, and distributed checkpointing support. Additionally, it includes documentation, a launch script, unit tests, and a Buildkite smoke test. Feedback suggests improving the robustness of the merge_gdn_qkv helper by dynamically using the length of split_sizes instead of a hardcoded range of 3.
| def merge_gdn_qkv(partitions, split_sizes): | ||
| size = len(partitions) | ||
| local_sizes = tuple(n // size for n in split_sizes) | ||
| parts = [p.split(local_sizes, dim=0) for p in partitions] | ||
| return torch.cat([torch.cat([p[i] for p in parts], dim=0) for i in range(3)], dim=0).contiguous() |
There was a problem hiding this comment.
The merge_gdn_qkv function currently hardcodes range(3) when concatenating the split Q, K, and V segments. To make this helper more robust and generic for any number of split segments, we should use range(len(split_sizes)) instead of the hardcoded 3.
| def merge_gdn_qkv(partitions, split_sizes): | |
| size = len(partitions) | |
| local_sizes = tuple(n // size for n in split_sizes) | |
| parts = [p.split(local_sizes, dim=0) for p in partitions] | |
| return torch.cat([torch.cat([p[i] for p in parts], dim=0) for i in range(3)], dim=0).contiguous() | |
| def merge_gdn_qkv(partitions, split_sizes): | |
| size = len(partitions) | |
| local_sizes = tuple(n // size for n in split_sizes) | |
| parts = [p.split(local_sizes, dim=0) for p in partitions] | |
| return torch.cat([torch.cat([p[i] for p in parts], dim=0) for i in range(len(split_sizes))], dim=0).contiguous() |
|
Overview
Adds Qwen3.8-27B GRPO support on Ascend with opt-in GDN head tensor parallelism.
Head-local projections, convolution and recurrent computation are partitioned
across TP ranks. The checkpoint retains the Qwen3.5 hybrid-attention architecture,
so this reuses the existing model bridge. This PR targets
ascendand follows#403's script, E2E and training-evidence format.
Changes
qwen3_5.pyandhf_attention.pyadd head sharding,SP/non-SP collectives, shared-normalization gradient reduction and checkpoint
factories. Enable with
--qwen-gdn-tensor-parallel; CP=1 is required.The replicated GDN path remains the default.
unequal Q/K/V segments through
vime/utils/gdn_tp.pyand parameter metadata.reuse existing dependency HCCL/IPC patches.
regression tests and
tests/test_qwen3.8_27B_npu.py, registered asnpu-16in Buildkite NPU
smk. The launcher isolates Ray ports/session without killing other jobs.Usage
Use the NPU image/setup, then:
Override
MODEL_DIR,DATASET_DIR,CHECKPOINT_DIRorNUM_ROLLOUTas needed.Model/optimizer checkpoints are saved every 25 rounds; four require approximately
1.4 TiB plus input storage. Use unused Ray/HCCL ports in a dedicated NPU container.
Testing
100-round hardware training log
The saved 2026-10-30/31 run completed 100 rollout/train/post-training-sync
cycles, 25,600 samples and 54,285,777 response tokens. Ray finished
SUCCEEDED, driver exit 0, and saved the final model/optimizer checkpoint at iteration 99.Evidence scope: this adaptation snapshot predates the PR's additional
input-layernorm TP-gradient fix. It used the 0.30rc1 unpacked-HCCL compatibility
adapter; current-head hardware verification belongs to PR CI. Repository dependency pins are unchanged.
Hardware, environment and training configuration
Ascend910fromnpu-smi; 16 logical devices / 8 physical cards+empty) / vLLM-Ascend 0.30.0rc1Raw values, trailing 5-step means and full-run means, following #403:
Full-resolution SVG ·
100-step metrics log (JSONL) ·
CSV ·
Run summary ·
Provenance/timing notes ·
SHA256
DAPO scores +1/−1, so the judged-correct fraction is
(raw_reward + 1) / 2.These are shuffled training batches, with late-stage fluctuations; they do
not establish held-out accuracy or convergence.
perf/step_timeWall time uses successive post-training-sync ends, including checkpoint/phase
overhead; native
perf/step_timehas different boundaries. The sync chart usespost-training Timers rather than lagged
perf/update_weights_time. Shorterresponses and concurrent late-run backup I/O preclude isolating kernel speedup from these timings.
Selected native log values and source traceability
The JSONL preserves all 100 native scalar dictionaries, Timer values and source
line numbers. Source
train.log: 18,183,197 bytes / 132,386 lines, checkedagainst the original evidence archive:
train/kl_losswas absent with the reference KL penalty disabled; the dashboarduses the logged
train/ppo_kl. Near-zero normalized PG loss should be read withnonzero gradient norms. Native generic FLOPs are not interpreted as hybrid-GDN MFU.
CPU checks and current-head E2E
Megatron cases skipped because Megatron is not installed locally.
smoke argument construction and
npu-16suite registration were checked.thinking disabled, 16 devices, TP4/PP2/SP/CP1, V2, no MTP:
Registered in NPU
smk; results are tracked by the PR checks.AI assistance: OpenAI Codex assisted with implementation, tests, documentation and log visualization.