Skip to content

[Feature][Ascend] Add sparse HCCL weight synchronization - #541

Open
wangx700 wants to merge 8 commits into
vllm-project:ascendfrom
wangx700:feat/sparse-hccl-vime-converter
Open

wangx700 wants to merge 8 commits into
vllm-project:ascendfrom
wangx700:feat/sparse-hccl-vime-converter

Conversation

@wangx700

@wangx700 wangx700 commented Nov 20, 2026 •

Copy link
Copy Markdown

Implement sparse HCCL weight synchronization using VIME's existing HF conversion code. This is an alternative to #492, based on its head f5a6181, targeting ascend.

Implementation

  • Reuse [Feature][Ascend] Add sparse HCCL weight synchronization #492's sparse transport, snapshot comparison, payload validation, batching, version lifecycle, and synchronized failure handling.
  • Export full seeds through HfWeightIteratorDirect.
  • Convert local shard changes through the same native convert_to_hf() used by full export. Local NaN placeholders replay VIME's existing TP merge, including gate/up sections and expert-TP layouts, without dense weight collectives during delta conversion.
  • Build the ordered parameter directory from native VIME metadata. Preserve distinct TP shards and EP experts, deduplicate replicas, and retain empty PP rows for collective ordering.
  • Remove the NVIDIA Megatron Bridge adapter and its explicit image installation block. Dependency patches are unchanged from the existing [NPU] Sync Ascend with main rollout, training and serving updates #517 baseline.
  • Replace Bridge-dependent fixtures with native VIME loading/conversion and add CPU differential coverage to CI.

Interface flow

Pause generation → flush the rollout cache → start the weight update → send the initial full seed through VIME's HfWeightIteratorDirect, or diff local training shards against pinned snapshots → convert shard changes into HF checkpoint coordinates with VIME's native convert_to_hf() and NaN placeholder probes → gather the ordered trainer-rank payload → validate it → broadcast sparse coordinates and values through native sparse HCCL → apply them with the native checkpoint loader → finish the update and advance the weight version → resume generation.

Sparse updates carry absolute target values for changed coordinates, not arithmetic deltas. If a transmission fails after a partial update, generation remains paused; reconnect and send a fresh full seed before resuming.

Validation

The native implementation at PR head 41e86290b1af1f4dd4dfe448947abce7208a1245 was validated on Ascend using a runtime reused from the existing environment. Megatron Bridge was absent and unimportable throughout the run. The validation-only commit 69374bedb408fb86c758b26d48fdc1e8b594d11f adds only pipeline_dtype=torch.bfloat16 to the PP=2 test fixture; production code matches the PR implementation.

Test results

Table note: Counts are distinct selected pytest cases from the final root JUnit results; every group completed with zero failures, errors, or skips.

Test group Passed Coverage
VIME sparse / factory / comparison 128 Native coordinate conversion, updater lifecycle, sparse/full equivalence, including a real four-rank NPU comparison
vLLM checkpoint loader 1 Checkpoint-coordinate writes
vLLM sparse transport 18 Native sparse protocol and error handling
Ascend weight transfer 178 Platform contracts and transfer boundary cases
VIME shared regression 122 Existing updater, launch lifecycle, and synchronization regressions
Native receiver integration 2 Real NPU TP/DP/EP mapping and weight installation
Megatron argument validation 82 Argument parsing and validation
Total 531

Key passing output:

SPARSE VS FULL PASS: tensors=45 updates=1975 bitwise_equal=True
128 passed
178 passed
122 passed
82 passed
SPARSE HCCL E2E PASS: token IDs and logprobs match exactly

The standalone Qwen3-30B-A3B run completed an initial seed and three optimizer updates. An independently exported full checkpoint preserved all 18,867 keys, shapes, and dtypes. The sparse-updated service and a fresh service loaded from that checkpoint produced identical 12 token IDs, every generated-token logprob, and finish reason at weight version 4. Maximum absolute logprob error was 0.0.

Three-round weight-update comparison

Table note: Each row measures one training update after the initial seed. Seconds are the complete actor.update_weights() wall time. MiB = 2^20 bytes. Payload is logical tensor data measured at sender HCCL broadcast calls, including sparse indices; it is not physical link traffic. Changed-element percentages use 30,532,122,624 HF elements as the denominator. The full-unpacked run replays the same saved rollout samples as the sparse run.

Round Full unpacked (s) Sparse (s) Speedup Full (MiB) Sparse (MiB) Payload reduction Changed elements (full / sparse)
1 51.768 35.159 1.472× 58235.402 278.571 99.5216% 0.1594585% / 0.1594510%
2 51.503 39.612 1.300× 58235.402 421.555 99.2761% 0.2497942% / 0.2412938%
3 51.626 39.912 1.293× 58235.402 463.758 99.2036% 0.2685359% / 0.2654503%

Across the three post-seed updates, mean update time decreased from 51.632 s to 38.228 s (1.351×) and aggregate logical payload decreased 99.3338%. Including the initial seed, cumulative payload reduction was 24.5004%. Updates accounted for 39.44% versus 17.46% of training-plus-update wall time; rollout generation/replay and audits are excluded. The first training round includes lazy optimizer-state initialization, so its training duration is not attributable to weight transport.

Training checks and limitations

Table note: Metrics are measured during the three optimizer updates. grad_norm is the optimizer-reported gradient norm. The same rollout data is replayed by both modes.

Mode Round Loss PG loss Entropy loss Gradient norm
Sparse 1 -0.0001569554734 0 0.1569554657 0.003942197189
Sparse 2 -0.0001267921907 0 0.1267921776 0.004972146358
Sparse 3 -9.985768702e-05 0 0.0998576805 0.005099223461
Full unpacked 1 -0.0001569554734 0 0.1569554657 0.003942442592
Full unpacked 2 -0.0001257484400 0 0.1257484406 0.005902990233
Full unpacked 3 -9.549617971e-05 0 0.09549617767 0.004477841314

All four training ranks completed three successful optimizer steps with finite, nonzero gradients. Policy loss was zero in this short workload; nonzero loss and gradients came from entropy regularization. These measurements demonstrate actual backward, optimizer, and synchronization activity, not reward-driven policy improvement or convergence. Paired samples did not yield bitwise-identical training trajectories or final weights.

Sparse conversion uses dense local probe buffers and output scans to reuse VIME's existing conversion rules. It avoids dense distributed weight gathers during delta conversion, but scratch memory is not O(delta) and can grow with TP size. Quantized and draft/MTP conversions are rejected. Qwen3-MoE has differential and end-to-end coverage; other model layouts need equivalent validation before support is claimed.

The full-unpacked measurement explicitly used HCCL with packed=False. Initial seed costs are excluded from the three-round means; sparse seed synchronization sends the complete weights and coordinate indices. These results cover this Qwen3-30B-A3B configuration and three consecutive updates, not independent repeated trials.

Signed-off-by: wangx700 <wangxin700@huawei.com>
Signed-off-by: wangx700 <wangxin700@huawei.com>
Signed-off-by: wangx700 <wangxin700@huawei.com>
Signed-off-by: wangx700 <wangxin700@huawei.com>
@read-the-docs-community

read-the-docs-community Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new sparse_hccl weight update transport mechanism for online transfer between Ascend trainers and vLLM-Ascend rollout workers, complete with documentation, a run script, and extensive unit/integration tests. The feedback highlights a potential distributed hang/crash in megatron_delta_export.py due to using the default HCCL/NCCL group for all_gather_object instead of Gloo, a performance bottleneck in _ensure_export_index when offload_train is enabled, and a recommendation to double-quote Bash array expansions in the run script to prevent word splitting.

Comment on lines +139 to +141
world = torch.distributed.get_world_size()
gathered: list = [None] * world
torch.distributed.all_gather_object(gathered, local_rows)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Potential crash or hang in distributed environments: all_gather_object is called on the default process group (which is HCCL/NCCL). NCCL/HCCL backends do not natively support object serialization collectives, which can lead to runtime errors or hangs. It should use the Gloo process group instead.

Suggested change
world = torch.distributed.get_world_size()
gathered: list = [None] * world
torch.distributed.all_gather_object(gathered, local_rows)
from vime.utils.distributed_utils import get_gloo_group
world = torch.distributed.get_world_size(group=get_gloo_group())
gathered: list = [None] * world
torch.distributed.all_gather_object(gathered, local_rows, group=get_gloo_group())

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No change needed. The real four-rank NPU sparse/full comparison exercised this export-index exchange on the target HCCL runtime and passed bitwise equality (45 tensors, 1,975 updates).

Comment on lines +216 to +219
def _ensure_export_index(self) -> None:
if self._index is not None and not self.args.offload_train:
return
self._index = build_export_index(self._iterator, self.model)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Performance Bottleneck: When offload_train is True, _ensure_export_index rebuilds the entire export index and performs a distributed all_gather_object on every single weight update step. Since the model structure (parameter names, shapes, dtypes, and slots) is static, doing a distributed exchange on every step is redundant and introduces significant overhead. Consider caching the slot tables or updating the parameter references in-place without re-running _exchange_slot_tables.

        if self._index is not None:
            if self.args.offload_train:
                local_weights = self._iterator.local_weights()
                for record in self._index:
                    record.param = local_weights.get(record.megatron_name)
            return
        self._index = build_export_index(self._iterator, self.model)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No change needed. Rebuilding the export index when offload_train is enabled reacquires the current local parameter objects after training offload; this path is separate from the validated default training path, and the suggested caching optimization is not required for correctness.

Comment on lines +120 to +128
${MODEL_ARGS[@]} \
${CKPT_ARGS[@]} \
${ROLLOUT_ARGS[@]} \
${OPTIMIZER_ARGS[@]} \
${GRPO_ARGS[@]} \
${PERF_ARGS[@]} \
${VLLM_ARGS[@]} \
${UPDATE_WEIGHT_ARGS[@]} \
${MISC_ARGS[@]}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In Bash, expanding arrays without double quotes (e.g., ${MODEL_ARGS[@]}) triggers word splitting and globbing on the elements. If any argument contains spaces or special characters, it will be split into multiple arguments, leading to parsing errors. It is highly recommended to double-quote array expansions to safely preserve individual elements.

Suggested change
${MODEL_ARGS[@]} \
${CKPT_ARGS[@]} \
${ROLLOUT_ARGS[@]} \
${OPTIMIZER_ARGS[@]} \
${GRPO_ARGS[@]} \
${PERF_ARGS[@]} \
${VLLM_ARGS[@]} \
${UPDATE_WEIGHT_ARGS[@]} \
${MISC_ARGS[@]}
"${MODEL_ARGS[@]}" \
"${CKPT_ARGS[@]}" \
"${ROLLOUT_ARGS[@]}" \
"${OPTIMIZER_ARGS[@]}" \
"${GRPO_ARGS[@]}" \
"${PERF_ARGS[@]}" \
"${VLLM_ARGS[@]}" \
"${UPDATE_WEIGHT_ARGS[@]}" \
"${MISC_ARGS[@]}"

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No change needed. The current script arrays contain separately tokenized CLI arguments with no embedded whitespace, so the existing expansion preserves the arguments used by this script.

Signed-off-by: wangx700 <wangxin700@huawei.com>
@wangx700 wangx700 changed the title [Feature][Ascend] Reuse native VIME HF conversion for sparse HCCL [Feature][Ascend] Add sparse HCCL weight synchronization Nov 20, 2026
Signed-off-by: wangx700 <wangxin700@huawei.com>
Signed-off-by: wangx700 <wangxin700@huawei.com>
@wangx700
wangx700 marked this pull request as ready for review November 20, 2026 06:54
floatlibai

This comment was marked as low quality.

@CalvinXKY

Copy link
Copy Markdown
Collaborator

Request changes.

Performance expectation (as claimed for resident actor, Qwen3-30B-A3B):

  • Benefit: ~1.35× faster post-seed updates (~51.6s → ~38.2s), ~99% logical payload cut on sparse rounds, and update share of train+update wall time ~39% → ~17%. Correctness vs independent full export looks good in the reported E2E.
  • Scope: this win assumes a live trainer session (no per-step disconnect), sparse parameter churn, and excludes the initial full seed (seed is still dense + coordinates). Not yet a general claim for all models/layouts.

Risk / gating before merge:

  • With --use-critic (forces offload_train) the path reconnects and clears _seeded every step → falls back to full seed and can be slower than dense NCCL. Same class of issue for index rebuild under offload.
  • Please either keep the sparse session across sleep, or explicitly reject/document critic+offload+sparse as unsupported, and tighten arg validation (NPU-only / no colocate / no full+sparse_hccl). Also confirm NPU CI actually ran the sparse suite.

if int(config.get("prefill_context_parallel_size", config.get("pcp_size", 1)) or 1) != 1:
raise ValueError("Sparse HCCL requires inference prefill_context_parallel_size=1")
self.disconnect_rollout_engines()
self._seeded = False

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking for real delta: connect_rollout_engines clears _seeded. With --use-critic (forces offload_train), actor reconnects every step → every update becomes _send_seed, which can be slower than dense NCCL. Please keep the session/_seeded across sleep, or reject/document critic+sparse as unsupported.

def pop_metrics(self) -> dict[str, float]:
return {}

def _ensure_export_index(self) -> None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

under offload_train, _ensure_export_index rebuilds the full directory every step (NaN probes + all_gather_object). That can erase the sparse win. Cache layout/slot tables across sleep (e.g. gather_group=None) unless the topology actually changes.

Comment on lines +76 to +78
def get_hf_weight_chunks(self):
# Export the live model, never a possibly stale training CPU backup.
yield from self.direct.get_hf_weight_chunks(self.local_weights())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we restore the actor weights before the initial export and snapshot capture? With offload_train=False, initialization leaves reference/teacher weights in the live model, and this updater ignores the supplied actor weights_getter. When the checkpoints differ, the first seed therefore sends the wrong weights. A regression with different actor and reference weights would help cover this.

Comment on lines +170 to +174
self.disconnect_rollout_engines()
self._seeded = False
self._failed = False
self._snapshots.clear()
self._index = None

@Meihan-chen Meihan-chen Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearing _failed here can allow generation to resume with partially updated weights. After one chunk is applied and a later trainer-side validation fails, the worker session remains active. The next start after reconnect then fails, but _recover_before_payload() finishes the old session and resumes generation. The incomplete-update state needs to survive reconnects until a full seed succeeds; please cover this recovery path in a regression.

Comment on lines +299 to +312
for chunk in self._iterator.get_hf_weight_chunks():
if dist.get_rank() != 0 or not chunk:
continue
for name, tensor in chunk:
flat = tensor.reshape(-1)
if flat.numel() >= 2**31:
raise ValueError(f"Sparse loader requires checkpoint tensors smaller than 2**31 elements: {name}")
max_elements = max(self.args.update_weight_buffer_size // (4 + flat.element_size()), 1)
for start in range(0, flat.numel(), max_elements):
end = min(start + max_elements, flat.numel())
indices = torch.arange(start, end, dtype=torch.int32, device=flat.device)
bucket.add(name, tensor.shape, indices, flat[start:end])
# The native converter may return views into temporary export buffers.
bucket.flush()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we synchronize the publication result across all ranks before advancing to the next export chunk? If bucket.add() or flush() raises on rank 0, it leaves the loop and enters the error reduction while other ranks enter the next HF export collective. This mismatches collective ordering and times out. Please cover a first-chunk send failure with at least two export chunks.

Comment thread .buildkite/npu_suites.py
("test_qwen3_4B_npu.py", "npu-8", "", {}),
("test_qwen3_30B_A3B_npu.py", "npu-16", "", {}),
("test_qwen3_30B_A3B_sparse_hccl_npu.py", "npu-8", "", {}),
("test_sparse_hccl.py", "npu-4", "", {}),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to keep only test_qwen3_30B_A3B_sparse_hccl_npu.py in smk and remove the test_sparse_hccl.py entry along with this filename-specific branch? The E2E test covers normal training, seed and delta updates, and inference through a real vLLM receiver. Coordinate, validation, and failure-state tests could run as unit tests, with the four-rank integration test run on demand. This would keep the shared runner unchanged.

Signed-off-by: wangx700 <wangxin700@huawei.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants