Repository navigation
Conversation
Signed-off-by: wangx700 <wangxin700@huawei.com>
Signed-off-by: wangx700 <wangxin700@huawei.com>
Signed-off-by: wangx700 <wangxin700@huawei.com>
Signed-off-by: wangx700 <wangxin700@huawei.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a new sparse_hccl weight update transport mechanism for online transfer between Ascend trainers and vLLM-Ascend rollout workers, complete with documentation, a run script, and extensive unit/integration tests. The feedback highlights a potential distributed hang/crash in megatron_delta_export.py due to using the default HCCL/NCCL group for all_gather_object instead of Gloo, a performance bottleneck in _ensure_export_index when offload_train is enabled, and a recommendation to double-quote Bash array expansions in the run script to prevent word splitting.
| world = torch.distributed.get_world_size() | ||
| gathered: list = [None] * world | ||
| torch.distributed.all_gather_object(gathered, local_rows) |
There was a problem hiding this comment.
Potential crash or hang in distributed environments: all_gather_object is called on the default process group (which is HCCL/NCCL). NCCL/HCCL backends do not natively support object serialization collectives, which can lead to runtime errors or hangs. It should use the Gloo process group instead.
| world = torch.distributed.get_world_size() | |
| gathered: list = [None] * world | |
| torch.distributed.all_gather_object(gathered, local_rows) | |
| from vime.utils.distributed_utils import get_gloo_group | |
| world = torch.distributed.get_world_size(group=get_gloo_group()) | |
| gathered: list = [None] * world | |
| torch.distributed.all_gather_object(gathered, local_rows, group=get_gloo_group()) |
There was a problem hiding this comment.
No change needed. The real four-rank NPU sparse/full comparison exercised this export-index exchange on the target HCCL runtime and passed bitwise equality (45 tensors, 1,975 updates).
| def _ensure_export_index(self) -> None: | ||
| if self._index is not None and not self.args.offload_train: | ||
| return | ||
| self._index = build_export_index(self._iterator, self.model) |
There was a problem hiding this comment.
Performance Bottleneck: When offload_train is True, _ensure_export_index rebuilds the entire export index and performs a distributed all_gather_object on every single weight update step. Since the model structure (parameter names, shapes, dtypes, and slots) is static, doing a distributed exchange on every step is redundant and introduces significant overhead. Consider caching the slot tables or updating the parameter references in-place without re-running _exchange_slot_tables.
if self._index is not None:
if self.args.offload_train:
local_weights = self._iterator.local_weights()
for record in self._index:
record.param = local_weights.get(record.megatron_name)
return
self._index = build_export_index(self._iterator, self.model)There was a problem hiding this comment.
No change needed. Rebuilding the export index when offload_train is enabled reacquires the current local parameter objects after training offload; this path is separate from the validated default training path, and the suggested caching optimization is not required for correctness.
| ${MODEL_ARGS[@]} \ | ||
| ${CKPT_ARGS[@]} \ | ||
| ${ROLLOUT_ARGS[@]} \ | ||
| ${OPTIMIZER_ARGS[@]} \ | ||
| ${GRPO_ARGS[@]} \ | ||
| ${PERF_ARGS[@]} \ | ||
| ${VLLM_ARGS[@]} \ | ||
| ${UPDATE_WEIGHT_ARGS[@]} \ | ||
| ${MISC_ARGS[@]} |
There was a problem hiding this comment.
In Bash, expanding arrays without double quotes (e.g., ${MODEL_ARGS[@]}) triggers word splitting and globbing on the elements. If any argument contains spaces or special characters, it will be split into multiple arguments, leading to parsing errors. It is highly recommended to double-quote array expansions to safely preserve individual elements.
| ${MODEL_ARGS[@]} \ | |
| ${CKPT_ARGS[@]} \ | |
| ${ROLLOUT_ARGS[@]} \ | |
| ${OPTIMIZER_ARGS[@]} \ | |
| ${GRPO_ARGS[@]} \ | |
| ${PERF_ARGS[@]} \ | |
| ${VLLM_ARGS[@]} \ | |
| ${UPDATE_WEIGHT_ARGS[@]} \ | |
| ${MISC_ARGS[@]} | |
| "${MODEL_ARGS[@]}" \ | |
| "${CKPT_ARGS[@]}" \ | |
| "${ROLLOUT_ARGS[@]}" \ | |
| "${OPTIMIZER_ARGS[@]}" \ | |
| "${GRPO_ARGS[@]}" \ | |
| "${PERF_ARGS[@]}" \ | |
| "${VLLM_ARGS[@]}" \ | |
| "${UPDATE_WEIGHT_ARGS[@]}" \ | |
| "${MISC_ARGS[@]}" |
There was a problem hiding this comment.
No change needed. The current script arrays contain separately tokenized CLI arguments with no embedded whitespace, so the existing expansion preserves the arguments used by this script.
Signed-off-by: wangx700 <wangxin700@huawei.com>
Signed-off-by: wangx700 <wangxin700@huawei.com>
Signed-off-by: wangx700 <wangxin700@huawei.com>
|
Request changes. Performance expectation (as claimed for resident actor, Qwen3-30B-A3B):
Risk / gating before merge:
|
| if int(config.get("prefill_context_parallel_size", config.get("pcp_size", 1)) or 1) != 1: | ||
| raise ValueError("Sparse HCCL requires inference prefill_context_parallel_size=1") | ||
| self.disconnect_rollout_engines() | ||
| self._seeded = False |
There was a problem hiding this comment.
Blocking for real delta: connect_rollout_engines clears _seeded. With --use-critic (forces offload_train), actor reconnects every step → every update becomes _send_seed, which can be slower than dense NCCL. Please keep the session/_seeded across sleep, or reject/document critic+sparse as unsupported.
| def pop_metrics(self) -> dict[str, float]: | ||
| return {} | ||
|
|
||
| def _ensure_export_index(self) -> None: |
There was a problem hiding this comment.
under offload_train, _ensure_export_index rebuilds the full directory every step (NaN probes + all_gather_object). That can erase the sparse win. Cache layout/slot tables across sleep (e.g. gather_group=None) unless the topology actually changes.
| def get_hf_weight_chunks(self): | ||
| # Export the live model, never a possibly stale training CPU backup. | ||
| yield from self.direct.get_hf_weight_chunks(self.local_weights()) |
There was a problem hiding this comment.
Could we restore the actor weights before the initial export and snapshot capture? With offload_train=False, initialization leaves reference/teacher weights in the live model, and this updater ignores the supplied actor weights_getter. When the checkpoints differ, the first seed therefore sends the wrong weights. A regression with different actor and reference weights would help cover this.
| self.disconnect_rollout_engines() | ||
| self._seeded = False | ||
| self._failed = False | ||
| self._snapshots.clear() | ||
| self._index = None |
There was a problem hiding this comment.
Clearing _failed here can allow generation to resume with partially updated weights. After one chunk is applied and a later trainer-side validation fails, the worker session remains active. The next start after reconnect then fails, but _recover_before_payload() finishes the old session and resumes generation. The incomplete-update state needs to survive reconnects until a full seed succeeds; please cover this recovery path in a regression.
| for chunk in self._iterator.get_hf_weight_chunks(): | ||
| if dist.get_rank() != 0 or not chunk: | ||
| continue | ||
| for name, tensor in chunk: | ||
| flat = tensor.reshape(-1) | ||
| if flat.numel() >= 2**31: | ||
| raise ValueError(f"Sparse loader requires checkpoint tensors smaller than 2**31 elements: {name}") | ||
| max_elements = max(self.args.update_weight_buffer_size // (4 + flat.element_size()), 1) | ||
| for start in range(0, flat.numel(), max_elements): | ||
| end = min(start + max_elements, flat.numel()) | ||
| indices = torch.arange(start, end, dtype=torch.int32, device=flat.device) | ||
| bucket.add(name, tensor.shape, indices, flat[start:end]) | ||
| # The native converter may return views into temporary export buffers. | ||
| bucket.flush() |
There was a problem hiding this comment.
Could we synchronize the publication result across all ranks before advancing to the next export chunk? If bucket.add() or flush() raises on rank 0, it leaves the loop and enters the error reduction while other ranks enter the next HF export collective. This mismatches collective ordering and times out. Please cover a first-chunk send failure with at least two export chunks.
| ("test_qwen3_4B_npu.py", "npu-8", "", {}), | ||
| ("test_qwen3_30B_A3B_npu.py", "npu-16", "", {}), | ||
| ("test_qwen3_30B_A3B_sparse_hccl_npu.py", "npu-8", "", {}), | ||
| ("test_sparse_hccl.py", "npu-4", "", {}), |
There was a problem hiding this comment.
Would it make sense to keep only test_qwen3_30B_A3B_sparse_hccl_npu.py in smk and remove the test_sparse_hccl.py entry along with this filename-specific branch? The E2E test covers normal training, seed and delta updates, and inference through a real vLLM receiver. Coordinate, validation, and failure-state tests could run as unit tests, with the four-rank integration test run on demand. This would keep the shared runner unchanged.
Signed-off-by: wangx700 <wangxin700@huawei.com>
Implement sparse HCCL weight synchronization using VIME's existing HF conversion code. This is an alternative to #492, based on its head
f5a6181, targetingascend.Implementation
HfWeightIteratorDirect.convert_to_hf()used by full export. Local NaN placeholders replay VIME's existing TP merge, including gate/up sections and expert-TP layouts, without dense weight collectives during delta conversion.Interface flow
Pause generation → flush the rollout cache → start the weight update → send the initial full seed through VIME's
HfWeightIteratorDirect, or diff local training shards against pinned snapshots → convert shard changes into HF checkpoint coordinates with VIME's nativeconvert_to_hf()and NaN placeholder probes → gather the ordered trainer-rank payload → validate it → broadcast sparse coordinates and values through native sparse HCCL → apply them with the native checkpoint loader → finish the update and advance the weight version → resume generation.Sparse updates carry absolute target values for changed coordinates, not arithmetic deltas. If a transmission fails after a partial update, generation remains paused; reconnect and send a fresh full seed before resuming.
Validation
The native implementation at PR head
41e86290b1af1f4dd4dfe448947abce7208a1245was validated on Ascend using a runtime reused from the existing environment. Megatron Bridge was absent and unimportable throughout the run. The validation-only commit69374bedb408fb86c758b26d48fdc1e8b594d11fadds onlypipeline_dtype=torch.bfloat16to the PP=2 test fixture; production code matches the PR implementation.Test results
Table note: Counts are distinct selected pytest cases from the final root JUnit results; every group completed with zero failures, errors, or skips.
Key passing output:
The standalone Qwen3-30B-A3B run completed an initial seed and three optimizer updates. An independently exported full checkpoint preserved all 18,867 keys, shapes, and dtypes. The sparse-updated service and a fresh service loaded from that checkpoint produced identical 12 token IDs, every generated-token logprob, and finish reason at weight version 4. Maximum absolute logprob error was 0.0.
Three-round weight-update comparison
Table note: Each row measures one training update after the initial seed. Seconds are the complete
actor.update_weights()wall time. MiB = 2^20 bytes. Payload is logical tensor data measured at sender HCCL broadcast calls, including sparse indices; it is not physical link traffic. Changed-element percentages use 30,532,122,624 HF elements as the denominator. The full-unpacked run replays the same saved rollout samples as the sparse run.Across the three post-seed updates, mean update time decreased from 51.632 s to 38.228 s (1.351×) and aggregate logical payload decreased 99.3338%. Including the initial seed, cumulative payload reduction was 24.5004%. Updates accounted for 39.44% versus 17.46% of training-plus-update wall time; rollout generation/replay and audits are excluded. The first training round includes lazy optimizer-state initialization, so its training duration is not attributable to weight transport.
Training checks and limitations
Table note: Metrics are measured during the three optimizer updates.
grad_normis the optimizer-reported gradient norm. The same rollout data is replayed by both modes.All four training ranks completed three successful optimizer steps with finite, nonzero gradients. Policy loss was zero in this short workload; nonzero loss and gradients came from entropy regularization. These measurements demonstrate actual backward, optimizer, and synchronization activity, not reward-driven policy improvement or convergence. Paired samples did not yield bitwise-identical training trajectories or final weights.
Sparse conversion uses dense local probe buffers and output scans to reuse VIME's existing conversion rules. It avoids dense distributed weight gathers during delta conversion, but scratch memory is not O(delta) and can grow with TP size. Quantized and draft/MTP conversions are rejected. Qwen3-MoE has differential and end-to-end coverage; other model layouts need equivalent validation before support is claimed.
The full-unpacked measurement explicitly used HCCL with
packed=False. Initial seed costs are excluded from the three-round means; sparse seed synchronization sends the complete weights and coordinate indices. These results cover this Qwen3-30B-A3B configuration and three consecutive updates, not independent repeated trials.