Repository navigation
[Sync] Update Megatron 0.20 compatibility and packed weight-transfer coverage - #518
Conversation
Co-authored-by: Shifang Xu <shifangx@nvidia.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Shifang Xu <shifangx@nvidia.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Junting Lv <49749126+sheep0923@users.noreply.github.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
There was a problem hiding this comment.
Code Review
This pull request updates the codebase to align with a newer Megatron-LM commit, including support for the 'nvrx' async checkpointing strategy via nvidia-resiliency-ext, handling of run configurations, and updated Multi-Token Prediction (MTP) loss tracking. It also updates documentation, adds corresponding unit tests, and introduces model-specific compatibility fixes. The review feedback suggests adding defensive None checks on the MTP logging tracker to prevent potential runtime errors, as well as wrapping the nvidia-resiliency-ext import in a try-except block to provide a more descriptive error message if the dependency is missing.
| def _async_writer_class(async_strategy): | ||
| if async_strategy == "nvrx": | ||
| from nvidia_resiliency_ext.checkpointing.async_ckpt.filesystem_async import FileSystemWriterAsync | ||
| else: | ||
| from megatron.core.dist_checkpointing.strategies.filesystem_async import FileSystemWriterAsync | ||
| return FileSystemWriterAsync |
There was a problem hiding this comment.
If nvidia-resiliency-ext is not installed in the environment, attempting to use the 'nvrx' async strategy will result in a generic ImportError. It is highly recommended to wrap this import in a try-except block and raise a more descriptive error message to guide the user to install the missing dependency.
| def _async_writer_class(async_strategy): | |
| if async_strategy == "nvrx": | |
| from nvidia_resiliency_ext.checkpointing.async_ckpt.filesystem_async import FileSystemWriterAsync | |
| else: | |
| from megatron.core.dist_checkpointing.strategies.filesystem_async import FileSystemWriterAsync | |
| return FileSystemWriterAsync | |
| def _async_writer_class(async_strategy): | |
| if async_strategy == "nvrx": | |
| try: | |
| from nvidia_resiliency_ext.checkpointing.async_ckpt.filesystem_async import FileSystemWriterAsync | |
| except ImportError as e: | |
| raise ImportError( | |
| "nvidia-resiliency-ext is required when using the 'nvrx' async strategy. " | |
| "Please install it or choose a different strategy." | |
| ) from e | |
| else: | |
| from megatron.core.dist_checkpointing.strategies.filesystem_async import FileSystemWriterAsync | |
| return FileSystemWriterAsync |
There was a problem hiding this comment.
Thanks. The supported image explicitly installs nvidia-resiliency-ext==0.7.0, and the original eight-GPU async checkpoint save/load test passed with this dependency. We retain the direct import rather than wrap an otherwise actionable ImportError; recommending another strategy would also be misleading for the new MCore path.
| tracker = MTPLossLoggingHelper.tracker | ||
| # Megatron ae76c60e6 accumulates into "loss_sums" and reduces across ranks itself; | ||
| # "values" only exists after reduce_loss_in_tracker(). | ||
| reduced_by_megatron = "loss_sums" in tracker and hasattr(MTPLossLoggingHelper, "reduce_loss_in_tracker") |
There was a problem hiding this comment.
To prevent potential TypeError or AttributeError if MTPLossLoggingHelper.tracker is None or uninitialized, we should defensively check that tracker is not None before performing membership checks.
| reduced_by_megatron = "loss_sums" in tracker and hasattr(MTPLossLoggingHelper, "reduce_loss_in_tracker") | |
| reduced_by_megatron = tracker is not None and "loss_sums" in tracker and hasattr(MTPLossLoggingHelper, "reduce_loss_in_tracker") |
There was a problem hiding this comment.
Checked both supported MCore implementations: MTPLossLoggingHelper.tracker is initialized to {}, and its cleanup routines clear tensor contents rather than replacing the dictionary with None. The original eight-GPU MTP-only training test also passed. A None guard would cover an unsupported state, so we keep the direct dictionary contract.
| loss_key = "loss_values" if "loss_values" in tracker else "values" | ||
| if loss_key in tracker: |
There was a problem hiding this comment.
Similarly, defensively check that tracker is not None before checking for "loss_values" or accessing keys to avoid a TypeError if the tracker is uninitialized.
| loss_key = "loss_values" if "loss_values" in tracker else "values" | |
| if loss_key in tracker: | |
| loss_key = "loss_values" if tracker is not None and "loss_values" in tracker else "values" | |
| if tracker is not None and loss_key in tracker: |
There was a problem hiding this comment.
Same dictionary invariant as the preceding comment: tracker remains a dictionary through initialization and cleanup. Keeping this unchanged avoids a redundant guard; the MTP-only GPU regression passed.
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Preserve upstream model and training contracts, translate native multimodal handling, and include validated reload and dependency compatibility fixes. Co-authored-by: Shifang Xu <shifangx@nvidia.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: hotragn <hotragn.pettugani_2024@woxsen.edu.in> Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: aoshen02 <aoshen@inferact.ai>
0b02e4d to
233ad1d
Compare
| for mm_input_dict in multimodal_train_inputs: | ||
| if mm_input_dict is not None: | ||
| for key, mm_tensor in mm_input_dict.items(): | ||
| mm_tensor = torch.atleast_1d(torch.as_tensor(mm_tensor)) |
There was a problem hiding this comment.
Blocking: when one prompt has multiple Nemotron images at different target resolutions, the HF processor returns pixel_values as a Python list of CHW tensors (it only stacks when every tile matches). torch.as_tensor(list_of_tensors) always raises ValueError, so training dies here before NemotronHVLModel can iterate the list.
Same-size multi-image (stacked NCHW) and cross-sample mixed sizes (one NCHW tensor per sample) are fine; this is the within-sample mixed-size path that nemotron_h_vl.py and test_nemotron_vl_image_expansion already claim.
Please keep a list/tuple of tensors as-is (or stack only equal shapes) instead of unconditional as_tensor.
There was a problem hiding this comment.
Confirmed and fixed in 74ac164. Nonempty tensor lists/tuples now bypass as_tensor, while empty sequences and ordinary numeric metadata retain the existing normalization. Audio padding is unchanged.
The real processor produces CHW tensors of shapes (3,480,576) and (3,704,384). We reproduced the conversion failure. Continuing through the real GPU vision path also exposed CPU image tensors reaching CUDA weights, so the model's existing per-image dtype conversion now also specifies embeddings.device.
Validation on H200 under Slurm:
- 168 existing CPU tests and changed-file pre-commit passed.
- Original eight-GPU proxy regression: 7 passed, 1 optional full-checkpoint header check skipped.
- Additional two-image/mixed-resolution eight-GPU run with the final source: 7 passed, the same optional check skipped. Two steps had train/rollout logprob differences 0.06275282 and 0.05960483, below the unchanged 0.1 threshold. TP2/CP2/EP8 training, TP8/EP8 rollout, micro-batch size 1; existing training parameters and assertions unchanged. Diagnostic inputs duplicate four Geo3K images at different aspect ratios with two image markers each.
This verifies mixed-size images within one sample. Multiple samples containing nested image lists in the same micro-batch remain a separate batching limitation; this fix does not claim to support that case. The random proxy's gradients were zero, so this is execution/alignment evidence, not learning-quality validation. Candidate image/remote CI refresh is still pending.
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Integrate upstream vllm-project#513 and vllm-project#518. Use the shared MTP expert-name handling while preserving existing NPU integration and the ROCm Dockerfile deletion. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Released after merge
With explicit release authorization,
vllm/vime:latestand the immutablevllm/vime:slime-79aa41a884ab-vime-74ac1641d993now point to the verified AMD64/ARM64 indexsha256:1087f4d6f2408c750f78de2de14d87621f163be4b6e286e28f8cb7098a65cbc9. Remote payloads/digests were read back and matched. Built source74ac1641dand merge2ce39aa2have identical trees; no rebuild or dependency upgrade. Validation scope and limitations below remain unchanged; earlier publication-pending statements describe pre-release history.Review follow-up — current head
74ac1641dFixed mixed-resolution image tensor-list normalization and model-local CPU→GPU transfer. Changed-file pre-commit, CPU regression and the final-source eight-H200 mixed-image diagnostic passed (7 passed, 1 optional full-checkpoint header check skipped; logprob differences 0.06275282/0.05960483 against unchanged 0.1 threshold). Micro-batch size was 1; general nested multi-sample batching and learning quality are not claimed. See review reply.
Current-head targeted CI1892 passed: all seven CPU checks and the selected Nemotron GPU regression (7 passed, 1 optional full-checkpoint header check skipped). It uses
vllm/vime@sha256:6d13d2f23ea6a50a0ea2bf08f445026ce8089386a810af91820cfd948581ca1a, built from74ac1641d. This is targeted follow-up, not a new full GPU matrix run. ARM source-only refresh passed 206+168 CPU checks and the original four-GPU Qwen three-step smoke (not ARM Nemotron mixed-image coverage). Both architectures are published asvllm/vime:test-slime-79aa41a884ab-vime-74ac1641d993, indexsha256:1087f4d6f2408c750f78de2de14d87621f163be4b6e286e28f8cb7098a65cbc9; both bind the same source and preserve their parent compiled dependencies. Experimental only;latestremains unchanged. Older digests/full CI results below remain historical evidence. No release or latest promotion.Previous candidate evidence (
233ad1dc)Current PR head
233ad1dcadds Nemotron Super VL model/conversion support, native image resizing and the original eight-GPU proxy test to the prior training-compatibility sync below. It differs from tested/built0b02e4dbonly by adding the missing DCO sign-off to the old R3 commit: both trees are exactlyece6e84be6dde650d63bc3697f72442b5084f815. DCO now passes. Image tags retain their actual built-source SHA; fixed-image CPU-only CI1886 passed all7 CPU checks on the rewritten head. Its GPU gate stays intentionally blocked: unchanged GPU evidence comes from CI1879, not a second GPU run.Previous candidate validation
CI attempt history: Qwen30B DeepEP/FP8, Moonlight and Agent each passed after an automatic retry. Qwen and Agent received cancellation signals (exit-1; exact external causes unavailable); Moonlight failed Ray job submission with HTTP500/no available agent before training. These are not first-attempt passes; logs are retained.
vllm/vime:test-slime-79aa41a884ab-vime-0b02e4dbdf0b-amd64, immutable digestsha256:ec18cfb1082c560169fd143a2afb52f731b8019316ef338978c0f1704268160d. Remote digest verified; installed tracked source, selected dependencies and FA2 hash checked. Incremental source binding over tested layers, not a full recipe rebuild.latestpromotion or automatic merge.vllm/vime:test-slime-79aa41a884ab-vime-0b02e4dbdf0b-arm64, digestsha256:24016abb21c3f304848a4fb9b576b5cc0a4f6b7c5786e5de5b4080a4fd8743e8, passed206 CPU tests and the original four-GPU Qwen3.5 test on GB200. Three logprob differences:0.007006/0.008015/0.008975; gradients nonzero. Dependency GPU smoke also passed. Source/modes, dependency versions and selected preserved binaries were verified; task cleanup passed. This is not full Nemotron model validation on ARM, and existing compiler warnings remain..buildkite/pipeline-npu.yamlon a main-target PR. This is the previously recorded trigger-scope issue, not an NPU test pass. It is not included in the CPU/H100 coverage claim.Current dual-architecture experimental image
vllm/vime:test-slime-79aa41a884ab-vime-0b02e4dbdf0bIndex:
sha256:5592215935e7f7cc59f77186d6b8fe292517a38ab2e6213addfc131d3b2bc46d.It contains exactly the tested AMD64 and ARM64 platform manifests; registry JSON
and source labels independently verified. The tag identifies the actual built
commit0b02; current head233ad has exactly the same source tree. No latest promotion.
Previous candidate evidence — historical bindings
The following evidence remains attached to its original revisions and image digests. It is not a claim that the new candidate passed those runs.
Previous summary
Synchronize training compatibility and weight-broadcast contracts on top of main720d223, candidate1d9a3e73.
/dev/shmand read it from the same path. No training/precision/assertion changes.Validation
CI runtime image
vllm/vime:test-slime-f22a2b2226d4-vime-21e5791bb36d-amd64Immutable digest:
sha256:316f66e4aed0c71fcd4c710083d8808d946cced22393289406e76ba184b9b82dSource/labels match21e5791; complete tracked-source and symlink checks pass. Dependency versions and FA2 binary hash match the tested parent. Source-only incremental build; no FA/TE rebuild. No latest promotion.
Final dual-architecture experimental image
vllm/vime:test-slime-f22a2b2226d4-vime-1d9a3e73ff5bIndex:
sha256:413268094c3e3e090b8291839076ceea4b3b601a5b44f6e01b1387ffd76b66ddsha256:0be8db78169c92db437b3c9f5cfe30a64791842d5c337f2e926e894b4148e02fsha256:f74efafd485c01cc2c3a893f9f17516e5e39cd9332eb3ccb298551514439e104Both architectures contain final source1d9a3e73, Megatron0.20.0+ae76c60 and resiliency0.7.0. Their tracked-source bytes/modes/symlinks match the same frozen manifest. Selected compiled dependencies remain unchanged relative to their respective validated parents. Index annotations identify this PR and source revision; the ARM config retains a disclosed historical PR473 URL label, not its candidate identity.
ARM's original four-GPU Qwen3.5-0.8B test passed three training steps: logprob differences0.00720709/0.00794848/0.00815425, gradient norms4.10798/4.11254/3.98829. Test and job exited0, task container removed and GPU process query empty. Teardown Inductor warnings remain; this is not a warning-free claim. ARM also passed75+25 CPU tests. No latest promotion.
Remaining
The local checks listed above passed. The additional original fully-async 9B example completed all three training steps, model checkpoint, final weight synchronization and natural exit0; container/GPU cleanup was verified. It retained the eight-GPU 4+4 layout, TP2/CP2, 128K response limit and top128 score centering. Response draining/CPU decoding took substantial time after model saving; this was not a hang. Native layerwise warnings match the untouched vision tower and derived buffers retained by the loader in this text-only example. This is not an independent checkpoint-reload or tensor-by-tensor update verification.
The original eight-GPU Qwen3.5-35B-A3B multimodal regression also passed with TP2/CP2/EP8, fresh checkpoint conversion and unchanged Geo3K/test configuration. Exit and cleanup verified; logprob difference0.0229956. All four responses hit the original512-token limit, with reward/gradient0, so this verifies the lifecycle and original assertions—not learning efficacy. Startup health timeouts recovered; teardown subprocess warnings remain in the logs. Its first local attempt failed while copying protected cached metadata, before training; only the launcher copy permissions were corrected.
All five checkpoint CI variants passed, including async-save. GPU-save/CPU-load needed one automatic retry after Ray returned HTTP500 while submitting the first training job; that failed attempt is retained, not reported as first-attempt success.
Full CI1866 passed CPU7/7 and GPU56/57 on21e5791. The remaining R3 DeepEP/FP8 variant lost its agent four times during checkpoint conversion/save. An exact Pod eviction reason is unavailable; the successful storage adaptation does not uniquely establish the infrastructure root cause.
Isolated storage validation1871 passed both original training steps and assertions after only the checkpoint path adaptation. The tested change is now adopted as1d9a3e73; the disposable single-job experimental pipeline is not included in this PR. Targeted final-head CI1873 passed all7 CPU checks and both affected R3 variants. The other55 GPU cases retain1866's unchanged-code evidence: all57 GPU cases are covered across these bound runs, not one full rerun or a claim that1866 turned green. Unpinned automatic1872 was canceled.
CI1873 uses the CI runtime image316f above plus the exact1d9a3e73 repository checkout; that image's baked source/labels remain21e5791. Only the R3 test file differs. Final AMD64 source rebinding completed in19 seconds and is now published, along with the validated ARM image and dual-architecture candidate above. Final human review/merge and authorized release remain; no automatic merge or latest promotion.
Separately, the current-head NPU pipeline build582 (like earlier577) fails before running tests: its shared bootstrap loads
.buildkite/pipeline-npu.yaml, which is absent on both main and this candidate. This is a pipeline trigger-scope issue, not an NPU test pass or a failure in the CPU/H100 build above.The unchanged legacy build-cu13 alias still overrides the fixed base with floating nightly; the user explicitly chose to record it without changing it this round. It is not used by the fixed-digest experimental build or CI.