Skip to content

[Sync] Update Megatron 0.20 compatibility and packed weight-transfer coverage - #518

Merged
CalvinXKY merged 7 commits into
vllm-project:mainfrom
aoshen02:codex/slime-sync-20261105
Nov 9, 2026
Merged

CalvinXKY merged 7 commits into
vllm-project:mainfrom
aoshen02:codex/slime-sync-20261105

Conversation

@aoshen02

@aoshen02 aoshen02 commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Released after merge

With explicit release authorization, vllm/vime:latest and the immutable vllm/vime:slime-79aa41a884ab-vime-74ac1641d993 now point to the verified AMD64/ARM64 index sha256:1087f4d6f2408c750f78de2de14d87621f163be4b6e286e28f8cb7098a65cbc9. Remote payloads/digests were read back and matched. Built source 74ac1641d and merge 2ce39aa2 have identical trees; no rebuild or dependency upgrade. Validation scope and limitations below remain unchanged; earlier publication-pending statements describe pre-release history.

Review follow-up — current head 74ac1641d

Fixed mixed-resolution image tensor-list normalization and model-local CPU→GPU transfer. Changed-file pre-commit, CPU regression and the final-source eight-H200 mixed-image diagnostic passed (7 passed, 1 optional full-checkpoint header check skipped; logprob differences 0.06275282/0.05960483 against unchanged 0.1 threshold). Micro-batch size was 1; general nested multi-sample batching and learning quality are not claimed. See review reply.

Current-head targeted CI1892 passed: all seven CPU checks and the selected Nemotron GPU regression (7 passed, 1 optional full-checkpoint header check skipped). It uses vllm/vime@sha256:6d13d2f23ea6a50a0ea2bf08f445026ce8089386a810af91820cfd948581ca1a, built from 74ac1641d. This is targeted follow-up, not a new full GPU matrix run. ARM source-only refresh passed 206+168 CPU checks and the original four-GPU Qwen three-step smoke (not ARM Nemotron mixed-image coverage). Both architectures are published as vllm/vime:test-slime-79aa41a884ab-vime-74ac1641d993, index sha256:1087f4d6f2408c750f78de2de14d87621f163be4b6e286e28f8cb7098a65cbc9; both bind the same source and preserve their parent compiled dependencies. Experimental only; latest remains unchanged. Older digests/full CI results below remain historical evidence. No release or latest promotion.


Previous candidate evidence (233ad1dc)

Current PR head 233ad1dc adds Nemotron Super VL model/conversion support, native image resizing and the original eight-GPU proxy test to the prior training-compatibility sync below. It differs from tested/built 0b02e4db only by adding the missing DCO sign-off to the old R3 commit: both trees are exactly ece6e84be6dde650d63bc3697f72442b5084f815. DCO now passes. Image tags retain their actual built-source SHA; fixed-image CPU-only CI1886 passed all7 CPU checks on the rewritten head. Its GPU gate stays intentionally blocked: unchanged GPU evidence comes from CI1879, not a second GPU run.

  • Preserve existing Omni audio behavior and source model/training contracts.
  • Add source-pinned Mamba/causal-conv1d dependencies. The approved compiler compatibility patch removes two CUDA C++17 overrides, leaving PyTorch to choose its supported standard.
  • Use native vLLM multimodal handling, the existing upstream placeholder-mask fix #54548, and the reviewed model lifecycle fix fork #247.
  • Multiple images sharing one image marker remain an explicitly accepted SGLang-only engine difference; no new vLLM behavior was invented for it.

Previous candidate validation

CI attempt history: Qwen30B DeepEP/FP8, Moonlight and Agent each passed after an automatic retry. Qwen and Agent received cancellation signals (exit-1; exact external causes unavailable); Moonlight failed Ray job submission with HTTP500/no available agent before training. These are not first-attempt passes; logs are retained.

  • Changed-file pre-commit passed. Independent shared-core and Docker recipe reviews completed.
  • Current source and installed-image CPU suites each passed 206 tests, no skips, including all2702 Geo3K images and native image-expansion checks.
  • Clean original eight-H200 proxy E2E passed: trainer TP2/CP2/EP8, rollout TP8/EP8, two rollouts, unchanged logprob-difference bound0.1; observed0.06197/0.05875. This is a random reduced proxy, not full-model quality evaluation.
  • The same original proxy also passed on eight H100s in current CI1879 with the final source/image:7 passed/1 optional checkpoint-header skip,592.28s; logprob differences0.05634/0.06499. Gradient norm remains0 in this random/truncated proxy, so this is not learning-quality evidence. Teardown compiler-worker warnings remain in the log.
  • Extra MTP-enabled random-proxy experiment did not pass: nonzero MTP gradients were observed, then the inherited loss<1 assertion failed at12.59. Complete MTP update/quality validation is not claimed.
  • Current AMD64 experimental image: vllm/vime:test-slime-79aa41a884ab-vime-0b02e4dbdf0b-amd64, immutable digest sha256:ec18cfb1082c560169fd143a2afb52f731b8019316ef338978c0f1704268160d. Remote digest verified; installed tracked source, selected dependencies and FA2 hash checked. Incremental source binding over tested layers, not a full recipe rebuild.
  • Same-tree CI1879 passed:7 CPU checks and all58 GPU tests on tested head0b02 and the immutable image above. The executed GPU labels exactly match the candidate's six-suite matrix. Unpinned automatic builds1878/1885 were canceled. No latest promotion or automatic merge.
  • Current ARM image vllm/vime:test-slime-79aa41a884ab-vime-0b02e4dbdf0b-arm64, digest sha256:24016abb21c3f304848a4fb9b576b5cc0a4f6b7c5786e5de5b4080a4fd8743e8, passed206 CPU tests and the original four-GPU Qwen3.5 test on GB200. Three logprob differences:0.007006/0.008015/0.008975; gradients nonzero. Dependency GPU smoke also passed. Source/modes, dependency versions and selected preserved binaries were verified; task cleanup passed. This is not full Nemotron model validation on ARM, and existing compiler warnings remain.
  • Separate NPU check589 fails before tests because the shared pipeline uploads absent .buildkite/pipeline-npu.yaml on a main-target PR. This is the previously recorded trigger-scope issue, not an NPU test pass. It is not included in the CPU/H100 coverage claim.

Current dual-architecture experimental image

vllm/vime:test-slime-79aa41a884ab-vime-0b02e4dbdf0b

Index: sha256:5592215935e7f7cc59f77186d6b8fe292517a38ab2e6213addfc131d3b2bc46d.
It contains exactly the tested AMD64 and ARM64 platform manifests; registry JSON
and source labels independently verified. The tag identifies the actual built
commit0b02; current head233ad has exactly the same source tree. No latest promotion.

Previous candidate evidence — historical bindings

The following evidence remains attached to its original revisions and image digests. It is not a claim that the new candidate passed those runs.

Previous summary

Synchronize training compatibility and weight-broadcast contracts on top of main720d223, candidate1d9a3e73.

  • Upgrade Megatron to core0.20 (ae76c60) and nvidia-resiliency-ext0.7.0; refresh the selected Megatron patch.
  • Mirror run-config/Gloo/checkpoint/MTP/value-head and Qwen constructor/mRoPE compatibility changes.
  • Remove obsolete local FA4 import-order and packed-mRoPE hunks now provided by the pinned native code.
  • Mixed-dtype flattened weight broadcasting already uses native vLLM packed NCCL. Add the matching parser/adapter assertions, not another transport implementation or selector.
  • Correct the CI guides to name the actual Mooncake recovery connector.
  • Keep the vLLM83c base and compiled dependencies unchanged. Preserve source authorship and existing native behavior.
  • Approved Vime CI adaptation: place the R3 test's converted checkpoint in /dev/shm and read it from the same path. No training/precision/assertion changes.

Validation

  • Whole-tree audit plus source-delta/CI-registration review completed; existing unsupported/deferred features remain explicitly tracked.
  • All-file pre-commit and14 existing docs checks passed.
  • Original130 installed-image CPU cases passed; final candidate additionally passes75 parser and25 adapter/factory cases (overlapping counts, not additive).
  • Qwen3.5 four-GPU short training passed, including fault recovery; three logprob differences0.006436–0.008380.
  • MiMo eight-GPU MTP-only test passed: all ranks11 MTP parameters with nonzero gradients, zero non-MTP; MTP loss0.536925, logprob difference0.01007124.
  • Four-GPU recovery passed: real OOM, trainer TP1→TP2 restart, serving retained, data replayed, weight versions2→4. Task containers/GPU allocations cleaned.
  • Original eight-GPU async checkpoint save/load test passed (TP2, CP2, CPU optimizer): real NVRX async save finalized and the subsequent job loaded iteration1 successfully; container/GPU cleanup verified. The first local attempt exhausted a32GiB shared-memory cap; the successful retry changed only that container limit to256GiB. Original training configuration/assertions unchanged. This verifies save/load, not additional optimizer steps after loading.

CI runtime image

vllm/vime:test-slime-f22a2b2226d4-vime-21e5791bb36d-amd64

Immutable digest: sha256:316f66e4aed0c71fcd4c710083d8808d946cced22393289406e76ba184b9b82d

Source/labels match21e5791; complete tracked-source and symlink checks pass. Dependency versions and FA2 binary hash match the tested parent. Source-only incremental build; no FA/TE rebuild. No latest promotion.

Final dual-architecture experimental image

vllm/vime:test-slime-f22a2b2226d4-vime-1d9a3e73ff5b

Index: sha256:413268094c3e3e090b8291839076ceea4b3b601a5b44f6e01b1387ffd76b66dd

  • AMD64 platform manifest: sha256:0be8db78169c92db437b3c9f5cfe30a64791842d5c337f2e926e894b4148e02f
  • ARM64 platform manifest: sha256:f74efafd485c01cc2c3a893f9f17516e5e39cd9332eb3ccb298551514439e104

Both architectures contain final source1d9a3e73, Megatron0.20.0+ae76c60 and resiliency0.7.0. Their tracked-source bytes/modes/symlinks match the same frozen manifest. Selected compiled dependencies remain unchanged relative to their respective validated parents. Index annotations identify this PR and source revision; the ARM config retains a disclosed historical PR473 URL label, not its candidate identity.

ARM's original four-GPU Qwen3.5-0.8B test passed three training steps: logprob differences0.00720709/0.00794848/0.00815425, gradient norms4.10798/4.11254/3.98829. Test and job exited0, task container removed and GPU process query empty. Teardown Inductor warnings remain; this is not a warning-free claim. ARM also passed75+25 CPU tests. No latest promotion.

Remaining

The local checks listed above passed. The additional original fully-async 9B example completed all three training steps, model checkpoint, final weight synchronization and natural exit0; container/GPU cleanup was verified. It retained the eight-GPU 4+4 layout, TP2/CP2, 128K response limit and top128 score centering. Response draining/CPU decoding took substantial time after model saving; this was not a hang. Native layerwise warnings match the untouched vision tower and derived buffers retained by the loader in this text-only example. This is not an independent checkpoint-reload or tensor-by-tensor update verification.

The original eight-GPU Qwen3.5-35B-A3B multimodal regression also passed with TP2/CP2/EP8, fresh checkpoint conversion and unchanged Geo3K/test configuration. Exit and cleanup verified; logprob difference0.0229956. All four responses hit the original512-token limit, with reward/gradient0, so this verifies the lifecycle and original assertions—not learning efficacy. Startup health timeouts recovered; teardown subprocess warnings remain in the logs. Its first local attempt failed while copying protected cached metadata, before training; only the launcher copy permissions were corrected.

All five checkpoint CI variants passed, including async-save. GPU-save/CPU-load needed one automatic retry after Ray returned HTTP500 while submitting the first training job; that failed attempt is retained, not reported as first-attempt success.

Full CI1866 passed CPU7/7 and GPU56/57 on21e5791. The remaining R3 DeepEP/FP8 variant lost its agent four times during checkpoint conversion/save. An exact Pod eviction reason is unavailable; the successful storage adaptation does not uniquely establish the infrastructure root cause.

Isolated storage validation1871 passed both original training steps and assertions after only the checkpoint path adaptation. The tested change is now adopted as1d9a3e73; the disposable single-job experimental pipeline is not included in this PR. Targeted final-head CI1873 passed all7 CPU checks and both affected R3 variants. The other55 GPU cases retain1866's unchanged-code evidence: all57 GPU cases are covered across these bound runs, not one full rerun or a claim that1866 turned green. Unpinned automatic1872 was canceled.

CI1873 uses the CI runtime image316f above plus the exact1d9a3e73 repository checkout; that image's baked source/labels remain21e5791. Only the R3 test file differs. Final AMD64 source rebinding completed in19 seconds and is now published, along with the validated ARM image and dual-architecture candidate above. Final human review/merge and authorized release remain; no automatic merge or latest promotion.

Separately, the current-head NPU pipeline build582 (like earlier577) fails before running tests: its shared bootstrap loads .buildkite/pipeline-npu.yaml, which is absent on both main and this candidate. This is a pipeline trigger-scope issue, not an NPU test pass or a failure in the CPU/H100 build above.

The unchanged legacy build-cu13 alias still overrides the fixed base with floating nightly; the user explicitly chose to record it without changing it this round. It is not used by the fixed-digest experimental build or CI.

aoshen02 and others added 4 commits November 5, 2026 12:41
Co-authored-by: Shifang Xu <shifangx@nvidia.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Shifang Xu <shifangx@nvidia.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Junting Lv <49749126+sheep0923@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the codebase to align with a newer Megatron-LM commit, including support for the 'nvrx' async checkpointing strategy via nvidia-resiliency-ext, handling of run configurations, and updated Multi-Token Prediction (MTP) loss tracking. It also updates documentation, adds corresponding unit tests, and introduces model-specific compatibility fixes. The review feedback suggests adding defensive None checks on the MTP logging tracker to prevent potential runtime errors, as well as wrapping the nvidia-resiliency-ext import in a try-except block to provide a more descriptive error message if the dependency is missing.

Comment on lines +106 to +111
def _async_writer_class(async_strategy):
if async_strategy == "nvrx":
from nvidia_resiliency_ext.checkpointing.async_ckpt.filesystem_async import FileSystemWriterAsync
else:
from megatron.core.dist_checkpointing.strategies.filesystem_async import FileSystemWriterAsync
return FileSystemWriterAsync

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If nvidia-resiliency-ext is not installed in the environment, attempting to use the 'nvrx' async strategy will result in a generic ImportError. It is highly recommended to wrap this import in a try-except block and raise a more descriptive error message to guide the user to install the missing dependency.

Suggested change
def _async_writer_class(async_strategy):
if async_strategy == "nvrx":
from nvidia_resiliency_ext.checkpointing.async_ckpt.filesystem_async import FileSystemWriterAsync
else:
from megatron.core.dist_checkpointing.strategies.filesystem_async import FileSystemWriterAsync
return FileSystemWriterAsync
def _async_writer_class(async_strategy):
if async_strategy == "nvrx":
try:
from nvidia_resiliency_ext.checkpointing.async_ckpt.filesystem_async import FileSystemWriterAsync
except ImportError as e:
raise ImportError(
"nvidia-resiliency-ext is required when using the 'nvrx' async strategy. "
"Please install it or choose a different strategy."
) from e
else:
from megatron.core.dist_checkpointing.strategies.filesystem_async import FileSystemWriterAsync
return FileSystemWriterAsync

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The supported image explicitly installs nvidia-resiliency-ext==0.7.0, and the original eight-GPU async checkpoint save/load test passed with this dependency. We retain the direct import rather than wrap an otherwise actionable ImportError; recommending another strategy would also be misleading for the new MCore path.

tracker = MTPLossLoggingHelper.tracker
# Megatron ae76c60e6 accumulates into "loss_sums" and reduces across ranks itself;
# "values" only exists after reduce_loss_in_tracker().
reduced_by_megatron = "loss_sums" in tracker and hasattr(MTPLossLoggingHelper, "reduce_loss_in_tracker")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To prevent potential TypeError or AttributeError if MTPLossLoggingHelper.tracker is None or uninitialized, we should defensively check that tracker is not None before performing membership checks.

Suggested change
reduced_by_megatron = "loss_sums" in tracker and hasattr(MTPLossLoggingHelper, "reduce_loss_in_tracker")
reduced_by_megatron = tracker is not None and "loss_sums" in tracker and hasattr(MTPLossLoggingHelper, "reduce_loss_in_tracker")

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked both supported MCore implementations: MTPLossLoggingHelper.tracker is initialized to {}, and its cleanup routines clear tensor contents rather than replacing the dictionary with None. The original eight-GPU MTP-only training test also passed. A None guard would cover an unsupported state, so we keep the direct dictionary contract.

Comment on lines 946 to 947
loss_key = "loss_values" if "loss_values" in tracker else "values"
if loss_key in tracker:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Similarly, defensively check that tracker is not None before checking for "loss_values" or accessing keys to avoid a TypeError if the tracker is uninitialized.

Suggested change
loss_key = "loss_values" if "loss_values" in tracker else "values"
if loss_key in tracker:
loss_key = "loss_values" if tracker is not None and "loss_values" in tracker else "values"
if tracker is not None and loss_key in tracker:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same dictionary invariant as the preceding comment: tracker remains a dictionary through initialization and cleanup. Keeping this unchanged avoids a redundant guard; the MTP-only GPU regression passed.

@read-the-docs-community

read-the-docs-community Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

aoshen02 and others added 2 commits November 7, 2026 14:41
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Preserve upstream model and training contracts, translate native multimodal handling, and include validated reload and dependency compatibility fixes.

Co-authored-by: Shifang Xu <shifangx@nvidia.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: hotragn <hotragn.pettugani_2024@woxsen.edu.in>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
@aoshen02
aoshen02 force-pushed the codex/slime-sync-20261105 branch from 0b02e4d to 233ad1d Compare November 7, 2026 14:41

@CalvinXKY CalvinXKY left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One merge blocker on the claimed multi-image Nemotron path (inline). Geo3K / single-image e2e are fine.

Comment thread vime/backends/megatron_utils/data.py Outdated
for mm_input_dict in multimodal_train_inputs:
if mm_input_dict is not None:
for key, mm_tensor in mm_input_dict.items():
mm_tensor = torch.atleast_1d(torch.as_tensor(mm_tensor))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: when one prompt has multiple Nemotron images at different target resolutions, the HF processor returns pixel_values as a Python list of CHW tensors (it only stacks when every tile matches). torch.as_tensor(list_of_tensors) always raises ValueError, so training dies here before NemotronHVLModel can iterate the list.

Same-size multi-image (stacked NCHW) and cross-sample mixed sizes (one NCHW tensor per sample) are fine; this is the within-sample mixed-size path that nemotron_h_vl.py and test_nemotron_vl_image_expansion already claim.

Please keep a list/tuple of tensors as-is (or stack only equal shapes) instead of unconditional as_tensor.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 74ac164. Nonempty tensor lists/tuples now bypass as_tensor, while empty sequences and ordinary numeric metadata retain the existing normalization. Audio padding is unchanged.

The real processor produces CHW tensors of shapes (3,480,576) and (3,704,384). We reproduced the conversion failure. Continuing through the real GPU vision path also exposed CPU image tensors reaching CUDA weights, so the model's existing per-image dtype conversion now also specifies embeddings.device.

Validation on H200 under Slurm:

  • 168 existing CPU tests and changed-file pre-commit passed.
  • Original eight-GPU proxy regression: 7 passed, 1 optional full-checkpoint header check skipped.
  • Additional two-image/mixed-resolution eight-GPU run with the final source: 7 passed, the same optional check skipped. Two steps had train/rollout logprob differences 0.06275282 and 0.05960483, below the unchanged 0.1 threshold. TP2/CP2/EP8 training, TP8/EP8 rollout, micro-batch size 1; existing training parameters and assertions unchanged. Diagnostic inputs duplicate four Geo3K images at different aspect ratios with two image markers each.

This verifies mixed-size images within one sample. Multiple samples containing nested image lists in the same micro-batch remain a separate batching limitation; this fix does not claim to support that case. The random proxy's gradients were zero, so this is execution/alignment evidence, not learning-quality validation. Candidate image/remote CI refresh is still pending.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

Signed-off-by: aoshen02 <aoshen@inferact.ai>
@CalvinXKY
CalvinXKY merged commit 2ce39aa into vllm-project:main Nov 9, 2026
3 of 4 checks passed
Meihan-chen added a commit to Meihan-chen/vime that referenced this pull request Nov 9, 2026
Integrate upstream vllm-project#513 and vllm-project#518. Use the shared MTP expert-name handling while preserving existing NPU integration and the ROCm Dockerfile deletion.

Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants