Repository navigation
[None][feat] Support MiniMax-M3 sparse KV offload - #22410
Draft
cascade812 wants to merge 3 commits into
Draft
cascade812 wants to merge 3 commits into
cascade812 wants to merge 3 commits into
Conversation
Collaborator
|
Hi, heads-up: #22286 has merged, so ModelingV2 now lives at Since this PR touches modeling_v2, please rebase it onto the latest main. You can use these migration commands to move any files you added under the old directory and update any remaining references to the old path. |
Keep sparse M3 K/V history and NVFP4 scales on host and fetch selected blocks into a bounded GPU arena. Restore and flush partial blocks, preserve per-head selections, and align resident/offloaded transfer pool layouts. Add configuration schemas, documentation, and CI coverage. Validate exact resident/offload tokens for FP8 and NVFP4 on nine requests each, including repeated and concurrent LongBench continuations with native host-fetch evidence. Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com> Signed-off-by: Guiju Zhang <guijuz@login-lyris02.lyris.clusters.nvidia.com>
Cover full-document replays, KV page boundaries, and concurrent decoding with request retirement for FP8 and NVFP4 KV. Keep LongBench answer scores informational and require exact token and finish-reason parity. Fix prefill chunk shapes and synchronize concurrent decode admission so paired cache modes use identical batch schedules. Signed-off-by: Guiju Zhang <guijuz@login-lyris02.lyris.clusters.nvidia.com>
Cover reused prompt K/V, NVFP4 scales, and index keys through native cache transitions and exact attention parity. Add FP8 and NVFP4 serving parity cases with block reuse enabled and require cache hits for every replayed document. Compare the same received-prompt reuse policy in both modes so generation-time rebasing cannot change the reference. Document generation-worker prompt reuse and the deterministic-mode override instead of requiring block reuse to be disabled. Signed-off-by: Guiju Zhang <guijuz@login-lyris02.lyris.clusters.nvidia.com>
cascade812
force-pushed
the
guiju/minimax-m3-sparse-kv-offload
branch
from
November 17, 2026 03:34
76ca3fc to
ae88318
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
MiniMax-M3 disaggregated generation keeps sparse main K/V history on the GPU. This adds opt-in sparse KV offload to the built-in MSA backend and KV cache manager V2: sparse main K/V and NVFP4 block scales live in host memory, selected blocks are fetched into a bounded GPU cache, and dense layers and index keys stay resident.
This extends the shared offload implementation from #21350. The change handles the union of selections from local KV heads, restores each head's original selection for attention, and preserves the growing 128-token block through transfer, eviction, and writeback. Native cache handling, MSA decode, and transfer layouts are included together because they must agree on the same block ownership and storage layout. Configuration validation, generated serving schemas, routing, documentation, and CI entries cover the new path.
Enable
sparse_attention_config.enable_kv_cache_offloadon the generation worker withimplementation: msa, KV cache manager V2, 128-token pages, and pipeline parallel size 1. Offload supports single-token, single-beam generation without speculative decoding. Received-prompt block reuse is supported for FP8 and NVFP4 KV on the generation worker; offloaded generated history is not published to the reuse tree. The context worker remains resident and uses matching KV precision.The integration profiles cover block reuse enabled and disabled, checking 30 paired requests for each of FP8 and NVFP4 KV: full-document prompts, replays, page-boundary prefixes, concurrent decoding, and request retirement. Paired runs use identical prefill and decode schedules and require exact token IDs, finish reasons, and evidence of native host fetches. The reuse profile also requires cache hits for all six replayed documents. LongBench answer scores are informational and do not impose an accuracy threshold.
The reuse comparison sets
kv_cache_config.block_reuse_config.policy: per_requestin both generation modes. Residentall_reusablecan substitute cached prompt pages into newly completed generation blocks; offload publishes only received prompts, so both modes must use the same publication policy for this parity check. Context-worker reuse stays disabled, andFORCE_DETERMINISTIC=0allows generation-worker reuse.Test Coverage
The branch is based on
mainat2978f8c57eand contains only the three MiniMax-M3 commits.After rebasing, 38 CPU parity-check tests and 24 M3 schema snapshot cases passed. All 22 changed Python files/stubs parse, and the feature edits across all 40 files match the previously validated changes after accounting for the upstream
modeling_v2directory move. The routing test import was updated for that move and passed its pre-commit hooks. DCO and whitespace checks passed.GPU/native tests and full schema regeneration were not rerun after this rebase. The validation below was collected before the history rewrite:
76ca3fc08d, prompt reuse passed 30/30 exact-token parity pairs with FP8 KV and 30/30 with NVFP4 KV, using MiniMax-M3-NVFP4 weights on four GB300 GPUs per run, TP2/EP2 context and generation workers, KV cache manager V2, and 128-token pages. Each precision reused 99,456 prompt tokens in each resident/offload replay round. CUDA graphs were disabled. All 30 M3 offload component tests passed, including 12 reuse cases across FP8/NVFP4, 1/2/4 local KV heads, and identical or shorter-prefix requests.8e0b1232a5, block reuse disabled passed 30/30 paired requests with FP8 KV and 30/30 with NVFP4 KV. Both profiles cover full-document input lengths of 10,120–24,749 tokens; boundaries at 4,479/4,480/4,481, 8,191/8,192/8,193, and 16,383/16,384/16,385 tokens; concurrent equal/mixed output lengths; and request retirement.b2849b824bpassed 1,037 focused tests: 321 M3/cache/transfer tests, 549 CPU serving-schema tests, 129 native KV cache manager V2 tests, and 38 parity-check tests. Schemas and the LLM-args golden manifest regenerated without changes; schema freshness, pre-commit, and DCO checks passed.The GB300 end-to-end CI entries cover
TestMiniMaxM3::test_kv_cache_offloadandTestMiniMaxM3::test_kv_cache_offload_block_reuse, each withkv_dtype=fp8andkv_dtype=nvfp4, intests/integration/defs/accuracy/test_disaggregated_serving.py.PR Checklist