Skip to content

[None][feat] Support MiniMax-M3 sparse KV offload - #22410

Draft
cascade812 wants to merge 3 commits into
NVIDIA:mainfrom
cascade812:guiju/minimax-m3-sparse-kv-offload
Draft

cascade812 wants to merge 3 commits into
NVIDIA:mainfrom
cascade812:guiju/minimax-m3-sparse-kv-offload

Conversation

@cascade812

@cascade812 cascade812 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Description

MiniMax-M3 disaggregated generation keeps sparse main K/V history on the GPU. This adds opt-in sparse KV offload to the built-in MSA backend and KV cache manager V2: sparse main K/V and NVFP4 block scales live in host memory, selected blocks are fetched into a bounded GPU cache, and dense layers and index keys stay resident.

This extends the shared offload implementation from #21350. The change handles the union of selections from local KV heads, restores each head's original selection for attention, and preserves the growing 128-token block through transfer, eviction, and writeback. Native cache handling, MSA decode, and transfer layouts are included together because they must agree on the same block ownership and storage layout. Configuration validation, generated serving schemas, routing, documentation, and CI entries cover the new path.

Enable sparse_attention_config.enable_kv_cache_offload on the generation worker with implementation: msa, KV cache manager V2, 128-token pages, and pipeline parallel size 1. Offload supports single-token, single-beam generation without speculative decoding. Received-prompt block reuse is supported for FP8 and NVFP4 KV on the generation worker; offloaded generated history is not published to the reuse tree. The context worker remains resident and uses matching KV precision.

The integration profiles cover block reuse enabled and disabled, checking 30 paired requests for each of FP8 and NVFP4 KV: full-document prompts, replays, page-boundary prefixes, concurrent decoding, and request retirement. Paired runs use identical prefill and decode schedules and require exact token IDs, finish reasons, and evidence of native host fetches. The reuse profile also requires cache hits for all six replayed documents. LongBench answer scores are informational and do not impose an accuracy threshold.

The reuse comparison sets kv_cache_config.block_reuse_config.policy: per_request in both generation modes. Resident all_reusable can substitute cached prompt pages into newly completed generation blocks; offload publishes only received prompts, so both modes must use the same publication policy for this parity check. Context-worker reuse stays disabled, and FORCE_DETERMINISTIC=0 allows generation-worker reuse.

Test Coverage

The branch is based on main at 2978f8c57e and contains only the three MiniMax-M3 commits.

After rebasing, 38 CPU parity-check tests and 24 M3 schema snapshot cases passed. All 22 changed Python files/stubs parse, and the feature edits across all 40 files match the previously validated changes after accounting for the upstream modeling_v2 directory move. The routing test import was updated for that move and passed its pre-commit hooks. DCO and whitespace checks passed.

GPU/native tests and full schema regeneration were not rerun after this rebase. The validation below was collected before the history rewrite:

  • At 76ca3fc08d, prompt reuse passed 30/30 exact-token parity pairs with FP8 KV and 30/30 with NVFP4 KV, using MiniMax-M3-NVFP4 weights on four GB300 GPUs per run, TP2/EP2 context and generation workers, KV cache manager V2, and 128-token pages. Each precision reused 99,456 prompt tokens in each resident/offload replay round. CUDA graphs were disabled. All 30 M3 offload component tests passed, including 12 reuse cases across FP8/NVFP4, 1/2/4 local KV heads, and identical or shorter-prefix requests.
  • At 8e0b1232a5, block reuse disabled passed 30/30 paired requests with FP8 KV and 30/30 with NVFP4 KV. Both profiles cover full-document input lengths of 10,120–24,749 tokens; boundaries at 4,479/4,480/4,481, 8,191/8,192/8,193, and 16,383/16,384/16,385 tokens; concurrent equal/mixed output lengths; and request retirement.
  • A fresh native build at b2849b824b passed 1,037 focused tests: 321 M3/cache/transfer tests, 549 CPU serving-schema tests, 129 native KV cache manager V2 tests, and 38 parity-check tests. Schemas and the LLM-args golden manifest regenerated without changes; schema freshness, pre-commit, and DCO checks passed.

The GB300 end-to-end CI entries cover TestMiniMaxM3::test_kv_cache_offload and TestMiniMaxM3::test_kv_cache_offload_block_reuse, each with kv_dtype=fp8 and kv_dtype=nvfp4, in tests/integration/defs/accuracy/test_disaggregated_serving.py.

PR Checklist

  • Description, coding guidelines, tests, and documentation reviewed.
  • New paths have component and integration coverage.
  • Configuration is opt-in and backward compatible.
  • Generated schemas and manifest are current.
  • No dependency or ownership changes.

@cascade812 cascade812 added the api-compatible Accepted LLM API contract change that is backwards-compatible label Nov 17, 2026 — with ChatGPT Codex Connector
@ruocheng-nv

Copy link
Copy Markdown
Collaborator

Hi, heads-up: #22286 has merged, so ModelingV2 now lives at tensorrt_llm/_torch/modeling_v2 (module tensorrt_llm._torch.modeling_v2), and the old _torch/_experimental/modeling_v2 path no longer exists.

Since this PR touches modeling_v2, please rebase it onto the latest main. You can use these migration commands to move any files you added under the old directory and update any remaining references to the old path.

cascade812 and others added 3 commits November 16, 2026 19:33
Keep sparse M3 K/V history and NVFP4 scales on host and fetch selected blocks into a bounded GPU arena. Restore and flush partial blocks, preserve per-head selections, and align resident/offloaded transfer pool layouts.

Add configuration schemas, documentation, and CI coverage. Validate exact resident/offload tokens for FP8 and NVFP4 on nine requests each, including repeated and concurrent LongBench continuations with native host-fetch evidence.

Signed-off-by: Guiju Zhang <7135567+cascade812@users.noreply.github.com>
Signed-off-by: Guiju Zhang <guijuz@login-lyris02.lyris.clusters.nvidia.com>
Cover full-document replays, KV page boundaries, and concurrent decoding with request retirement for FP8 and NVFP4 KV. Keep LongBench answer scores informational and require exact token and finish-reason parity.

Fix prefill chunk shapes and synchronize concurrent decode admission so paired cache modes use identical batch schedules.

Signed-off-by: Guiju Zhang <guijuz@login-lyris02.lyris.clusters.nvidia.com>
Cover reused prompt K/V, NVFP4 scales, and index keys through native
cache transitions and exact attention parity. Add FP8 and NVFP4 serving
parity cases with block reuse enabled and require cache hits for every
replayed document. Compare the same received-prompt reuse policy in
both modes so generation-time rebasing cannot change the reference.

Document generation-worker prompt reuse and the deterministic-mode
override instead of requiring block reuse to be disabled.

Signed-off-by: Guiju Zhang <guijuz@login-lyris02.lyris.clusters.nvidia.com>
@cascade812
cascade812 force-pushed the guiju/minimax-m3-sparse-kv-offload branch from 76ca3fc to ae88318 Compare November 17, 2026 03:34

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-compatible Accepted LLM API contract change that is backwards-compatible

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants