Repository navigation
feat(kv_connector): mooncake store support GLM-5.3-Flash - #5034
Merged
RunningLeon merged 82 commits intoOct 16, 2026
Merged
RunningLeon merged 82 commits into
RunningLeon merged 82 commits into
Conversation
Reuse MLA/MoE loaders, FA3, mHC, sparse Top-K and compact DeepGEMM. Add KDA state adaptation and multimodal GLM configuration/processing. Preserve existing defaults and the positional MoE prefix argument.
Reuse GDN state helpers and channelwise TileLang verification; honor MTP index sharing and invalidate graphs after buffer growth. Share AR sampling filters with rejection/recovery and stabilize GLM TP/mHC and sparse-index arithmetic. Keep EP/DP, prefix-cache and performance experiments outside this patch.
Reject unsupported DLINFER MoE reduction options instead of ignoring them. Leave NCCL NVLS policy to the runtime and remove model-specific environment plumbing. Document GLM FP32 operator usage and cover dtype, reduction, and environment contracts. Validation: 110 focused tests passed on CPU/H200; Ruff passed for lmdeploy and tests/pytorch. DLINFER rejection regressions failed before the fix and pass afterward.
Apply the configured string and docstring formatters to GLM PR files. Verified executable AST is unchanged; both previously failing hooks and Ruff pass locally.
Replace the compiled GLM helper with an opt-in FP32 compute contract through ApplyRotaryEmb and its backend build spec. CUDA retains FP32 arithmetic inside the fused kernel; defaults remain unchanged and unsupported Dlinfer requests fail explicitly. Validated 73 focused tests and actual vLLM rotary kernel parity with identical inputs and tables. FP32-table outputs can differ from the old compiled helper at rounding boundaries.
Reuse the existing strided TileLang recurrent inputs instead of copying Q/K/V in AR and speculative decode. Preserve contiguous prefill inputs and explicitly allocate contiguous recurrent output for channel-major views. Validated 25 new stride/state/graph tests, 25 shared GDR tests, and 12 complete KDA adapter comparisons. TP4 MTP-off text regression matches the previous head exactly at 124 full-vocabulary logits positions and all 29 requests.
Consolidate PR-based P2 and the staged selective opt port before rebasing. Original HEAD: 7ef51dce043d3655a356ed687cadc5922196c115 Pre-rebase snapshot: ff1b28d18307cfbcdb6640e5718daafa88d7ec4c
Handle unsupported AutoProcessor import/value errors with the existing GLM-4V compatibility builder. Cover native, missing-modality, error propagation and trust-remote-code paths. Update DeepEP FP32 option and FlashInfer dtype/workspace fixtures, and repair the two dead Llama 3.1 documentation links.
Fuse target raw-ring updates and restore fused draft decoding while preserving cache state semantics.
Integrate PR2 with PR4968 development branch
…_p/min_p" This reverts commit 387e561.
Keep unified MTP prefill and Mooncake state-cache semantics while integrating DCP attention and the draft DP-phase transition fix. Validation on H200: 847 changed/core tests, 132 GPU tests, and 419 additional owner-level regression tests passed without skips.
RunningLeon
requested review from
RunningLeon and
yidingcheng0206
and
a balanced review from Copilot
October 16, 2026 04:08
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Checkpoint planning can stall multimodal prefill, and local-only checkpoint chunking can lose convolution history without TileLang.
2 open findings
What changed in this PR
Extends Mooncake Store prefix reuse to GLM-5.3-Flash’s hybrid attention and MTP caches, building on the existing distributed KV-cache integration.
Changes:
- Adds asynchronous hybrid-state transfers and coordinated target/draft checkpoints.
- Unifies prefill alignment and adds multimodal cache identity.
- Adds external-cache metrics, configuration options, documentation, and regression tests.
| File | Description |
|---|---|
| tests/test_lmdeploy/test_metrics_loggers.py | Tests external-cache logs and counters. |
| tests/test_lmdeploy/serve/parsers/test_required_tool_choice.py | Separates test tool calls with newlines. |
| tests/pytorch/spec_decode/test_spec_agent.py | Tests shifted MTP chunks and multimodal equivalence. |
| tests/pytorch/paging/test_scheduler.py | Tests aligned checkpoints and cache metrics. |
| tests/pytorch/paging/test_scheduler_kv_transfer.py | Tests remote-hit accounting and rollback. |
| tests/pytorch/paging/test_prefill_scheduler.py | Tests alignment, reservations, and scheduling estimates. |
| tests/pytorch/kv_connector/test_mooncake_store_lookup.py | Tests lookup rewind metadata. |
| tests/pytorch/kv_connector/test_mooncake_store_connector.py | Tests state-pool registration forwarding. |
| tests/pytorch/kernel/test_kpool.py | Tests restored target/MTP tail reconstruction. |
| tests/pytorch/kernel/test_kda.py | Tests restored recurrent-state correctness. |
| tests/pytorch/engine/test_ray_executor_kv_connector.py | Tests request-level transfer failures. |
| tests/pytorch/engine/test_model_agent.py | Tests short convolution chunks and stream safety. |
| tests/pytorch/engine/test_model_agent_kv_connector.py | Tests registration and target/MTP transfer ordering. |
| tests/pytorch/engine/test_inputs_maker.py | Tests chunk boundaries and checkpoint-copy inputs. |
| tests/pytorch/engine/test_executor_base.py | Tests configuration validation and snapshot budgets. |
| tests/pytorch/engine/test_cache_engine/test_state_engine.py | Tests transferable state pools and local copies. |
| tests/pytorch/engine/test_cache_engine/test_schema.py | Tests cache persistence flags. |
| tests/pytorch/engine/test_cache_engine_connector_caches.py | Tests exclusion of local-only MTP tails. |
| tests/pytorch/config/test_hf_overrides.py | Uses the public speculative configuration. |
| lmdeploy/pytorch/strategies/base/sequence.py | Defines draft-input shift metadata. |
| lmdeploy/pytorch/strategies/ar_spec/sequence.py | Makes rewind and input-shift policies configurable. |
| lmdeploy/pytorch/strategies/ar_spec/__init__.py | Selects method-specific sequence policies. |
| lmdeploy/pytorch/spec_decode/spec_agent.py | Replaces chunk carry with prompt lookahead. |
| lmdeploy/pytorch/prefix_cache_state.py | Clarifies block identity typing. |
| lmdeploy/pytorch/paging/state_manager.py | Reserves Store snapshots outside runtime allocation. |
| lmdeploy/pytorch/paging/scheduler.py | Enables hybrid loads, alignment, and external metrics. |
| lmdeploy/pytorch/paging/prefill_scheduler.py | Updates aligned scheduling and load admission. |
| lmdeploy/pytorch/paging/kv_load_coordinator.py | Coordinates hybrid restore, rollback, and accounting. |
| lmdeploy/pytorch/paging/eviction_helper/recompute_eviction_helper.py | Excludes recomputation from external statistics. |
| lmdeploy/pytorch/nn/kpool.py | Normalizes error-string quoting. |
| lmdeploy/pytorch/models/glm5_next.py | Marks reconstructible MTP tails local-only. |
| lmdeploy/pytorch/model_inputs.py | Carries non-final prompt lookahead. |
| lmdeploy/pytorch/messages.py | Adds alignment metadata and Store multimodal identity. |
| lmdeploy/pytorch/long_context.py | Plans aligned, shifted multimodal chunks. |
| lmdeploy/pytorch/kv_connector/mooncake/store/worker.py | Registers state pools and finds complete checkpoints. |
| lmdeploy/pytorch/kv_connector/mooncake/store/worker_threads.py | Transfers state snapshots alongside KV blocks. |
| lmdeploy/pytorch/kv_connector/mooncake/store/scheduler.py | Manages hybrid snapshots, hashes, and failures. |
| lmdeploy/pytorch/kv_connector/mooncake/store/protocol.py | Documents lookup rewind framing. |
| lmdeploy/pytorch/kv_connector/mooncake/store/lookup.py | Transmits rewind policy to workers. |
| lmdeploy/pytorch/kv_connector/mooncake/store/data.py | Defines state-transfer records and multimodal hashing. |
| lmdeploy/pytorch/kv_connector/mooncake/store/connector.py | Forwards state-pool registration. |
| lmdeploy/pytorch/kv_connector/mooncake/store/__init__.py | Exports state registration metadata. |
| lmdeploy/pytorch/kv_connector/base.py | Adds state pools and request-level failure reporting. |
| lmdeploy/pytorch/kv_connector/__init__.py | Exports the cache-pool interface. |
| lmdeploy/pytorch/engine/model_agent/agent.py | Coordinates MTP checkpoint copies and registration. |
| lmdeploy/pytorch/engine/inputs_maker.py | Builds aligned inputs and state-save plans. |
| lmdeploy/pytorch/engine/executor/base.py | Validates hybrid support and budgets snapshot memory. |
| lmdeploy/pytorch/engine/engine.py | Computes multimodal identity for Store reuse. |
| lmdeploy/pytorch/engine/config_builder.py | Propagates new cache and speculative options. |
| lmdeploy/pytorch/engine/cache_engine/state.py | Exposes transferable state pools. |
| lmdeploy/pytorch/engine/cache_engine/schema.py | Propagates cache persistence flags. |
| lmdeploy/pytorch/engine/cache_engine/layout.py | Defines transferable byte ranges. |
| lmdeploy/pytorch/engine/cache_engine/engine.py | Excludes local-only pools from transfers. |
| lmdeploy/pytorch/configurations/glm5_next.py | Excludes reconstructible KPool state tails. |
| lmdeploy/pytorch/config.py | Defines alignment, snapshot, and rewind settings. |
| lmdeploy/pytorch/backends/gated_delta_rule.py | Tracks short-chunk convolution history. |
| lmdeploy/pytorch/backends/default/cache.py | Separates persisted and local-only block pools. |
| lmdeploy/pytorch/backends/cuda/causal_conv1d.py | Preserves history when saving short-chunk state. |
| lmdeploy/metrics/stats.py | Carries cumulative external-cache statistics. |
| lmdeploy/metrics/loggers.py | Reports external-cache logs and Prometheus metrics. |
| lmdeploy/messages.py | Exposes public configuration and metric fields. |
| lmdeploy/cli/utils.py | Defines alignment, snapshot, and rewind flags. |
| lmdeploy/cli/serve.py | Wires new serving options. |
| lmdeploy/cli/cli.py | Wires new chat options. |
| docs/zh_cn/advance/spec_decoding.md | Documents shared checkpoint alignment in Chinese. |
| docs/zh_cn/advance/metrics.md | Documents external-cache metrics in Chinese. |
| docs/en/advance/spec_decoding.md | Documents shared checkpoint alignment. |
| docs/en/advance/metrics.md | Documents external-cache metrics and queries. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+64
to
+69
| if (cache_config.use_mooncake_store and model_config.states_shapes | ||
| and model_config.is_gated_delta and device_type == 'cuda'): | ||
| from lmdeploy.pytorch.backends.cuda.utils import has_tilelang | ||
| if not has_tilelang(): | ||
| raise RuntimeError('Hybrid Mooncake Store requires TileLang causal convolution on CUDA: ' | ||
| 'the Dao backend does not restore prefill initial states') |
Comment on lines
105
to
106
| if seq.num_history_ids < step and seq.is_prefix_cache_boundary_safe(step): | ||
| limit = min(limit, step - seq.num_history_ids) |
wanfengcxz
added a commit
to wanfengcxz/lmdeploy
that referenced
this pull request
Oct 27, 2026
Rebase fallout from upstream InternLM#5034 (mooncake store support GLM-5.3-Flash), which reworked the connector around linear-attention state pools: * start_load_kv\x27s new state-load fence went through torch.cuda.Event(); route it through TransferBackend.make_ready_event() like the save path so state loads can record on Ascend. * _build_state_registrations required tensor.is_cuda; apply the same cuda/npu device rule as KV rows so hybrid (GLM-5.3-Flash) state pools can register NPU HBM. * the KV-row device check additionally honors is_cuda-only tensors again: upstream tests expose CUDA registration over host memory through a forced is_cuda on a device=\x27cpu\x27 tensor subclass, which the pure device-string check would reject. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Motivation
This PR extends the Mooncake Store integration introduced in #4903 to support GLM-5.3-Flash with hybrid attention and multi-token prediction (MTP).
GLM-5.3-Flash requires both attention KV caches and linear-attention states to resume from a cached prefix. With MTP enabled, the corresponding target and draft caches must also be consistent at the restored position. Transferring KV blocks alone is therefore insufficient.
This PR adds:
These changes allow GLM-5.3-Flash to reuse cached prefixes after local eviction, reducing repeated prefill computation in long-context, multi-turn workloads.
Deploy
The benchmark deployment uses 8 × NVIDIA H200, the PyTorch backend, TP=8, Ray, local prefix caching, and MTP with 3 draft tokens.
1. Start the Mooncake master
2. Prepare the Mooncake config
Save the following as
mooncake_config.json. Adjust the master address and RDMA device names for your environment.{ "mode": "embedded", "metadata_server": "P2PHANDSHAKE", "master_server_address": "127.0.0.1:50051", "global_segment_size": "160GB", "local_buffer_size": "4GB", "protocol": "rdma", "device_name": "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7", "enable_offload": false }3. Launch LMDeploy
--prefill-save-alignment 1024configures the prefill state-checkpoint alignment shared by local prefix caching and Mooncake Store. Keep--prefix-cache-decode-state-intervalat its default value of0for this configuration.For the Mooncake OFF baseline, remove only
--kv-transfer-config; retain local prefix caching, MTP, and the other settings.The benchmark intentionally uses
--cache-max-entry-count 0.3to expose eviction pressure at higher concurrency. The resulting GPU cache capacities are 33,699 blocks with Mooncake ON and 33,906 blocks with Mooncake OFF, with 64 tokens per block.Test
1. Precision
GLM-5.3-Flash accuracy results, with vLLM as the reference:
The reported differences from vLLM are within 0.52 percentage points across these benchmarks.
2. Performance
Both configurations use GLM-5.3-Flash with TP=8, Ray, local prefix caching, and MTP with 3 draft tokens.
The workload is multi-turn dialogue using SWE_Smith: a 32K-token first turn, 4K additional input tokens per subsequent turn, 512 generated tokens per turn, and 5 turns per conversation.
2.1 Sufficient GPU KV cache
8 concurrent conversations, totaling 40 requests, with an approximately 400K-token cache working set.
With sufficient local cache capacity, the measured hit rate is unchanged. Enabling Mooncake introduces 1.36% throughput overhead, while subsequent-turn TTFT increases by 20.57% in this measurement.
2.2 Insufficient GPU KV cache
80 concurrent conversations, totaling 400 requests, with an approximately 4M-token cache working set that exceeds the configured GPU cache capacity.
Under eviction pressure, Mooncake increases total throughput to 2.54× the OFF baseline, reduces total duration by 60.66%, and reduces subsequent-turn TTFT by 74.96%. External cache reuse allows evicted prefixes to be restored instead of recomputed.
TODO
prefix_cache_token_lookaheaddefault from 0 to 1