Skip to content

feat(kv_connector): mooncake store support GLM-5.3-Flash - #5034

Merged
RunningLeon merged 82 commits into
InternLM:mainfrom
caikun-pjlab:feat/mooncake_store_support_glm5.3_flash
Oct 16, 2026
Merged

RunningLeon merged 82 commits into
InternLM:mainfrom
caikun-pjlab:feat/mooncake_store_support_glm5.3_flash

Conversation

@caikun-pjlab

@caikun-pjlab caikun-pjlab commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

This PR extends the Mooncake Store integration introduced in #4903 to support GLM-5.3-Flash with hybrid attention and multi-token prediction (MTP).

GLM-5.3-Flash requires both attention KV caches and linear-attention states to resume from a cached prefix. With MTP enabled, the corresponding target and draft caches must also be consistent at the restored position. Transferring KV blocks alone is therefore insufficient.

This PR adds:

  • Registration, lookup, asynchronous save, and restore of hybrid state caches alongside attention KV caches.
  • MTP-aware cache reuse and chunked prefill, with consistent checkpoint boundaries for local prefix caching and Mooncake Store.
  • Multimodal content identity in cache keys to distinguish prefixes with different image/video inputs.
  • External cache-hit statistics in logs and Prometheus metrics.

These changes allow GLM-5.3-Flash to reuse cached prefixes after local eviction, reducing repeated prefill computation in long-context, multi-turn workloads.

Deploy

The benchmark deployment uses 8 × NVIDIA H200, the PyTorch backend, TP=8, Ray, local prefix caching, and MTP with 3 draft tokens.

1. Start the Mooncake master

mooncake_master \
  -rpc_port=50051 \
  -rpc_thread_num=4 \
  -default_kv_lease_ttl=30000 \
  -eviction_high_watermark_ratio=0.95 \
  -eviction_ratio=0.1 \
  -logtostderr

2. Prepare the Mooncake config

Save the following as mooncake_config.json. Adjust the master address and RDMA device names for your environment.

{
  "mode": "embedded",
  "metadata_server": "P2PHANDSHAKE",
  "master_server_address": "127.0.0.1:50051",
  "global_segment_size": "160GB",
  "local_buffer_size": "4GB",
  "protocol": "rdma",
  "device_name": "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7",
  "enable_offload": false
}

3. Launch LMDeploy

export MOONCAKE_CONFIG_PATH=mooncake_config.json

lmdeploy serve api_server /path/to/GLM-5.3-Flash \
  --tp 8 \
  --backend pytorch \
  --server-name 0.0.0.0 \
  --server-port 9000 \
  --model-name GLM-5.3-Flash \
  --enable-prefix-caching \
  --max-batch-size 80 \
  --cache-max-entry-count 0.3 \
  --reasoning-parser default \
  --tool-call-parser glm47 \
  --max-prefill-token-num 8192 \
  --session-len 1048576 \
  --log-level INFO \
  --distributed-executor-backend ray \
  --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both"}' \
  --prefill-save-alignment 1024 \
  --prefix-cache-state-budget 32 \
  --speculative-algorithm deepseek_mtp \
  --speculative-num-draft-tokens 3 \
  --speculative-disable-prefix-cache-block-drop

--prefill-save-alignment 1024 configures the prefill state-checkpoint alignment shared by local prefix caching and Mooncake Store. Keep --prefix-cache-decode-state-interval at its default value of 0 for this configuration.

For the Mooncake OFF baseline, remove only --kv-transfer-config; retain local prefix caching, MTP, and the other settings.

The benchmark intentionally uses --cache-max-entry-count 0.3 to expose eviction pressure at higher concurrency. The resulting GPU cache capacities are 33,699 blocks with Mooncake ON and 33,906 blocks with Mooncake OFF, with 64 tokens per block.

Test

1. Precision

GLM-5.3-Flash accuracy results, with vLLM as the reference:

Dataset vLLM (%) LMDeploy + Mooncake (%) Difference (pp)
GPQA-Diamond — average of 8 runs 91.35 91.86 +0.51
AIME 2025 — average of 32 runs 96.56 96.04 −0.52
GSM8K 97.65 97.27 −0.38
MMMU-Pro 77.69 77.86 +0.17

The reported differences from vLLM are within 0.52 percentage points across these benchmarks.

2. Performance

Both configurations use GLM-5.3-Flash with TP=8, Ray, local prefix caching, and MTP with 3 draft tokens.

The workload is multi-turn dialogue using SWE_Smith: a 32K-token first turn, 4K additional input tokens per subsequent turn, 512 generated tokens per turn, and 5 turns per conversation.

2.1 Sufficient GPU KV cache

8 concurrent conversations, totaling 40 requests, with an approximately 400K-token cache working set.

Metric Mooncake OFF Mooncake ON ON vs OFF
Success / failed 40 / 0 40 / 0 —
Duration 45.01 s 45.63 s +1.38%
Request throughput 0.8887 req/s 0.8766 req/s −1.36%
Total throughput 36,901.06 tok/s 36,399.34 tok/s −1.36%
Output throughput 455.00 tok/s 448.81 tok/s −1.36%
Avg latency 8.562 s 8.731 s +1.98%
TTFT 2,496.79 ms 2,648.03 ms +6.06%
TPOT 11.87 ms 11.90 ms +0.25%
KV cache hit rate 75.9033% 75.9033% 0 pp
Subsequent-turn TTFT 1,079.16 ms 1,301.12 ms +20.57%

With sufficient local cache capacity, the measured hit rate is unchanged. Enabling Mooncake introduces 1.36% throughput overhead, while subsequent-turn TTFT increases by 20.57% in this measurement.

2.2 Insufficient GPU KV cache

80 concurrent conversations, totaling 400 requests, with an approximately 4M-token cache working set that exceeds the configured GPU cache capacity.

Metric Mooncake OFF Mooncake ON ON vs OFF
Success / failed 400 / 0 400 / 0 —
Duration 808.73 s 318.15 s −60.66%
Request throughput 0.4946 req/s 1.2573 req/s +154.21%
Total throughput 20,536.69 tok/s 52,204.46 tok/s +154.20%
Output throughput 253.24 tok/s 643.73 tok/s +154.20%
Avg latency 154.281 s 61.838 s −59.92%
TTFT 79,779.24 ms 30,139.00 ms −62.22%
TPOT 145.80 ms 62.03 ms −57.46%
KV cache hit rate 5.3435% 77.1562% +71.81 pp
Subsequent-turn TTFT 82,901.88 ms 20,759.12 ms −74.96%

Under eviction pressure, Mooncake increases total throughput to 2.54× the OFF baseline, reduces total duration by 60.66%, and reduces subsequent-turn TTFT by 74.96%. External cache reuse allows evicted prefixes to be restored instead of recomputed.

TODO

  • set prefix_cache_token_lookahead default from 0 to 1

qescccczmr and others added 30 commits September 14, 2026 12:43
Reuse MLA/MoE loaders, FA3, mHC, sparse Top-K and compact DeepGEMM. Add KDA state adaptation and multimodal GLM configuration/processing. Preserve existing defaults and the positional MoE prefix argument.
Reuse GDN state helpers and channelwise TileLang verification; honor MTP index sharing and invalidate graphs after buffer growth. Share AR sampling filters with rejection/recovery and stabilize GLM TP/mHC and sparse-index arithmetic. Keep EP/DP, prefix-cache and performance experiments outside this patch.
Reject unsupported DLINFER MoE reduction options instead of ignoring them. Leave NCCL NVLS policy to the runtime and remove model-specific environment plumbing. Document GLM FP32 operator usage and cover dtype, reduction, and environment contracts.

Validation: 110 focused tests passed on CPU/H200; Ruff passed for lmdeploy and tests/pytorch. DLINFER rejection regressions failed before the fix and pass afterward.
Apply the configured string and docstring formatters to GLM PR files. Verified executable AST is unchanged; both previously failing hooks and Ruff pass locally.
Replace the compiled GLM helper with an opt-in FP32 compute contract through ApplyRotaryEmb and its backend build spec. CUDA retains FP32 arithmetic inside the fused kernel; defaults remain unchanged and unsupported Dlinfer requests fail explicitly.

Validated 73 focused tests and actual vLLM rotary kernel parity with identical inputs and tables. FP32-table outputs can differ from the old compiled helper at rounding boundaries.
Reuse the existing strided TileLang recurrent inputs instead of copying Q/K/V in AR and speculative decode. Preserve contiguous prefill inputs and explicitly allocate contiguous recurrent output for channel-major views.

Validated 25 new stride/state/graph tests, 25 shared GDR tests, and 12 complete KDA adapter comparisons. TP4 MTP-off text regression matches the previous head exactly at 124 full-vocabulary logits positions and all 29 requests.
Consolidate PR-based P2 and the staged selective opt port before rebasing.
Original HEAD: 7ef51dce043d3655a356ed687cadc5922196c115
Pre-rebase snapshot: ff1b28d18307cfbcdb6640e5718daafa88d7ec4c
qescccczmr and others added 20 commits October 8, 2026 05:13
Handle unsupported AutoProcessor import/value errors with the existing GLM-4V compatibility builder. Cover native, missing-modality, error propagation and trust-remote-code paths. Update DeepEP FP32 option and FlashInfer dtype/workspace fixtures, and repair the two dead Llama 3.1 documentation links.
Fuse target raw-ring updates and restore fused draft decoding while preserving cache state semantics.
Integrate PR2 with PR4968 development branch
Keep unified MTP prefill and Mooncake state-cache semantics while
integrating DCP attention and the draft DP-phase transition fix.

Validation on H200: 847 changed/core tests, 132 GPU tests, and 419
additional owner-level regression tests passed without skips.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Checkpoint planning can stall multimodal prefill, and local-only checkpoint chunking can lose convolution history without TileLang.

2 open findings
What changed in this PR

Extends Mooncake Store prefix reuse to GLM-5.3-Flash’s hybrid attention and MTP caches, building on the existing distributed KV-cache integration.

Changes:

  • Adds asynchronous hybrid-state transfers and coordinated target/draft checkpoints.
  • Unifies prefill alignment and adds multimodal cache identity.
  • Adds external-cache metrics, configuration options, documentation, and regression tests.
File Description
tests/​test_lmdeploy/​test_metrics_loggers.py Tests external-cache logs and counters.
tests/​test_lmdeploy/​serve/​parsers/​test_required_tool_choice.py Separates test tool calls with newlines.
tests/​pytorch/​spec_decode/​test_spec_agent.py Tests shifted MTP chunks and multimodal equivalence.
tests/​pytorch/​paging/​test_scheduler.py Tests aligned checkpoints and cache metrics.
tests/​pytorch/​paging/​test_scheduler_kv_transfer.py Tests remote-hit accounting and rollback.
tests/​pytorch/​paging/​test_prefill_scheduler.py Tests alignment, reservations, and scheduling estimates.
tests/​pytorch/​kv_connector/​test_mooncake_store_lookup.py Tests lookup rewind metadata.
tests/​pytorch/​kv_connector/​test_mooncake_store_connector.py Tests state-pool registration forwarding.
tests/​pytorch/​kernel/​test_kpool.py Tests restored target/MTP tail reconstruction.
tests/​pytorch/​kernel/​test_kda.py Tests restored recurrent-state correctness.
tests/​pytorch/​engine/​test_ray_executor_kv_connector.py Tests request-level transfer failures.
tests/​pytorch/​engine/​test_model_agent.py Tests short convolution chunks and stream safety.
tests/​pytorch/​engine/​test_model_agent_kv_connector.py Tests registration and target/MTP transfer ordering.
tests/​pytorch/​engine/​test_inputs_maker.py Tests chunk boundaries and checkpoint-copy inputs.
tests/​pytorch/​engine/​test_executor_base.py Tests configuration validation and snapshot budgets.
tests/​pytorch/​engine/​test_cache_engine/​test_state_engine.py Tests transferable state pools and local copies.
tests/​pytorch/​engine/​test_cache_engine/​test_schema.py Tests cache persistence flags.
tests/​pytorch/​engine/​test_cache_engine_connector_caches.py Tests exclusion of local-only MTP tails.
tests/​pytorch/​config/​test_hf_overrides.py Uses the public speculative configuration.
lmdeploy/​pytorch/​strategies/​base/​sequence.py Defines draft-input shift metadata.
lmdeploy/​pytorch/​strategies/​ar_spec/​sequence.py Makes rewind and input-shift policies configurable.
lmdeploy/​pytorch/​strategies/​ar_spec/​__init__.py Selects method-specific sequence policies.
lmdeploy/​pytorch/​spec_decode/​spec_agent.py Replaces chunk carry with prompt lookahead.
lmdeploy/​pytorch/​prefix_cache_state.py Clarifies block identity typing.
lmdeploy/​pytorch/​paging/​state_manager.py Reserves Store snapshots outside runtime allocation.
lmdeploy/​pytorch/​paging/​scheduler.py Enables hybrid loads, alignment, and external metrics.
lmdeploy/​pytorch/​paging/​prefill_scheduler.py Updates aligned scheduling and load admission.
lmdeploy/​pytorch/​paging/​kv_load_coordinator.py Coordinates hybrid restore, rollback, and accounting.
lmdeploy/​pytorch/​paging/​eviction_helper/​recompute_eviction_helper.py Excludes recomputation from external statistics.
lmdeploy/​pytorch/​nn/​kpool.py Normalizes error-string quoting.
lmdeploy/​pytorch/​models/​glm5_next.py Marks reconstructible MTP tails local-only.
lmdeploy/​pytorch/​model_inputs.py Carries non-final prompt lookahead.
lmdeploy/​pytorch/​messages.py Adds alignment metadata and Store multimodal identity.
lmdeploy/​pytorch/​long_context.py Plans aligned, shifted multimodal chunks.
lmdeploy/​pytorch/​kv_connector/​mooncake/​store/​worker.py Registers state pools and finds complete checkpoints.
lmdeploy/​pytorch/​kv_connector/​mooncake/​store/​worker_threads.py Transfers state snapshots alongside KV blocks.
lmdeploy/​pytorch/​kv_connector/​mooncake/​store/​scheduler.py Manages hybrid snapshots, hashes, and failures.
lmdeploy/​pytorch/​kv_connector/​mooncake/​store/​protocol.py Documents lookup rewind framing.
lmdeploy/​pytorch/​kv_connector/​mooncake/​store/​lookup.py Transmits rewind policy to workers.
lmdeploy/​pytorch/​kv_connector/​mooncake/​store/​data.py Defines state-transfer records and multimodal hashing.
lmdeploy/​pytorch/​kv_connector/​mooncake/​store/​connector.py Forwards state-pool registration.
lmdeploy/​pytorch/​kv_connector/​mooncake/​store/​__init__.py Exports state registration metadata.
lmdeploy/​pytorch/​kv_connector/​base.py Adds state pools and request-level failure reporting.
lmdeploy/​pytorch/​kv_connector/​__init__.py Exports the cache-pool interface.
lmdeploy/​pytorch/​engine/​model_agent/​agent.py Coordinates MTP checkpoint copies and registration.
lmdeploy/​pytorch/​engine/​inputs_maker.py Builds aligned inputs and state-save plans.
lmdeploy/​pytorch/​engine/​executor/​base.py Validates hybrid support and budgets snapshot memory.
lmdeploy/​pytorch/​engine/​engine.py Computes multimodal identity for Store reuse.
lmdeploy/​pytorch/​engine/​config_builder.py Propagates new cache and speculative options.
lmdeploy/​pytorch/​engine/​cache_engine/​state.py Exposes transferable state pools.
lmdeploy/​pytorch/​engine/​cache_engine/​schema.py Propagates cache persistence flags.
lmdeploy/​pytorch/​engine/​cache_engine/​layout.py Defines transferable byte ranges.
lmdeploy/​pytorch/​engine/​cache_engine/​engine.py Excludes local-only pools from transfers.
lmdeploy/​pytorch/​configurations/​glm5_next.py Excludes reconstructible KPool state tails.
lmdeploy/​pytorch/​config.py Defines alignment, snapshot, and rewind settings.
lmdeploy/​pytorch/​backends/​gated_delta_rule.py Tracks short-chunk convolution history.
lmdeploy/​pytorch/​backends/​default/​cache.py Separates persisted and local-only block pools.
lmdeploy/​pytorch/​backends/​cuda/​causal_conv1d.py Preserves history when saving short-chunk state.
lmdeploy/​metrics/​stats.py Carries cumulative external-cache statistics.
lmdeploy/​metrics/​loggers.py Reports external-cache logs and Prometheus metrics.
lmdeploy/​messages.py Exposes public configuration and metric fields.
lmdeploy/​cli/​utils.py Defines alignment, snapshot, and rewind flags.
lmdeploy/​cli/​serve.py Wires new serving options.
lmdeploy/​cli/​cli.py Wires new chat options.
docs/​zh_cn/​advance/​spec_decoding.md Documents shared checkpoint alignment in Chinese.
docs/​zh_cn/​advance/​metrics.md Documents external-cache metrics in Chinese.
docs/​en/​advance/​spec_decoding.md Documents shared checkpoint alignment.
docs/​en/​advance/​metrics.md Documents external-cache metrics and queries.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +64 to +69
if (cache_config.use_mooncake_store and model_config.states_shapes
and model_config.is_gated_delta and device_type == 'cuda'):
from lmdeploy.pytorch.backends.cuda.utils import has_tilelang
if not has_tilelang():
raise RuntimeError('Hybrid Mooncake Store requires TileLang causal convolution on CUDA: '
'the Dao backend does not restore prefill initial states')
Comment on lines 105 to 106
if seq.num_history_ids < step and seq.is_prefix_cache_boundary_safe(step):
limit = min(limit, step - seq.num_history_ids)

@RunningLeon RunningLeon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@RunningLeon
RunningLeon merged commit 30fd106 into InternLM:main Oct 16, 2026
3 of 4 checks passed
wanfengcxz added a commit to wanfengcxz/lmdeploy that referenced this pull request Oct 27, 2026
Rebase fallout from upstream InternLM#5034 (mooncake store support GLM-5.3-Flash),
which reworked the connector around linear-attention state pools:

* start_load_kv\x27s new state-load fence went through torch.cuda.Event();
  route it through TransferBackend.make_ready_event() like the save path so
  state loads can record on Ascend.
* _build_state_registrations required tensor.is_cuda; apply the same
  cuda/npu device rule as KV rows so hybrid (GLM-5.3-Flash) state pools can
  register NPU HBM.
* the KV-row device check additionally honors is_cuda-only tensors again:
  upstream tests expose CUDA registration over host memory through a forced
  is_cuda on a device=\x27cpu\x27 tensor subclass, which the pure device-string
  check would reject.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants