Skip to content

feat(kv_connector): support mooncake store - #4903

Merged
lvhan028 merged 17 commits into
InternLM:mainfrom
caikun-pjlab:feat/support-mooncake-store-v2
Aug 28, 2026
Merged

lvhan028 merged 17 commits into
InternLM:mainfrom
caikun-pjlab:feat/support-mooncake-store-v2

Conversation

@caikun-pjlab

@caikun-pjlab caikun-pjlab commented Aug 26, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

In production LLM serving, a large fraction of requests share common prompt prefixes — system prompts, few-shot examples, RAG context, multi-turn conversation history, etc. LMDeploy's local prefix caching reuses KV cache within a single engine instance, but it cannot help when:

  • Cross-instance reuse: multiple LMDeploy instances serving similar traffic recompute the same KV blocks independently.
  • Post-eviction recomputation: after KV cache eviction under memory pressure, the same blocks must be recomputed from scratch.
  • Cold-start warm-up: a newly launched instance has an empty KV cache and must compute every block, even those computed elsewhere.

This PR addresses these gaps by integrating Mooncake's distributed store as a shared KV-cache pool. Computed KV blocks are stored with content-addressable keys (block hashes); before computing a prefill, the engine queries the store and, on a hit, loads the blocks directly via zero-copy RDMA, skipping redundant prefill computation. The capability is orthogonal to PD disaggregation and benefits any deployment topology.

The overall design is referenced from vLLM RFC #38474 and the vllm implementation.

Deploy

This section shows a complete deployment of GLM-5.2-FP8 with TP=8, Ray as the distributed executor, prefix caching enabled, and Mooncake Store acting as a shared KV-cache pool (kv_role=kv_both).

1. Start the Mooncake master

The master coordinates the distributed store metadata. Start it on a host reachable from every LMDeploy worker:

mooncake_master \
  -rpc_port=50051 \
  -rpc_thread_num=4 \
  -default_kv_lease_ttl=30000 \
  -eviction_high_watermark_ratio=0.95 \
  -eviction_ratio=0.1 \
  -logtostderr

2. Prepare the Mooncake config

Each LMDeploy worker reads this JSON config. It declares the embedded store mode, the master address, the RDMA devices to use, and the global / local buffer sizes:

{
  "mode": "embedded",
  "metadata_server": "P2PHANDSHAKE",
  "master_server_address": "127.0.0.1:50051",
  "global_segment_size": "160GB",
  "local_buffer_size": "4GB",
  "protocol": "rdma",
  "device_name": "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7",
  "enable_offload": false
}

3. Launch LMDeploy with --kv-transfer-config

export MOONCAKE_CONFIG_PATH=mooncake_config.json

lmdeploy serve api_server /path/to/GLM-5.2-FP8 \
    --tp 8 \
    --backend pytorch \
    --server-name 0.0.0.0 \
    --model-name GLM-5.2-FP8 \
    --server-port 9000 \
    --enable-prefix-caching \
    --log-level INFO \
    --quant-policy fp8 \
    --max-batch-size 64 \
    --cache-max-entry-count 0.9 \
    --reasoning-parser default \
    --tool-call-parser glm47 \
    --trust-remote-code \
    --max-prefill-token-num 8192 \
    --session-len 262144 \
    --distributed-executor-backend ray \
    --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both"}'

Test

Verified on GLM-5.2-FP8, PyTorch backend, TP=8, Ray executor, prefix caching enabled, Mooncake kv_both.

1. Precision

Dataset mean_acc baseline
GSM8K 0.9773 -
GPQA-Diamond 0.9091 0.912

2. Performance

2.1 GPU KV cache sufficient (no eviction pressure)

Test method: multi-turn dialogue, SWE_Smith dataset, 8 requests, first turn length 32K, generating 512 tokens per turn, increasing by 4K per turn, for a total of 5 turns.

Metric Mooncake OFF Mooncake ON ON vs OFF
Success / failed 40 / 0 40 / 0 —
Duration 99.23 s 99.42 s +0.2%
Total throughput 16737.51 tok/s 16704.71 tok/s -0.20%
Output throughput 206.39 tok/s 205.99 tok/s -0.20%
Avg latency 19.67 s 19.71 s +0.19%
TTFT 4330.30 ms 4301.49 ms -0.67%
TPOT 30.02 ms 30.15 ms +0.43%
KV cache hit rate 75.9287% 75.9287% 0%
Subsequent-turn TTFT 2015.47 ms 1981.83 ms -1.67%

Conclusion: with sufficient GPU KV cache the connector only saves blocks asynchronously and never loads — overhead is ~0.20%, within measurement noise.

2.2 GPU KV cache insufficient (eviction pressure)

Test method: multi-turn dialogue, SWE_Smith dataset, 32 requests, first turn length 32K, generating 512 tokens per turn, increasing by 4K per turn, for a total of 5 turns.

Metric Mooncake OFF Mooncake ON ON vs OFF
Success / failed 160 / 0 160 / 0 —
Duration 618.52 s 294.48 s -52.39%
Request throughput 0.26 req/s 0.54 req/s +107.69%
Total throughput 10742.05 tok/s 22562.08 tok/s +110.04%
Output throughput 132.45 tok/s 278.18 tok/s +110.04%
Avg latency 113.69 s 55.15 s -51.49%
TTFT 44846.20 ms 24193.99 ms -46.05%
TPOT 134.73 ms 60.58 ms -55.04%
KV cache hit rate 21.5917% 75.6952% +54.10%
Subsequent-turn TTFT 42845.15 ms 17098.32 ms -60.09%

Conclusion: total throughput reaches 2.03× the OFF baseline (+103.3%). The gain comes from external cache hits on subsequent turns — once the local L1 is evicted, blocks are reloaded from Mooncake instead of being recomputed.

BC-breaking

No backward-compatibility break for existing users:

  • kv_transfer_config defaults to None and the connector is never imported or built when disabled — existing engine configurations and CLI invocations are unchanged.
  • num_replicate_key_value_heads defaults to 1 and is only consumed when a connector is enabled.
  • The new SchedulerOutput fields, Scheduler / ExecutorBase / BaseModelAgent methods, and CacheEngine.connector_kv_caches property are additive.
  • The only behavioral change visible without a connector is a longer worker-shutdown timeout in RayExecutor and ZMQMPEngine, and only when kv_transfer_config is set.

Use cases (Optional)

  • Cross-instance prefix reuse: a fleet of LMDeploy instances serving the same model + system prompt share a Mooncake store; each newly computed block is published once and reused by every instance.
  • Post-eviction recovery: when local KV cache is evicted under memory pressure, the next request with the same prefix loads blocks from the store instead of recomputing them.
  • Cold-start warm-up: a new instance joins the fleet and immediately benefits from blocks already in the store, eliminating cold-start prefill latency for common prefixes.

Future work

  • Support speculative decoding.
  • Support more attention variants (SWA, linear attention, etc.).
  • Support Mooncake standalone-store mode.
  • Support SSD-based KV cache tiering.
  • external cache hit statistics and logs

@lvhan028
lvhan028 requested review from grimoire and irexyc and a lite review from Copilot August 26, 2026 12:01
@lvhan028 lvhan028 added the enhancement New feature or request label Aug 26, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a distributed KV-cache transfer pathway to LMDeploy’s PyTorch engine by integrating Mooncake Store as an external shared KV pool, enabling cross-instance prefix reuse and avoiding redundant prefill computation under eviction pressure.

Changes:

  • Introduces a new external KV connector abstraction (KVTransferConfig, connector base/factory, connector metadata/output aggregation) and implements MooncakeStoreConnector (scheduler + worker components, lookup RPC, async load/save threads).
  • Wires connector lifecycle into the PyTorch engine execution flow (scheduler paging ownership for async loads/saves, connector-only executor steps, Ray aggregation, shutdown/sleep sequencing).
  • Adds substantial test coverage for config normalization/CLI plumbing and Mooncake connector behavior (lookup protocol, scheduler planning, worker delegation, engine/executor wiring).

Reviewed changes

Copilot reviewed 49 out of 49 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/test_lmdeploy/test_kv_transfer_config.py Validates KVTransferConfig normalization, defaults, CLI parsing, and backend gating.
tests/pytorch/kv_connector/test_mooncake_store_scheduler.py Covers scheduler-side hash planning, lookup behavior, load/save planning, and edge cases.
tests/pytorch/kv_connector/test_mooncake_store_lookup.py Verifies ZMQ lookup protocol, async polling, timeouts, and idempotent close semantics.
tests/pytorch/kv_connector/test_mooncake_store_connector.py Tests connector role delegation and fail-fast behavior on wrong-side calls.
tests/pytorch/engine/test_zmq_engine_kv_connector.py Ensures MP engine shutdown timeout changes when KV connector is enabled.
tests/pytorch/engine/test_ray_executor_kv_connector.py Validates Ray executor release timeout and all-TP connector output aggregation.
tests/pytorch/engine/test_model_agent.py Updates existing model-agent tests to account for kv_connector presence.
tests/pytorch/engine/test_model_agent_kv_connector.py Adds tests for connector lifecycle in model agent and connector-only step behavior.
tests/pytorch/engine/test_kv_connector_wiring.py End-to-end wiring tests for config prep, scheduler shutdown ordering, and mp-backend rejection.
tests/pytorch/engine/test_inputs_maker.py Tests connector-only routing and correct token-boundary propagation to connector metadata.
tests/pytorch/engine/test_engine_sleep.py Extends sleep test fakes to support deferred KV transfer drain hooks.
tests/pytorch/engine/test_cache_engine_connector_caches.py Tests CacheEngine.connector_kv_caches layout/validation for connector registration.
tests/pytorch/config/test_model_config.py Tests KV head replication bookkeeping in ModelConfig.
lmdeploy/pytorch/paging/seq_states/states.py Adds remote-loading state to protect sequences during async external KV writes.
lmdeploy/pytorch/paging/kv_save_coordinator.py New coordinator to pin/release logical block leases for async saves.
lmdeploy/pytorch/paging/kv_load_coordinator.py New coordinator managing async load lifecycle, rollback/publish, and soft admission budgets.
lmdeploy/pytorch/paging/eviction_helper/recompute_eviction_helper.py Prevents eviction of remote-ready loads and keeps load admission accounting in sync.
lmdeploy/pytorch/paging/block_manager/base_block_manager.py Adds logical-block truncate and pin/release helpers for async ownership.
lmdeploy/pytorch/messages.py Introduces WAITING_FOR_REMOTE_KVS status and notifies connector on new requests.
lmdeploy/pytorch/paging/scheduler.py Adds connector metadata building/leasing, connector output handling, and deferred drain hooks.
lmdeploy/pytorch/kv_connector/base.py Defines connector interfaces, metadata/output/result types, and all-rank output aggregator.
lmdeploy/pytorch/kv_connector/factory.py Adds connector config preparation and lazy connector construction.
lmdeploy/pytorch/kv_connector/init.py Exposes KV connector public API for engine integration.
lmdeploy/pytorch/kv_connector/mooncake/init.py Package marker for Mooncake connector implementation.
lmdeploy/pytorch/kv_connector/mooncake/store/init.py Exposes Mooncake store connector components.
lmdeploy/pytorch/kv_connector/mooncake/store/connector.py Role adapter dispatching calls to scheduler vs worker implementations.
lmdeploy/pytorch/kv_connector/mooncake/store/scheduler.py Scheduler-side lookup planning, load binding, and save request construction.
lmdeploy/pytorch/kv_connector/mooncake/store/worker.py Worker-side Mooncake store integration, registration, lookup, and async transfer orchestration.
lmdeploy/pytorch/kv_connector/mooncake/store/worker_threads.py Async sender/receiver threads implementing multi-buffer RDMA/tcp store operations.
lmdeploy/pytorch/kv_connector/mooncake/store/lookup.py ZMQ-based lookup RPC (server/client) for scheduler-to-worker key existence queries.
lmdeploy/pytorch/kv_connector/mooncake/store/protocol.py Defines lookup wire protocol constants.
lmdeploy/pytorch/kv_connector/mooncake/store/data.py Shared data types, config parsing, and stable block-hash construction.
lmdeploy/pytorch/engine/mp_engine/zmq_engine.py Extends shutdown grace period when connectors may need to release external registrations.
lmdeploy/pytorch/engine/model_agent/kv_connector.py Adds model-agent hooks to bind/launch/poll connector work around forwards.
lmdeploy/pytorch/engine/model_agent/agent.py Integrates connector-only steps, attaches connector progress to outputs, and manages connector lifecycle.
lmdeploy/pytorch/engine/inputs_maker.py Always builds connector metadata (even without model work) and routes it to workers.
lmdeploy/pytorch/engine/executor/ray_executor.py Tracks connector steps, aggregates all-TP connector outputs, and adjusts worker release timeouts.
lmdeploy/pytorch/engine/engine.py Prepares connector config before worker fan-out and injects scheduler-side connector instance.
lmdeploy/pytorch/engine/engine_loop.py Handles connector-only outputs, prioritizes pending lookup polling, and releases load reservations post-output.
lmdeploy/pytorch/engine/config_builder.py Deep-copies kv_transfer_config into cache config to avoid shared mutation.
lmdeploy/pytorch/engine/cache_engine.py Adds connector_kv_caches ordered, read-only mapping for raw cache-row registration.
lmdeploy/pytorch/configurations/builder.py Records num_replicate_key_value_heads on model config for KV-head shard replication.
lmdeploy/pytorch/config.py Adds kv_transfer_config to cache config and KV-head replication field to model config.
lmdeploy/messages.py Adds KVTransferConfig and normalizes PytorchEngineConfig.kv_transfer_config.
lmdeploy/cli/utils.py Adds --kv-transfer-config CLI argument with JSON parsing and usage guidance.
lmdeploy/cli/serve.py Plumbs --kv-transfer-config into PyTorch engine config and rejects it for TurboMind.
lmdeploy/init.py Exports KVTransferConfig from the top-level package.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@grimoire grimoire left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lvhan028
lvhan028 merged commit 8fd55ae into InternLM:main Aug 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants