Repository navigation
feat(kv_connector): support mooncake store - #4903
Merged
lvhan028 merged 17 commits intoAug 28, 2026
Merged
Conversation
lvhan028
requested review from
grimoire and
irexyc
and
a lite review from Copilot
August 26, 2026 12:01
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a distributed KV-cache transfer pathway to LMDeploy’s PyTorch engine by integrating Mooncake Store as an external shared KV pool, enabling cross-instance prefix reuse and avoiding redundant prefill computation under eviction pressure.
Changes:
- Introduces a new external KV connector abstraction (
KVTransferConfig, connector base/factory, connector metadata/output aggregation) and implementsMooncakeStoreConnector(scheduler + worker components, lookup RPC, async load/save threads). - Wires connector lifecycle into the PyTorch engine execution flow (scheduler paging ownership for async loads/saves, connector-only executor steps, Ray aggregation, shutdown/sleep sequencing).
- Adds substantial test coverage for config normalization/CLI plumbing and Mooncake connector behavior (lookup protocol, scheduler planning, worker delegation, engine/executor wiring).
Reviewed changes
Copilot reviewed 49 out of 49 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_lmdeploy/test_kv_transfer_config.py | Validates KVTransferConfig normalization, defaults, CLI parsing, and backend gating. |
| tests/pytorch/kv_connector/test_mooncake_store_scheduler.py | Covers scheduler-side hash planning, lookup behavior, load/save planning, and edge cases. |
| tests/pytorch/kv_connector/test_mooncake_store_lookup.py | Verifies ZMQ lookup protocol, async polling, timeouts, and idempotent close semantics. |
| tests/pytorch/kv_connector/test_mooncake_store_connector.py | Tests connector role delegation and fail-fast behavior on wrong-side calls. |
| tests/pytorch/engine/test_zmq_engine_kv_connector.py | Ensures MP engine shutdown timeout changes when KV connector is enabled. |
| tests/pytorch/engine/test_ray_executor_kv_connector.py | Validates Ray executor release timeout and all-TP connector output aggregation. |
| tests/pytorch/engine/test_model_agent.py | Updates existing model-agent tests to account for kv_connector presence. |
| tests/pytorch/engine/test_model_agent_kv_connector.py | Adds tests for connector lifecycle in model agent and connector-only step behavior. |
| tests/pytorch/engine/test_kv_connector_wiring.py | End-to-end wiring tests for config prep, scheduler shutdown ordering, and mp-backend rejection. |
| tests/pytorch/engine/test_inputs_maker.py | Tests connector-only routing and correct token-boundary propagation to connector metadata. |
| tests/pytorch/engine/test_engine_sleep.py | Extends sleep test fakes to support deferred KV transfer drain hooks. |
| tests/pytorch/engine/test_cache_engine_connector_caches.py | Tests CacheEngine.connector_kv_caches layout/validation for connector registration. |
| tests/pytorch/config/test_model_config.py | Tests KV head replication bookkeeping in ModelConfig. |
| lmdeploy/pytorch/paging/seq_states/states.py | Adds remote-loading state to protect sequences during async external KV writes. |
| lmdeploy/pytorch/paging/kv_save_coordinator.py | New coordinator to pin/release logical block leases for async saves. |
| lmdeploy/pytorch/paging/kv_load_coordinator.py | New coordinator managing async load lifecycle, rollback/publish, and soft admission budgets. |
| lmdeploy/pytorch/paging/eviction_helper/recompute_eviction_helper.py | Prevents eviction of remote-ready loads and keeps load admission accounting in sync. |
| lmdeploy/pytorch/paging/block_manager/base_block_manager.py | Adds logical-block truncate and pin/release helpers for async ownership. |
| lmdeploy/pytorch/messages.py | Introduces WAITING_FOR_REMOTE_KVS status and notifies connector on new requests. |
| lmdeploy/pytorch/paging/scheduler.py | Adds connector metadata building/leasing, connector output handling, and deferred drain hooks. |
| lmdeploy/pytorch/kv_connector/base.py | Defines connector interfaces, metadata/output/result types, and all-rank output aggregator. |
| lmdeploy/pytorch/kv_connector/factory.py | Adds connector config preparation and lazy connector construction. |
| lmdeploy/pytorch/kv_connector/init.py | Exposes KV connector public API for engine integration. |
| lmdeploy/pytorch/kv_connector/mooncake/init.py | Package marker for Mooncake connector implementation. |
| lmdeploy/pytorch/kv_connector/mooncake/store/init.py | Exposes Mooncake store connector components. |
| lmdeploy/pytorch/kv_connector/mooncake/store/connector.py | Role adapter dispatching calls to scheduler vs worker implementations. |
| lmdeploy/pytorch/kv_connector/mooncake/store/scheduler.py | Scheduler-side lookup planning, load binding, and save request construction. |
| lmdeploy/pytorch/kv_connector/mooncake/store/worker.py | Worker-side Mooncake store integration, registration, lookup, and async transfer orchestration. |
| lmdeploy/pytorch/kv_connector/mooncake/store/worker_threads.py | Async sender/receiver threads implementing multi-buffer RDMA/tcp store operations. |
| lmdeploy/pytorch/kv_connector/mooncake/store/lookup.py | ZMQ-based lookup RPC (server/client) for scheduler-to-worker key existence queries. |
| lmdeploy/pytorch/kv_connector/mooncake/store/protocol.py | Defines lookup wire protocol constants. |
| lmdeploy/pytorch/kv_connector/mooncake/store/data.py | Shared data types, config parsing, and stable block-hash construction. |
| lmdeploy/pytorch/engine/mp_engine/zmq_engine.py | Extends shutdown grace period when connectors may need to release external registrations. |
| lmdeploy/pytorch/engine/model_agent/kv_connector.py | Adds model-agent hooks to bind/launch/poll connector work around forwards. |
| lmdeploy/pytorch/engine/model_agent/agent.py | Integrates connector-only steps, attaches connector progress to outputs, and manages connector lifecycle. |
| lmdeploy/pytorch/engine/inputs_maker.py | Always builds connector metadata (even without model work) and routes it to workers. |
| lmdeploy/pytorch/engine/executor/ray_executor.py | Tracks connector steps, aggregates all-TP connector outputs, and adjusts worker release timeouts. |
| lmdeploy/pytorch/engine/engine.py | Prepares connector config before worker fan-out and injects scheduler-side connector instance. |
| lmdeploy/pytorch/engine/engine_loop.py | Handles connector-only outputs, prioritizes pending lookup polling, and releases load reservations post-output. |
| lmdeploy/pytorch/engine/config_builder.py | Deep-copies kv_transfer_config into cache config to avoid shared mutation. |
| lmdeploy/pytorch/engine/cache_engine.py | Adds connector_kv_caches ordered, read-only mapping for raw cache-row registration. |
| lmdeploy/pytorch/configurations/builder.py | Records num_replicate_key_value_heads on model config for KV-head shard replication. |
| lmdeploy/pytorch/config.py | Adds kv_transfer_config to cache config and KV-head replication field to model config. |
| lmdeploy/messages.py | Adds KVTransferConfig and normalizes PytorchEngineConfig.kv_transfer_config. |
| lmdeploy/cli/utils.py | Adds --kv-transfer-config CLI argument with JSON parsing and usage guidance. |
| lmdeploy/cli/serve.py | Plumbs --kv-transfer-config into PyTorch engine config and rejects it for TurboMind. |
| lmdeploy/init.py | Exports KVTransferConfig from the top-level package. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
grimoire
reviewed
Aug 27, 2026
grimoire
reviewed
Aug 27, 2026
lvhan028
reviewed
Aug 27, 2026
lvhan028
approved these changes
Aug 28, 2026
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
In production LLM serving, a large fraction of requests share common prompt prefixes — system prompts, few-shot examples, RAG context, multi-turn conversation history, etc. LMDeploy's local prefix caching reuses KV cache within a single engine instance, but it cannot help when:
This PR addresses these gaps by integrating Mooncake's distributed store as a shared KV-cache pool. Computed KV blocks are stored with content-addressable keys (block hashes); before computing a prefill, the engine queries the store and, on a hit, loads the blocks directly via zero-copy RDMA, skipping redundant prefill computation. The capability is orthogonal to PD disaggregation and benefits any deployment topology.
The overall design is referenced from vLLM RFC #38474 and the
vllmimplementation.Deploy
This section shows a complete deployment of GLM-5.2-FP8 with TP=8, Ray as the distributed executor, prefix caching enabled, and Mooncake Store acting as a shared KV-cache pool (
kv_role=kv_both).1. Start the Mooncake master
The master coordinates the distributed store metadata. Start it on a host reachable from every LMDeploy worker:
2. Prepare the Mooncake config
Each LMDeploy worker reads this JSON config. It declares the embedded store mode, the master address, the RDMA devices to use, and the global / local buffer sizes:
{ "mode": "embedded", "metadata_server": "P2PHANDSHAKE", "master_server_address": "127.0.0.1:50051", "global_segment_size": "160GB", "local_buffer_size": "4GB", "protocol": "rdma", "device_name": "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7", "enable_offload": false }3. Launch LMDeploy with
--kv-transfer-configTest
Verified on GLM-5.2-FP8, PyTorch backend, TP=8, Ray executor, prefix caching enabled, Mooncake
kv_both.1. Precision
2. Performance
2.1 GPU KV cache sufficient (no eviction pressure)
Test method: multi-turn dialogue, SWE_Smith dataset, 8 requests, first turn length 32K, generating 512 tokens per turn, increasing by 4K per turn, for a total of 5 turns.
Conclusion: with sufficient GPU KV cache the connector only saves blocks asynchronously and never loads — overhead is ~0.20%, within measurement noise.
2.2 GPU KV cache insufficient (eviction pressure)
Test method: multi-turn dialogue, SWE_Smith dataset, 32 requests, first turn length 32K, generating 512 tokens per turn, increasing by 4K per turn, for a total of 5 turns.
Conclusion: total throughput reaches 2.03× the OFF baseline (+103.3%). The gain comes from external cache hits on subsequent turns — once the local L1 is evicted, blocks are reloaded from Mooncake instead of being recomputed.
BC-breaking
No backward-compatibility break for existing users:
kv_transfer_configdefaults toNoneand the connector is never imported or built when disabled — existing engine configurations and CLI invocations are unchanged.num_replicate_key_value_headsdefaults to1and is only consumed when a connector is enabled.SchedulerOutputfields,Scheduler/ExecutorBase/BaseModelAgentmethods, andCacheEngine.connector_kv_cachesproperty are additive.RayExecutorandZMQMPEngine, and only whenkv_transfer_configis set.Use cases (Optional)
Future work