Skip to content

[None][feat] Mooncake safe shared-pool reuse and asynchronous loading - #22416

Draft
brb-nv wants to merge 4 commits into
NVIDIA:mainfrom
brb-nv:user/brb/mooncake-shared-pool-async-load-21656-mirror
Draft

brb-nv wants to merge 4 commits into
NVIDIA:mainfrom
brb-nv:user/brb/mooncake-shared-pool-async-load-21656-mirror

Conversation

@brb-nv

@brb-nv brb-nv commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Description

  • Load prefixes asynchronously, with configurable host staging.
  • Make shared-pool KV reuse safe.
  • Support attention DP in the Mooncake store.
  • Add speculative prefill on disagg context servers.
  • Add soft-pinned and per-block grouped puts.
  • Publish only computed blocks, and count a save once it is queued.
  • Add prefix-hole and load-phase stats (stats_interval).
  • Add local_reuse_backoff for one-model drafts, and hand failed loads back for a restart.
  • Encode block tokens in one numpy pass (perf).

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

SimengLiu-nv and others added 4 commits November 16, 2026 23:09
…e contracts

Support block-aligned prefix reuse, per-request failed-load recovery, context-only speculative prefill, and disaggregated completion reservations.

Account for retained padding allocations when admitting completion bounds and reject requests that cannot fit. Preserve published generation pages and connector save ownership across skipped iterations by reusing the pending allocation on retry.

Signed-off-by: Simeng Liu <simengl@nvidia.com>
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Apply clang-format to the nanobind binding and tidy comments and docstrings added by the previous commit (single backticks, no em-dashes, no history wording).

Restore the @pytest.mark.cpu_only decorator on test_KvCacheConfig_disagg_completion_reservation, which was lost in the rebase.

Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Enable safe shared-pool KV reuse and optional asynchronous loading in the Mooncake store connector.

Signed-off-by: Simeng Liu <simengl@nvidia.com>
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Apply ruff-format to the connector spec-prefill test and reword one comment.

Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
@brb-nv brb-nv changed the title User/brb/mooncake shared pool async load 21656 mirror [None][feat] Mooncake safe shared-pool reuse and asynchronous loading Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants