prefix-caching

🔥🔥 The extremely fast inference engine for decision model

OrbitKV: A KV cache for vLLM and SGLang.

Artifact for "Marconi: Prefix Caching for the Era of Hybrid LLMs" [MLSys '25 Outstanding Paper Award, Honorable Mention]

Serving Qwen3.8-27B-FP8 on a single DGX Spark (GB10): 7.88 to 58.5 tok/s single-stream from decode strategy alone, weights untouched. Speculative decoding and prefix caching benchmarked, plus DFlash 2 — the only Qwen3.8-27B build that can serve it under vLLM.

📚 An educational, step-by-step implementation of LLM KV cache reuse — learn prefix caching, chunk keys, storage & eviction, from pure Python to GPU tensors and vLLM integration. Inspired by LMCache.

Reproducing agentic KV-cache policy claims on real traces. 68k requests from 393 Claude Code sessions. LRU is harder to beat than the papers suggest.

Reproducible llama.cpp CPU inference profiling and a deterministic LLM serving simulator with continuous batching, KV cache, prefix caching, and workload-driven latency analysis.

DeepSeek 1017 EXL3 on 3-4 DGX Sparks. TP4 4.0.4: 4,277 tok/s cold prefill; C16 code 331.7 / prose 287.5 native decode sum. Smart CUDA Graph, KV Cache and Adaptive Speculation controllers. C16/R48, 9M KV, 3.5 GiB checkpoints. TP3 1.2.8: up to +58% cold prefill. Exact recipes, source and benchmarks.

Models Take Notes at Prefill: KV Cache Can Be Editable and Composable (arXiv:2606.17107) — paper, code, results, and interactive companion.

Miniature LLM serving runtime with continuous batching, chunked prefill, KV-cache management, preemption, prefix caching, and real Llama-family execution.

Adaptive Disaggregated Inference on a Role-free Fleet!

Persistent KV cache sessions for LLM serving on edge and on-premises hardware: restore the KV cache instead of recomputing it, lossless long-context overflow archive, cross-engine sessions (llama.cpp, vLLM, SGLang). IEEE Access 2026.

Minimal, educational continuous-batching LLM inference engine with Paged KV-cache, prefix caching, and speculative decoding for Qwen2.5 in pure PyTorch.

A prefix-cache advisor for LLM serving infrastructure that recommends KV-cache capacity and eviction policies from your request traces/logs.

Single-GPU NInfer + KVMem integration: multi-lane long contexts, DFlash2 + vision, native prefix reuse and reproducible measurements.

KV-cache-aware inference router for self-hosted LLM pools. Envoy ext_proc Endpoint Picker + standalone proxy, in Go.

Cache-aware router for OpenAI-compatible LLM servers, in Go. Per-worker radix trees route each request to the worker holding its KV prefix. Validated on 4x A100 + vLLM and Apple Silicon + llama.cpp.

Unified execution runtime for LLM and ML programs.

High-fidelity discrete-event simulator for LLM serving schedulers with first-principles roofline models, SGLang-style Radix prefix caching, and Perfetto trace export.

Fast local LLM / VLM inference engine for Apple Silicon (MLX). OpenAI- and Anthropic-compatible, lossless speculative decoding (DFlash2 / MTP), hybrid-model prefix cache with RAM/SSD tiers, native Claude Code / Codex / opencode support. Qwen3.8-27B tuned first.