Long-context inference is becoming essential for agentic AI, where assistants may need to reason over large code repositories and long chat histories. Agent-trace benchmarks now run from 64K all the way to 1M tokens and their KV caches are correspondingly large. Under a baseline tensor-parallel (TP) setup, this KV cache is partitioned by attention head, which puts a hard floor on how much it can shrink.

Modern models use one of two attention schemes, and both hit this floor. Grouped-query attention (GQA) models store a small number of KV heads, and TP can only split the KV cache down to one head per GPU; once TP exceeds the number of KV heads, the cache starts duplicating across GPUs. Multi-head latent attention (MLA) models make this even worse: MLA compresses the Key/Value into a single low-rank latent vector shared across all query heads, so it effectively has only one KV head. Under normal TP there is nothing to split by head, meaning the latent KV cache is replicated in full across every TP rank. In both cases the duplicated KV cache eats into GPU memory, leaving very little room to serve additional requests. This caps the number of concurrent requests the system can handle, driving down throughput and pushing up cost per token.

Decode Context Parallelism addresses this by splitting KV cache across the GPUs so each GPU stores and reads only part of the KV cache. This frees up GPU memory, allowing each GPU to take on more requests and thus run at a larger batch size. On systems with high-bandwidth GPU-to-GPU interconnects, this helps preserve interactive responsiveness while serving many long-context agents at once.

vLLM has supported DCP for almost a year, but we are writing this blog now to highlight the feature, along with the recent improvements and advancements we have made to it, because the rise of long-context agentic use cases has made its benefits more relevant than ever.

Under plain TP, both attention schemes waste memory on duplicated KV cache: GQA can only split down to one KV head per GPU before it starts replicating, and MLA behaves like a single KV head, so its latent cache is replicated on every rank. DCP instead shards the KV cache along the sequence dimension, so each GPU holds a unique slice and no capacity is wasted on duplicates.

2. Performance Results

To quantify the benefit of Decode Context Parallelism, we compared a baseline tensor-parallel deployment against DCP on an identical set of GPUs, holding the model, hardware, and workload fixed and varying only how the KV cache is sharded during decode.

2.1 Dataset

The dataset is a publicly available agentic long-context trace in Mooncake trace format, published here. See this section for more details on the dataset. It ships as JSONL where each line is a single request with input_length, output_length, and hash_ids fields, so it can be replayed directly with any Mooncake-compatible harness (e.g. aiperf --custom-dataset-type mooncake_trace). The hash_ids field encodes shared prefix blocks, making it well-suited for benchmarking KV-cache reuse and prefix-caching behavior.

It’s an agentic multi-turn workload of long inputs paired with short generations, chosen to reflect realistic long-horizon agent behavior. Inputs are centered around a median of ~67k tokens and paired with short ~400-token outputs, but the input distribution is bimodal rather than uniformly huge: roughly half the requests sit at 64k+ (≈53%, with a heavy tail reaching ~1M tokens) and half are short-to-mid (≈47% under 64k, ~18% under 8k). About 8% of requests exceed 128k and ~3–4% exceed 256k.

2.2 Benefits of Decode Context Parallelism

We ran an experiment on a single 8×B200 node serving Kimi K2.6 in NVFP4 with vLLM, sweeping request concurrency from 16 to 512 (see table below). DCP sustains far higher concurrency and delivers markedly higher throughput per GPU across the entire throughput–interactivity Pareto frontier.

The difference comes down to where the KV cache lives. Baseline TP replicates the KV cache on every GPU, so peak memory fills quickly. It reaches 100% at a concurrency of 64 and hits a wall, and throughput plateaus near 1,863 tok/s/GPU because no additional requests can fit. On the other hand, DCP shards the KV cache along the sequence dimension, so each GPU stores only 1/N of every request’s KV. This allows space on the GPU to support more incoming requests. As a result, even at high concurrencies DCP keeps scaling where TP hits a wall. DCP reaches 6,091 tok/s/GPU at c512 while still sitting at just 82% KV usage. The core value of DCP is that it sustains far higher concurrency, even on long-context runs, precisely the regime where replicated-KV TP runs out of memory first.

2.3 Comparison by Sequence Length

We also plotted performance against full sequence length (input + output). The figure shows a single throughput–interactivity Pareto frontier with requests grouped into five length bands (<32k, 32–64k, 64–128k, 128–200k, and 200k+) so we can see how performance shifts with context length. DCP keeps a high, stable frontier even in the 200k+ range, with the curves for short and long buckets nearly overlapping: throughput scales with concurrency while per-user speed stays usable at the long context lengths where the replicated-KV baseline runs out of memory and cannot scale.

3. Challenges of Serving Long Contexts

Under tensor parallelism, the KV cache is partitioned by the attention head. Each KV head owns its own separate K and V tensors, and the head is the smallest unit you can hand to a GPU. A standard TP has no mechanism to slice a single head’s KV cache. So if you have K KV heads, you can give each GPU a distinct subset of those heads, but only down to the point where every GPU holds one head. Once TP goes beyond K, there aren’t enough distinct heads to go around, so two or more GPUs end up holding a copy of the same head’s KV cache instead of a unique slice.

4. What is DCP?

Unlike pure TP methods, DCP is able to split KV cache across GPUs by sequence (context) dimension. Each GPU is made responsible for the KV cache of a chunk of token positions from the same sequence. For a single 200K-token request, GPU 0 might hold the cache for tokens 0–50K, GPU 1 for tokens 50K–100K, GPU 2 for 100K–150K, and GPU 3 for 150K–200K. By sharding KV cache, the KV cache footprint per GPU keeps shrinking as you add GPUs, freeing the memory that lets you raise the batch size and serve higher concurrencies.

4.1 Decode Context Parallelism Process

Standard Decode Context Parallelism keeps the communication pattern simple, following the rhythm AllGather Q → Compute → AllGather + ReduceScatter.

• AllGather Q: Each GPU has computed only a fragment of the query, but attention requires the full query vector to score against any key. An all-gather across the DCP group assembles a complete copy of the query on every GPU. This is cheap during decode because the query is a single token. As an opt-in alternative for MLA, vLLM #45964 can replicate the (small) query projection within each DCP group at load time so decode skips this query all-gather entirely (VLLM_DCP_Q_REPLICATE=1).

• Compute: Each GPU runs attention between the gathered query and its local slice of the KV cache. In vLLM this is k_up for MLA or tensor_broadcast for GQA.

• AllGather + ReduceScatter (cp_lse_ag_out_rs): The partial results are combined into the true output. AllGather shares each GPU’s partial output and LSE; the LSE values reweight and merge the partials (the online-softmax trick), and ReduceScatter sums them while handing each GPU back only its own head-slice.

5. vLLM Usage

DCP is enabled with a single extra argument, decode_context_parallel_size, alongside your existing tensor-parallel setting.

5.1 Offline

from vllm import LLM, SamplingParams

prompts = [
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(
    model="deepseek-ai/DeepSeek-V2-Lite",
    tensor_parallel_size=2,
    decode_context_parallel_size=2,
)
outputs = llm.generate(prompts, sampling_params)

5.2 Online

vllm serve deepseek-ai/DeepSeek-V2-Lite \
    --tensor-parallel-size 2 \
    --decode-context-parallel-size 2

5.3 MLA Backend

Models: DeepSeek-V2 / V3 / R1, Kimi K2.6 models using Multi-head Latent Attention.

Why it’s different. MLA compresses the Key/Value into a single low-rank latent vector that is shared across all query heads — effectively one KV “head.” Under pure tensor parallelism there’s nothing to split by head, so that latent KV cache is replicated in full on every TP rank. TP does nothing to shrink it, which makes MLA the ideal candidate for DCP: the whole cache is redundant, so the whole cache can be sequence-split.

What they do. DCP splits the latent KV cache along the sequence dimension, so each rank stores only its chunk of the latent; at attention time each rank up-projects its latent slice (the k_up step) to reconstruct the Keys/Values it needs. Because the effective KV-head count is 1, the sequence can be split up to the full TP degree — hence the constraints:

• tensor_parallel_size >= decode_context_parallel_size
• tensor_parallel_size % decode_context_parallel_size == 0

vllm serve deepseek-ai/DeepSeek-R1 \
    --tensor-parallel-size 8 \
    --decode-context-parallel-size 8

5.4 GQA Backend

Example models: Qwen3-235B, and other Grouped-Query-Attention models (Llama-family, etc.).

Why it’s different. GQA stores num_key_value_heads KV heads, and TP splits the KV cache by those heads first. That works cleanly only up to num_key_value_heads; once tensor_parallel_size exceeds it, the KV cache begins duplicating, with tp // num_key_value_heads identical copies across ranks.

What they do. DCP takes those would-be-duplicate copies and fills them with different sequence chunks instead, while the shared KV heads are broadcast across their query heads (the “tensor broadcast for GQA” step). So the sequence-split degree is capped by the duplication factor tp // num_key_value_heads:

• (tensor_parallel_size // num_key_value_heads) >= decode_context_parallel_size
• (tensor_parallel_size // num_key_value_heads) % decode_context_parallel_size == 0

# Qwen3-235B has num_key_value_heads = 4; tp=8 gives 8//4 = 2 redundant copies,
# so dcp can be up to 2.
vllm serve Qwen/Qwen3-235B-A22B \
    --tensor-parallel-size 8 \
    --decode-context-parallel-size 2

6. Future Work

Looking ahead, we plan to extend DCP along several main directions. We will add support for finer-grained parallelism sizes for both TP and DCP, giving users more precise control over their parallelism layout and reclaiming efficiency lost to over-provisioned sharding. We are also developing better DCP all-to-all (A2A) communication kernels for both multinode and single-node settings, reducing exposed communication and improving overlap with compute as context length and device count grow. We are working on better support for MTP and speculative decoding, so that DCP can deliver its efficiency gains without sacrificing the latency benefits of speculative methods, as well as hardening prefill/decode (P/D) disaggregation support to make DCP robust in disaggregated serving deployments. Finally, we aim to broaden DCP’s reach by extending support to a wider variety of backends and integrating it with hybrid models and Dynamic Chunked Pipeline Parallelism, so a much wider range of workloads can benefit from context-parallel efficiency gains.

The community is also expanding DCP to additional models such as GLM-5.2 and Kimi K3, and there is a longer roadmap for Prefill Context Parallelism (PCP). We are working on DCP performance benchmarking for the Kimi K3 model and plan to share those results as that work matures. For deployment guidance and historical notes on DCP, see the vLLM Decode Context Parallel docs.

7. Conclusion

Decode Context Parallelism represents a fundamental rethinking of how GPUs are organized for long-context inference. Rather than forcing GPUs to duplicate KV cache or sit underutilized, DCP puts every GPU to work: sharding the sequence during attention, then immediately reconfiguring those same GPUs to amortize FFN weight loading across the full pool. The result is a system that scales gracefully with context length rather than degrading under it.

With native support in vLLM, Decode Context Parallelism is ready to power the next generation of long-context agentic applications, from document reasoning to multi-session agentic pipelines, at the throughput and latency that production demands. It joins a broader industry move toward Decode Context Parallelism, a direction NVIDIA has also pursued with Helix Parallelism in TensorRT-LLM. We are also working on DCP performance benchmarking for the Kimi K3 model and plan to share those results as that work matures.

About Us

Special thanks to the NVIDIA team Anahita Bhiwandiwalla, Xin Li, Pavani Majety, Nidhi Bhatia, Roman Ageev, Pen Chung Li, and Chris Hoge for their reviews, benchmarking support, and engineering input throughout this study. We also thank Moonshot AI for the initial Decode Context Parallel work upstreamed in vLLM #23734, and Lucas Wilkinson for substantial follow-up contributions that helped harden and extend DCP. We also thank the broader vLLM community, whose open-source engine and continued collaboration made this benchmarking effort possible. For more on DCP deployment and related history, see the vLLM Decode Context Parallel docs.

The DCP results in this post were measured on NVIDIA B200 GPUs with Kimi K2.6 in NVFP4, and the recipes can be reproduced with current vLLM releases that support --decode-context-parallel-size. We are also working on DCP performance benchmarking for the Kimi K3 model and plan to share those results as that work matures.

Qwen3.5 was released in early 2026 and remains one of the most widely used models among customers. Because of its novel hybrid attention architecture, serving it in disaggregated mode brings in additional challenges and performance optimization opportunities. Thanks to the continuous contributions from the vLLM community, the disaggregated serving path for Qwen3.5 is now mature, and we are excited to report on the major contributions, latest performance on GB200 NVL72 systems, and recipes and best practices for you to reproduce. In this blog post we show how you can get over 25K total TPS/GPU performance.

Challenges and Key Optimizations

Qwen3.5’s hybrid architecture combines full-attention layers with Gated Delta Network (GDN) layers. This creates two distinct optimization challenges: accelerating GDN computation on Blackwell GPUs and transferring heterogeneous attention/GDN state correctly between prefill and decode workers.

SSM support for P/D serving was driven by the vLLM community through the NIXL disaggregation roadmap. For a deeper discussion of heterogeneous cache layouts, logical and physical block mapping, and tensor-parallel state transfer, see the detailed hybrid SSM disaggregation blog post. We highlight the following contributions as particularly important to Qwen3.5 performance.

1. Blackwell-Optimized GDN Prefill

FlashInfer: Add Blackwell GDN prefill kernel #3001

Compared with the previous FLA/Triton implementation, the new GDN kernel improves performance by approximately 1.02× to 5.78× across Qwen3.5 model sizes, tensor-parallel configurations, sequence lengths, and batch shapes.

The kernel was subsequently enabled on the prefill side in vLLM by vLLM PR #40717. On an 8×B200 system running Qwen3.5-397B-A17B-NVFP4, the vLLM integration delivered:

• Up to 5.92× higher GDN kernel performance in the tested microbenchmarks.
• 1.13× higher end-to-end prefill throughput on a prefill-only workload (ISL/OSL = 8192/1).
• A 12% reduction in mean TTFT (prefill-only workload 8K/1).

On supported Blackwell configurations, vLLM automatically selects the FlashInfer path when the GDN backend is set to auto. It can also be requested explicitly with:

--gdn-prefill-backend flashinfer

2. Hybrid Cache and GDN-State Transfer

Disaggregated serving for hybrid SSM-attention models builds on [Core][KVConnector] Support HMA+NixlConnector #35758 and a stack of connector changes described in the hybrid SSM disaggregation blog post. This PR is a necessary prerequisite: it maps HMA’s logical blocks onto the correct physical memory regions so NIXL can transfer only the cache regions belonging to each layer type, reducing transferred descriptors from 4,284 to 1,650 and improving throughput by up to approximately 7% in a small-scale intra-node H100 setup. But Mamba-style state differs enough in layout, size, and transfer semantics that HMA support alone would not have been sufficient for correct or efficient P/D serving.

The main PR for hybrid SSM-FA disaggregation is [PD][Nixl] Add support for hybrid SSM-FA models #36687, which adds dual descriptor views and homogeneous-TP support so prefill and decode workers can transfer both full-attention KV cache and Mamba-style SSM state over NIXL. Related follow-ups in the same stack include:

• [Kernel] Mamba support different layout for Conv state #37416
• [NIXL][Mamba][3/N] Heterogeneous TP: 3-read conv state transfer #37635
• [SSM/Mamba] Follow-up: N-1 prefill for P/D disaggregation #37310

See the hybrid SSM disaggregation blog post for how dual descriptor views, physical/logical block bridging, and conv-state transfer fit together.

For Qwen3.5 specifically, PD disagg with NIXL Connector: GDN support (Qwen3.5) #41869 extends this path to GDN layers.

3. Race-Free Async Scheduling

These two patches fix race conditions in KV block transfer that made async scheduling unusable — accuracy collapsed to zero with it enabled. Async scheduling turned out to be one of the key features behind crossing 25K tok/s/GPU, so both races had to be resolved.

• [KV Connector] Fix PD async scheduling race condition for hybrid attn models #48481
• [Bugfix] Defer block freeing until in-flight steps finish under async scheduling + PD KV consumer #45357

Performance

1. Environment Setup

Measurements were conducted on a GB200 cluster connected via NVLink72. We used ISL/OSL = 8192/1024. The evaluated model was Qwen3.5-397B-A17B-NVFP4. Performance was measured on a fixed decode topology and a constant number of decode endpoints. In this setup, the decode side used one endpoint with DEP8 (Data Parallel + Expert Parallel across 8 GPUs). On the prefill side, we evaluated configurations ranging from 4 to 8 endpoints, each using a fixed DEP2 topology.

To reproduce the results, use the latest vLLM vllm/vllm-openai:nightly-d223c90 Docker image, Dynamo 1.2.0.dev20260526, and srt-slurm v1.0.32. All recipes used in this article are available in the srt-slurm-recipes repository.

Accuracy Results

First, we measured accuracy for all serving configurations to ensure that the performance results are valid. For this purpose, we used the standard GSM8K (Grade School Math 8K) benchmark. Running GSM8K with srt-slurm is straightforward. To enable it, add the following benchmarking block to your recipe file:

benchmark:
  type: "gsm8k"

Accuracy results for all five configurations are 88%, which matches the accuracy we observe for the aggregated Qwen3.5 run.

2. Comments on Recipe Settings Choice

For performance measurements we used a fixed input sequence length / output sequence length benchmark, on a random dataset with random_range_ratio=0.8. The recipes themselves, and the settings that matter most, are covered in Recipes & best practices below.

3. Performance Results

Pareto curves for the individual configurations are shown in Figure 1, and the final Pareto frontier obtained after combining all configurations is shown in Figure 2. Total TPS per GPU reaches 25,000 tokens per second. Concurrency was swept from 64 up to 5120. We did not measure low concurrencies in the range of 1 to 32, since our focus here was the left part of the Pareto curve — maximizing the total TPS per GPU metric. At the other end, we did not go beyond 5120 because that is where we started running out of KV cache capacity on the decode side, which we deliberately fixed at a single 8×GB200 endpoint throughout these measurements. Pushing concurrency higher is entirely possible, but it requires adding GPUs on the decode side.

Recipes & best practices

All recipes used in this article live in the srt-slurm-recipes repository, and each one is launched with a single command:

srtctl run --file <recipe>.yaml

The naming scheme is NxDEP2-1xDEP8, where N is the number of prefill endpoints running DEP2 against a single DEP8 decode endpoint; there are five base configurations, from 4×DEP2 to 8×DEP2. Each comes with three derived variants: the base file sweeps sa-bench over concurrencies 64…3072, the -acc variant runs GSM8K five times on the same topology, and -cc4096 / -cc5120 each capture a single high-concurrency point with the decode-side max-cudagraph-capture-size raised to 640 and 768 respectively.

Most settings in the recipes are standard and shared across all configurations, but several are worth calling out:

• VLLM_SSM_CONV_STATE_LAYOUT=DS — mandatory for SSM models in disaggregated serving; conv-state transfer does not work without it. Our recipes also passed --no-disable-hybrid-kv-cache-manager; HMA has since been enabled by default in vLLM for several versions, so that flag is no longer required.
• --async-scheduling — one of the key features behind reaching 25K tok/s per GPU. It requires a vLLM build that already contains the race-condition fixes discussed above.
• --mamba-ssm-cache-dtype bfloat16 — significantly increases the effective KV cache capacity on the decode endpoint.
• --language-model-only — Qwen3.5 is a multimodal model, and for a purely textual workload this flag not only disables multimodal inputs but also unlocks the fused QK-norm + RoPE + gate path in the attention layers.
• --max-num-batched-tokens 16384 on the prefill side, i.e. 2× ISL. With fewer prefill endpoints ({4, 5, 6}×DEP2) prefill became the bottleneck and left the decode side idling below its peak throughput, so we let each prefill step batch two full prompts instead of one — worth about +8% of total TPS per GPU at high concurrencies.
• --max-cudagraph-capture-size on decode — raised to cc/8 + 128 for the two highest concurrency points (640 at cc=4096, 768 at cc=5120), where 8 is the number of DP ranks on the decode endpoint. The vLLM default caps captured graphs at 512, which is enough up to cc=3072. We are not certain this is actually required for the Pareto numbers reported here, but we set it as a precaution.
• Prefix caching is disabled: it buys nothing on a random dataset.
• --stream-interval 100 — reduces frontend overhead at high concurrency. Note that it buffers streamed output in 100-token chunks, so it does affect measured per-token latency; keep that in mind if you are optimizing for ITL/TPOT rather than aggregate throughput.

Finally, a couple of practical things that saved us a lot of time.

--api-server-count 1 is very useful while you are investigating a particular configuration. On a data-parallel endpoint vLLM defaults the API server count to the data-parallel size, and with more than one API server it disables its default stats logging altogether in order not to report incomplete numbers. Forcing the count to 1 brings that logging back: every 10 seconds — the interval is configurable through VLLM_LOG_STATS_INTERVAL — the server prints prompt and generation throughput along with KV cache utilization. Without these metrics we would hardly have identified the bottlenecks of the individual configurations, or understood why a particular option helps on our workload.

It is also worth setting three environment variables: DYN_LOG=error, DYN_SDK_DISABLE_ANSI_LOGGING=1, and VLLM_LOGGING_COLOR=0. The first one drastically cuts down the amount of Dynamo logs, while the other two suppress some (not all!) ANSI escape sequences in the log output. Without them your log files are very likely to be unreadable for a human, mostly because Dynamo produces an enormous amount of logging by default.

What’s next

Our measurements so far have concentrated mostly on the left part of the Pareto curve, squeezing out as much total TPS per GPU as possible. Next, we plan to sweep for the PD configurations that maximize Gen TPS per user instead. Reaching that regime will require shifting away from DEP topologies towards TEP (Tensor Parallel + Expert Parallel) or just TP, which as a rule deliver better per-user performance. Increasing the number of GPUs in use is another lever we expect to pay off here.

Acknowledgements

Artem Perevedentsev (NVIDIA), Vadim Gimpelson (NVIDIA), Jiangyun Zhu (Inferact), Nicolò Lucchesi (Mistral), Zhanqiu Hu (Red Hat), Nick Hill (Inferact), Linxuan Li (Alibaba), JingZe Cui (NVIDIA), Cyrus Chang (NVIDIA), Xin Li (NVIDIA)

Large language model serving on CPUs is an important deployment option because CPUs offer lower deployment cost, simpler infrastructure, and broad availability across cloud and enterprise data centers. As Arm® Neoverse™-based servers become more widely deployed, improving the usability, feature coverage, and performance of open-source serving frameworks such as vLLM on Arm CPUs has become increasingly important.

Over the last several months, we have worked with the vLLM, PyTorch, oneDNN, and KleidiAI communities to make upstream improvements across the Arm CPU serving stack. The result is improved usability, broader model and feature support, and substantial performance gains that benefit any Arm Neoverse-based server running vLLM.

In this blog, we walk through the usability and coverage improvements first, then dig into the main performance optimizations and the end-to-end serving results.

Enablement

Alongside performance optimizations, we improved the usability and feature completeness of vLLM on Arm®-based CPUs, making it easier to deploy vLLM on Arm® servers.

Key enablement improvements include:

• Pre-built wheels and Docker images.
• Bug fixes for crashes, accuracy issues, threading, and CPU utilization.
• Support for chunked prefill and prefix caching.
• Support for INT8 W8A8 and INT8 W4A8 inference.
• Model enablement for GPT-OSS, Whisper, and Qwen 3.5 / 3.6.
• Better integration with the PyTorch and UXL ecosystems.

With these enablement improvements in place, we turned our attention to understanding and eliminating the performance bottlenecks.

Performance Improvements

When we first benchmarked vLLM on Arm-based CPUs in October 2025, performance was much lower than expected given that roughly 80% of model runtime was spent in dense layers dispatched to highly optimized BF16 GEMMs. The standalone GEMM kernels behind those layers were already close to expected hardware efficiency, so the biggest gains were unlikely to come from GEMM kernels alone.

The profiles instead pointed to a broader optimization problem: allocator behavior, runtime synchronization, framework overheads, attention kernels, and quantized execution.

Memory Allocation

LLM serving puts significant pressure on the CPU memory allocator. During prefill and decode, vLLM repeatedly allocates and releases tensors for scheduling, KV-cache management, and intermediate operator outputs. In our initial benchmarks, memory allocation showed up as a bottleneck, with poor reuse of large allocations causing a high number of page faults.

The root cause was PyTorch’s use of glibc malloc. Large allocations were not reused effectively across repeated inference steps, and allocation/free paths became a source of contention as thread counts increased. As a workaround, we initially recommended preloading a caching allocator, but that added manual setup and made performance depend on runtime configuration.

To improve out-of-the-box performance, we enabled mimalloc as the default allocator on Arm-based CPUs in PyTorch. Mimalloc is a caching allocator designed to scale under multi-threaded allocation pressure. We chose it because it delivered strong performance across a broad range of TorchBench workloads and was already integrated as a PyTorch dependency for non-Arm Linux builds.

This improved Llama 3.1 8B out-of-the-box offline throughput by 2.3× and delivered gains of approximately 7× in low-concurrency serving scenarios.

Synchronization at High Core Counts

After improving memory allocation, the next bottleneck appeared when scaling inference to higher core counts. Beyond a certain point, adding more cores did not improve throughput and could even regress performance.

To understand where the scaling broke down, we profiled individual layers at high thread counts. One profile showed that 74% of the paged attention time was spent in OpenMP dynamic scheduling:

97.94% gomp_thread_start
  90.08% paged_attention_v1_impl
    74.07% gomp_iter_dynamic_next
     7.00% reduceValueBlock::lambda(int)

gomp_iter_dynamic_next is part of libgomp’s dynamic loop scheduling path. In this path, the runtime uses an atomic fetch-add to assign loop chunks to worker threads. The libgomp runtime used by the PyTorch wheels implemented that atomic update with a load-linked / store-conditional retry loop:

for (;;) {
    long old = LDXR(p);
    long newv = old + delta;
    int fail = STLXR(p, newv);
    if (fail == 0) {
        DMB_ISH();
        return old;
    }
}

At high core counts, many worker threads contend on the same atomic update, leading to repeated failed store attempts and retry traffic.

Tracing this down to the assembly revealed a missed hardware optimization opportunity. The benchmark system used Neoverse™ V2 cores, which support Arm Large System Extensions (LSE). LSE provides hardware atomic instructions, such as LDADDAL, that replace the inefficient loop above. However, the OpenMP runtime used by PyTorch did not leverage LSE atomics.

We addressed this by building a libgomp runtime in PyTorch that uses LSE atomics on capable CPUs.

This improved Llama 3.1 8B offline throughput by 9% and reduced Time Per Output Token (TPOT) latency by 15% in low-concurrency serving scenarios.

Dense-Layer Layout Overhead

Even after the allocator and runtime improvements, dense layers still left performance on the table. High-performance GEMM kernels are sensitive to weight layout: to run efficiently, weights need to be in a blocked format that matches the kernel’s vectorization and cache-access pattern. Without prepacking, each call can pay the cost of transforming weights from the framework tensor layout into the kernel-friendly format.

This is especially expensive at low concurrency, where the packing cost is not amortized over large batches. We addressed this by enabling a fast oneDNN path for dense layers, accelerated by the Compute Library for Arm Architecture. This path lets vLLM pack BF16 weights during model warmup into the format expected by the kernel, then reuse that packed representation during inference.

This improved Llama 3.1 8B offline throughput by 16% and reduced TPOT latency by 60% in low-concurrency serving scenarios.

Paged Attention

The CPU paged attention kernel was not optimized for Arm-based CPUs. The QK and PV matrix multiplications, along with the exponential in softmax, were falling back to reference implementations. As a result, we relied on PyTorch’s Scaled Dot-Product Attention kernel for prefill, which meant chunked prefill and prefix caching were not supported on the Arm CPU path.

We optimized the QK and PV paths with custom GEMM kernels using Arm BFMMLA Advanced SIMD instructions. We also optimized the softmax exponential with a fast vectorized third-degree polynomial approximation.

These changes made paged attention up to 4× faster and improved Llama 3.1 8B offline throughput by 12%. Furthermore, this allowed us to enable paged attention for prefill on Arm-based CPUs, unlocking support for chunked prefill and prefix caching.

BF16 Performance Improvements

The synchronization, weight prepacking, and paged attention optimizations combine to create a stronger BF16 serving baseline than the one we started with in October 2025.

INT8 W8A8 (8-bit weights and activations)

LLM inference repeatedly reads large weight matrices during prefill and decode. Storing weights in INT8 instead of BF16 reduces memory bandwidth pressure and can allow larger models to fit within the same memory budget.

On Arm-based CPUs with I8MM, W8A8 also maps to SMMLA, Arm’s signed INT8 matrix multiply-accumulate instruction, which provides twice the theoretical matrix-multiply throughput of BF16.

To take advantage of this, we accelerated the W8A8 quantization path with oneDNN JIT kernels that use SMMLA instructions on SVE128 and SVE256.

As a result, multiple Hugging Face INT8 W8A8 checkpoints, including RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 and RedHatAI/whisper-large-v3-quantized.w8a8, now perform well out of the box.

Compared to our optimized BF16 baseline, W8A8 with per-token activation quantization and channelwise weight quantization delivers as much as 88% higher throughput, 45% lower TPOT, and 54% lower TTFT, depending on concurrency.

INT8 W4A8 (4-bit weights, 8-bit activations)

W4A8 pushes the same idea further: quantize weights to INT4 to lower memory bandwidth pressure during inference. This is especially useful at low concurrency, where there is less batching to amortize the cost of reading model weights.

This path is accelerated through KleidiAI’s INT4 micro-kernels.

Compared to the W8A8 baseline above, W4A8 with per-token activation quantization and channelwise weight quantization delivers as much as 29% higher throughput, 26% lower TPOT, and 18% lower TTFT, depending on concurrency.

As expected, the biggest W4A8 speedups appear in low-concurrency scenarios where inference is mostly memory-bound.

Summary

vLLM on Arm-based CPUs has seen dramatic improvements in usability, robustness, model and feature coverage, and performance.

Relative to the October 2025 BF16 baseline, the optimized BF16 path delivers up to 2.7× the serving throughput. INT8 W8A8 reaches up to 4.8× the baseline throughput and a 5.7× TPOT speedup, while INT8 W4A8 delivers the best results with up to 6.2× the baseline throughput, a 7.8× TPOT speedup, and a 2.6× TTFT speedup.

The gains came from optimizing the full CPU inference stack: memory allocation, OpenMP synchronization, dense-layer prepacking, paged attention, and quantization.

Beyond the measured performance gains, these improvements make vLLM a more complete and production-ready inference stack for Arm Neoverse-based servers through broader feature coverage, better out-of-the-box usability, upstream integration, and expanded model support.

Acknowledgements

We thank the vLLM community for their continued support and collaboration.

Special thanks to Li Jiang (Intel®) for maintaining the vLLM CPU backend and implementing much of the infrastructure this work builds on. We also thank Sanket Kale (Fujitsu) for the initial Arm CPU enablement in vLLM, and Shreyas (Fujitsu) for contributing SVE256 INT8 kernels to oneDNN.

Arm is a registered trademark of Arm Limited (or its subsidiaries or affiliates).
PyTorch is a trademark of The Linux Foundation.
Intel and oneDNN are trademarks of Intel Corporation or its subsidiaries.

This blog post is Copyright 2026 Arm Limited and/or its affiliates <open-source-office@arm.com>

Speculative decoding has emerged as a core optimization technique for mitigating memory-bandwidth bottlenecks in Large Language Model (LLM) serving. By validating multiple candidate tokens in a single verifier-model forward pass, it allows production systems to achieve substantial inference speedups.

However, as serving infrastructure evolves, traditional speculative frameworks face a structural ceiling rooted in the way draft tokens are generated. Today, we are excited to showcase how Speculators and vLLM are moving beyond these limitations by providing full open-source support for three state-of-the-art parallel drafting algorithms: P-EAGLE, DFlash and DSpark.

2. The Limits of Recursive Drafting

The introduction of frameworks like EAGLE and MTP marked a major paradigm shift in speculative decoding. Instead of forcing the speculator model to guess blindly from surface-level text, EAGLE demonstrated that a speculator architecture could tap directly into the verifier model’s rich internal hidden states, dramatically increasing token acceptance rates.

Despite this breakthrough, advanced iterations like EAGLE-3 still operate under a fundamental constraint: auto-regressive drafting. To propose a sequence of candidate tokens, the speculator architecture must generate them sequentially, executing a separate forward pass for every single token.

This auto-regressive design introduces two major trade-offs in production:

• Constraints on Model Size: Because the drafting cost scales linearly with the speculation length, speculator models are forced to remain extremely small and lightweight to avoid consuming the execution time saved during verifier-model verification.
• Complex Operational Tuning: Linear scaling heavily limits the number of drafted tokens in practice. Choosing the optimal speculation length (K) becomes a sensitive variable that engineering teams must constantly adjust depending on the specific use case and real-time server loading.

Figure 2. Parallel drafting generates multiple draft tokens in a single step, whereas auto-regressive drafting generates one draft token per step.

3. The Shift to Parallel Drafting

Parallel drafting fundamentally re-engineers this trade-off by eliminating sequential execution from the drafting phase entirely. Rather than looping through single-token generation steps, parallel drafting algorithms predict an entire candidate block of tokens concurrently.

By flattening the drafting phase into a single forward pass, the latency of generating proposals is decoupled from the number of tokens speculated. This architectural shift simplifies production serving in two distinct ways:

• Capacity for Expressiveness: Because the speculator model only runs once per block, developers can utilize larger, more robust, and more expressive draft architectures. These deeper speculator models capture more complex context and yield higher acceptance rates without introducing a sequential latency penalty.

• Simplified Parameter Tuning: Decoupling drafting cost from block length removes the operational burden of hyper-tuning speculation parameters based on fluctuating server loads.

Parallel drafting as a concept has been explored before — Medusa and PARD are notable earlier examples. P-EAGLE, DFlash, and DSpark build on this foundation by combining parallel execution with deep verifier-state conditioning, the insight that made EAGLE so successful.

4. Under the Hood: Inference & Training Architecture

P-EAGLE, DFlash, and DSpark all build upon the verifier model’s hidden states to generate draft tokens in parallel, but each takes a different path to get there. Figure 3 illustrates their architectures side-by-side.

Figure 3. Comparison between P-EAGLE, DFlash and DSpark. P-EAGLE ingests hidden states from the verifier as part of the speculator model inputs. DFlash projects hidden states into KV-cache. DSpark builds on a DFlash backbone and adds sequential correction and confidence estimator.

A shared challenge across all three is training. Any parallel speculator must perform next-K prediction at every token position along a training sequence. For a sequence of length N and a lookahead window of K, naively computing losses across the full matrix causes memory and compute costs to scale prohibitively. Each algorithm addresses this differently.

P-EAGLE

P-EAGLE builds directly on EAGLE’s foundation of using the verifier model’s hidden states as input features. Instead of consuming those features to predict tokens sequentially, P-EAGLE maps them across multiple future positions simultaneously, outputting an entire sequence of candidate tokens in a single parallel step.

To keep training tractable, P-EAGLE implements draft block sparsification: it drops tokens along the lookahead dimension (K) according to a decaying rate, concentrating optimization on the most critical immediate tokens while pruning distant future positions from the loss calculation.

DFlash

DFlash routes verifier features differently. Rather than feeding hidden states in as standard inputs, DFlash projects them and injects them directly into the KV-cache of the speculator model. This tightly conditions the speculator’s attention mechanism on the verifier’s exact state without expanding the input sequence length, enabling it to generate a highly accurate block of candidate tokens via block diffusion.

For training, DFlash implements sequence length sparsification. Instead of calculating block loss at every token position across a sequence of length N, it selects random anchor points along the timeline and computes block predictions exclusively at these intersections — preserving GPU memory while maintaining representative coverage.

DSpark

DSpark takes DFlash’s parallel backbone and layers two additional innovations on top. First, it augments the architecture with a lightweight autoregressive correction head, allowing future tokens to be more strongly conditioned on past tokens. This combines the throughput benefits of parallel generation with the sequential coherence of autoregressive refinement.

Second, DSpark addresses a downstream bottleneck: verification cost. Parallel drafting can generate many draft tokens inexpensively, but the verifier must still process all of them. DSpark introduces a confidence head that scores draft tokens before they reach the verifier, selectively forwarding only those likely to be accepted. This reduces wasted verification compute and improves end-to-end throughput.

5. Inference Performance

Figure 1 illustrates the performance gains provided by parallel drafting algorithms when compared to EAGLE-3. Three distinct models and parallel drafting algorithms are displayed:

In all cases, parallel drafting shows significant improvement over EAGLE-3. Performance will vary across models, tasks, and hardware configurations — we encourage the community to benchmark on their own workloads.

6. Production Serving with vLLM and Speculators

Integrating state-of-the-art parallel drafting algorithms into production requires a stable, optimized infrastructure stack. The Speculators repository provides a unified ecosystem to train and evaluate these next-gen models, fully integrated with vLLM.

Launching a parallel-backed speculative engine is as straightforward as passing the appropriate configuration flags at initialization:

vllm serve Qwen/Qwen3-30B-A3B \
  --tensor-parallel-size 2 \
  --reasoning-parser qwen3 \
  --speculative-config '{
    "model": "RedHatAI/Qwen3-30B-A3B-speculator.dflash",
    "num_speculative_tokens": 7,
    "method": "dflash"
  }'

By moving from single-token generation to block-level parallel drafting, your inference pipeline becomes parallel all the way down—maximizing hardware utilization and delivering sustained, lossless acceleration. (Speculative decoding preserves the verifier model’s output distribution exactly via rejection sampling, so quality is mathematically identical to standard decoding.)

7. Get Started

Parallel drafting is fully supported, open-source, and production-ready today. We invite the community to explore the repository, utilize our documented training pathways to build your own parallel speculators, and benchmark them natively in vLLM.

• Repository: Speculators
• Pre-trained speculators: Speculators Collection on HuggingFace
• Training guides: Speculator tutorials

Errata

The plots in Figure 1 were updated on 7/29/26. The numbers in the original plots proved to be inconsistent with the reported benchmarking conditions due to an erroneous environment setup. However, the relative behavior between models was consistent and the conclusions in the blog are not changed.

Last week, we previewed the production-scale integration work for Kimi K3; today, Moonshot AI’s weights are public and the support is live.

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model (16 of 896 experts active per token) built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context window and native vision. For us, the most exciting challenge Kimi K3 brings is making KDA, MXFP4 MoE, KV cache management, prefill/decode disaggregation, speculative decoding, and long-context deployment recipes work together in a runnable serving engine.

The preview post explained the kernel and cache architecture, in particular the challenge of prefix caching that works on recurrent state. This release post is the practical guide: how vLLM adapts to Kimi K3’s architecture, the kernel work behind the numbers, and what is ready on day 0.

Quick start

# See the linked recipes for the exact Docker command.
vllm serve moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

The easiest way to run Kimi K3 is to use 8 NVIDIA B300 GPUs or 8 AMD MI355X GPUs with the above command.

Inferact has also trained and open-sourced a DSpark speculator for Kimi K3. Enable it by adding the following option to the serve command:

--speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'

For more details, including Docker images for various platforms and deployment strategies, refer to the detailed recipes. Because of complicated dependencies, only Docker images are usable now. The Docker images depend on several pre-release dependencies, including FlashInfer.

TL;DR

• A 2.8-trillion-parameter multimodal MoE: Kimi K3 activates 16 of 896 experts per token, supports a context window of up to 1M tokens, and is built on Kimi Delta Attention, Attention Residuals, LatentMoE, and native MXFP4 (4-bit) weights.
• Up to 370 tok/s per user: vLLM serves Kimi K3 at 118 tok/s without speculative decoding and 370 tok/s (a 3.14× improvement) with DSpark on 16 NVIDIA GB300 NVL72 GPUs, powered by extensive optimizations for Kimi K3’s architecture.
• Broad production feature support: vLLM supports speculative decoding, prefill/decode disaggregation, agentic KV caching with Mooncake, tool calling, reasoning output, and structured output, with NVIDIA (Hopper and Blackwell) and AMD (MI355X) support at launch.
• Open-source DSpark support: vLLM supports the state-of-the-art block-diffusion speculative decoding algorithm for Kimi K3, trained with vLLM and TorchSpec and open-sourced by Inferact.
• Hybrid prefix caching: serving Kimi K3’s recurrent and full-attention design required a redesign of hybrid prefix caching over recurrent KDA state. This change now benefits every hybrid linear model.

Kimi K3’s architecture, and how vLLM serves it

Kimi K3 architecture innovations, from the original release blog post.

Kimi K3’s architecture departs from a standard Transformer in a few ways, and each one changes what a serving engine has to do. The preview post covers the internals in depth; here we recap what’s new and focus on how vLLM adapts to serve it.

Kimi Delta Attention: a hybrid recurrent + full-attention stack

What’s new: Most of Kimi K3’s layers are KDA, a linear-attention mechanism that keeps a fixed-size recurrent state instead of a growing KV cache, interleaved with periodic full-attention layers that preserve exact global recall. That is what makes a 1M-token context affordable.

How vLLM serves it: A single hybrid KV-cache manager holds two kinds of memory side by side under one scheduler: paged KV blocks for the full-attention layers, and compact recurrent-state blocks for the KDA layers. A dedicated KDA attention backend runs FlashKDA for prefill and a fused CUDA kernel (or the Flash-Linear-Attention/Triton path when running speculative decoding) for decode.

The hardest part is prefix caching across Kimi K3’s hybrid cache: full-attention layers store per-token KV, while KDA layers update recurrent and convolution state at every token but cannot afford to retain a snapshot at every possible prefix boundary. vLLM decouples the large physical KDA state blocks from fine-grained prefix matching, registering state snapshots within those blocks and copying them before extension so long shared prompts can reuse both KDA state and paged KV. This hybrid-cache machinery is new in vLLM core and now benefits every hybrid model similar to Kimi K3.

Kimi K3 interleaves Kimi Delta Attention layers with periodic full-attention layers; vLLM’s hybrid cache manages recurrent state and paged KV together.

Attention Residuals: learned mixing of residual contributions across depth

What’s new: For each token, Block AttnRes replaces ordinary residual accumulation with depth-wise attention: every Transformer sublayer uses a learned pseudo-query to weight RMS-normalized residual states from preceding layer blocks, then receives the corresponding weighted combination as its input.

How vLLM serves it: vLLM uses optimized Triton and CUDA kernels to compute the depth-wise attention logits, softmax, and hidden-state aggregation in a single fused operation. Residual updates and output RMSNorm are folded into the same kernel where supported, reducing intermediate memory traffic and kernel-launch overhead in both prefill and decode.

Stable LatentMoE: quantile-balanced latent-space experts at 16-of-896 sparsity

What’s new. LatentMoE, introduced by NVIDIA, projects dispatched token activations into a narrower latent dimension for routed-expert computation, then projects the combined expert output back to the model width—reducing expert-weight bandwidth and all-to-all traffic so more experts can be used at similar inference cost. Kimi K3’s Stable LatentMoE scales this design to 896 experts with 16 active per token and uses Quantile Balancing to derive expert allocation from router-score quantiles instead of heuristic balancing updates.

How vLLM serves it: Experts are sharded with expert parallelism. vLLM offers two MoE backends tuned for different topologies: TRT-LLM-Gen for tensor-parallel (TP > 1) and MegaMoE for disaggregated/expert-parallel (DEP). It also supports optional Expert-Parallel Load Balancing (EPLB) to ensure each rank has a similar amount of compute. The weights execute natively in MXFP4 on the MoE path.

Chat template: a render program, not a Jinja template

What’s new: Kimi K3’s chat template must encode system, user, and assistant messages, multimodal content, tool definitions, and tool results using exact control tokens. Instead of the common approach of a Jinja chat template that renders the request as text before tokenization, Kimi K3 uses a Python program to build the prompt token sequence directly. Its output likewise contains distinct regions for reasoning, answer text, and tool calls that must be parsed into an API response.

How vLLM serves it: vLLM implements both the input renderer and streaming output parser in its Python and Rust frontends, preserving control-token boundaries while treating user-supplied and tool-supplied text as ordinary content. For tool calls and structured outputs, vLLM integrates Kimi K3’s format with XGrammar so structured regions are constrained during decoding and returned as separate reasoning, content, and tool-call fields.

Built for production

Serving a 2.8T hybrid MoE well means being fast for each user, efficient for many concurrent sessions, and scalable for agents. vLLM ensures Kimi K3 is ready on all three.

Ultra-low latency: speculative decoding with DSpark

To reach ultra-low latency on a 2.8T-parameter model like Kimi K3 without accuracy loss, speculative decoding is the natural choice. That is why vLLM supports DSpark, a state-of-the-art speculative decoding algorithm, from day 0—and why we trained and released a DSpark speculator for Kimi K3. The draft model is trained with vLLM using TorchSpec to achieve full numerical parity between speculator inference and training.

DSpark uses a block-diffusion backbone to generate multiple speculative tokens in one parallel pass based on Kimi K3’s rich intermediate states, so drafting cost stays flat as the block deepens. A low-rank Markov head supplies the intra-block dependency, and a confidence head predicts the likelihood for each draft to be accepted. We made the draft MLA-native, mirroring Kimi K3’s own attention, so draft and target share a similar KV layout to be maximally compatible with advanced KV management and P/D-disaggregated setups.

Kimi K3 DSpark positional acceptance rates across various datasets.

With DSpark, we achieve a 3.14× speedup on a single-user request, from 118 tok/s to 370 tok/s, measured using SPEED Bench. We also benchmarked the acceptance rate and speedups on different tasks, with the results shown above. For coding and other low-entropy tasks, we achieve around 4.73 accepted tokens per step. For high-entropy tasks such as creative writing, we achieve around 2.61 accepted tokens per step.

Confidence-based scheduling with DSpark is an ongoing effort in vLLM. Once enabled, it uses the confidence head included in the DSpark model to predict how likely each drafted token is to be accepted, prioritizing strong proposals and pruning weak ones so verification is not spent on tokens that will not survive.

Both the draft model and the inference support are open source as of this release. See the deployment guide below to enable it.

A lightweight DSpark draft proposes candidate tokens that Kimi K3 verifies in a single parallel pass, accelerating single-stream decode.

Sequence parallelism for TEP prefill

Sequence parallelism shards token ownership across ranks; the attention residual is applied per shard, and one all-gather rebuilds the full batch before the next layer’s QKV projection.

For the prefill phase, we combine attention tensor parallelism with MoE expert parallelism (TEP). Compared with pure TP, TEP reduces communication overhead and keeps whole experts on each rank, yielding more efficient expert GEMM shapes.

However, the naive TEP implementation performs two all-reduce operations per layer—one after the attention output projection and one after the MoE—so every rank materializes the full batch and redundantly applies the attention residual to all of it. To address this, we implement sequence parallelism: the all-reduce after o_proj is replaced with a reduce-scatter so that each rank owns a shard of the tokens, the attention residual is applied per shard, the MoE’s all-to-all performs dispatch and combine, and a single all-gather restores the full batch before the next layer’s QKV projection.

This design provides two key advantages:

• Reduced communication overhead: Reduce-scatter + all-to-all dispatch + all-to-all combine + all-gather are theoretically cheaper than two all-reduces. In practice, however, NCCL’s reduce-scatter and all-gather are not optimized for prefill’s message sizes and yield no speedup. We therefore implement custom reduce-scatter and all-gather kernels that are 1.7×–4.5× faster than NCCL, especially at small-to-medium message sizes.
• Sharded attention residual: The attention residual stays sharded across ranks throughout the layer, so each rank computes and maintains only its shard of the tokens rather than the entire batch. This matters especially for Kimi K3, where AttnRes turns the residual stream into persistent cross-layer state with its own compute and memory footprint.

Sequence parallelism is enabled by default when appropriate: when using TP with the MegaMoE kernel, or when combining TP + DP + EP. No extra flags are needed.

Large-scale serving: prefill/decode disaggregation

For high-throughput settings, vLLM serves Kimi K3 with expert and data parallelism across nodes and with prefill/decode (PD) disaggregation, which runs prefill-heavy and decode-heavy work on separate replicas so each is sized for its own bottleneck. One of our validated topologies routes TEP8 prefill to DEP16 decode, with NIXL as the KV transfer engine.

PD disaggregation is unforgiving for a hybrid model: the recurrent KDA state, the full-attention paged KV, and the block tables all have to arrive correctly. The NIXL connector treats the shared KV-cache page as two logical views: token-level MLA cache and request-level KDA state, including convolution and recurrent state. During the handshake, it exchanges the MLA/KDA metadata, then builds separate transfer descriptors for each transfer.

Under heterogeneous TP, vLLM’s hybrid allocator uses different block sizes for prefill and decode. To support that case, vLLM’s NIXL connector tracks the logical-to-physical block mapping and zeroes any untransferred tail regions, preventing stale data from previous requests from leaking through padding or layout gaps.

Reconciling partial block cache hits and KV cache offloading

As described in the Kimi K3 preview, vLLM supports fine-grained prefix hits that may end inside a physical cache block. This introduces a subtle challenge for KV offloading: vLLM may first find a local GPU hit with a partial tail, then discover a longer prefix in an external store such as Mooncake. With full-block hits, remote reuse extends cleanly beyond the local prefix. A partial tail, however, can overlap with the remote result.

The vLLM scheduler therefore compares the exact reusable token lengths from both tiers and selects the longer prefix. If the remote hit wins, it releases the block reserved for the shorter local tail and reconciles all cache groups to the new prefix length.

Importantly, we built this mechanism entirely through the existing KV Connector APIs, which already provide all the required semantics. This allows MooncakeStoreConnector, SimpleCPUOffloadConnector, and other connectors to support multi-tier partial-prefix reuse without model-specific integration paths.

The design is tracked in the RFC and implemented across PR #45939, PR #46384, and PR #49502.

Agentic serving: smarter cache retention policies

Kimi K3’s linear-attention layers require only a constant-size KDA state, making them memory-efficient at long context lengths. A single layer’s KDA state is roughly equivalent to the MLA cache for a few thousand tokens. Although large, this state does not grow with sequence length, unlike a conventional KV cache. That distinction becomes significant for agentic workloads spanning hundreds of thousands to one million tokens.

The same design complicates prefix caching. KDA state is updated in place during decoding, so vLLM must copy the state at a selected prefix boundary before the next forward pass overwrites it. Caching at every token position would be prohibitively expensive: each KDA checkpoint is much larger than one token’s MLA cache and would quickly exhaust even a distributed cache pool.

To improve cache-space efficiency while preserving useful prefixes, vLLM supports two complementary retention policies.

Interval-based retention

Caching every KDA state is wasteful, but caching too sparsely forces the next request to recompute a large suffix. Interval-based retention balances these costs by treating selected positions as checkpoints—for example, one every 32K tokens.

Prompt boundaries are even better checkpoints. In agentic workloads, the next turn usually begins by replaying the previous turn’s prompt, so the state at the end of that prompt is especially likely to be reused. vLLM detects and retains these boundaries automatically.

Users can control periodic checkpointing with VLLM_PREFIX_CACHE_RETENTION_INTERVAL. Setting it to 0 disables periodic checkpoints and retains only prompt-end states, which is a good fit for workloads dominated by multi-turn conversations. Larger intervals trade some recomputation for lower cache usage.

Interval-based retention was introduced for DeepSeek V4 and hybrid sliding-window-attention models in PR #43447, with day-0 support for Kimi K3 and hybrid linear-attention models added in PR #45845.

Interval-based cache retention. MLA caches KV for every block, while a KDA state is kept only at checkpoints: prompt ends (green) are always retained, and fixed-interval checkpoints (orange) are configurable.

Marconi-style selective retention

Prompt-end retention works well for conversational state, but valuable shared prefixes can appear elsewhere. A system prompt, repository snapshot, or tool specification may be reused across many requests without aligning with a prompt boundary.

Marconi-style retention (MLSys ‘25) handles these cases with a simple rule: cache on the second hit. The first observation provides evidence that the prefix exists; the second shows that it is actually shared. Only then does vLLM spend cache capacity on its KDA state.

This turns retention into a demand-driven decision. One-off prefixes do not crowd the cache, while recurring prefixes are promoted automatically—without requiring users to predict which parts of their workload will become hot.

Selective retention was introduced in PR #37898, with day-0 Kimi K3 support added in PR #47782.

Selective cache retention. Request 1 keeps a KDA state only at its own prompt end, past the shared prefix, so request 2 gets a KV hit but a KDA miss. That second sighting is evidence the prefix is shared, so a state is cached at the prefix boundary, and request 3 reuses it.

Together, the policies cover both predictable and emergent reuse: interval retention checkpoints structurally important boundaries, while Marconi-style retention learns which other prefixes are worth keeping.

Performance optimizations

Serving large models like Kimi K3 brings its own challenges because of its size. The entire model can barely fit in a single NVIDIA DGX B300 and requires a minimum of 16 NVIDIA B200/GB200 GPUs to serve on that hardware generation. Serving must trade off interactivity against total system throughput: tensor parallelism is good for interactivity but offers low overall throughput because effective KV cache size is limited, while large-scale expert parallelism can limit per-user output-token speed because of network-bandwidth bottlenecks. Here we highlight optimizations that improve performance in both cases so users can choose the recipe that suits their workload. Many of these optimizations are already covered in our preview blog.

Attention Residuals

Kimi K3 uses Block AttnRes, attending over up to eight cached block representations plus the current within-block residual. For each token, vLLM computes logits from RMS-normalized sources, applies softmax across these depth-wise candidates, and aggregates their representations. Its implementation resembles FlashAttention’s online-softmax strategy but operates across model depth rather than sequence positions and has at most nine sources. vLLM performs this mixing in a single fused kernel, incorporating the residual update at the input and optionally applying RMSNorm to the output. A portable Triton implementation covers the general path, while a specialized CUDA kernel accelerates supported Blackwell configurations.

KDA decode

The fused KDA decode kernel folds the causal convolution, recurrent update, and RMSNorm into a single launch instead of a chain of separate kernels.

A KDA layer involves many operations: input projections, causal 1D convolutions, QK norm, gate computation, KDA recurrent update, and output gated RMSNorm. On supported configurations, vLLM fuses the post-projection decode path—from the causal convolutions through gated RMSNorm—into a single specialized CUDA kernel. The kernel updates the convolution and recurrent states in place and writes the normalized output directly, avoiding intermediate tensors, repeated state traffic, and per-operation launch overhead across Kimi K3’s many KDA layers. Portable Triton fallback paths cover unsupported configurations.

KDA prefill

KDA prefill became one of our favorite examples of open-source development in practice. Moonshot AI first released FlashKDA, a high-performance CUTLASS implementation of KDA. We quickly integrated it into vLLM and worked through less glamorous production details: broader GPU coverage, metadata dtypes, tensor layouts, and reliable vendoring. Shikhar Mishra then optimized the kernels for H100 and published Flash-Flash-KDA, improving data movement while preserving numerical correctness. Within a day, we validated the improvements on GB300 NVL72, refined the recurrence pipeline and synchronization, and folded them into our FlashKDA integration. The result was not a one-way handoff, but a continuous loop in which an open kernel was extended by the serving community, improved by an independent contributor, and quickly carried into production.

KDA metadata builder

During Kimi K3 DSpark bring-up, KDA metadata preparation emerged as a significant source of overhead. Kimi K3 initially reused the generic GDN metadata builder, which prepared FLA metadata that K3 does not consume and used sequences of small eager PyTorch operations to assemble and stage GPU metadata. We introduced a dedicated Kimi K3 KDA metadata builder that prunes the unused paths and replaces those operation sequences with fused Triton kernels, reducing each sequence to a single launch. At batch size 1, this reduced metadata-preparation latency by 96%, from 870 μs to 34 μs, and improved end-to-end DSpark latency by 6%.

Low-latency BF16 GEMM

In low-batch-size, latency-sensitive settings, we replace generic BF16 GEMM—used in several linear projection layers—with our own skinnyGEMM implementation. Generic cuBLAS kernels do not achieve the best performance here because they are optimized for more general shapes. In the kernel, we bypass shared-memory data staging, load activations and weights directly into registers, and use CUDA Core FMA instructions to perform the math. This avoids the heavy TMA and Tensor Core setup phase used to achieve maximum throughput. Our microbenchmarks show kernel-level speedups ranging from 8% to 100% and an end-to-end latency reduction of about 10% in small-batch settings.

Low-latency MoE tail fusion

The LatentMoE tail optimization replaces two all-reduces, RMSNorm, latent up-projection, and an elementwise add with three kernels to reduce compute and better overlap communication and computation.

vLLM uses a novel strategy to reduce latent-MoE tail latency in ultra-low-latency serving. At the end of LatentMoE, the reduced activation from routed experts must be normalized with RMSNorm and up-projected before it is added to the shared-expert output. In the normal TP case, this requires two all-reduces on the routed and shared experts—or one all-reduce with concatenation—and replicates the up-projection.

To avoid redundant compute in the replicated linear projection, vLLM instead performs reduce-scatter on the shared experts and keeps all-reduce on the routed experts because their activations need to be normalized. The replicated routed-expert activation then performs matrix multiplication with the up-projection in a column-parallel fashion and is added elementwise to the already-sharded shared-expert output. Finally, the results are all-gathered onto each rank using broadcast. We observe about a 20% latency reduction in this step and about a 7%–8% end-to-end speedup.

Quality and Performance Benchmarks

Accuracy and correctness evaluation

vLLM takes accuracy as seriously as speed. We validated Kimi K3 end to end through a served OpenAI-compatible endpoint, with exact configurations in the recipes, and it passes the accuracy evaluations cleanly. At the maximum reasoning-effort level, Kimi K3 on vLLM scores 0.976 on GSM8K, 0.939 on GPQA-Diamond, 0.889 on OCRBench, and 0.818 on MMMU Pro Vision.

One caveat worth knowing for evaluation: Kimi K3 thinks a lot before it answers. A low score is more often a truncated answer than a wrong one, so increase the reasoning effort, set max_tokens generously, and check for cut-off generations before debugging anything else.

Serving performance

Kimi K3 decode throughput at batch size 1, measured on GB300 NVL72 GPUs in TP8 and TP16 configurations.

At launch, vLLM achieves 111 tok/s per user on TP8 and 118 tok/s per user on TP16 at batch size 1. DSpark speculative decoding boosts interactivity by roughly 3×, reaching 331 tok/s per user on TP8 and 370 tok/s per user on TP16.

We also present initial Pareto-frontier performance results for serving Kimi K3 on GB300 NVL72 across a range of scenarios, from high-throughput serving at 2K+ TPGS to low-latency serving at 100+ TPS/user.

Reproduce our benchmark

Here are the full recipes to reproduce the decode throughput numbers above for TP8 with DSpark:

export NCCL_DMABUF_ENABLE=0
export VLLM_ALLREDUCE_USE_FLASHINFER=1
export VLLM_USE_RUST_FRONTEND=1
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export HEAD_ADDR=127.0.0.1  # Change if vllm-bench runs on another host.

vllm serve moonshotai/Kimi-K3 \
  --enable-prefix-caching \
  --tensor-parallel-size 8 \
  --nnodes 2 \
  --node-rank 0 \
  --moe-backend auto \
  --trust-remote-code \
  --load-format fastsafetensors \
  --max-num-seqs 512 \
  --gpu-memory-utilization 0.9 \
  --max-model-len auto \
  --max-cudagraph-capture-size 256 \
  --kv-cache-dtype fp8 \
  --attention-config '{"mla_prefill_backend":"FLASHINFER","use_prefill_query_quantization":true}' \
  --speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'

# Batch size = 1, 8K/1K random (no speculative decoding)
vllm-bench \
  --backend openai \
  --base-url "http://${HEAD_ADDR}:8000" \
  --model moonshotai/Kimi-K3 \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --random-range-ratio 0.8 \
  --prompt-token-ids \
  --ignore-eos \
  --sweep-max-concurrency 1 \
  --sweep-num-prompts-factor 10 \
  --seed 42 \
  --percentile-metrics "ttft,tpot,itl,e2el" \
  --metric-percentiles "50,90,99" \
  --save-result

# Batch size = 1, SPEED Bench (speculative decoding)
vllm-bench \
  --backend openai \
  --base-url "http://${HEAD_ADDR}:8000" \
  --model moonshotai/Kimi-K3 \
  --dataset-name speed-bench \
  --speed-bench-config throughput_16k \
  --speed-bench-max-input-len 10240 \
  --speed-bench-category low_entropy \
  --output-len 1536 \
  --num-prompts 10 \
  --no-oversample \
  --max-concurrency 1 \
  --temperature 1.0 \
  --top-p 0.95 \
  --save-result \
  --save-detailed

Full recipes, including multi-node, expert-parallel, and vision configurations, are in the Kimi K3 recipes.

Important Deployment Tips

1. Prefix caching: --enable-prefix-caching turns prefix caching on. Prefix caching is typically enabled by default in vLLM, but it is currently disabled by default for Kimi K3 while the hybrid-cache design continues to evolve. Pass the flag explicitly.
2. Tool calling: Validate on your own traffic before depending on it. We’ve occasionally seen K3 emit a tool-call format its own parser does not expect, yielding an empty tool_calls result, while clean probes on the same setup parse perfectly. It is prompt- and run-dependent, not a blanket failure, but production agents should validate against your schema, retry or fall back when tool_calls comes back empty, and consider strict or structured tool calling, which constrains the output grammar during generation.
3. All-to-all backend: --all2all-backend determines how the MoE backend communicates during expert parallelism. Use flashinfer_nvlink_one_sided for NVIDIA NVLink and deepep_v2 for RDMA.
4. MoE backend: vLLM has several MoE backends for different scenarios. We recommend deep_gemm_mega_moe for any DEP environment.
5. Rust frontend: Set VLLM_USE_RUST_FRONTEND=1 to enable the Rust frontend, which fully supports this model.
6. ViT parallelism: --mm-encoder-tp-mode=data is enabled by default. K3’s vision encoder has head_size=12, which cannot be sharded evenly under TP=8. Because K3’s vision encoder has fewer than 1B parameters while the backbone has about 2T, we enable ViT DP by default to avoid all-reduce overhead from the encoder.

Kimi K3 vLLM FAQ

How many GPUs do I need to serve Kimi K3?

At least one 8× B300 (or GB300 NVL72) node is required; 16× B200 is also supported. Most production deployments run multi-node with expert and data parallelism, connected over RDMA or NVLink.

How do I enable DSpark speculative decoding?

Add:

--speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'

It roughly triples single-stream decode on reasoning and coding workloads.

Which MoE and all-to-all backend should I use?

Use deep_gemm_mega_moe for disaggregated or expert-parallel (DEP) deployments and flashinfer_trtllm for TP > 1. Choose the all-to-all backend to match your interconnect: flashinfer_nvlink_one_sided for NVLink and deepep_v2 for RDMA.

Does Kimi K3 support prefix caching, and is it on by default?

It supports prefix caching over both full-attention KV and recurrent KDA state, but it is not enabled by default, so pass --enable-prefix-caching.

Does vLLM support Kimi K3 on AMD GPUs?

Yes. ROCm support ships at launch, with broader tuning on the roadmap.

How is this different from the Kimi K3 preview post?

The preview is the architecture and kernel deep dive, including how KDA prefix caching and the kernels are built. This post is the practical launch guide and includes the artifacts: how vLLM adapts to Kimi K3, recipes, flags, performance, and what Kimi K3 is ready for in production.

Roadmap and Future Work

• RL support for Kimi K3: vLLM rollout support has already been added. We will work closely with RL ecosystem projects to support end-to-end RL training for Kimi K3.
• Continuous performance improvement: continue improving performance after day 0.
• Decode Context Parallelism (DCP): our prototype shows good speedup, and we will soon upstream the support. Early experiments show 40% higher throughput than TP8 under selected workloads.
• Expert-Parallel Load Balancing (EPLB): improve EPLB performance.
• Confidence-based scheduling: use the confidence head in DSpark to prune the number of draft tokens to verify.
• Broader AMD ROCm tuning.

Quick links

• Model: moonshotai/Kimi-K3
• DSpark draft: Inferact/Kimi-K3-DSpark
• Recipes and Docker images: recipes.vllm.ai/moonshotai/Kimi-K3
• Kimi K3 technical blog: kimi.com/blog/kimi-k3
• vLLM design for K3: the preview post

Acknowledgements

Thank you to Moonshot AI for creating K3, sharing the architecture ahead of release, and co-designing the KDA-aware caching; to the Inferact team for the end-to-end vLLM integration and deployment validation; to NVIDIA for the fused KDA decode, KDA prefill, and Attention Residual kernels and the MXFP4 MoE collaboration; to AMD for ROCm bring-up; to our inference partners, including Alibaba Cloud, Baseten, DigitalOcean, and Modal; to Shikhar for Flash-Flash-KDA; and to the vLLM community. The cache infrastructure built for Kimi K3 now belongs to every hybrid model with a similar architecture. We can’t wait to see what you serve.

We deployed GLM-5.2-NVFP4 across three 8-GPU B300 servers (24 GPUs in total) using a disaggregated 4-Prefill + 1-Decode topology. Under our production SLA targets of mean TTFT ≤ 2.5 s and mean TPOT ≤ 20 ms, we measured the following:

Our starting point on the same hardware was a mean TPOT of nearly 40 ms for 16K-token inputs — roughly twice our SLA limit. This post documents the full journey from 40 ms to 17 ms: what we changed at each step, how much each change contributed, and why the parallelism strategy we shipped is not the one with the highest raw throughput. A complete, reproducible set of vllm serve commands is included at the end.

1. Why We Optimized for SLA Compliance, Not Peak Throughput

In colocated serving, prefill chunks are interleaved into decode batches, and every long prompt entering a batch stretches the inter-token latency of every request already decoding. TPOT tail latency therefore becomes a function of the incoming prompt length distribution — something a serving system does not control. Disaggregation removes prefill work from the decode critical path entirely, making TPOT a function of decode batch composition alone. That is what makes a tight TPOT SLA achievable in the first place, and it is why the rest of this post treats P/D as the starting point rather than as one optimization among many.

Production services are not accepted on peak throughput alone. The real question is how much traffic the system can sustain without violating latency targets. Our requirements were explicit:

• Typical context length: 16K–256K tokens
• Mean TTFT ≤ 2.5 s — the maximum acceptable delay between a user action and the first visible token
• Mean TPOT ≤ 20 ms — roughly 50 tokens/s of streaming output; below this rate, the reading experience degrades noticeably
• Subject to those two hard constraints, maximize throughput

A note on batch size: we did not fix one. Load is applied as a request rate, and concurrency is whatever that rate produces under the SLA — deliberately as large as the latency budget allows. Because prefill cost grows with context length, concurrency settles lower as input length rises: roughly 700 concurrent requests at 8K, 300 at 16K, and 25 at 256K, all under a --max-concurrency 1024 cap.

This objective shaped the methodology throughout: every configuration search was constraint-aware. A configuration that improves throughput by 30% but pushes mean TPOT past the SLA is not useful to us. Section 4 contains a representative example.

GLM-5.2 is a 744B-parameter MoE model with 40B active parameters. It uses DSA sparse attention and natively supports MTP speculative decoding, and vLLM already provides mature support for all three. Our work was to combine them effectively with P/D disaggregation, then tune parameters, topology, and scheduling around production SLA targets.

2. Starting Point: Improving Decode Performance under P/D Disaggregation

With the initial configuration, the Prefill side already met its target and had ample TTFT headroom. The bottleneck was Decode: with 16K input tokens and 1K output tokens, mean TPOT was close to 40 ms, with substantial P99 jitter.

2.1 Root Cause: Mixed Batches at the P/D Handoff

Speculative decoding has become a standard inference-time optimization for large MoE models, and production Decode deployments increasingly run it by default. Profiling revealed an issue that sits precisely at the interaction between P/D disaggregation and speculative decoding.

After a request transfers its prompt KV cache to a Decode node through KVConnector, its first Decode step needs to compute only one token. Existing requests on that Decode node, however, are scheduled with 1 + N tokens per step when MTP is enabled. Because the two request types have different shapes, the step becomes a mixed batch. It can no longer take the uniform-decode full-CUDA-Graph fast path and instead falls back to the more expensive piecewise or eager execution path.

Data parallelism amplifies the impact. Under DP, CUDA Graph mode and padding require coordination across ranks, so if any DP rank receives a newly transferred request, the remaining ranks follow it onto the same execution path. In steady-state P/D operation, new requests arrive at the Decode instance continuously, so the slow path is triggered constantly.

2.2 Optimization: Speculative Padding on the Decode Side

The fix is conceptually simple. On the first Decode step after a request arrives, dummy speculative tokens pad its shape to 1 + N, matching the other requests already in the Decode worker. This preserves uniform Decode execution and keeps the workload on the full-CUDA-Graph fast path.

It requires no transfer of generated tokens or draft tokens from the Prefill node. The optimization was merged by the vLLM community in PR #45237.

2.3 Performance Gain

With the execution-path regression caused by mixed batches eliminated, end-to-end mean TPOT dropped from approximately 40 ms to approximately 22 ms — the single largest improvement of the entire effort.

The result carries a broader lesson for deployments that combine P/D disaggregation with speculative decoding: the largest performance loss may not live in any individual kernel. It can arise at the boundary between subsystems, where small inconsistencies in request state, scheduling shape, and CUDA Graph execution mode are amplified by DP scale and continuous traffic.

3. Further Decode-Side Optimizations

At 22 ms we were close to the SLA but had no safety margin, so we ran another round of configuration search.

3.1 Model Runner V2: 11% Lower TPOT

vLLM Model Runner V2 (MRV2) refactors the runtime execution path. Since v0.25.0 it is the default for all dense models; GLM-5.2 is an MoE model, so it is not enabled by default and must be activated explicitly with VLLM_USE_V2_MODEL_RUNNER=1.

On our Decode configuration, MRV2 improved TPOT by approximately 11% over MRV1. Beyond the shorter execution path, MRV2 brings several capabilities that matter for predictable production latency:

1. PR #47285 adds the GLM-5.2 DSA indexer prefill-metadata kernel to startup warmup, so the first production request no longer triggers Triton JIT compilation and a latency spike. This is easy to miss in benchmarks, where warmup absorbs it; in production it shows up as a cold-start spike after every rolling deployment.
2. PR #46448 adds local argmax reduction for multi-GPU MTP. With use_local_argmax_reduction enabled, draft-token generation no longer AllGathers full-vocabulary logits, reducing TP communication from a volume proportional to vocabulary size to approximately 2 × TP size. MTP, EAGLE, DFlash, and other speculators running under MRV2 all benefit.
3. PR #45953 lets dynamic speculative lengths work with full CUDA Graphs, reducing graph misses and eager fallbacks caused by changes in draft length.

3.2 All-to-All Backend: 4% Lower TPOT

Because GLM-5.2 is an MoE model, the Decode side runs DEP8, which places expert dispatch and combine communication directly on the critical path. We replaced the default AllGather/ReduceScatter-based EP backend with the FlashInfer NVLink A2A backend; in our measurements, flashinfer_nvlink_two_sided cut TPOT by a further 4%.

vLLM now also ships the newer flashinfer_nvlink_one_sided backend, which is expected to perform better. This post keeps the two-sided backend because that is the configuration we actually measured. Evaluating the one-sided backend under the same Decode workload is on our list.

3.3 CUDA Graph Mode

The Decode instance runs --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' together with --max-num-batched-tokens 1024. The Decode side does not need graphs compiled for prefill shapes, and FULL_DECODE_ONLY gives complete graph coverage of the Decode path while cutting startup compilation time significantly.

3.4 MTP Speculative Decoding

We use num_speculative_tokens=3 on the Decode side and 1 on the Prefill side. (MTP only becomes cost-effective on GLM-5.2 together with IndexerCache, discussed in Section 5.) The asymmetry is intentional: Prefill nodes should produce and hand off KV cache as fast as possible, so deeper speculation buys little there. Decode nodes sit on the latency-critical path, where deeper speculation amortizes the execution cost per token — provided the acceptance rate stays high.

4. Prefill Parallelism: Why We Did Not Choose the Highest-Throughput Configuration

On the Prefill side we compared several parallelism strategies at 8K- and 32K-token inputs. Since the configurations use different numbers of GPUs, TGS — throughput per GPU — is the meaningful metric.

Per-GPU prefill throughput for four parallelism strategies, at 8K and 32K input lengths.

In absolute terms TP1 DP4 EP is the fastest instance — 47,806 tok/s at 8K inputs — but only because it uses twice as many GPUs as the others; per GPU it lands second. Two conclusions stand out:

1. TP2 + EP performed worse than plain TP2. At a scale of only two GPUs, the all-to-all overhead introduced by EP exceeds its benefit. EP needs enough experts spread across enough devices to amortize its communication cost.
2. TP1 DP2 EP achieved the best TGS, but we shipped TP1 DP4 EP.

The second point is where production engineering parts ways with benchmark chasing. TP1 DP2 EP delivered the best per-GPU efficiency, but each instance had only two GPUs, leaving too little KV-cache capacity for GLM-5.2’s 1M-token context capability. We were not willing to give up one of the model’s most important features for an 8% TGS advantage, so we chose TP1 DP4 EP — trading roughly 8% of per-GPU efficiency for the KV-cache capacity of four GPUs per instance.

5. MTP + IndexerCache: How We Improved the Acceptance Rate

vLLM has been extensively validated in production under conventional configurations. This deployment enabled three relatively new capabilities at once — P/D disaggregation, MTP, and MRV2 — which put us on a less frequently exercised combination path, and the tuning process surfaced several long-tail issues. Most fixes went from report to release within a few days; anyone on v0.26.0 already has all of them. This section records the work to show how MTP acceptance was stabilized step by step.

5.1 IndexerCache: Making MTP Cost-Effective with DSA

IndexerCache (PR #44420) is not a conventional KV cache — it reuses the Top-K sparse indices produced by the DSA indexer. A naive implementation reruns the indexer for every MTP draft step, and since sparse-retrieval cost grows with context length, that can consume much of the benefit expected from speculative decoding. PR #44420 introduced index_share_for_mtp_iteration, which lets the first draft step compute Top-K indices and subsequent draft steps reuse them. This is a prerequisite for MTP to be worthwhile on GLM-5.2 at all.

The community then completed three improvements around this mechanism, which together stabilized acceptance under high concurrency:

• PR #45895 improves indexer initialization when Top-K layers are skipped and fixes the GLM-5.2 MTP normalization loop. The PR reports that on GLM-5.2-FP8 with TP=8, mean accepted length rose from approximately 3 to approximately 4, at an average acceptance rate of about 60%, while IFBench held at 74.62.
• PR #47238 optimizes the layout of the shared index buffer for batched requests: after the first draft step, it retains only the Top-K indices corresponding to each request’s final query token. This was the key step in extending index sharing from single-request execution to high-concurrency batching.
• PR #47448 ensures the MTP loop reuses the post-final-norm hidden state.

Taken together, IndexerCache is not merely a compute-saving optimization. It is also a key mechanism for holding MTP acceptance rates up under high concurrency.

5.2 Two Additional Fixes for the Combined Configuration

MRV2 scheduling classification. After switching to MRV2, we observed excessive TPOT variance under a specific benchmark pattern and reported it in Issue #47239. The community quickly traced it to uniform-decode ordering: speculative-decoding steps were being classified as prefill and therefore took a slower execution path. PR #47381 fixed it.

Lookahead handling for asynchronous KV loading in P/D deployments. PR #46694 improves slot-allocation timing for the combined GLM-5.2 + NIXL P/D + MTP configuration. The Decoder now waits until remote KV transfer completes before allocating speculative-token slots, correctly handling the boundary case of a final partial KV block. This partial-block handoff is specific to the interaction between P/D disaggregation and speculative decoding, and fixing it was another step toward making the path production-ready.

Both fixes are included in v0.25.0 and later releases.

5.3 Accuracy Validation

We ran a set of public benchmarks with the final configuration to verify that combining NVFP4 quantization, MTP, and P/D disaggregation did not reduce output quality:

The results are consistent with community-reported GLM-5.2 numbers. LongBench V2 mattered most to us because it directly exercises DSA sparse attention and IndexerCache index sharing under long contexts. A score of 64.01 indicates that index sharing holds up in long-context workloads, and that the speedup from speculative decoding did not come at the expense of output quality.

6. Upstream Progress

The capabilities used in this deployment build on a broader set of optimizations coordinated by the vLLM community under the GLM-5.2 optimization tracking issue, Issue #46654. Beyond the PRs referenced above, two developments are especially relevant.

A secondary tier for P/D disaggregation. PR #42285 introduces a unified CPU KV-cache layout as an intermediate layer. TieringManager coordinates the primary cache tier and the P/D connector, reducing coupling between the transfer backend and the model execution path. Merged in v0.25.0.

PCP virtual batching for long-context Prefill. PR #46570 splits requests into multiple virtual batch rows processed in parallel across context-parallel ranks, aggregating only the MLA latent cache and DSA indexer cache. In an initial 4-GPU GLM-5.2-NVFP4 / 32K Prefill test, TP=2 with PCP=2 raised prompt throughput from approximately 20.1K tok/s to approximately 27.3K tok/s. The PR was merged into main on July 19, 2026, and will ship in the next release. Since KV-cache capacity is exactly why we rejected the TGS-optimal configuration in Section 4, PCP may change that trade-off, and it is a key focus of our next validation phase.

7. Observability: Verifying the SLA after Production Launch

Passing a benchmark does not make a system production-ready. Observability is harder for a disaggregated P/D deployment than for a single instance, because request latency is split across two resource pools: TTFT is determined mostly by the Prefill pool, TPOT by the Decode pool, with a KV transfer in between. When any stage degrades, users see only one symptom — the service got slower.

We built monitoring for the P/D topology with Prometheus and Grafana. In production we watch the following metric groups:

• Per-pool TTFT and TPOT percentiles, not just end-to-end aggregates. This is the first place to look to decide whether a problem belongs to Prefill or Decode.
• MTP acceptance rate and mean accepted length. Easy to overlook, yet among the earliest warning signals available. A declining acceptance rate produces no error; TPOT simply degrades gradually. In practice this is the first dashboard we open for any Decode-side anomaly, and we recommend treating MTP acceptance rate as a first-class alerting metric.
• KV-cache utilization versus GPU utilization, on both the Prefill and Decode sides. The central benefit of P/D disaggregation is that the two resource types scale independently; the relative levels of these curves are the scaling signal.
• KV-transfer latency and queue depth, to determine whether the inter-node network has become the bottleneck.

7.1 A Problem Visible Only during Long-Running Stability Tests

Short benchmarks validate performance, not long-term stability. Our production acceptance process includes multi-day continuous runs, and that step revealed persistent host-memory growth: vLLM process RSS increased linearly over tens of hours without reaching a plateau.

Grafana Memory Usage (WSS) panel: vLLM container memory grew linearly from approximately 721 GiB to approximately 800 GiB over tens of hours.

Several characteristics explain why this issue needed production monitoring, rather than a benchmark, to be discovered:

• The growth rate was slow and only visible over hours. A vllm bench serve run lasting seconds or minutes could not reveal it.
• The growth affected host memory, not GPU memory. Every GPU-side metric looked normal.
• Conventional memory-analysis tools could not see it. EngineCore calls gc.freeze() during startup, so the leaked objects never appeared in gc.get_objects() or tracemalloc; the symptom looked like allocator fragmentation.

We reported the behavior and our initial diagnosis to the community in PR #47723, and a maintainer incorporated it into PR #44490.

The root cause was inconsistent gating between a producer and a consumer. PR #35219 had introduced SingleTypeKVCacheManager.new_block_ids for clearing Mamba SSM cache state. Entries were recorded based on the KV-cache spec type — FullAttentionSpec, MLAAttentionSpec, and so on — but cleared only when the model contained Mamba layers. For models without Mamba layers, which includes most standard attention models and GLM-5.2 with MLA, every block allocation was recorded and the list was never drained, so it grew without bound as request volume increased. The fix was to drain take_new_block_ids() unconditionally on every scheduling step and use its result only when clearing is actually required. Mamba behavior is unchanged.

8. Complete Deployment Recipe

8.1 Environment

8.2 Prefill Node

export VLLM_USE_V2_MODEL_RUNNER=1

vllm serve /mnt/model/glm/GLM-5.2-NVFP4 \
    --trust-remote-code \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}' \
    --chat-template-content-format=string \
    -ep \
    -tp 1 \
    -dp 4 \
    --tool-call-parser glm47 \
    --enable-auto-tool-choice \
    --reasoning-parser glm45 \
    --gpu-memory-utilization 0.92 \
    --enable-prompt-tokens-details \
    --speculative-config='{"method":"mtp","num_speculative_tokens":1}' \
    --shutdown-timeout 300 \
    --fingerprint-mode=none

8.3 Decode Node

export VLLM_USE_V2_MODEL_RUNNER=1

vllm serve /mnt/model/glm/GLM-5.2-NVFP4 \
    --trust-remote-code \
    --chat-template-content-format=string \
    --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}' \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --max-num-batched-tokens 1024 \
    -ep \
    -tp 1 \
    -dp 8 \
    --tool-call-parser glm47 \
    --enable-auto-tool-choice \
    --reasoning-parser glm45 \
    --gpu-memory-utilization 0.90 \
    --enable-prompt-tokens-details \
    --all2all-backend=flashinfer_nvlink_two_sided \
    --speculative-config='{"method":"mtp","num_speculative_tokens":3}' \
    --shutdown-timeout 300 \
    --fingerprint-mode=none

8.4 Benchmark Methodology

All performance data was collected with the random dataset in vllm bench serve. There were no prefix-cache hits, so the TTFT figures represent worst-case, compute-only latency. Requests were injected at a fixed --request-rate, computed as target TPS divided by (input_len + output_len) and multiplied by a tuning factor.

vllm bench serve \
  --backend openai-chat \
  --model /mnt/model/glm/GLM-5.2-NVFP4 \
  --endpoint /v1/chat/completions \
  --dataset-name random \
  --random-input-len 16384 \
  --random-output-len 1000 \
  --request-rate <target TPS / (input + output) × tuning factor> \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90 \
  --save-result

9. What Comes Next

For GLM-5.2 and the larger MoE models that will follow, we see two major directions for Decode optimization. The following are forward-looking assessments rather than conclusions measured in this post.

First, reduce the cost of each target forward pass. Beyond core kernels such as GEMM, Attention, and the Indexer, this includes exploring PDL, persistent kernels, localized megakernels, and coordinated computation and communication across MoE Dispatch, Expert GEMM, and Combine. As deployments scale across nodes, WideEP, hierarchical all2all, and overlap between inter-node communication and computation become increasingly important. The objective is to shorten the end-to-end critical path of a complete Decode step.

Second, advance model–runtime co-optimization for speculative decoding. On the model side, stronger drafters such as DSpark can be trained on broader datasets to improve the accuracy of consecutive proposal tokens. On the runtime side, dynamic speculative decoding, per-request proposal lengths, and compact verification can keep verification cost in check across different workloads. The draft model determines how far the system can predict; the serving runtime determines how far prediction is actually economical.

In addition, the PCP virtual-batch work described in Section 6 has already landed in main. We will evaluate whether it removes the KV-cache-capacity versus per-GPU-efficiency trade-off discussed in Section 4.

About Us

This work was completed by the DaoCloud team. We deliver full-stack platforms for enterprise LLM training and inference, including heterogeneous accelerator scheduling, inference-service orchestration, and observability. We shared our observations and validation results with the vLLM community throughout the tuning process, and the resulting improvements were merged upstream. The configuration in this post can be used directly with v0.26.0.

Special thanks to Nicolò Lucchesi (@NickLucche) from Mistral AI, who suggested writing this post in the first place, reviewed it in detail, and kept it moving from a rough set of notes to what you have just read.

We also thank the vLLM community for its efficient collaboration under Issue #46654. In most cases, fixes moved from initial report to a released version within one week.

vLLM AFD Plugin brings AFD into Mixture-of-Experts (MoE) models by separating Attention and FFN into independently deployed services. The plugin preserves vLLM’s existing request lifecycle and OpenAI-compatible serving interface, while allowing the Attention and FFN paths to scale independently.

The project currently supports NVIDIA GPUs and Ascend NPUs, synchronous and asynchronous connectors, DeepSeek V2/V3-family model wrappers, and eager, graph, and dual-batch execution paths within clearly validated limits.

Why Attention-FFN Disaggregation?

Mixture-of-Experts (MoE) inference combines two very different kinds of work inside every transformer layer. Attention is stateful and closely coupled to request scheduling and the KV cache, while the FFN or expert path is dominated by routed expert computation and all-to-all communication. When both paths share the same worker topology, the serving system must make one set of scaling and execution choices for workloads with very different requirements.

Making this separation practical requires addressing several system design challenges:

1. Attention and FFN have different scaling requirements. Attention capacity follows request state, sequence length, and KV-cache pressure. Expert capacity follows token routing and expert load. The serving system should support independent scaling by allowing both paths to use different rank topologies, instead of requiring one shared layout.
2. Attention and FFN have different runtime responsibilities. Attention needs scheduling, KV-cache coordination, and sampling. FFN execution only needs activations, routing metadata, and a way to return expert outputs. Splitting the services lets the FFN side run as a lightweight connector-driven daemon.
3. Communication is backend-specific. CUDA and Ascend expose different collective libraries, graph runtimes, and optimized MoE operators. A common connector contract keeps the model-facing flow stable while allowing each backend to own its data path.
4. Communication and computation benefit from overlap. Asynchronous dispatch and MoE ubatching can overlap independent stages instead of serializing all expert work behind the Attention path.

Together, these challenges define the core design goal of AFD: keep vLLM’s request-facing Attention path intact, while moving FFN execution behind a narrow connector interface that can scale, communicate, and execute independently.

Inside the Architecture

The plugin integrates through vLLM’s vllm.general_plugins entry point and the standard --additional-config channel. It does not require edits to the vLLM source tree.

The runtime has three main parts:

• Attention service. The Attention worker retains vLLM’s scheduler, KV cache, batching, model lifecycle, and sampling path. A plugin-owned model runner installs AFD metadata into the forward context and publishes data-parallel, ubatch, layer, and graph state to the FFN side.
• FFN service. The FFN worker has no request traffic, scheduler, or KV cache. A background loop receives metadata and activations, invokes compute_ffn_output() on the plugin-owned model wrapper, and sends the result back to Attention. Requests are always sent to the Attention API server.
• Connector layer. At each split layer, the connector transfers Attention hidden states together with the execution metadata required by the FFN service, then returns the computed FFN outputs. A backend-neutral connector interface defines this exchange while allowing each backend to implement its own communication and runtime optimizations.

This integration surface is designed to be intentionally small. vLLM continues to own the serving control plane where its existing abstractions fit, while the plugin provides the implementations of AFD workers, model runners, connectors, metadata, model split points, and a small set of version-scoped compatibility patches.

Connector and backend support

The same high-level exchange - Attention output to FFN, FFN output back to Attention - is shared across connectors. Backend packages remain separate so CUDA graph behavior, ACL graph behavior, NCCL communication, and Ascend custom operators do not leak into one another.

Supported features

• Native vLLM serving surface. Existing vLLM users still launch with vllm serve, send requests to an OpenAI-compatible endpoint, and configure the runtime through --additional-config.
• GPU and NPU implementations. GPU workers extend vLLM v1 classes, while NPU workers extend vLLM-Ascend classes directly. Shared behavior lives in configuration, topology, metadata, and connector contracts rather than cross-device inheritance.
• Synchronous AFD for decode throughput. P2pNcclAFDConnector and CAMP2pAFDConnector synchronously exchange Attention activations and FFN outputs, allowing the two roles to scale independently in throughput-oriented decode deployments. Their current graph paths use FULL_DECODE_ONLY semantics on CUDA and ACL, respectively.
• Asynchronous AFD for prefill. CAMAsyncAFDConnector uses CAM asynchronous dispatch and combine operators to decouple prefill Attention ranks from expert workers. Together with AFD-managed MoE ubatching, it overlaps independent Attention and FFN stages to reduce pipeline stalls. This path currently targets the prefill stage in a prefill/decode-disaggregated deployment and does not yet support graph execution.
• MoE model integration. The plugin registers wrappers for DeepSeek V2/V3-family architectures, including DeepSeek V3.2, and GLM MoE DSA. The wrapper exposes separate Attention and FFN computations while reusing upstream layer implementations.
• Graph and ubatching paths. The synchronous GPU and NPU connectors support decode-only graph capture. Dual Batch Overlap is supported with exactly two ubatches, and CAM async provides AFD-managed MoE ubatching for its prefill path.

A Performance Snapshot

Synchronous AFD Decode Throughput with CAMP2pAFDConnector

The synchronous decode recipe in vllm-project/afd-plugin#67 compares a conventional EP64 deployment with CAMP2pAFDConnector-based AFD deployments for DeepSeek-V3.2 W8A8 on Ascend 910C. The benchmark measures saturated decode throughput rather than online-serving latency.

Throughput is normalized by the total number of deployed dies:

tokens/s/die = aggregate output token throughput / total deployed dies

Both workloads use fixed-length inputs and uniformly distributed outputs from 512 to 1,536 tokens.

16K fixed input

EP64 achieves 232.6 tokens/s/die, 48A16F achieves 220.3 tokens/s/die, and 64A16F achieves 258.9 tokens/s/die. Relative to EP64, the AFD results are -5.3% for 48A16F and +11.3% for 64A16F.

32K fixed input

EP64 achieves 168.2 tokens/s/die, 48A16F achieves 151.4 tokens/s/die, and 64A16F achieves 183.3 tokens/s/die. Relative to EP64, the AFD results are -10.0% for 48A16F and +9.0% for 64A16F.

Across both input lengths, 48A16F is below the EP64 baseline, while 64A16F delivers the highest normalized throughput: +11.3% at 16K and +9.0% at 32K. This result shows that the Attention-to-FFN allocation matters; disaggregation alone does not guarantee a throughput gain.

Due to limited machine availability, we did not evaluate deployments with higher Attention-to-FFN ratios. The observed trend suggests that, at the ratios tested, the FFN ranks still have compute headroom rather than being compute-bound. Increasing the proportion of Attention ranks may therefore reveal further throughput gains.

Asynchronous AFD Prefill Performance with CAMAsyncAFDConnector

The repository includes an early CAM async experiment on two Ascend 910C nodes using a DeepSeek V3.2 W8A8 model reduced to 10 layers. The comparison uses forced expert balancing and contrasts a DP4PCP8 TP1 baseline with an AFD layout consisting of Attention DP3PCP8 TP1 plus FFN EP8.

Across the measured request rates, the AFD configuration lowers median/P50 time to first token. At 12 requests per second, median TTFT decreases from 15.1 seconds to 8.0 seconds, a reduction of approximately 47%. At both 10 and 12 requests per second, the measured gap is about 7.2 seconds.

Note: These numbers are a focused validation of the CAM async execution path, not a general performance claim for full DeepSeek V3.2 or every AFD topology. The performance gains may also vary across workloads.

Getting Started

The current implementation requires Python 3.10–3.13 and targets vLLM 0.19.1.

Install

Check out the installation steps in our README for details.

Deployment Recipes

Deployment commands depend on the backend, connector, model, and rank topology. Instead of duplicating configurations here, use the maintained AFD Plugin recipes:

• GPU synchronous AFD: the DeepSeek V2 Lite P2P NCCL recipes cover decode-oriented colocated and prefill/decode-disaggregated deployments, eager and CUDA graph execution, and multiple DP/TP layouts.
• NPU asynchronous prefill AFD: the DeepSeek V3.2 CAM async recipe documents the required environment, topology, AFD configuration, benchmark setup, and current limitations.

Refer to the repository README and recipe directory for the latest supported connector matrix, configuration fields, and complete launch commands.

Current Scope and Roadmap

The project intentionally exposes its current boundaries: exact vLLM version pinning, model runner v1 only, full weights on both roles, decode-only graph modes, exactly two ubatches for DBO, and hardware-gated end-to-end testing.

The next phase of development will focus on:

• Broader vLLM compatibility and upstream alignment: track newer vLLM releases, evaluate model runner v2, keep compatibility patches minimal, and contribute generally useful abstractions upstream as they mature.
• More flexible execution: extend graph modes, ubatch counts, asynchronous stages, and validated rank topologies.
• Production-scale validation: publish repeatable accuracy, latency, throughput, stability, and multi-node results on full models and realistic workloads.
• Expanded model and connector coverage: add MoE architectures and backend transports through the existing model-wrapper and connector interfaces, together with corresponding deployment recipes for each newly supported model and connector.
• Multimodal and vLLM-Omni integration: explore how AFD can integrate with vLLM-Omni and heterogeneous multimodal pipelines, including its application within autoregressive (AR), Diffusion Transformer (DiT), and other stages that can benefit from independently scaled Attention and FFN execution.
• Heterogeneous hardware and low-latency serving: explore deploying Attention and FFN roles across different accelerator types and interconnects, together with connector, scheduling, placement, and computation-communication overlap optimizations that reduce time to first token and inter-token latency.

Join the Community

vLLM AFD Plugin is at an early stage, and feedback from model, serving, and hardware communities will shape its direction.

• Code and documentation: github.com/vllm-project/afd-plugin
• Runtime design docs: GPU Attention/FFN and Ascend Attention/FFN designs
• Issues and feature requests: GitHub Issues

Let’s build a more composable and hardware-aware future for MoE serving together.

Moonshot AI has announced that the full model weights will be released by July 27, 2026. In the meantime, vLLM, Moonshot AI, NVIDIA, AMD, and the broader community are working through the final integration and validation so the open-source community can serve Kimi K3 from day 0.

This post is a preview and performance optimization is ongoing, but the core model path, KDA-aware prefix caching, multimodal integration, tool calling parsers, and hardware-specific optimizations are already taking shape. Selected trusted partners, approved by both Moonshot AI and the vLLM/Inferact team, have also begun deployment validation using the same code that is being prepared for open source.

As stated in the announcement blog, KDA poses new challenges for conventional prefix caching, and the Moonshot AI team has contributed a corresponding implementation to the vLLM project, to be released alongside the model weights. We will dedicate a future blog post to explaining the design.

TL;DR

• Day-0 open-source serving: vLLM is preparing model implementation, Docker images, deployment recipes, and production validation for the Kimi K3 weight release.
• A new hybrid architecture: Kimi K3 combines KDA-dominant linear attention with periodic full-attention layers, AttnRes across depth, Stable LatentMoE, and native vision support.
• Prefix caching required core changes: vLLM now separates the physical KDA state-block size from prefix-match granularity, enabling useful partial prefix-cache hits without storing recurrent state at every small attention block.
• Kernel work across the stack: the release branch includes FlashKDA integration, fused KDA decode, fused KDA projections and convolution, fused AttnRes, reimplemented MLA module, SiTU-enabled MXFP4 MoE execution, and optimized expert routing.
• NVIDIA and AMD support: NVIDIA-specific kernels are under final tuning, while an initial AMD implementation with a FlyDSL MoE kernel is already in place and moving through broader validation.

Kimi K3 at a Glance

Kimi K3 is not a larger version of Kimi K2. Kimi K3 changes the serving problem in several dimensions at once.

For inference systems, each of these choices moves cost somewhere new. KDA reduces the need to retain a conventional KV pair for every past token, but introduces a large recurrent state. AttnRes reduces the limitations of a single residual stream, but creates additional cross-layer memory traffic. Extreme MoE sparsity avoids activating all 2.8T parameters for every token, but raises the stakes for routing and communication. vLLM’s job is to make all of these pieces work together behind one familiar serving API.

A Collaboration Built Over Multiple Kimi Generations

Kimi K3 continues a long collaboration between Moonshot AI and the vLLM community.

• At GOSIM 2024, Moonshot AI engineers presented how vLLM was used at scale inside Moonshot AI and discussed the vLLM + Mooncake prefill/decode-disaggregated architecture.
• Moonshot AI later shared Kimi K2 training and inference practices at the vLLM Beijing Meetup, including operating under strict SLOs while serving online traffic and supporting reinforcement-learning workloads.
• vLLM has been a day-0 launch partner for Kimi K2, Kimi K2-Thinking, Kimi K2.5, Kimi Linear, and so on.
• vLLM has deep technical collaboration with Moonshot AI engineers, including Kimi K2 tool-calling accuracy for correctness, improved CUDA debugging for development, decode context parallelism, Mooncake-based PD disaggregation, and large-scale performance validation. Kimi K2.5 has also appeared in public InferenceX serving results.

That history matters. Day-0 support is rarely one pull request written after a release announcement. It comes from model and inference teams sharing architecture details early, testing real checkpoints under realistic parallelism, identifying gaps in the serving engine, and upstreaming improvements that remain useful after one launch. vLLM is proud to be a long-term partner of Moonshot AI and a popular inference engine for Kimi-series models.

Now, let’s dive into one of the most interesting technical challenges we ran into.

The Hardest Part: Prefix Caching for KDA

Conventional full attention and KDA remember a prefix in very different ways.

In full attention, a prefix is represented by per-token key and value vectors. vLLM stores those vectors in paged blocks, hashes complete token blocks, and can reuse a matching sequence of blocks for another request.

KDA is recurrent. Instead of retaining a conventional KV pair for every token, each KDA layer advances a matrix-like recurrent state, together with a short convolution state. To resume from a cached prefix, the engine needs the KDA state at the exact prefix boundary. Replaying an earlier state to reach that boundary would erase much of the benefit of prefix caching.

The straightforward solution—store KDA state at every small attention-cache boundary—is too expensive. A KDA state is much larger than one ordinary token’s KV entry, so implementations use a relatively large physical state block to amortize storage. Before the current work, that physical block size also constrained where a prefix-cache hit could land. With a multi-thousand-token state block, two requests sharing almost the entire prompt could still miss the reusable prefix because their common boundary did not fill the same physical block.

The new vLLM design separates three concepts that used to move together:

• Physical block size: how KDA state and full-attention KV are allocated on the GPU.
• Scheduler alignment: where execution must stop so all cache groups remain consistent.
• Prefix-match unit: the finer token interval at which a shared prefix is hashed and may be matched.

This lets vLLM register a valid KDA state at a fine-grained boundary inside a larger physical state block. When a later request hits that partial block, the cached state is copied into a private destination before the request extends it. This copy-on-write rule preserves the shared cached prefix while allowing the new request to continue generation safely.

The implementation also handles details that are easy to miss:

• The scheduler stops at the right block and hash boundaries so the recurrent state being registered really corresponds to the advertised token prefix.
• Full-attention and KDA cache groups agree on one num_computed_tokens, even though their physical block sizes differ.
• Partial cache entries use chained, fine-grained hashes so a boundary identifies the entire prefix, not only the tail tokens.
• Same-step reuse is deferred until the state copy is safe, avoiding races between cache registration and extension.
• Cache transfer and disaggregated prefill/decode paths can carry the same logical prefix across workers.

This work was motivated by Kimi K3 and many other hybrid attention models, but it is core vLLM infrastructure rather than a model-specific shortcut. The vLLM team and the Moonshot AI team collaborated deeply on the design. The two teams will publish a separate post with the design, invariants, and benchmarks in more detail.

Performance Work: Removing the New Bottlenecks

Our current progress can be summarized into this table:

Kimi K3 changes the hot path, so the team has optimized more than the attention kernel itself. Below are details of the progress in each area.

KDA prefill and decode

The prefill path integrates FlashKDA and Flash Linear Attention (FLA). Around the core recurrence, vLLM fuses the input projections and causal convolution, and gathers initial recurrent states in one operation.

Decode uses a fused NVIDIA kernel on supported architectures and shapes. Instead of launching separate operations for the short convolution, KDA state update, output gate, and normalization for every generated token, the fused path performs them together. This is especially important because Kimi K3 contains many KDA layers; a small per-layer launch or memory penalty quickly becomes a large TPOT penalty.

Attention Residuals

AttnRes retrieves from representations written by earlier layer blocks rather than relying on only one uniformly accumulated residual stream. A naive implementation creates extra reads, writes, reductions, and normalization launches throughout the 93-layer network.

The release branch includes a Triton implementation and an NVIDIA kernel that fuse residual update, AttnRes mixing, and output RMSNorm for supported cases. Sequence-parallel work also shards the attention-residual traffic across ranks. Early kernel-level results are encouraging, while end-to-end gains are still being measured across prefill lengths and parallel configurations.

Optimized MLA module for native PD disaggregation deployment

Kimi K3 still uses MLA attention every four layers. In the previous model, vLLM relied heavily on a torch.compile custom-fusion path to map small kernels into fused kernels, which slowed startup and still left many kernels unfused. In this release, we implement a new MLA module that fuses these kernels manually. MLA also requires different kernel launch orders for prefill and decode, so we implement two code paths with different fusion patterns, specialized for PD-disaggregated deployment. Furthermore, Kimi K3 introduces a gate projection that can execute in parallel with the main attention path. We optionally add multi-stream support for the gate projection in the decode path, while in the prefill path—where multi-stream overlap is not optimal—we fuse the elementwise multiply and sigmoid into the gate-projection epilogue.

MXFP4 MoE

Kimi K3’s release configuration uses MXFP4 weights and the SiTU activation. Before this work, the MXFP4 TRTLLM-Gen path did not support SiTU and would fall back to a slower implementation. vLLM now maps Kimi K3’s SiTU parameters into the optimized FP4 expert path and also handles large token-by-top-k launch grids by safely chunking the workload.

This has already been validated on a 16-GPU DP16+EP16 configuration, where all ranks selected the optimized MXFP4 backend and passed correctness checks.

On the AMD side, Kimi K3 MoE is supported on FlyDSL’s MLIR Python kernel stack. This includes hardware-tuned A16W4/A8W4 quantized fused operators and a SiTU activation implementation, all built on FlyDSL’s modular abstractions.

What to Expect on Open-Source Day

The planned day-0 package includes:

• vLLM model, parser, cache, and kernel integration;
• initial open-source Docker images;
• validated launch recipes for NVIDIA configurations;
• an initial AMD path with FlyDSL MoE kernel, with more ROCm tuning to follow;
• multimodal, tool-use, reasoning, and structured-output examples;
• initial performance results.

Trusted deployment partners are already exercising the release candidate under a dual-approval process from Moonshot AI and vLLM/Inferact. This provides real production feedback without distributing prerelease model artifacts broadly. It also gives us a chance to test the complete serving system—frontend semantics, batching, cache transfer, expert parallelism, observability, and failure handling—not only isolated kernels.

Acknowledgements

Kimi K3 day-0 support is a joint effort across the model vendor, inference engine, and hardware communities.

We thank the Moonshot AI team for creating Kimi K3, sharing architecture details ahead of the weight release, contributing the initial model integration and KDA prefix-caching work, and collaborating closely on correctness and production validation.

We thank the Inferact team for integrating the model into vLLM, extending the core cache manager for partial hybrid prefix hits, implementing serving semantics and multimodal support, building deployment recipes, and driving end-to-end performance optimization.

We thank the NVIDIA team for KDA decode and Attention Residual kernels, MXFP4 MoE collaboration, and performance work across the board.

We thank the AMD team for initial day-0 ROCm support and for continuing to expand Kimi K3 across AMD GPUs.

Most importantly, we thank the broader open-source community for the anticipation, testing, and feedback already surrounding Kimi K3. We look forward to putting the weights and the inference engine support in your hands.

One More Thing: Why the Announcement and Open-Source Release Are Separated

Kimi K3 also features a release process that we hope more model vendors will consider: announce the model first, then release the weights and inference engine support later.

The vLLM team proposed this separation, and Moonshot AI agreed and executed. The reason is practical. A frontier-model announcement has unavoidable last-mile uncertainty. The model team is simultaneously stabilizing its own products, APIs, evaluations, safety work, documentation, and commercial launch. If open-source weights and open-source support must land at the exact same moment, a community project such as vLLM suffers from the moving deadline.

Separating the two timelines gives both sides a better contract:

1. The model vendor can concentrate on its product launch and freeze the final checkpoint, configuration, tokenizer, and serving semantics.
2. The open-source inference engine team gets a stable integration window for correctness tests, performance tuning, Docker builds, and recipe validation.
3. The community gets a public, bounded expectation instead of an ambiguous “coming soon.”

The separation is not a retreat from day-0 support. It is a more sustainable way to deliver day-0 support against the artifact that users will actually download. We encourage more model vendors to follow!

In less than a year since its public launch, vLLM Semantic Router has reached 5,000 stars, 150+ contributors, and more than 300,000 cumulative downloads across our Hugging Face model family. Across three major releases—Iris, Athena, and Themis—the system boundary moved from choosing a model, to governing multi-model inference, to preserving state and coordination across sessions. Those releases built the foundation for the MoM architecture envisioned from day 0.

This post describes the next step for vLLM Semantic Router: moving from routing among models to building dependable model systems from them. Under one versioned contract, independent models, policies, preferences, and execution paths become a system that can be trained, evaluated, exported, imported, deployed, and invoked through one interface. Our goal is to make vLLM Semantic Router a training, evaluation, and inference engine for Mixture-of-Models.

Figure 1: A Mixture-of-Models turns a heterogeneous model portfolio into one model experience.

How vLLM-SR Got Here

The first vLLM Semantic Router post asked a practical question: why give simple and difficult requests the same reasoning budget? A lightweight classifier used fixed domain labels to choose between fast and reasoning paths, helping vLLM spend inference compute more selectively.

Production traffic quickly exposed the limit of that design. Domain alone could not represent privacy, safety, context, language, modality, tools, preferences, latency, and authorization. A static label also could not account for an endpoint that was cheap but overloaded, capable but remote, or unsafe to switch into midway through an agent session.

We rebuilt the classifier layer around modular model support, shared LoRA computation, Rust/Candle inference, and Go integration. We then replaced fixed classification with a Signal–Decision architecture that separated observed evidence from policy and execution. This became the spine of the next three releases.

Figure 2: Each stage changed the unit of control: model, decision, system, session, and finally the complete model lifecycle.

Iris made routing composable. Domain, keyword, embedding, factuality, feedback, and preference signals fed explicit decisions, while safety, PII protection, caching, hallucination detection, and tool selection became route-scoped behavior. Iris also introduced the MoM model family and described vLLM-SR as “System Level Intelligence for Mixture-of-Models.”

Athena added first-class model selection, memory and RAG, a multilingual and multimodal model stack, ROCm acceleration, and an operating dashboard. The project was becoming the control system around multi-model inference, not just a classifier in front of vLLM.

Themis turned that broader system into an operable contract:

Signals become projections. Projections feed decisions. Decisions choose algorithms. Algorithms select models.

Themis added session-aware agentic routing, replayable traces, stronger protocol support, an operator console, and runtime paths across AMD ROCm, NVIDIA CUDA, Intel OpenVINO, and CPU environments. It also made a route explainable: operators can see the evidence, policy, algorithm, and physical model behind each decision.

From Signal–Decision to Workload–Router–Pool

The releases built the runtime. Two project papers explained the architecture behind it.

The white paper, Signal Driven Decision Routing for Mixture-of-Modality Models, formalized the separation between neural evidence and symbolic policy. Fast heuristics and learned classifiers turn prompts, context, identity, safety, and modality into a structured signal vector; a Boolean engine then composes those signals into auditable policy. A typed neural-symbolic DSL parses and validates that policy before compiling it into deployable configuration. When the paper was published, the system covered thirteen signal types and thirteen model-selection algorithms, with per-decision plugins for caching, RAG, memory, safety, provider handling, and response validation.

The vision paper, The Workload–Router–Pool Architecture for LLM Inference Optimization, widened the frame. It argues that three variables have to be designed together:

• Workload: chat or agent, single-turn or multi-turn, warm or cold, prefill-heavy or decode-heavy
• Router: static semantic policy, online feedback or bandit adaptation, RL-based selection, and quality-aware cascades
• Pool: homogeneous or heterogeneous accelerators, prefill/decode topology, model placement, and KV-cache management

Those variables cannot be optimized independently. Workload shape changes which routing policy works; routing policy changes the required pool size and topology; pool state changes which route is efficient. Safety and privacy cut across all three dimensions, while cost, quality, latency, and energy define the optimization frontier. The paper maps the project’s research into a 3 × 3 WRP matrix and identifies twenty-one open directions where those dimensions still need to meet.

Figure 3: The white paper defines the programmable routing engine; the vision paper connects it to workload and physical pool design.

Together, the papers made routing programmable and tied it to workload and hardware—the two foundations MoM brings under one model contract.

Meanwhile, the runtime was already moving beyond single-model selection. Fusion, ReMoM, Confidence, Ratings, and bounded Workflows let one request invoke a controlled collaboration among models. As the Micro-Agent work showed, a client can call one model name while the serving layer selects a recipe, fans out to workers, verifies or synthesizes their results, and returns one ordinary response.

Routing remains fundamental. It is how a Mixture-of-Models allocates work, applies policy, and coordinates its parts. But routing is the mechanism. The model system is the product.

Why the Model Boundary Has to Move

Today’s AI stack is fragmented along four axes:

• Models are fragmented. Closed frontier models, open general models, domain experts, compact local models, verifiers, and multimodal models will coexist. None wins simultaneously on quality, cost, latency, trust, privacy, and domain fit.

• Compute is fragmented. GPUs, CPUs, specialized accelerators, edge devices, cloud capacity, and private clusters differ in memory, kernels, availability, price, and energy use. Model choice and placement are becoming the same decision.

• Location is fragmented. Inference spans cloud, data center, and edge. Privacy or residency may rule out a stronger remote model, while a local workload may still need an on-demand cloud expert.

• Preference is fragmented. There is no universal “best.” Products and users make different tradeoffs among accuracy, latency, price, privacy, safety, style, and multimodality. Those choices should shape execution directly.

Today, each application has to reconcile these fragments on its own.

Figure 4: Before MoM, fragmented intelligence becomes application-side routing glue.

Mixture-of-Models moves that responsibility behind one model boundary.

At that boundary, intelligent allocation becomes part of the model. The engine determines which models are eligible, where execution can run, whether models should collaborate, and how to satisfy hard constraints.

Energy makes allocation inseparable from efficiency. Hardware and inference engines improve the supply side by producing more tokens per watt per dollar. The allocation layer controls demand: which work deserves those tokens, and which model or collaboration can provide them within the required quality, latency, and energy budget.

The application selects one versioned model identity and receives one attributable response. Its physical realization can still span open and closed models, cloud and edge, and different accelerator generations. The fragmentation remains, but it becomes internal to the model system instead of leaking into every application.

Figure 5: With MoM, the same fragmented resources become the internal realization of one model.

What We Mean by Mixture-of-Models

A Mixture-of-Models is a versioned composite model whose engine realizes each request through a preference-conditioned, resource-bounded path across independent models and operators. It is presented to the user through one model interface and returns one attributable result.

A multi-upstream gateway can forward traffic without owning system quality. An MoM owns an objective, an evaluation contract, a reproducible composition, and the runtime that executes it.

MoM also differs from Mixture-of-Experts. MoE routes tokens among internal experts during one forward pass; MoM coordinates independent models that may differ in architecture, owner, license, modality, protocol, context window, and hardware. An MoE checkpoint can itself be one MoM component.

Figure 6: Selection is one MoM topology. Cascades, parallel fusion, and bounded workflows share the same model boundary.

A portable MoM therefore needs more than weights and configuration: it needs a component manifest, capability metadata, routing and collaboration recipes, policies, preferences, evaluation suites, runtime constraints, provenance, and version history.

Open checkpoints can travel with the artifact; closed models remain authenticated external references with explicit capability and policy contracts. Exporting an MoM does not make a proprietary checkpoint portable. It makes the model system reproducible.

Turn Preferences into Models

Preferences become concrete when they are published as model identities. One MoM family can offer several operating points:

Each name is a versioned model contract, not a router preset. The application chooses the behavior it needs; vLLM-SR selects and coordinates the models that deliver it while preserving hard privacy, residency, authorization, and safety constraints.

To an application, the full system remains an ordinary model call:

{
  "model": "vllm-sr/mom-v1-ultra",
  "messages": [
    {"role": "user", "content": "Review this design and identify its weakest assumption."}
  ]
}

That identity may select one model, escalate through a cascade, compare parallel answers, require grounding, or run a bounded workflow—without changing the external interface, version, or response contract.

Figure 7: Preferences are published as bounded, versioned model contracts—not hidden application-side routing presets.

Four planes separate ownership:

Figure 8: A complete MoM spans four planes: artifact, learning, execution, and physical realization.

A deployment must map logical requirements onto the models and machines available in its environment. The proposal uses four objects:

1. The bundle fixes the interface, graph, policies, behavior variant, bounds, and immutable semantic assets.
2. The binding maps logical components to eligible deployments without changing the model’s decision semantics.
3. The resolution lock freezes the constituent revisions, runtimes, images, accelerators, and provider observations.
4. The run record attributes every decision, call, constraint check, cost, and outcome to the bundle, binding, and lock that produced it.

Figure 9: One stable model identity, from portable contract to attributable run.

This separation keeps portability honest. The same mom-v1-ultra can bind to ROCm, CUDA, a private CPU or NPU node, or a hybrid deployment without promising identical outputs from opaque providers. Instead, it preserves control semantics, exposes substitutions, and gives serving and evaluation the same resolved system.

vLLM-SR as the MoM Engine

Training, evaluation, and inference must share one contract; otherwise research, benchmarks, and production drift into different systems.

Training allocation, not only weights

MoM training covers router-owned embeddings, signal encoders, preference and safety models, and selectors. It also learns allocation and collaboration: which path fits a workload and budget, when a cascade should stop, how a panel should judge or synthesize, and when an agent session should switch models. Because constituents may be independent or closed, progress does not require gradients through all of them; policies, thresholds, pools, prompts, contracts, and topology can be optimized from traces and outcomes.

The target is a frontier across quality, latency, cost, safety, privacy, reliability, locality, and energy. Replay and outcomes feed production experience back into offline training without letting the hot path silently rewrite policy.

Evaluating the MoM as one model

Evaluation must score the model identity end to end; backend benchmarks are inputs, not the result. A versioned scorecard should measure routing regret, collaboration gain, recovery, session continuity, tail latency, cost, safety, privacy, and energy. It should stress provider failures, device loss, model disagreement, workload drift, and preference changes. Each declared operating point also needs its own test: flash on its latency–quality frontier, light against its quality floor, and ultra within its budget.

The scientific test is stricter than asking whether more calls improve a benchmark. Under matched active compute, can a conditional system exploit complementary strengths and failure modes better than the best fixed model? Without that control, MoM can hide brute-force scaling behind a clever graph. Evaluations must report calls, tokens, cost, latency, and energy alongside quality—and publish when composition does not help.

Figure 10: Composition gain is meaningful only under matched active compute, with quality reported alongside calls, tokens, cost, latency, and energy.

Executing intelligence at inference time

At inference time, the engine decides whether one model is enough. It may choose a local specialist, preserve a warm session, escalate through a confidence cascade, require retrieval or verification, run a Fusion panel, or execute a bounded workflow. The runtime owns the budget, topology, fallback, trace, and response contract; the application makes a normal model call.

Figure 11: MoM is a closed lifecycle: train the allocation policy, evaluate the full system, execute it, and turn outcomes into the next validated version.

One Model That Can Move

Our target is a complete MoM that can be built, exported, imported, versioned, evaluated, deployed, and invoked as a unified model. A logical specification compiles into an immutable bundle, binds to an environment, resolves the concrete deployment, and retains the same identity for serving and evaluation.

The artifact should run across developer machines, private clusters, cloud fleets, and edge environments while its physical realization changes. A specialist may resolve to an admissible local checkpoint or managed endpoint; an accelerator runtime may be replaced. If privacy makes a remote expert unavailable, the engine follows a declared fallback or abstention path. A binding cannot silently rewrite the graph, relax a guard, or turn a panel into a cascade—those changes require a new model version.

“Run on any hardware” is an architectural requirement, not a claim that every component is portable today. The project already supports paths across ROCm, CUDA, OpenVINO, and CPU. Next, hardware capability and placement become part of the MoM contract, allowing the engine to map the model system onto what is available.

The standard for the user experience is simple:

One model identity. Many models. Any hardware.

Figure 12: One logical model identity can be realized across developer, data-center, cloud, and edge hardware.

If the application needs to know which provider owns every submodel, which device runs it, or which fallback graph to execute, the abstraction has leaked.

What Changes Now

The next stage focuses on four connected areas:

1. Define a portable MoM specification. Package components, objectives, policy, preferences, evaluation, constraints, and execution semantics as one versioned artifact.
2. Close the training–evaluation–inference loop. Improve models and recipes from evaluation and replay, then ship them through reviewable, rollback-safe releases.
3. Build a heterogeneous runtime. Map one MoM across cloud, data center, and edge using hardware, locality, energy, and data boundaries as inputs.
4. Keep the model interface boring. Make an MoM as easy to import, deploy, and invoke as a single model.

Figure 13: Four connected workstreams turn Mixture-of-Models from an execution pattern into the next model architecture.

This is a research program for how independent models should specialize, compete, verify, and collaborate; how to measure the resulting system; and how one model contract can survive across devices and environments. Our mission is:

Advancing the science of intelligence across models, devices, and environments.

We will study when composition produces capabilities beyond a single checkpoint, treat placement and energy as part of intelligence, and carry the same model contract from edge to cloud and from research to production.

Build It With Us

Building Mixture-of-Models requires more than routing. The work spans model training, evaluation, serving systems, hardware, and production operations.

Iris, Athena, and Themis improved because contributors brought real workloads, added backends, trained models, published benchmarks, found failure cases, and argued for better interfaces. MoM needs the same range of work: learned allocation, preference optimization, model cooperation, energy-aware inference, portable artifacts, open evaluation, and heterogeneous runtimes.

If you work on these problems, we want to learn from your workloads and measurements. Build an operating point, add a runtime, test a collaboration recipe, or publish a case where composition fails. MoM will be stronger if its assumptions are tested in the open.

Acknowledgments

vLLM-SR has grown through work across engineering, research, and the wider ecosystem. We thank Xunzhuo Liu, Huamin Chen, Bowei He, Yankai Chen, Fuyuan Lyu, and Steve Liu for helping shape its technical and research direction. We also thank Andy Luo and Haichen Zhang for their work on ROCm enablement, router-model training, and open MoM experimentation.

The work has also been carried by FAUST, David Shrader, Yang Wu, Ramakrishnan Sathyavageeswaran, Kuntai Wu, Aayush Saini, siloteemu, Chen Wang, Yue Zhu, Senan Zedan, Yossi Ovadia, Samzong Lu, Liav Weiss, Asaad Balum, Yehudit, Noa Limoy, Marina Koushnir, Jared Wen, Abdallah Samara, Hen Schwartz, Srinivas A, Yang Zhu, Jintao Zhang, yuluo-yx, cryo, Bishen Yu, Zhijie Wang, Hao Wu, and Qiping Pan. Their code, reviews, testing, documentation, and stewardship carried the project from one release to the next.

At this milestone, the project stands at 1,734 commits and 150+ contributors. We thank collaborators at MBZUAI, McGill University, Mila, and Rice University, and the broader vLLM, AMD, Intel, Meta, Red Hat, Microsoft, Google, IBM, NVIDIA, Hugging Face, NASA, Nutanix, DaoCloud, and open-source communities. This milestone belongs to everyone who helped turn an early router into a real system.

Figure 14: Building the MoM engine is an open systems problem that needs the full model and infrastructure community.

Join us on GitHub, explore the documentation, try the MoM model family, and meet the community in the #semantic-router channel on vLLM Slack.

vLLM Semantic Router began by helping infrastructure choose the right model for each request.

Now we are extending that foundation beyond a single model: toward systems that can coordinate, evaluate, and operate multiple models across devices and environments.

We invite the community to help build and test that approach in the open.

Intro

vLLM is the most widely used open-source LLM inference engine. 86K+ GitHub stars. 5.6M+ monthly pip installs. 2.5M+ monthly image pulls, with support for 1000+ model architectures and 600+ accelerator types.

Supporting this many models and accelerators has always been one of vLLM’s biggest strengths. It’s also what makes it so hard to keep stable.

In June 2026, vLLM merged 1,918 commits into main — 64 a day on average, on par with other big OSS projects like PyTorch or Kubernetes. During this time, our CI ran 13 million job minutes with 1400 concurrent runners at peak.

Testing vLLM, especially at this pace, gets harder every day. A change that’s clean on an H100 might fail to compile on AMD, lose throughput on B200, or nudge a model’s outputs just enough on one backend to matter. The surface area that makes vLLM worth using is the exact surface area we have to defend on every commit.

In this post, I want to share how we keep vLLM releases stable at this pace: what works, what we’ve learned, and where we still fall short. It will mostly cover high-level processes rather than technical details; I’ll save those for another post.

A journey from pull requests to a new version release on vLLM has to go through three layers:

• CI — how we catch what breaks loudly, on every PR

• Performance benchmarking & accuracy evaluation — how we catch what breaks silently, beyond what CI can cover

• Release process — how we evaluate the signals, make the call, then build and ship artifacts to users safely.

Layer 1: CI

Extensive unit testing on every component of the codebase

Every PR starts with lightweight GitHub Actions checks—linting, formatting, and similar guardrails. Once a committer thinks the PR is ready to merge, the heavier unit testing then starts running on Buildkite, our CI platform.

Buildkite assembles each PR’s testing pipeline dynamically: a bootstrap step reads the job definitions, inspects the diff, and schedules only the relevant groups. Change only documentation and you may get a handful of jobs. Touch a few important kernels? Buckle up for 100+ jobs launching in parallel.

In total, the vLLM CI suite runs 37 test groups and 266 jobs, covering every major component and feature—from different kernels to speculative decoding to LoRA. Groups range from a couple of jobs to a few dozen, and many tests exercise several components at once. Here is a subset:

Ensuring test environment is consistent

A test only means something if it runs the same way every time. We usually see two kinds of drift get in the way: the environment can differ across our CI runners, and dependencies can change under us over time. A shared container image removes the first; a pinned dependency graph removes the second.

Same container image, every machine. With 266 jobs fanning out across dozens of machine types, the fastest way to a flaky, untrustworthy result is to let each job set up its own slightly different environment. To avoid this, the majority of our jobs run inside the same container image, built once at the start of a run and reused everywhere. Our Dockerfile builds in stages, each adding to the one below it.

A base stage provides the CUDA toolchain; a build stage compiles the wheels on top of it; and a runtime stage installs those wheels with their runtime dependencies.

From there, the build forks: one image adds the serving entrypoint and becomes the release image, while a separate test image adds the test dependencies and becomes the image CI jobs pull. That shared ancestry keeps what we test close to what we ship.

For jobs using that shared image, a kernel test on a B200 and an entrypoints test on an L4 pull the same container image, byte for byte, while running on different hardware. Building it once removes a major source of variation: failures are much less likely to come from per-job setup drift.

Same versions, every run. Dependencies drift over time—and that’s the half we learned the hard way.

An unpinned dependency makes failures tricky to chase down: the same test passes on Monday and crashes on Wednesday. You read every code change in between; none of it looks related. Hours later, it clicks—FlashInfer shipped a new version on Wednesday, and the build quietly picked it up. And FlashInfer was never alone: nixl, transformers, and their transitive dependencies bit us the same way—each unannounced upgrade a fresh chance to break CI, with the cause buried a dependency layer down.

So vLLM CI locks its dependencies. We run the top-level dependencies through pip-compile to generate lock files that pin every package, including transitive dependencies.

We update the locks periodically and run the full CI suite each time. Since we started pinning the full graph, dependency-caused breakages are no longer a recurring headache.

Scaling CI compute across a heterogeneous, multi-provider fleet

Each job gets pushed to a runner queue on Buildkite — a pool of machines with a particular hardware profile. For example, the gpu_1 queue is backed by individual VMs with L4 GPU; the b200 queue is backed by a Kubernetes cluster with B200s inside. When a runner becomes available, it claims the next job in the queue, runs it, and reports the result back to Buildkite.

vLLM CI, at the time of writing, has 58 runner queues spanning a wide range of accelerators, and that hardware is provided by multiple partner organizations.

There’s a limit on how much we can spend on compute. Even if we can afford it, managing all of these machines ourselves is a pretty tough job. CI coverage with this much diversity is only possible because of the generous support and collaboration from many of our amazing partners.

However, integration is a big challenge. Every partner has different requirements: some simply hand us access to everything, some prefer to manage their own hardware, some have very tight security guardrails.

So how do we manage to plug them all into one CI pipeline?

This is where Buildkite agent comes in. It runs inside the provider’s environment and connects outbound to Buildkite over HTTPS to receive work. Because Buildkite does not need to initiate connections to the agent, providers do not have to expose inbound ports, configure a VPN, or give us access to their network.

When the agent accepts a job, it runs the command, streams the logs back, and reports the final exit status. A persistent agent then waits for more work, while an ephemeral agent exits after completing its job.

There’s more than one way to run that agent, and providers pick whatever fits their setup.

The simplest is a standalone machine — our 8xA100 machine or Arm server. The provider installs the agent, points it at a runner queue, and it runs that loop forever.

For machines in a Kubernetes cluster, the same model works through the Buildkite Agent Stack for Kubernetes. The controller turns each matching job into a Kubernetes Job with a single Pod, which runs the test and reports back. We always recommend this way because it’s very scalable: you don’t need to install Buildkite agents on every single node, just add them into the cluster.

Either way, onboarding is simple on our end: we create a queue, provide a token, and the provider starts the agent themselves. We don’t need access to the machine. That’s what lets vLLM test on more hardware than we could ever afford to own—a donated fleet worth millions of dollars a year.

Utilizing hardware is challenging

There’s a lot of demand and a hard limit on compute, so we need to make sure none of it goes to waste.

MIG-slice the big GPUs

Most CI jobs run with smaller models and need far less than a whole GPU. NVIDIA’s Multi-Instance GPU (MIG) lets us carve one card into several isolated slices — an H200 becomes seven 18 GB partitions — meaning 7 jobs can share one GPU at a time. We did some math here and realized that in many cases, slicing a big GPU is a lot cheaper than renting smaller GPUs for the same amount of workload!

Autoscale from zero, one job per machine

For the machines we rent by the hour, leaving them on and idle just wastes money. So each of those queues scales itself: when jobs are waiting, it starts more machines; when there’s nothing to run, it goes down to zero. Each machine picks up one job, runs it in a container, and shuts down. As a bonus, it also keeps tests clean: every job gets a fresh machine, so nothing left over from a past run can mess with it.

Don’t rebuild what you can reuse

The slowest, most repetitive, and most expensive parts of CI are:

1. Building the standard Docker image used by whole CI pipeline

   1. Compiling CUDA kernels

   2. Installing all dependencies

2. Downloading model weights from Hugging Face.

so we try not to:

• Docker layers: we apply registry caching and reuse cached layers instead of rebuilding. This would include the dependencies.

• Warm-cache AMI for builder: we have a nightly job to build the AMI used for our builder machines with the latest layers already pulled, so our builder machine starts as close to main as possible.

• Compiler cache: we leverage sccache so that compiled C++/CUDA outputs are cached in an S3 bucket and reused across builds. Every builder machine can read from this bucket, but only builder machines used for the main branch can write to it.

• Model weights: the models we test are huge, so for each of the clusters, we download them once to shared storage and every job reads from there, instead of pulling gigabytes each time.

Making CI health visible

With hundreds of CI runs a day, each running hundreds of jobs across different hardware, we also need to know whether the system itself is healthy.

A queue quietly backs up to hours of wait time. A test starts to flake one run in twenty. The job runs 10 minutes slower than last month. It’s not easy to track that.

We took inspiration from the incredible PyTorch CI HUD (hud.pytorch.org), built by our good friends at PyTorch, and created one of our own at ci.vllm.ai.

Every 15 minutes, data from our Buildkite pipelines is ingested into Databricks and ClickHouse.

With all the available data and full control of the dashboard, we have so much flexibility on building out our observability stack. It gives us an easier time answering these typical questions:

Is main branch healthy right now?

For the past 3 days, no. And why did jobs take 10 hours!?

Which test is broken or flaky, and since when?

This AMD hardware test group has been failing since PR #47329 was merged.

Basic correctness test failed once so it’s probably flaky.

Is any runner queue congested?

small_cpu_queue_premerge runner queue looks pretty congested… Its capacity probably maxed out at 5 instances, so let’s raise it.

Which job takes the longest in CI? What’s its duration trend over the past two weeks?

Those are just a few examples of what our dashboard can do. Modern coding agents have made this kind of tooling surprisingly approachable, even without deep front-end expertise.

Automating failure detection and response

The dashboard helps us see problems. The next step is shortening the time from detection to diagnosis, and of course we have to leverage the powerful AI agents here.

Every night, a CI-analyzer bot runs the full suite and compares the results with the previous night’s run. If something newly failed, it reads the error logs, classifies the failure, and walks the intervening commits to find the culprit. It then posts a report to Slack with an auto-revert PR ready for maintainers to review and merge. That’s about 1.5 auto-revert PRs a day, with the right failure and culprit commit identified around 70% of the time—so the on-call reviewer usually starts from a correct diagnosis instead of a blank page.

The bot has become essential to catching breakages fast, alongside the community effort to fix issues as they land—shout-out to everyone that helps, especially the on-call rotation at Red Hat!

What a green check cannot tell us

Put together, all of this is what lets us trust a green check on a PR: broad unit test coverage, run across a huge fleet of different accelerators, in consistent environments, and well-monitored. When CI passes, we’re confident merge risk is significantly reduced.

But CI doesn’t tell the whole story. A change can pass every test and still make a model slower or its output incorrect. To keep CI fast and affordable, we tend to skip a lot of e2e tests and, more importantly, not closely simulate what vLLM users go through every day. That’s what the next layer is for.

Layer 2: Performance benchmarking & accuracy evaluation

In May, we shipped v0.20.0 and within days had to cut two emergency patches, v0.20.1 and v0.20.2. Two problems had slipped through: one broke gpt-oss on Blackwell when split across multiple GPUs (tensor parallelism > 1), the other tanked DeepSeek V4 throughput on GB200.

At the time we had no benchmarking pipeline; nothing ran these models end to end on that hardware to confirm they still worked and ran fast before we shipped. So both problems sailed past CI and reached users.

Performance regressions rarely crash. The server starts and requests succeed; users simply get fewer tokens per second or wait longer for the first token. Accuracy regressions are quiet: the model returns a valid response, but the answer is wrong.

We realized how important it is to run models end to end, with performance benchmarks and accuracy checks, so we invested a lot of time building this layer.

It now provides a lot of signals for our release process and has already caught several major regressions. We built the system that would’ve caught the problems on v0.20.0 before it was shipped.

Running a matrix of models and accelerators every night

We maintain our pipeline at https://github.com/vllm-project/perf-eval. Each config file describes a workload: how to start vLLM server, which arguments to use, which model to serve, which accelerator, and which tasks to run.

Each workload generally runs three tasks:

• Performance benchmark — measuring time-to-first-token (TTFT), time-per-output-token (TPOT), and many other metrics — using vllm-bench

• Model accuracy on math and reasoning benchmarks (GSM8K, GPQA, AIME) using lm-eval

• Function-calling accuracy via the Berkeley Function-Calling Leaderboard (BFCL)

Every night, and for every release candidate, we run the full suite across selected models — DeepSeek V4 Pro/Flash, gpt-oss, Kimi K2.5, MiniMax M2.5 and M3, Qwen3.5, GLM 5.1, Gemma 4, and Nemotron 3 Super — on H200, B200, MI300X, and MI355X. That’s 17 model-hardware recipes in total right now, and the list keeps growing. We plan to add support for GB200/GB300, PD disaggregation, and more models very soon.

Is it always fast?

After every run, the results are ingested into our database. Remember the CI dashboard from earlier? It has perf results too!

We turn the nightly numbers into charts that make regressions easy to spot over time.

For example, this is a view of our Performance dashboard:

Performance history for gpt-oss 120B on H200 with tensor parallelism 8, split by concurrency.

The Compare view lets us compare two vLLM images head-to-head — say, a release candidate against the last release.

Is it always correct?

If your vLLM instance is blazing fast but its output is garbage, that speed is worthless. Beyond performance, we make sure the model’s answers still hold up.

The Evaluation dashboard stores aggregate scores and error bars, then lets us open a run and inspect the underlying question, reference answer, raw response, extracted answer, and correctness result. That sample-level evidence is far more useful than debugging from a single aggregate number.

An incorrect GSM8K sample exposes the exact question, expected answer, model response, and extraction result.

Layer 3: Release process

Shipping on a fast cadence

Since November 2025, we have been maintaining a two-week cadence on vLLM releases. Many projects of our size take a lot longer to ship. Why we keep this cadence:

• Changes reach users fast. A new release is never far behind main.

• It’s predictable. Users and downstream projects can plan around a steady schedule instead of guessing when the next release lands.

• Managing features and tracing regressions are easier when there are 500 commits to bisect rather than a few thousand.

• Less deadline pressure. Contributors no longer feel the need to rush their changes in before the train departs. They just catch the next one in two weeks.

• Cherry-picks stay clean. A fix from just days ago is usually a simple pick, not a merge-conflict mess.

Every other Monday, we kick off release week. Here’s what that looks like:

Start from the safest commit

On Monday, the release manager reviews the most recent full-CI runs on the main branch and chooses the greenest commit. That gives the release branch the healthiest available starting point before any release-specific changes are added.

We cut releases/vX.Y.Z at that exact commit and announce the branch and release window.

Heavy testing on every release candidate

From the branch cut through Wednesday, we review the cherry-pick requests, cherry-pick them into the release branch in batches, and tag the result as the next release candidate.

Every candidate goes through the same three gates:

• Full CI suite

• Performance benchmark suite

• Model accuracy evaluation suite

Each result is tied to a release candidate. When a later candidate changes CI health, performance, or evaluation quality, we can track down which candidate introduced the difference: they are just tens of commits away.

We end the cherry-pick window on Wednesday. Then, only fixes for existing issues on release candidates can be cherry-picked, followed by a new release-candidate tag and another run through the three gates, until one candidate meets the bar.

No compromise for the bar

A candidate qualifies only when all three gates pass.

Sometimes there’s no qualifying candidate at the end of the week, and that’s okay. We try our best to release on time, but never compromise our bar just to make it. We treat our new version the way Rockstar treats GTA 6: it’s done when it’s done. We don’t take a decade though…

Ship for every platform

Once a candidate qualifies, we take that commit and start building all the artifacts for different hardware platforms and CUDA versions, ensuring that everyone out there can use vLLM natively. And before anything ships, we smoke test the built artifacts themselves.

At the time of writing, we are shipping these for every release:

• 7 Python wheels:

  • CUDA 12.9 x86_64/arm64

  • CUDA 13.0 x86_64/arm64

  • CPU x86_64/arm64

  • ROCm

• 11 Docker images:

  • CUDA 12.9, x86_64/arm64, Ubuntu 22.04/24.04

  • CUDA 13.0, x86_64/arm64, Ubuntu 22.04/24.04

  • ROCm

  • CPU x86_64/arm64

What’s next

I’ve been bragging a lot about what we’ve built — but honestly, we still have a lot to do on our roadmap. Some of the big ones:

• Automatic test selection. Today we pick which tests run for each PR from a hand-maintained mapping, and it goes stale fast. We want this to be automatic, and we’re trying a few angles: LLM-based selection, static analysis, dynamic analysis, and labeling source paths to match them to tests.

• Faster time-to-signal. CI takes 1–2 hours on average to return a verdict; we’d love to get that under 30 minutes.

• Leaner unit tests. A lot of our “unit” tests actually spin up a full vLLM server and fire real requests at it, which slows down CI a lot.

• Better exit-code handling. Some jobs still return the wrong exit code when they fail, like reporting an infra problem as a failed test, making it hard to triage failures and alert/retry jobs.

• Faster flaky-test detection and quarantine. We have plenty of flaky tests — from infra, upstream packages, or tests that just aren’t written safely — and we’d like to catch and quarantine them automatically.

• Automatic detection for infra issues. Spot a bad machine quickly and pull it out of the CI fleet on its own, before it fails a pile of jobs.

• Better alerting. We have some basic alerts for congested runner queues and regressions. It’s always nice to have more: high disk pressure on CI runners, jobs suddenly failing far faster than usual, broken dependency installs, etc.

• Code-coverage reporting. Our coverage is broad, but we can’t yet say for sure that every corner of the codebase is actually exercised.

Working on CI is actually a lot more interesting than most people think. This post only covers the high-level process of how we keep vLLM releases stable; there are plenty of fun technical details I didn’t get to cover — maybe in another post :)

If any of these problems sound like your kind of fun, or you think we’re doing something wrong, come say hi in #sig-ci on the vLLM Slack. And if you’d like to work on this full-time, we’re hiring at Inferact~!

Acknowledgements

None of this is a solo effort. vLLM CI is built and kept alive by the whole community.

I’m deeply grateful to everyone who helped with CI along the way (listed alphabetically):

• Amazon: Junpu Fan, Liangfu Chen, Omri Shiv,

• AMD: Alexei Ivanov, Andreas Karatzas, Kenny Roche, Micah Williamson

• Arm: Fadi Arafeh, Ioana Ghiban

• EmbeddedLLM: Tun Jian Tan

• Google: Brittany Rockwell, Jincheng Chen, Ming Huang, Qiliang Cui, Yarong Mu, Yiwei Wang

• HuggingFace: Harry Mellor

• Inferact: Harry Chen, Jiangyun Zhu, Kaichao You, Nick Hill, Roger Wang, Simon Mo, Zhewen Li

• Intel: Chendi Xue, Jiang Li, Kunshang Ji, Wenjun Liu

• Meta: Andrey Talman, Charlotte Qi, Eli Uriegas, Huamin Li, Huy Do, Orion Reblitz-Richardson, Reza Barazesh

• NVIDIA: Alec Flowers, Benjamin Chislett, Mathew Wicks, Pen Chung Li, Stefano Castagnetta, Vadim Gimpelson, Xin Li

• Red Hat: Andy Linfoot, Avinash Singh, Doug Smith, Edward Quarm, Flora Feng, Lucas Wilkinson, Luka Govedic, Matt Bonanni, Michael Goin, Nicolo Lucchesi, Robert Shaw, Russell Bryant, Tarun Kumar, Tyler Michael Smith, Wentao Ye

• Reflection AI: Amr Mahdi (contribution made during his time at Meta)

• Independent contributors: Cyrus Leung (DarkLight1337), Yuqi Wang (noooop), haosdent, Mohammad Angkad

the amazing partners:

• AWS, Crusoe, LambdaLabs, Nebius, NVIDIA, Roblox, RunPod for sponsoring us with compute credits

• Buildkite for letting us run CI free of charge on their platform <3

and finally, two mentors who taught me a lot about CI during my time at Anyscale (Ray): Lonnie Liu (now at OpenAI) and Cuong Nguyen (now at NVIDIA).