One Long Prompt Stalls Every Reply: Disaggregating LLM Prefill and Decode

Here's a failure mode that doesn't show up where you'd look for it. Time-to-first-token (TTFT) sits comfortably inside its SLO. Throughput is fine. Error rate is flat. What breaks is the other latency metric: inter-token latency, the gap between generated tokens, also called TPOT (time per output token). Its p99 is fine most of the time, then jumps three or four times over for a few seconds and settles back down.

Plot TPOT against the request log and the pattern gives itself away: every spike lines up with a long prompt entering the running batch. One user pastes a 12k-token document to summarize, and for the duration of that request's prefill, every other user watching tokens stream back sees them stutter. The service isn't overloaded. Two fundamentally different workloads are being forced through the same GPU at the same instant.

Two phases, two bottlenecks, one accelerator

An LLM request runs in two phases with completely different hardware profiles. That split usually shows up as a fleet-sizing problem across different request types; here it bites inside a single batch.

• Prefill reads the whole prompt in one forward pass and builds the KV cache for every input token. It's a big, parallel, matmul-heavy operation that saturates the GPU's matrix units. It is compute-bound.
• Decode generates output tokens one at a time. Each step streams the entire model's weights (plus the growing KV cache) out of HBM to produce a single token. Arithmetic intensity is tiny, so the chip sits mostly idle waiting on memory. It is memory-bandwidth-bound.

Modern servers like vLLM use continuous batching (iteration-level scheduling): instead of waiting for a batch to finish, the scheduler rebuilds the batch every step and slots new work in as it arrives. That's exactly what keeps GPUs busy — and exactly what causes this problem. When a long prompt's prefill is scheduled, it dominates that step's compute, and the decode tokens of every other request in the batch wait behind it. One expensive prefill becomes head-of-line blocking for dozens of cheap decodes.

The first fix: chunked prefill

The standard first response — and the right one to try before anything more invasive — is chunked prefill. Instead of running a 12k-token prefill as one monolithic step, the scheduler slices it into fixed-size chunks and interleaves those chunks with decode steps from other requests. No single step is dominated by one giant prefill, so the decode stutter shrinks.

In vLLM this is on by default on recent versions, but the knob that matters is the token budget per step:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048 \
  --max-num-seqs 256

--max-num-batched-tokens caps how many tokens (prefill chunks + decode) one step may process. Lower it and prefill is sliced finer, so inter-token latency gets smoother — at the cost of prefill throughput, because you're paying scheduling overhead on more, smaller steps and leaving some compute on the table. It's a dial between TTFT and TPOT, and you tune it against your own traffic mix.

Chunked prefill genuinely helps. What it can't do is change the fact that prefill and decode still run on the same GPU, with the same tensor-parallel degree, out of the same memory budget. You're time-slicing a shared resource, not removing the contention. A compute-bound phase and a memory-bound phase are still bidding for the same silicon, and you still size one pool of hardware around a compromise between the two.

The structural fix: disaggregate prefill from decode

If the two phases want different things from the hardware, stop running them on the same hardware. Prefill/decode disaggregation does exactly that: a pool of prefill workers computes the KV cache for incoming prompts, hands that cache off to a separate pool of decode workers, and the decode pool streams tokens back to the client. The KV cache travels between the pools over a fast interconnect (NVLink/NVSwitch within a node, RDMA over InfiniBand or high-bandwidth Ethernet across nodes).

The payoff is independent tuning. The prefill pool can run big batches and aggressive tensor parallelism to feed its compute appetite. The decode pool optimizes for the opposite: pack many concurrent sequences to amortize the weight-streaming cost, hold large KV caches, and chase HBM bandwidth. Each pool batches on its own, hits its own SLO (TTFT for prefill, TPOT for decode), and — critically — scales on its own signal. Input-heavy traffic (RAG, summarization, long-context agents) adds prefill replicas; output-heavy traffic (chat, code generation) adds decode replicas. No more provisioning one blob of GPUs around an uneasy average of both.

This is the architecture under the recent production results people are citing — the llm-d project (a Kubernetes-native stack built on vLLM and the Gateway API Inference Extension) together with NVIDIA Dynamo, which handles disaggregated serving plus the KV-cache transfer layer. Google Cloud reported roughly 5x performance for Reflection's Beam model on that combination. The research lineage is the DistServe work (OSDI 2024), which framed the goal precisely: optimize for goodput — requests per second that meet both the TTFT and TPOT SLOs — rather than raw throughput that quietly violates one of them.

On vLLM you can wire a basic version of this with a KV connector that ships cache from a prefill instance to a decode instance:

# prefill instance: produce KV cache
vllm serve $MODEL --kv-transfer-config \
  '{"kv_connector":"PyNcclConnector","kv_role":"kv_producer","kv_rank":0}'

# decode instance: consume KV cache
vllm serve $MODEL --kv-transfer-config \
  '{"kv_connector":"PyNcclConnector","kv_role":"kv_consumer","kv_rank":1}'

What it costs, and when not to reach for it

Disaggregation is not free, and it is not the first thing to try. You've added a network hop on the critical path: the KV cache for a request has to move from prefill to decode before the first token can come out, so TTFT now includes transfer time. For a long prompt the KV cache is large, which is why this architecture effectively requires a real interconnect — RDMA-class networking or NVLink. On commodity networking the transfer can eat the very latency you were trying to win back.

You've also doubled the operational surface: two deployments to scale, two autoscalers to tune, a KV-transfer path to monitor, and failure modes where a decode worker is healthy but starved because prefill capacity lagged a traffic shift. Below a certain scale, none of that pays for itself. The honest decision order is:

• First, confirm the problem is actually phase interference — TPOT p99 spikes correlated with long prompts entering the batch, not just general overload.
• Turn on chunked prefill and tune --max-num-batched-tokens. For many services this is enough, and it costs you nothing but a config line.
• Reach for disaggregation when you're running a sizable multi-GPU fleet, your prefill and decode load genuinely move independently, and you have the interconnect to move KV cache cheaply.

Lesson

The TPOT spike isn't a capacity shortfall — it's two workloads with opposite hardware appetites sharing one accelerator and colliding whenever their timing lines up. Chunked prefill smooths the collision; disaggregation removes it by giving each phase its own pool to scale and tune against its own SLO. The broader point is one worth carrying into any serving design: "LLM inference" is not a single workload, and the moment you treat prefill and decode as one thing to provision for, you've already signed up to pay for the wrong bottleneck — or to watch one phase's spikes become the other's tail latency.

---

Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.