Round-Robin Is Wrecking Your LLM Throughput: Route by KV Cache, Not Connection Count

You went from 2 vLLM replicas to 8 because your queue was backing up. Eight GPUs now, four times the hardware. Aggregate throughput went up maybe 30%, and p99 time-to-first-token (TTFT) got worse. The per-GPU utilization graphs look busy, the autoscaler is happy, and the bill quadrupled for a fraction of the goodput you paid for.

Before you reach for bigger GPUs, look at the thing in front of them. A plain round-robin or least-request load balancer is the wrong tool for LLM inference, and it fails in a way that's invisible on a CPU/GPU utilization dashboard.

Why a normal load balancer is wrong here

Web load balancing works because requests are roughly interchangeable: each one is short-lived, stateless, and costs about the same. Spraying them round-robin across identical replicas is close to optimal. LLM serving violates every one of those assumptions.

• Requests are not uniform. A request that generates 2,000 tokens occupies KV cache in GPU HBM for its entire decode loop — far longer, and far more memory, than one that generates 50. Two "connections" can differ in cost by two orders of magnitude.
• There is server-side state that routing can exploit. With automatic prefix caching, a replica that already served a request with your 4k-token system prompt has those KV blocks resident. Send the next request sharing that prefix to the same replica and it skips prefill for the shared tokens. Send it elsewhere and that replica recomputes the whole prefix from scratch.
• The real capacity limit is KV cache, not connections. vLLM does continuous batching: it packs as many sequences as fit in the KV cache into each forward pass. When the cache fills, new requests queue, and in-flight ones can be preempted and recomputed later. Connection count doesn't see any of this.

Round-robin actively fights all three. It scatters same-prefix requests across every replica, so your prefix cache hit rate collapses and every GPU redundantly recomputes the same prefill. And because it's blind to KV cache occupancy, it cheerfully hands a new request to a replica that's already at 98% cache and preempting — while another replica sits half-empty.

The tell is in the model server's own metrics

vLLM exports the numbers that matter on /metrics. You don't need a distributed trace to diagnose this — scrape one replica under load:

$ curl -s localhost:8000/metrics | grep -E 'vllm:(num_requests|gpu_cache|prefix)'
vllm:num_requests_running 41.0
vllm:num_requests_waiting 132.0
vllm:gpu_cache_usage_perc 0.98
vllm:prefix_cache_hit_rate 0.07

Read that as a sentence: 41 sequences decoding, 132 waiting, KV cache 98% full, and a prefix cache hit rate of 7%. That last number is the smoking gun. If your traffic shares system prompts, few-shot examples, or a retrieved document — and most production traffic does — a 7% hit rate means routing is throwing away almost all the prefill reuse that's sitting right there. Now check a sibling replica and you'll often find gpu_cache_usage_perc at 0.4 with an empty queue. That imbalance, not raw GPU speed, is what capped your throughput.

Route on what the servers are actually telling you

The fix is to make the router inference-aware: have it read each replica's live KV cache utilization, queue depth, and prefix affinity, and pick the endpoint on those, per request. This is exactly what the Gateway API Inference Extension standardizes, and what GKE Inference Gateway implements on top of Envoy. Two pieces do the work: an InferencePool that groups your model-server pods, and an endpoint picker (EPP) the gateway consults before it forwards each request.

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: llama-pool
spec:
  targetPortNumber: 8000
  selector:
    app: vllm-llama
  extensionRef:
    name: llama-epp        # the endpoint picker for this pool
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llama-route
spec:
  parentRefs:
    - name: inference-gateway
  rules:
    - backendRefs:
        - group: inference.networking.x-k8s.io
          kind: InferencePool     # not a plain Service
          name: llama-pool

The point is the backendRef: instead of a Service doing kube-proxy round-robin, the gateway asks the EPP which pod should get this specific request. The EPP scrapes the vLLM metrics above and scores every candidate. Conceptually the scoring blends three signals — exact schemas still move between releases, so treat this as the shape, not a frozen API:

# endpoint-picker scoring profile (representative)
plugins:
  - type: prefix-cache-scorer         # prefer a replica that already
                                      # holds this prompt's prefix blocks
  - type: kv-cache-utilization-scorer # penalize replicas near cache-full
  - type: queue-depth-scorer          # penalize replicas with waiting reqs
schedulingProfile:
  - pluginRef: prefix-cache-scorer
    weight: 3
  - pluginRef: kv-cache-utilization-scorer
    weight: 2
  - pluginRef: queue-depth-scorer
    weight: 1

The prefix scorer is what recovers your throughput. Weighted highest, it pulls same-prefix requests toward the replica that can serve their prefill from cache — but the KV and queue scorers stop that from overloading a single hot replica, spilling to the next-best once it approaches cache-full. That's the balance the round-robin panel above never finds.

None of this works if the server isn't sharing state it can reuse, so make sure prefix caching is actually on and the server is configured to pack batches rather than run one sequence at a time:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching \
  --max-num-seqs 256 \
  --disable-log-requests

When your GPUs are in three regions, not three racks

Accelerator scarcity means capacity rarely lives in one cluster — you get a slice here, a slice an ocean away. Naively load-balancing across regions trades your KV cache problem for a latency problem: cross-region prefill adds tens of milliseconds before the model does any work, and you shred prefix locality by bouncing a conversation between continents.

The multi-cluster form of this keeps the same scoring but adds a locality term: prefer in-region replicas, and only spill to a remote cluster once the local pool is genuinely cache-saturated. Google reports the routing decision itself adds under 1% overhead on its multi-cluster Inference Gateway; the win is keeping the common case local and prefix-warm while still draining a regional spike into borrowed capacity instead of shedding it.

Lesson

Throughput problems in LLM serving are usually routing problems wearing a hardware costume. Before you add GPUs, check the prefix cache hit rate and the spread of gpu_cache_usage_perc across replicas — if the hit rate is single digits and the cache utilization is lopsided, you're paying for capacity a connection-counting load balancer is actively wasting. The router in front of an inference fleet has to understand KV cache and prefix locality, because those, not connection count, are what actually decide how much work each GPU can take.

---

Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.