One Model, Two Bottlenecks: LLM Classification Is Compute-Bound, Generation Is Memory-Bound

The symptom: one benchmark number, two very different bills

You get a new accelerator, run the vendor's throughput benchmark, and it posts a great tokens-per-second number. You size a fleet against that number. Then one of two things happens in production. Either your classification service — text in, one label out — runs at a fraction of the throughput the benchmark promised, or your chat/generation service leaves the accelerators sitting at 20% utilization while latency creeps up. Same model, same silicon, and the economics are nowhere near what you provisioned for.

The benchmark wasn't lying. It measured a workload whose bottleneck has almost nothing to do with yours. "LLM inference" is not one workload — it is two, and they live at opposite ends of the same accelerator's roofline.

Two phases hiding inside "inference"

Every request through a transformer has two distinct phases with different performance characteristics:

Prefill processes the entire input prompt in one shot. All prompt tokens go through the model together as a single large matrix-multiply. This is what determines time to first token (TTFT).
Decode generates the output one token at a time, autoregressively. Each step feeds the single most recent token back in, multiplies it against every weight in the model, and reads the growing KV cache. This is what determines time per output token (TPOT).

Now map real workloads onto those phases. A classification or extraction task — sentiment, routing, PII detection, "is this spam" — takes a potentially long input and emits one token, or a handful. It is almost entirely prefill. A generation task — a chatbot turn, a summary, a code completion — takes a short-ish prompt and emits hundreds or thousands of tokens. It is almost entirely decode.

Those two phases stress completely different parts of the chip.

The roofline tells you which one you're paying for

The clean way to reason about this is the roofline model. An accelerator has two ceilings: peak compute (FLOP/s) and peak memory bandwidth (bytes/s). Which one you hit depends on your kernel's arithmetic intensity — how many FLOPs it does per byte it reads from memory. Below the crossover ("ridge point"), you are memory-bound and adding FLOP/s buys you nothing. Above it, you are compute-bound and bandwidth is spare.

# Order-of-magnitude accelerator spec (illustrative, not a specific product)
PEAK_FLOPS = 200e12   # 200 TFLOP/s, bf16 matmul
HBM_BW     = 2.0e12   # 2 TB/s memory bandwidth

ridge = PEAK_FLOPS / HBM_BW          # FLOP/byte you must feed to stay compute-bound
print(f"ridge point: {ridge:.0f} FLOP/byte")   # -> 100

# For a dense transformer, the arithmetic intensity of a weight-bound step is
# roughly the number of tokens processed per weight load (bf16 = 2 bytes/param):
#   FLOPs ~= 2 * P * T     (P params, T tokens in the step)
#   bytes ~= 2 * P         (every weight read once)
#   intensity ~= T
#
# prefill: T = prompt_len * batch   -> hundreds of tokens per step
# decode:  T = batch                -> ONE token per sequence per step
for T in (1, 8, 64, 512):
    bound = "compute-bound" if T >= ridge else "MEMORY-bound"
    print(f"T={T:4d} tokens/step -> intensity ~{T:4d} FLOP/byte -> {bound}")

The output is the whole story:

ridge point: 100 FLOP/byte
T=   1 tokens/step -> intensity ~   1 FLOP/byte -> MEMORY-bound
T=   8 tokens/step -> intensity ~   8 FLOP/byte -> MEMORY-bound
T=  64 tokens/step -> intensity ~  64 FLOP/byte -> MEMORY-bound
T= 512 tokens/step -> intensity ~ 512 FLOP/byte -> compute-bound

A single 512-token prompt clears the ridge point on its own — prefill is compute-bound almost by default, which is why classification benchmarks post gaudy throughput numbers and high model-FLOPs-utilization. Decode is the opposite: at batch size 1 the intensity is about 1 FLOP/byte, two orders of magnitude below the ridge. To drag decode up to the compute ceiling you'd need to batch on the order of 100 concurrent sequences — and the KV cache, which you also have to stream from HBM every step and which grows with context length, only makes the memory pressure worse. In practice decode is memory-bandwidth-bound essentially all the time.

Measuring your actual workload, not the benchmark's

The mistake is benchmarking with a shape that doesn't match production. A generic "throughput" run with 128-in / 128-out tokens is a blend of both phases and tells you little about either. Drive the two phases separately. With vLLM's serving benchmark you can pin the input/output ratio to isolate each:

# Classification-shaped: long input, ~1 output token -> stresses PREFILL
python benchmarks/benchmark_serving.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random \
  --random-input-len 512 \
  --random-output-len 1 \
  --request-rate 200

# Generation-shaped: short input, long output -> stresses DECODE
python benchmarks/benchmark_serving.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random \
  --random-input-len 64 \
  --random-output-len 512 \
  --request-rate 200

And measure the metric that matches the SLO, not a single averaged number:

For the classification/prefill run, the number that matters is throughput (requests/sec, or input tokens/sec) and TTFT. You should be able to push utilization high and watch it scale with batch size until you saturate the matrix units.
For the generation/decode run, watch TPOT and its tail. Throughput will plateau well below the compute ceiling, and past a certain concurrency TPOT degrades because you're contending for HBM bandwidth and KV-cache capacity, not FLOP/s.

If you only ever report a blended tokens/sec, these two failure modes are invisible until they show up on the bill or the latency dashboard.

Provisioning for the bottleneck you actually have

Once you know which phase dominates, the levers are different for each.

Raise the decode batch. Decode only climbs toward the compute ceiling as concurrency rises, so continuous (in-flight) batching is the single biggest win for generation workloads — it keeps refilling the batch as sequences finish instead of waiting for a whole batch to complete. The knobs that cap it are worth setting deliberately:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-seqs 256 \            # ceiling on concurrent sequences (decode batch)
  --max-num-batched-tokens 8192 \ # prefill token budget per scheduler step
  --enable-chunked-prefill        # split big prefills so they co-schedule with decode

Stop letting prefill starve decode. A big prefill is a compute-heavy step; if it monopolizes the accelerator, every in-flight generation request stalls and TPOT tails blow out. Chunked prefill breaks a large prompt into pieces that get interleaved with ongoing decode steps, trading a little TTFT for a much steadier TPOT — usually the right call for interactive generation.

Separate the two workloads. Because prefill and decode want different things — prefill wants raw FLOP/s, decode wants HBM bandwidth and capacity — running them on one pool sized by a single benchmark guarantees you overpay for one of them. The current answer to this is prefill/decode disaggregation: run prefill on one pool, hand the KV cache off to a separate decode pool, and scale each independently. It shows up now in production-grade stacks (vLLM's disaggregated serving, NVIDIA Dynamo, and research systems like DistServe and Splitwise). If you have a mixed workload at scale, this is how you keep each phase on hardware that fits it.

Pick hardware by phase, not by headline. A classification-heavy fleet is compute-bound — it rewards chips with dense matmul throughput and can tolerate more modest memory bandwidth. A generation-heavy fleet is memory-bound — HBM bandwidth and the capacity to hold big KV caches matter far more than peak FLOP/s. The same accelerator that looks like a bargain per FLOP for classification can be the wrong buy for long-form generation, and vice versa.

The lesson

Before you size an inference fleet, decompose your traffic into prefill and decode and benchmark each in isolation — a single blended tokens/sec number hides the one thing you need to know, which ceiling you're actually hitting. Classification and generation can run the same weights on the same chip and still be different performance problems with different fixes. Provision for the phase your workload is made of, not for the phase the benchmark happened to measure.

Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.

Search This Blog

Lets Architect the Performance Engineering