Your RAG App Isn't Slow Because of the LLM. It's the Retrieval Path.

Four seconds to the first word

The complaint from the product team is always the same shape: "the assistant feels slow." Not broken, not wrong — slow. You open the traces and there it is: a median of around four seconds between the user hitting enter and the first word of the answer appearing. The reflex in the room is to blame the model. Someone suggests a smaller, faster model. Someone else suggests a bigger GPU. Both are usually wrong.

A Retrieval-Augmented Generation request is a pipeline, and the model is only the last stage of it. Before a single token is generated, you embed the query, search one or more vector indexes, often rerank the candidates, and assemble a prompt that can run to several thousand tokens. If you have never instrumented those stages separately, you are flying blind — and you will keep spending money on the one stage that probably isn't the problem.

Instrument the request as a waterfall

Before optimizing anything, break one request into spans. You don't need anything exotic — OpenTelemetry around each stage is enough to turn "it feels slow" into a budget you can argue with.

from opentelemetry import trace
tracer = trace.get_tracer("rag")

async def answer(query: str) -> str:
    with tracer.start_as_current_span("rag.request"):
        with tracer.start_as_current_span("embed_query"):
            qvec = await embed(query)
        with tracer.start_as_current_span("vector_search"):
            hits = await search(qvec, k=50)
        with tracer.start_as_current_span("rerank"):
            top = await rerank(query, hits, k=6)
        with tracer.start_as_current_span("build_prompt"):
            prompt = build_prompt(query, top)
        with tracer.start_as_current_span("llm_generate"):
            return await generate(prompt)

The first time most teams look at this waterfall, the shape is a surprise: the model's own time-to-first-token is a minority of the wall clock. The diagram below is illustrative, but the proportions match what you tend to find in an un-tuned pipeline — retrieval and reranking eat the budget, and because nothing is streamed, the user waits for the entire bar before seeing anything.

Fix 1: stream, so perceived latency becomes time-to-first-token

The single biggest perceived-speed win usually costs nothing in compute: stop waiting for the full answer before you send anything. If the user sees words appearing at 750ms, they do not care that the full response finishes at 2 seconds. A blank spinner for 2 seconds feels broken; a response that starts typing at 750ms feels fast.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

@app.post("/chat")
async def chat(q: str):
    async def tokens():
        async for chunk in generate_stream(build_prompt(q, await retrieve(q))):
            yield chunk
    return StreamingResponse(tokens(), media_type="text/event-stream")

Streaming doesn't reduce the total work, but it re-bases what the user is waiting on from total latency to time-to-first-token (TTFT). Every other fix below is about shrinking that TTFT.

Fix 2: parallelize retrieval instead of chaining it

The waterfall code above is written sequentially, and that's often how it runs in production too: embed, then search index A, then search index B, then rerank. If you retrieve from more than one source — a dense vector index, a keyword/BM25 index, a media-specific store — there is rarely a data dependency between them. Fire them concurrently.

import asyncio

async def retrieve(query: str):
    qvec = await embed(query)
    # dense, sparse, and image lookups have no dependency on each other
    dense, sparse, media = await asyncio.gather(
        vector_search(qvec, k=50),
        bm25_search(query, k=50),
        image_search(qvec, k=10),
    )
    fused = reciprocal_rank_fusion(dense, sparse, media)
    return await rerank(query, fused, k=6)

Three 300ms lookups run serially cost 900ms; run concurrently they cost roughly one 300ms lookup. That is the flattest, cheapest chunk of the "before" bar to reclaim.

Fix 3: cache at two levels

Consumer traffic is repetitive. People ask the same handful of questions in slightly different words, and your static system prompt is identical on every single request. Both are cacheable, at different layers.

Semantic cache on the query. Instead of an exact-string cache (which almost never hits for natural language), cache on the query embedding and treat a near-neighbor above a similarity threshold as a hit. This skips the entire retrieve-and-generate path for repeat questions.

async def cached_answer(query: str):
    qvec = await embed(query)
    hit = await cache_index.search(qvec, k=1)          # e.g. Redis vector / pgvector
    if hit and hit.score >= 0.95:                       # tune this threshold carefully
        return hit.payload["answer"]
    answer = await answer_uncached(query, qvec)
    await cache_index.upsert(qvec, {"answer": answer})
    return answer

Set that threshold conservatively — too loose and you return a confidently wrong cached answer to a subtly different question, which is worse than being slow. Give cached entries a TTL so answers that depend on changing data don't go stale.

Prefix caching in the serving engine. Your system prompt and few-shot examples are the same tokens on every request, yet a naive setup recomputes their attention KV cache every time. Modern inference servers can reuse it. In vLLM it's a flag:

vllm serve your-model \
  --enable-prefix-caching \
  --max-model-len 8192

With prefix caching on, the prefill for the shared prefix is computed once and reused across requests, so TTFT for a long, mostly-static prompt drops sharply. Put the invariant part of the prompt (system instructions, format examples) first and the volatile part (retrieved chunks, user query) last, so the cacheable prefix is as long as possible.

Fix 4: stop over-retrieving

Two habits quietly inflate both the rerank and prefill stages. The first is reranking far more candidates than you need: pulling 50 hits and running an expensive cross-encoder over all of them, when you only ever pass six to the model. Cut the reranker's input, or skip reranking entirely for queries where dense retrieval is already confident. The second is context stuffing — packing every plausibly-relevant chunk into the prompt "just in case." Prefill time scales with prompt length, so every chunk you add is TTFT you pay for on every request, and past a point extra context makes answer quality worse, not better, by burying the relevant passage. Fewer, better chunks is faster and usually more accurate.

The lesson

"The assistant feels slow" is a statement about time-to-first-token, and TTFT in a RAG system is dominated by everything that happens before the model runs. Instrument the pipeline as a waterfall first; the stage that's actually expensive is rarely the one you'd have guessed. Most of the wins — streaming, parallel retrieval, semantic and prefix caching, trimming the context — cost nothing in hardware and never touch the model. Reach for a bigger GPU only after the trace tells you the model is genuinely the tall bar, not before.

---

Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.