Posts

Round-Robin Is Wrecking Your LLM Throughput: Route by KV Cache, Not Connection Count

-
You went from 2 vLLM replicas to 8 because your queue was backing up. Eight GPUs now, four times the hardware. Aggregate throughput went up maybe 30%, and p99 time-to-first-token (TTFT) got worse . The per-GPU utilization graphs look busy, the autoscaler is happy, and the bill quadrupled for a fraction of the goodput you paid for. Before you reach for bigger GPUs, look at the thing in front of them. A plain round-robin or least-request load balancer is the wrong tool for LLM inference, and it fails in a way that's invisible on a CPU/GPU utilization dashboard. Why a normal load balancer is wrong here Web load balancing works because requests are roughly interchangeable: each one is short-lived, stateless, and costs about the same. Spraying them round-robin across identical replicas is close to optimal. LLM serving violates every one of those assumptions. Requests are not uniform. A request that generates 2,000 tokens occupies KV cache in GPU HBM for its entire decode l...
Post a Comment
Read more

You Gave an Agent Your Database via MCP, and Now the Connection Pool Is on Fire

-
The traffic graph is flat. Nobody deployed a schema change, nobody launched a campaign, the request rate to your API looks exactly like it did last week. And yet the database is throwing FATAL: sorry, too many clients already , and pg_stat_activity is full of sessions parked in the idle in transaction state. The one thing that changed: yesterday you wired an AI agent to the database through an MCP server so it could answer questions over your data. This is becoming a common way to melt a perfectly healthy Postgres. The Model Context Protocol makes it a few lines of config to hand an agent a query tool backed by AlloyDB, Cloud SQL, or plain Postgres. What the tutorial doesn't mention is that an LLM is a very strange database client, and the connection pool you sized for a web app makes assumptions the agent quietly violates. I have seen this movie before, without the AI Early in my career I ran a web API backend on GCP that behaved fine in staging and then, under real load,...
Post a Comment
Read more

Your Vector Search Got Faster and Recall Fell Off a Cliff — Nobody Got Paged

-
The latency graph looked like a win. Someone had tuned the vector index over the weekend, and p99 for semantic search dropped from around 200ms to 25ms. The dashboard was green, the change shipped, everyone moved on. Two weeks later a different graph started climbing: support tickets. Search "felt dumb." The RAG assistant kept missing answers that were obviously in the knowledge base — type in almost the exact wording of a document and it still wouldn't surface. Nothing had errored. Nothing had paged. The index was returning ten results for every query, fast, every time. They were just increasingly the wrong ten. Why approximate search fails quietly Any production vector search runs on an approximate nearest neighbor (ANN) index, because exact nearest-neighbor over millions of high-dimensional embeddings means scanning every row. ANN indexes buy their speed by not looking at most of your data on each query — and every one of them exposes a knob that controls exact...
Post a Comment
Read more

Your AI Agent Spends 8 Seconds Booting a Sandbox to Run 300ms of Code

-
An agent answers a question that requires it to run a three-line Python snippet — parse a CSV, sum a column, return the total. The code runs in about 300 milliseconds. The user waits eight seconds. Trace the request and the actual computation is a rounding error; everything else is the agent waiting for somewhere to run the code. The bill tells the same story from the other side. You provisioned a generously sized pool of execution environments so agents never queue, and the utilization dashboard shows them sitting idle most of the day, warm and billable, waiting for the next tool call. You are paying full compute prices to keep empty rooms lit. Both symptoms — the latency and the cost — come from the same design decision: one fresh, full-sized sandbox per agent, cold on the way in and idle on the way out. Where the eight seconds go An agent that executes code or calls tools needs an isolated environment to do it in — you cannot run model-generated code in ...
Post a Comment
Read more

Your Inference GPUs Are Starved, Not Slow: Finding the Idle Time in Your AI Bill

-
It usually shows up as a billing question, not an alert. The GPU line on the cloud bill has climbed quarter over quarter while throughput has stayed flat, and someone from finance wants to know why. You open the accelerator dashboard expecting pegged GPUs and instead find them hovering around 25–30% utilization. You are renting some of the most expensive compute available and using a third of it. This is the most common failure mode in production AI infrastructure right now, and it is rarely a "buy fewer GPUs" problem. The GPUs are not slow. They are starved, fragmented, or idle — and every one of those is fixable without touching the model. Step 1: distrust the utilization number you have The first mistake is trusting nvidia-smi . Its headline GPU-Util field does not mean what most people assume. The NVIDIA docs define it as the percent of time over the sample window during which one or more kernels was executing. A single tiny kernel copying data counts the same a...
Post a Comment
Read more

Your Dual-Write Migration Is Lying to You: Catching Silent Divergence Before Cutover

-
You flip the read path to the new database. Within a minute, a fraction of a percent of lookups start returning 404 , and a handful return data that's a few minutes stale. The write path has been dual-writing to both stores for three weeks. Every write returned 200 . The backfill job logged "complete." By every dashboard you had, the migration was done. It wasn't. The old store and the new store had been quietly drifting apart the entire time, and the cutover is exactly the moment you discover it — when the new store becomes the source of truth for reads and its gaps become user-visible. Why "both writes succeeded" doesn't mean "both stores match" The seductive thing about dual-write is that it looks correct locally. Your write path does something like this: def save_order(order): old_db.upsert(order) # legacy Postgres new_db.upsert(order) # Spanner return 200 Both calls return, you return 200 , and the r...
Post a Comment
Read more

The Dashboards Went Flat Mid-Incident. The Bottleneck Was Your Telemetry Gateway.

-
The flat line that isn't zero Traffic spikes, and the dashboards stop telling you anything. Not a crash — the lines just freeze at their last value, then go ragged, then open into a gap. The services behind those numbers are up and serving. The numbers simply stopped arriving. Every metric, log, and span in the platform passes through one component on its way to storage, and that component is now the choke point: the central telemetry gateway. Running a collector as a per-host sidecar is easy. Running one tier as a centralized gateway — a fleet of Grafana Alloy or OpenTelemetry Collector replicas that absorbs the whole platform's telemetry so you can do egress control, tail sampling, cost enforcement, and one place to change config — is a different animal. It concentrates your entire signal into a single pipe, and a pipe has a single point of backpressure. When ingest outruns drain, queues fill, the memory limiter starts refusing data, and points get dropped. The cruel par...
Post a Comment
Read more

The Build Was Green. p99 Climbed 40% Anyway. A Perf-Regression Gate an Agent Can Explain.

-
Green build, red latency graph The failure mode is quiet by design. A pull request runs the full suite, everything passes, and it merges. Two days later the p99 latency graph for a hot endpoint is 40% higher than it was on Monday, and nobody can point to the cause — because 60 PRs shipped that week and every one of them was green. The regression didn't break a test. It produced exactly the right answer, a little slower each time, and no assertion in the codebase was watching the clock. There's real momentum right now behind autonomous bug-finding harnesses — tools that fuzz, triage, reproduce, and even patch defects with an agent in the loop. That works because a memory-safety bug or a crash is a discrete event a harness can trip over. A performance regression is not that. It's a distribution shift in a number your tests never recorded. If you want to catch it before it reaches production, you have to build a gate that measures time on purpose, and — this is the part pe...
Post a Comment
Read more

Your RAG App Isn't Slow Because of the LLM. It's the Retrieval Path.

-
Four seconds to the first word The complaint from the product team is always the same shape: "the assistant feels slow." Not broken, not wrong — slow. You open the traces and there it is: a median of around four seconds between the user hitting enter and the first word of the answer appearing. The reflex in the room is to blame the model. Someone suggests a smaller, faster model. Someone else suggests a bigger GPU. Both are usually wrong. A Retrieval-Augmented Generation request is a pipeline, and the model is only the last stage of it. Before a single token is generated, you embed the query, search one or more vector indexes, often rerank the candidates, and assemble a prompt that can run to several thousand tokens. If you have never instrumented those stages separately, you are flying blind — and you will keep spending money on the one stage that probably isn't the problem. Instrument the request as a waterfall Before optimizing anything, break one request into ...
Post a Comment
Read more