Lets Architect the Performance Engineering

Database CPU sits at 95%. The app's request rate hasn't moved all week. The slow-query log, though, is full of statements no one on the team would ever write by hand: SELECT * FROM events WHERE payload::text LIKE '%refund%' ORDER BY created_at DESC; A leading-wildcard LIKE on a text-cast JSONB column, across a multi-terabyte table, selecting every column, sorting the entire result set, no LIMIT . It's a guaranteed sequential scan plus a disk sort. And it's running dozens of times an hour. Then you remember: last week you wired an AI agent into the database through an MCP server, and you gave it a single, flexible tool called execute_sql . The agent isn't broken. It's doing exactly what a general-purpose SQL tool invites it to do — exploring. The investigation The first instinct is to blame a traffic spike, but the app's own request metrics are flat. So the load is coming from somewhere that isn't the app. pg_stat_statements settles it...

Here is a failure mode that looks like an autoscaling bug but isn't. A traffic spike hits your model-serving deployment, the HPA (or KEDA, or your queue-depth scaler) correctly decides it needs three more replicas, and the new pods go Pending → ContainerCreating → eventually Running . The whole time, the queue keeps growing and p99 stays in the toilet. By the time the new replicas actually start serving tokens, the spike is half over. The scaler did its job in seconds. The pods took minutes to become useful. That gap — from "a replica was requested" to "that replica is serving real traffic at full speed" — is the cold-start tax, and on GPU model serving it is brutal: a multi-gigabyte container image, tens of gigabytes of weights, and a warmup pass all sit between the scale-up decision and the first good response. The usual reaction is to stop trusting the autoscaler and just run a fat idle buffer of hot replicas, which is the most expensive possible workarou...

Here is a billing situation that should make anyone running a training cluster uncomfortable: eight H100s, 100% allocated, reserved for the whole job, and the accelerators themselves working about a third of the time. You pay for every second the hardware is yours. The hardware spends most of each step waiting for data to show up. The reason this goes unnoticed for so long is that the obvious metric lies to you. nvidia-smi prints a column called GPU-Util , and during a stalled training run it will happily read 95%. The job looks healthy. The loss goes down. Nobody opens a ticket. Meanwhile the effective throughput — samples per second per dollar — is less than half of what the same silicon could do. Why the headline utilization number is misleading The GPU-Util field from nvidia-smi is defined as the fraction of the sampling window during which at least one kernel was executing. That is a very low bar. A training step that spends 600ms blocked on the dataloader and 400ms doin...

Count the inference endpoints in your org. There's the self-hosted vLLM deployment on GKE serving the model you fine-tuned. There's a managed endpoint — Vertex, Bedrock, whatever — for the one you didn't want to operate. There's OpenAI or Anthropic for the frontier models you haven't brought in-house. Three backends, three URLs, three auth schemes, and every one of them hardcoded into a dozen client services. That sprawl is not a cosmetic problem. It has concrete, recurring failure modes: Adding or renaming a model means a coordinated client redeploy, because the endpoint URL lives in application config. There's no shared rate limit, so one team's overnight batch job saturates the GPU pool and interactive traffic's time-to-first-token falls off a cliff — with no throttle to protect it. Retry, timeout, and fallback logic is reimplemented (badly, inconsistently) in every client. When finance asks which team spent the inference budge...

Memory is the first hard limit most Kubernetes clusters hit. Nodes run out of RAM long before they run out of CPU, and the scheduler stops placing pods the moment requests exhaust allocatable memory — even if the CPUs are half-idle. You end up paying for cores you can't use because there's no RAM left to hand out. Agentic AI workloads make this worse in a specific, annoying way. A typical agent pod loads a language runtime, maybe a model client, maybe a sandbox for executing untrusted code — a few gigabytes of resident footprint — spikes on startup, and then sits idle waiting for the next prompt. That memory is reserved and resident, but it's cold most of the time. Multiply by a fleet of agents and you've got a node that's RAM-bound on memory that nobody is actually touching. Kubernetes node swap just graduated to GA, and this is precisely the workload it was built for. It's also a very good way to wreck your tail latency if you treat it as free memory. Her...

The cost model is a page. The document isn't. The last post on batch-classifying 400 million documents got a sharp question in the comments: Gemini's native PDF handling gives you a fixed, predictable token cost per page — 258 tokens per page is the actual number Google publishes — so a 50-page document costs roughly 50x a 1-page one. That's what makes costing 400 million documents a multiplication instead of a guess. The question was: what happens to a table that starts on page 12 and ends on page 13, once you're chunking and classifying by page? Nothing good, by default. And the failure is quiet — it doesn't throw an error, it just returns a confident, wrong label. A page break is a rendering artifact, not a semantic one Nothing in the PDF format guarantees that a table, a clause, or even a sentence finishes before the page does. A contract's indemnification table can run four rows on page 12 and six more on page 13 with no repeated header row — it's...

The cluster is 60% idle and the important job has waited three days Here is a combination of facts that should not be able to coexist, but does, on GPU clusters everywhere. The accelerator bill is enormous. The "GPU allocated" line on the dashboard sits around 60%. And the one training run the research lead actually cares about has been stuck in Pending for three days while a dozen tiny experiments churn away on the hardware it needs. Everyone's first instinct is to buy more GPUs. But this is almost never a capacity problem. A cluster that is 60% allocated has room; the important job can't get in anyway. This is a scheduling problem, and you can usually fix it without adding a single accelerator. AI21, writing about their move to Google Cloud's AI Hypercomputer, reported cutting high-priority job wait times from 72 hours to 12 and manual scheduling interventions from roughly 20 per week to zero — the kind of swing you get from fixing how work is admitted, no...

The agent that remembered everything, and slowed to a crawl Here is a failure mode that shows up on a schedule once you give an agent persistent memory. In week one it feels sharp: it recalls what you told it yesterday, it picks up a multi-day task where it left off, memory lookups are a few milliseconds and nobody thinks about them. By week six the same agent is noticeably slower on every turn, and — more insidiously — its answers have started drifting toward stale or irrelevant facts it dug up from months ago. The reflex is to blame the vector index, or the model, or to move to a bigger database instance. Usually none of those is the real problem. The problem is architectural, and it is almost always the same one: all of the agent's memory lives in a single flat, append-only table that nothing ever consolidates or evicts. That design degrades on two axes at once, and you have to fix both. Why a flat memory table degrades on two axes The naive design is seductive because ...