Lets Architect the Performance Engineering

The suite is green and the agent is broken You change one tool definition in your agent — tighten a parameter description, swap the underlying model, reorder two functions in the system prompt. CI runs the eval suite. It comes back at 94%, same as last week. You ship. Two weeks later a support ticket arrives: for a particular class of request, the agent stopped calling the escalation tool entirely. It answers confidently, it never opens the ticket, and the customer quietly churns. You go back to the eval run that approved the change. It's still sitting there at 94%. The regression was in the suite the whole time; the suite just couldn't see it. This is the failure mode nobody warns you about when you start building agents. The eval harness that felt rigorous in month one — a few hundred cases, a pass rate, a green check — gives you confidence that is not earned. Here are four things that actually have to be true before a green eval run means anything. 1. One aggregate p...

A batch of agent jobs lands at the top of the hour. Workflows that normally finish in a few seconds sit there for ten minutes doing nothing. You open the worker dashboard braced for a fire — pegged CPU, OOM kills, thrashing — and instead everything is calm. CPU is at 20%. Memory is flat. The autoscaler is parked at its minimum replica count, perfectly content. Nothing is broken. Everything is waiting. The tasks are sitting in Temporal's task queue before any worker picks them up , and that wait is invisible to every dashboard you just looked at. The queue you can't see When you run workflows on Temporal (or any durable-execution engine — the same shape shows up in Cadence, and conceptually in Step Functions), your code doesn't run on the Temporal service. The service is a task broker. It persists workflow state and hands out two kinds of tasks on named task queues: workflow tasks (advance the state machine) and activity tasks (do the actual side-effecting wor...

Here is a failure mode that doesn't show up in any single-machine benchmark: a multi-day training job across 512 GPUs stops making progress at 3am. The loss curve flatlines. No step completes. Every one of those 512 accelerators is allocated, powered, and billed — and all of them are doing nothing, parked at a collective-communication barrier, waiting on a peer that is never going to answer. A few minutes later a watchdog fires and the whole job crashes with something like Watchdog caught collective operation timeout . If you were checkpointing every few hours, you've just lost everything computed since the last checkpoint across the entire cluster. The actual fault was one GPU throwing an uncorrectable memory error. One. The other 511 were fine. Why one GPU can wedge the whole job Synchronous data-parallel training — the default for anything large — runs a collective operation (an all-reduce of gradients) at the end of every step. Every rank contributes, and every rank b...

Database CPU sits at 95%. The app's request rate hasn't moved all week. The slow-query log, though, is full of statements no one on the team would ever write by hand: SELECT * FROM events WHERE payload::text LIKE '%refund%' ORDER BY created_at DESC; A leading-wildcard LIKE on a text-cast JSONB column, across a multi-terabyte table, selecting every column, sorting the entire result set, no LIMIT . It's a guaranteed sequential scan plus a disk sort. And it's running dozens of times an hour. Then you remember: last week you wired an AI agent into the database through an MCP server, and you gave it a single, flexible tool called execute_sql . The agent isn't broken. It's doing exactly what a general-purpose SQL tool invites it to do — exploring. The investigation The first instinct is to blame a traffic spike, but the app's own request metrics are flat. So the load is coming from somewhere that isn't the app. pg_stat_statements settles it...

Count the inference endpoints in your org. There's the self-hosted vLLM deployment on GKE serving the model you fine-tuned. There's a managed endpoint — Vertex, Bedrock, whatever — for the one you didn't want to operate. There's OpenAI or Anthropic for the frontier models you haven't brought in-house. Three backends, three URLs, three auth schemes, and every one of them hardcoded into a dozen client services. That sprawl is not a cosmetic problem. It has concrete, recurring failure modes: Adding or renaming a model means a coordinated client redeploy, because the endpoint URL lives in application config. There's no shared rate limit, so one team's overnight batch job saturates the GPU pool and interactive traffic's time-to-first-token falls off a cliff — with no throttle to protect it. Retry, timeout, and fallback logic is reimplemented (badly, inconsistently) in every client. When finance asks which team spent the inference budge...

Memory is the first hard limit most Kubernetes clusters hit. Nodes run out of RAM long before they run out of CPU, and the scheduler stops placing pods the moment requests exhaust allocatable memory — even if the CPUs are half-idle. You end up paying for cores you can't use because there's no RAM left to hand out. Agentic AI workloads make this worse in a specific, annoying way. A typical agent pod loads a language runtime, maybe a model client, maybe a sandbox for executing untrusted code — a few gigabytes of resident footprint — spikes on startup, and then sits idle waiting for the next prompt. That memory is reserved and resident, but it's cold most of the time. Multiply by a fleet of agents and you've got a node that's RAM-bound on memory that nobody is actually touching. Kubernetes node swap just graduated to GA, and this is precisely the workload it was built for. It's also a very good way to wreck your tail latency if you treat it as free memory. Her...

The agent that remembered everything, and slowed to a crawl Here is a failure mode that shows up on a schedule once you give an agent persistent memory. In week one it feels sharp: it recalls what you told it yesterday, it picks up a multi-day task where it left off, memory lookups are a few milliseconds and nobody thinks about them. By week six the same agent is noticeably slower on every turn, and — more insidiously — its answers have started drifting toward stale or irrelevant facts it dug up from months ago. The reflex is to blame the vector index, or the model, or to move to a bigger database instance. Usually none of those is the real problem. The problem is architectural, and it is almost always the same one: all of the agent's memory lives in a single flat, append-only table that nothing ever consolidates or evicts. That design degrades on two axes at once, and you have to fix both. Why a flat memory table degrades on two axes The naive design is seductive because ...

Your support agent decides a customer is owed a refund. It calls the issue_refund tool, the payment provider returns 200, and the agent writes status = "refunded" to its own task row. Clean. Except on about one run in a few thousand, the customer gets refunded twice — or, on a different unlucky run, the agent swears it refunded them and the money never moved. Nothing in the model is broken. The prompt is fine, the tool schema is fine, the eval suite is green. What's broken is older and more boring than anything about LLMs: the agent is doing two writes to two different systems and pretending they happen together. They don't. The symptom You notice it from a reconciliation job, not from the agent. A finance export shows two refund transactions against one order ID, minutes apart, both tagged with the same agent run. Or a customer emails back asking where their money is, and the trace shows the tool call succeeded. When you pull the logs for the double case, yo...

You ramp traffic onto an LLM serving deployment and the graph does not add up. Time-to-first-token, flat at 300ms all morning, starts climbing: 900ms, 2s, 5s. Users are waiting. And the HorizontalPodAutoscaler that is supposed to save you is sitting at 2 replicas, perfectly calm, because the metric it watches — CPU — reads 22%. Nothing is broken, exactly. The autoscaler is doing precisely what you told it to do. You just told it to watch the wrong number. The graph that doesn't add up Here is the HPA's view of the world during the incident: $ kubectl describe hpa vllm-llama Name: vllm-llama Reference: Deployment/vllm-llama Metrics: ( current / target ) resource cpu on pods (as a percentage of request): 22% (110m) / 70% Min replicas: 2 Max replicas: 12 Deployment pods: 2 current / 2 desired Conditions: Type Status Reason Message ScalingActive True ValidMetricFound the HPA was able to compute a target ...