Lets Architect the Performance Engineering

A batch of agent jobs lands at the top of the hour. Workflows that normally finish in a few seconds sit there for ten minutes doing nothing. You open the worker dashboard braced for a fire — pegged CPU, OOM kills, thrashing — and instead everything is calm. CPU is at 20%. Memory is flat. The autoscaler is parked at its minimum replica count, perfectly content. Nothing is broken. Everything is waiting. The tasks are sitting in Temporal's task queue before any worker picks them up , and that wait is invisible to every dashboard you just looked at. The queue you can't see When you run workflows on Temporal (or any durable-execution engine — the same shape shows up in Cadence, and conceptually in Step Functions), your code doesn't run on the Temporal service. The service is a task broker. It persists workflow state and hands out two kinds of tasks on named task queues: workflow tasks (advance the state machine) and activity tasks (do the actual side-effecting wor...

Here's a failure mode that doesn't show up where you'd look for it. Time-to-first-token (TTFT) sits comfortably inside its SLO. Throughput is fine. Error rate is flat. What breaks is the other latency metric: inter-token latency, the gap between generated tokens, also called TPOT (time per output token). Its p99 is fine most of the time, then jumps three or four times over for a few seconds and settles back down. Plot TPOT against the request log and the pattern gives itself away: every spike lines up with a long prompt entering the running batch. One user pastes a 12k-token document to summarize, and for the duration of that request's prefill, every other user watching tokens stream back sees them stutter. The service isn't overloaded. Two fundamentally different workloads are being forced through the same GPU at the same instant. Two phases, two bottlenecks, one accelerator An LLM request runs in two phases with completely different hardware profiles. That...

Here is a failure mode that doesn't show up in any single-machine benchmark: a multi-day training job across 512 GPUs stops making progress at 3am. The loss curve flatlines. No step completes. Every one of those 512 accelerators is allocated, powered, and billed — and all of them are doing nothing, parked at a collective-communication barrier, waiting on a peer that is never going to answer. A few minutes later a watchdog fires and the whole job crashes with something like Watchdog caught collective operation timeout . If you were checkpointing every few hours, you've just lost everything computed since the last checkpoint across the entire cluster. The actual fault was one GPU throwing an uncorrectable memory error. One. The other 511 were fine. Why one GPU can wedge the whole job Synchronous data-parallel training — the default for anything large — runs a collective operation (an all-reduce of gradients) at the end of every step. Every rank contributes, and every rank b...

Database CPU sits at 95%. The app's request rate hasn't moved all week. The slow-query log, though, is full of statements no one on the team would ever write by hand: SELECT * FROM events WHERE payload::text LIKE '%refund%' ORDER BY created_at DESC; A leading-wildcard LIKE on a text-cast JSONB column, across a multi-terabyte table, selecting every column, sorting the entire result set, no LIMIT . It's a guaranteed sequential scan plus a disk sort. And it's running dozens of times an hour. Then you remember: last week you wired an AI agent into the database through an MCP server, and you gave it a single, flexible tool called execute_sql . The agent isn't broken. It's doing exactly what a general-purpose SQL tool invites it to do — exploring. The investigation The first instinct is to blame a traffic spike, but the app's own request metrics are flat. So the load is coming from somewhere that isn't the app. pg_stat_statements settles it...

Here is a failure mode that looks like an autoscaling bug but isn't. A traffic spike hits your model-serving deployment, the HPA (or KEDA, or your queue-depth scaler) correctly decides it needs three more replicas, and the new pods go Pending → ContainerCreating → eventually Running . The whole time, the queue keeps growing and p99 stays in the toilet. By the time the new replicas actually start serving tokens, the spike is half over. The scaler did its job in seconds. The pods took minutes to become useful. That gap — from "a replica was requested" to "that replica is serving real traffic at full speed" — is the cold-start tax, and on GPU model serving it is brutal: a multi-gigabyte container image, tens of gigabytes of weights, and a warmup pass all sit between the scale-up decision and the first good response. The usual reaction is to stop trusting the autoscaler and just run a fat idle buffer of hot replicas, which is the most expensive possible workarou...

Here is a billing situation that should make anyone running a training cluster uncomfortable: eight H100s, 100% allocated, reserved for the whole job, and the accelerators themselves working about a third of the time. You pay for every second the hardware is yours. The hardware spends most of each step waiting for data to show up. The reason this goes unnoticed for so long is that the obvious metric lies to you. nvidia-smi prints a column called GPU-Util , and during a stalled training run it will happily read 95%. The job looks healthy. The loss goes down. Nobody opens a ticket. Meanwhile the effective throughput — samples per second per dollar — is less than half of what the same silicon could do. Why the headline utilization number is misleading The GPU-Util field from nvidia-smi is defined as the fraction of the sampling window during which at least one kernel was executing. That is a very low bar. A training step that spends 600ms blocked on the dataloader and 400ms doin...

Count the inference endpoints in your org. There's the self-hosted vLLM deployment on GKE serving the model you fine-tuned. There's a managed endpoint — Vertex, Bedrock, whatever — for the one you didn't want to operate. There's OpenAI or Anthropic for the frontier models you haven't brought in-house. Three backends, three URLs, three auth schemes, and every one of them hardcoded into a dozen client services. That sprawl is not a cosmetic problem. It has concrete, recurring failure modes: Adding or renaming a model means a coordinated client redeploy, because the endpoint URL lives in application config. There's no shared rate limit, so one team's overnight batch job saturates the GPU pool and interactive traffic's time-to-first-token falls off a cliff — with no throttle to protect it. Retry, timeout, and fallback logic is reimplemented (badly, inconsistently) in every client. When finance asks which team spent the inference budge...

Memory is the first hard limit most Kubernetes clusters hit. Nodes run out of RAM long before they run out of CPU, and the scheduler stops placing pods the moment requests exhaust allocatable memory — even if the CPUs are half-idle. You end up paying for cores you can't use because there's no RAM left to hand out. Agentic AI workloads make this worse in a specific, annoying way. A typical agent pod loads a language runtime, maybe a model client, maybe a sandbox for executing untrusted code — a few gigabytes of resident footprint — spikes on startup, and then sits idle waiting for the next prompt. That memory is reserved and resident, but it's cold most of the time. Multiply by a fleet of agents and you've got a node that's RAM-bound on memory that nobody is actually touching. Kubernetes node swap just graduated to GA, and this is precisely the workload it was built for. It's also a very good way to wreck your tail latency if you treat it as free memory. Her...