Lets Architect the Performance Engineering

"It works" is not a number An LLM agent ships. It passed the demo, it passed the handful of prompts in the eval notebook, and for two weeks nobody complains. Then a support ticket arrives: the agent confidently told a customer about a refund policy that doesn't exist. You go looking for a dashboard that tells you how often that happens and there isn't one. You have latency graphs, you have token-cost graphs, and for the thing that actually matters — is the agent right — you have a vibe. That gap is the whole problem. We instrument agents like web services (RPS, p99, error rate) and then act surprised that none of those numbers move when the agent hallucinates. A 200 response carrying a made-up fact is still a 200. The reliability machinery you already run is fine; it's just pointed at the wrong signals. Reframe: agent quality is a reliability problem An error budget is just the inverse of a target: if you promise 99%, you have a 1% budget to spend. That f...

You flip the read path to the new database. Within a minute, a fraction of a percent of lookups start returning 404 , and a handful return data that's a few minutes stale. The write path has been dual-writing to both stores for three weeks. Every write returned 200 . The backfill job logged "complete." By every dashboard you had, the migration was done. It wasn't. The old store and the new store had been quietly drifting apart the entire time, and the cutover is exactly the moment you discover it — when the new store becomes the source of truth for reads and its gaps become user-visible. Why "both writes succeeded" doesn't mean "both stores match" The seductive thing about dual-write is that it looks correct locally. Your write path does something like this: def save_order(order): old_db.upsert(order) # legacy Postgres new_db.upsert(order) # Spanner return 200 Both calls return, you return 200 , and the r...

Green build, red latency graph The failure mode is quiet by design. A pull request runs the full suite, everything passes, and it merges. Two days later the p99 latency graph for a hot endpoint is 40% higher than it was on Monday, and nobody can point to the cause — because 60 PRs shipped that week and every one of them was green. The regression didn't break a test. It produced exactly the right answer, a little slower each time, and no assertion in the codebase was watching the clock. There's real momentum right now behind autonomous bug-finding harnesses — tools that fuzz, triage, reproduce, and even patch defects with an agent in the loop. That works because a memory-safety bug or a crash is a discrete event a harness can trip over. A performance regression is not that. It's a distribution shift in a number your tests never recorded. If you want to catch it before it reaches production, you have to build a gate that measures time on purpose, and — this is the part pe...

The alert that lies to you The page fires at 02:14: API p99 latency > 2s . Your incident-response agent — the LLM you wired into the observability stack last quarter — does exactly what you asked. It pulls the API service's metrics, reads the last few hundred log lines, grabs a handful of exemplar traces, and concludes with total confidence: the API service is CPU-bound, scale it up. You scale it up. p99 doesn't move. The API service was never the problem. Three hops downstream, a connection pool is quietly handing out dead connections, and every request is blocking on a TCP handshake that will never complete. The agent never looked there, because nothing in the pile of telemetry you handed it said the API depends on this pool. It pattern-matched on the loudest signal, which is almost always the symptom, not the cause. Why raw telemetry defeats the agent The instinct when building an RCA agent is to give it everything: hook it up to Prometheus, Loki, and Tempo, let ...