Lets Architect the Performance Engineering

"It works" is not a number An LLM agent ships. It passed the demo, it passed the handful of prompts in the eval notebook, and for two weeks nobody complains. Then a support ticket arrives: the agent confidently told a customer about a refund policy that doesn't exist. You go looking for a dashboard that tells you how often that happens and there isn't one. You have latency graphs, you have token-cost graphs, and for the thing that actually matters — is the agent right — you have a vibe. That gap is the whole problem. We instrument agents like web services (RPS, p99, error rate) and then act surprised that none of those numbers move when the agent hallucinates. A 200 response carrying a made-up fact is still a 200. The reliability machinery you already run is fine; it's just pointed at the wrong signals. Reframe: agent quality is a reliability problem An error budget is just the inverse of a target: if you promise 99%, you have a 1% budget to spend. That f...

The flat line that isn't zero Traffic spikes, and the dashboards stop telling you anything. Not a crash — the lines just freeze at their last value, then go ragged, then open into a gap. The services behind those numbers are up and serving. The numbers simply stopped arriving. Every metric, log, and span in the platform passes through one component on its way to storage, and that component is now the choke point: the central telemetry gateway. Running a collector as a per-host sidecar is easy. Running one tier as a centralized gateway — a fleet of Grafana Alloy or OpenTelemetry Collector replicas that absorbs the whole platform's telemetry so you can do egress control, tail sampling, cost enforcement, and one place to change config — is a different animal. It concentrates your entire signal into a single pipe, and a pipe has a single point of backpressure. When ingest outruns drain, queues fill, the memory limiter starts refusing data, and points get dropped. The cruel par...

The alert that lies to you The page fires at 02:14: API p99 latency > 2s . Your incident-response agent — the LLM you wired into the observability stack last quarter — does exactly what you asked. It pulls the API service's metrics, reads the last few hundred log lines, grabs a handful of exemplar traces, and concludes with total confidence: the API service is CPU-bound, scale it up. You scale it up. p99 doesn't move. The API service was never the problem. Three hops downstream, a connection pool is quietly handing out dead connections, and every request is blocking on a TCP handshake that will never complete. The agent never looked there, because nothing in the pile of telemetry you handed it said the API depends on this pool. It pattern-matched on the loudest signal, which is almost always the symptom, not the cause. Why raw telemetry defeats the agent The instinct when building an RCA agent is to give it everything: hook it up to Prometheus, Loki, and Tempo, let ...