Your RCA Agent Blamed the Database. The Database Was Fine.
The alert that lies to you The page fires at 02:14: API p99 latency > 2s . Your incident-response agent — the LLM you wired into the observability stack last quarter — does exactly what you asked. It pulls the API service's metrics, reads the last few hundred log lines, grabs a handful of exemplar traces, and concludes with total confidence: the API service is CPU-bound, scale it up. You scale it up. p99 doesn't move. The API service was never the problem. Three hops downstream, a connection pool is quietly handing out dead connections, and every request is blocking on a TCP handshake that will never complete. The agent never looked there, because nothing in the pile of telemetry you handed it said the API depends on this pool. It pattern-matched on the loudest signal, which is almost always the symptom, not the cause. Why raw telemetry defeats the agent The instinct when building an RCA agent is to give it everything: hook it up to Prometheus, Loki, and Tempo, let ...