Your RCA Agent Blamed the Database. The Database Was Fine.

The alert that lies to you

The page fires at 02:14: API p99 latency > 2s. Your incident-response agent — the LLM you wired into the observability stack last quarter — does exactly what you asked. It pulls the API service's metrics, reads the last few hundred log lines, grabs a handful of exemplar traces, and concludes with total confidence: the API service is CPU-bound, scale it up.

You scale it up. p99 doesn't move. The API service was never the problem. Three hops downstream, a connection pool is quietly handing out dead connections, and every request is blocking on a TCP handshake that will never complete. The agent never looked there, because nothing in the pile of telemetry you handed it said the API depends on this pool. It pattern-matched on the loudest signal, which is almost always the symptom, not the cause.

Why raw telemetry defeats the agent

The instinct when building an RCA agent is to give it everything: hook it up to Prometheus, Loki, and Tempo, let it query freely, trust the model to find the needle. It doesn't work, for a reason that has nothing to do with how smart the model is.

Consider what "everything" weighs. A single service under load emits thousands of active time series; an hour of logs for one busy pod is easily tens of megabytes; a trace-heavy request path produces spans by the thousand per second. You cannot fit that into a context window, so you sample — and sampling throws away exactly the rare event you're hunting. What survives the truncation is the high-volume, high-amplitude signal: the elevated latency at the edge, the retries piling up. That is the symptom. The cause is usually a small, quiet anomaly on a service the agent was never even told to look at.

Feeding an agent more telemetry gives it volume, not causality. Correlation in a dashboard is not causation, and an LLM staring at correlated graphs will confidently narrate the correlation. The missing input isn't data. It's structure.

The missing input is topology

Watch how an experienced SRE actually debugs the same page. They do not read all the telemetry. They start at the alerting service, ask "what does this call, and what calls it," and walk that dependency graph one hop at a time, pruning healthy branches until they reach the service where the trouble originates. The map lives in their head. The agent has no such map — so give it one.

A knowledge graph is that map made queryable: entities (services, pods, deployments, databases, connection pools, nodes) as vertices, and relationships (calls, runs-on, connects-to, owns, deployed-by) as edges. Crucially, the edges you need are already latent in signals you're collecting:

CALLS edges come straight from OpenTelemetry span data — the service-graph / span-metrics connector emits caller/callee pairs.
RUNS_ON and DEPLOYED_BY come from the Kubernetes API — pods to nodes, pods to deployments.
CONNECTS_TO edges (service to pool to database) come from your config and instrumentation.

Modeling the graph

You don't need a bespoke graph database to start; a property-graph model expressed in Cypher (Neo4j, Memgraph) or even a well-indexed relational schema is enough. The shape matters more than the engine. A minimal model:

// Entities
(:Service {name})
(:ConnectionPool {name, max_size})
(:Database {name, engine})

// Relationships, populated from OTel + k8s
(api:Service {name:"api"})       -[:CALLS]->      (orders:Service {name:"orders"})
(orders:Service {name:"orders"}) -[:CONNECTS_TO]->(pool:ConnectionPool {name:"orders-pg-pool"})
(pool:ConnectionPool)            -[:BACKS]->      (db:Database {name:"orders-pg"})

// The query a human runs in their head, made explicit:
// "starting from the alerting service, what is reachable downstream?"
MATCH path = (start:Service {name:"api"})-[:CALLS|CONNECTS_TO|BACKS*1..5]->(dep)
RETURN path

That last query is the whole trick. It returns the blast-radius subgraph downstream of the symptom — a dozen nodes instead of ten thousand time series. That is a payload an agent can actually reason over.

Giving the agent a graph to walk

Instead of a "dump me all the metrics" tool, expose the graph as a traversal tool over something like MCP. The agent asks for neighbors, not for oceans of data:

{
  "name": "get_downstream_dependencies",
  "description": "Given a service, return its direct downstream dependencies (services, pools, databases) with a one-line health summary for each.",
  "input_schema": {
    "type": "object",
    "properties": {
      "entity": { "type": "string", "description": "entity name, e.g. 'api'" },
      "depth":  { "type": "integer", "default": 1, "maximum": 3 }
    },
    "required": ["entity"]
  }
}

Now the loop looks like disciplined debugging rather than a fishing expedition:

Start at the alerting node.
Fetch its immediate downstream neighbors from the graph.
For each neighbor, pull a small, targeted health slice — error rate, saturation, a saturation-specific metric like pool utilization — not the full firehose.
Prune the healthy branches. Follow the unhealthy edge one hop deeper.
Stop when a node is unhealthy but all of its dependencies are healthy. That node is your root cause.

The graph does the pruning; the telemetry is fetched lazily, one node at a time, so the context window never fills with noise. The diagram below shows a single walk: the alert lands on api, but the traversal steps past a healthy auth branch and follows the sick edge down to a saturated pool.

A concrete walk: from an edge timeout to a saturated pool

This isn't hypothetical for me. Early in my career I ran a web API backend on GCP that started fine and then, after it had been under load for a while, went from merely slow to timing out — not at startup, but well into a busy stretch. The alerting metric was request latency at the edge, which is to say: the loudest, least useful signal.

Finding the cause was manual dashboard archaeology. I checked the database's active connection count against the pool's configured max, dug through the pool's own metrics and logs, correlated the timeout spike against the latency and error-rate graphs, and finally reproduced it deliberately under load. Two things were compounding: the pool was sized for an estimate of load rather than the real concurrency, and it was handing out stale, dead connections without validating them first. The fix was unglamorous — size the pool to measured concurrency, add connection validation and eviction, and set a connect timeout so a request failed fast instead of hanging forever on a corpse of a connection.

None of that investigation required genius. It required knowing that api depends on orders, that orders connects through a pool, and that a pool is a resource that can saturate independently of the database behind it. That is precisely the knowledge a topology graph encodes. A graph-walking agent hitting this today would step from the edge alert to orders (slow but not saturated), then to the pool (utilization pegged at 100%, connection age climbing), then check the database behind it (healthy) — and stop, because the unhealthy node whose own dependencies are all healthy is the answer. Minutes of traversal instead of an hour of correlating graphs by eye.

What this doesn't fix

A knowledge graph makes localization tractable; it does not make it infallible, and it's worth being honest about the sharp edges:

A stale graph lies as confidently as stale telemetry. If a new dependency isn't reflected in your CALLS edges, the agent will prune the very branch the cause lives on. The graph has to be rebuilt continuously from live signals, not hand-maintained.
Localization is not remediation. The agent can correctly find the saturated pool and still propose a bad fix. Pointing at the right node is the win; deciding what to do about it still deserves human judgment.
Some failures aren't on the dependency graph at all — a noisy neighbor on the same node, a bad kernel, clock skew. You need RUNS_ON and infrastructure edges too, or those causes stay invisible.

The direction is promising, and not just anecdotally. Grafana's team recently replayed a single real incident 16 times each way and reported that an agent given knowledge-graph context localized the cause far more consistently than the same agent working from raw telemetry alone. That matches the intuition: the graph isn't smarter than the model, it just stops the model from having to guess at structure it was never given.

Lesson

An incident agent is only as good as the structure you hand it, and telemetry alone has no structure — it's a haystack with the causality removed. The lever that turns an LLM from a confident guesser into a useful first responder isn't a bigger context window or a better model; it's a fresh, machine-readable map of what depends on what. Build the graph first, then let the agent walk it. The one that reads all your telemetry will keep blaming the database, and the database will keep being fine.

Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.

Search This Blog

Lets Architect the Performance Engineering