Your AI Agent Ran a Full Table Scan: Governing What Agents Can Ask the Database
Database CPU sits at 95%. The app's request rate hasn't moved all week. The slow-query log, though, is full of statements no one on the team would ever write by hand:
SELECT * FROM events
WHERE payload::text LIKE '%refund%'
ORDER BY created_at DESC;
A leading-wildcard LIKE on a text-cast JSONB column, across a
multi-terabyte table, selecting every column, sorting the entire result set,
no LIMIT. It's a guaranteed sequential scan plus a disk sort.
And it's running dozens of times an hour.
Then you remember: last week you wired an AI agent into the database
through an MCP server, and you gave it a single, flexible tool called
execute_sql. The agent isn't broken. It's doing exactly what a
general-purpose SQL tool invites it to do — exploring.
The investigation
The first instinct is to blame a traffic spike, but the app's own request
metrics are flat. So the load is coming from somewhere that isn't the app.
pg_stat_statements settles it quickly — it ranks query
fingerprints by total time, independent of who issued them:
SELECT
calls,
round(mean_exec_time) AS mean_ms,
round(total_exec_time) AS total_ms,
rows,
query
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;
The top few rows are the culprits: high mean_ms, modest
calls, enormous rows scanned. Running
EXPLAIN on one confirms the shape — Seq Scan on the
big table, a Sort node spilling to disk, no index in sight.
Checking pg_stat_activity during a spike shows the sessions all
belong to the role the MCP server connects as. The queries aren't coming
from a feature anyone shipped. They're coming from the agent reasoning its
way through a question like "has this customer had any refunds?" — and
deciding, reasonably enough for a text model, that a wildcard search of the
whole events table is a fine way to find out.
This is a different failure mode from the usual "an agent exhausted the connection pool" story, where the queries are cheap but the connections are held across the model's thinking time. Here the connections are fine. The queries themselves are the load.
Root cause: a raw-SQL tool makes the planner's worst case reachable
Every database has a worst case. There is always some query — a full scan, a cartesian join, an unbounded sort — that will saturate CPU or I/O. In a normal application you never hit it, because a human wrote every query, a code reviewer read it, and an index was added before it shipped. The set of query shapes that can reach production is small, reviewed, and effectively frozen.
Handing an agent an execute_sql tool deletes that entire
safety property. The set of possible query shapes is now "anything a
language model might emit given a schema and a goal" — which is to say, the
planner's full worst case is reachable on every turn, at machine speed, with
no reviewer in the loop. The model isn't malicious or even bad at SQL. It's
that an unbounded tool surface plus a probabilistic generator eventually
produces the pathological query, and then produces it again.
You can feel the pull to fix this with the prompt — "only write efficient queries, always add a LIMIT, never use leading wildcards." Don't. A prompt is a suggestion to a sampler, not an invariant. The query surface is the thing to constrain, not the model's intentions.
The fix: curated, parameterized tools instead of raw SQL
The architectural move is to stop exposing SQL to the model at all. Instead
of one execute_sql god-tool, you expose a small catalog of
named tools, each backed by a single vetted statement with typed parameters.
The agent chooses a tool and supplies the blanks; it never writes the query
body, the FROM clause, or the join.
MCP Toolbox for Databases makes this a config file rather than a service you have to build. Each tool is a statement with bind parameters, and the source it runs against is yours to pick — so you point it at a read replica, not the primary:
sources:
analytics-ro:
kind: postgres
host: 10.0.0.12 # read replica, never the primary
port: 5432
database: app
user: agent_ro
password: ${AGENT_DB_PASSWORD}
tools:
recent_refund_events:
kind: postgres-sql
source: analytics-ro
description: >
Most recent refund events for one customer, newest first.
Use this to answer questions about a customer's refunds.
parameters:
- name: customer_id
type: string
description: The customer's UUID.
- name: max_rows
type: integer
description: Rows to return (1-100).
default: 20
statement: |
SELECT id, customer_id, amount_cents, status, created_at
FROM events
WHERE type = 'refund'
AND customer_id = $1
ORDER BY created_at DESC
LIMIT LEAST($2, 100);
Every property that made the original query dangerous is now fixed at
config time, not left to the sampler. The predicate is an equality on an
indexed column instead of a leading-wildcard scan. The column list is
explicit. The LIMIT LEAST($2, 100) caps result size even if the
agent asks for a million rows. And because the description tells the model
what the tool is for, it reliably picks this over improvising — the
curated tool is the path of least resistance.
Defense in depth at the role level
Curated tools constrain the common path, but you still want the database itself to refuse anything expensive, so that a future tool with a sloppy statement — or an escape-hatch tool that does allow some ad-hoc querying — can't pin CPU. Push the guardrails onto the role the agent connects as:
CREATE ROLE agent_ro LOGIN PASSWORD '...';
GRANT CONNECT ON DATABASE app TO agent_ro;
GRANT USAGE ON SCHEMA public TO agent_ro;
-- grant SELECT only on the tables the tools actually touch
GRANT SELECT ON events, customers TO agent_ro;
ALTER ROLE agent_ro SET statement_timeout = '5s';
ALTER ROLE agent_ro SET idle_in_transaction_session_timeout = '10s';
ALTER ROLE agent_ro SET default_transaction_read_only = on;
Now the worst a single query can do is run for five seconds before the server cancels it. The role can't write, can't touch tables outside its grants, and can't hold a transaction open across the model's thinking time. These are the same hygiene settings you'd reach for against any untrusted caller — the agent is just the newest one.
When you genuinely need ad-hoc SQL
Sometimes the whole point of the agent is open-ended data exploration, and
a fixed catalog of tools defeats it. If you must expose an ad-hoc query tool,
gate it rather than trust it: run the statement through EXPLAIN (FORMAT
JSON) first, parse the top-level Total Cost, and reject
the query before executing it if the estimate blows a threshold:
plan = run("EXPLAIN (FORMAT JSON) " + sql) # cheap; doesn't execute
cost = plan[0]["Plan"]["Total Cost"]
if cost > MAX_COST:
return f"Query rejected: estimated cost {cost:.0f} exceeds {MAX_COST}. " \
"Add a more selective filter or a smaller time range."
Handing that rejection message back to the agent as a tool result is
usually enough — it reads the feedback, adds a predicate, and tries again,
which is exactly the loop you want it in. The expensive query never reaches
the executor, and the statement_timeout is there as a backstop
for anything the cost estimate underestimates.
Watching for the next one
Because the agent's query shapes evolve as it's used, the useful monitor is
"a query fingerprint we've never seen before just entered the top of
pg_stat_statements." That's a cheap thing to diff on a schedule,
and it's the kind of pattern-change signal an AIOps or anomaly-detection
layer flags well — a brand-new, high-cost fingerprint from the agent's role
is worth an alert long before it shows up as a CPU page. The point isn't to
catch the model being clever; it's to notice the query surface drifting and
pull the newcomer into a curated tool before it becomes load.
Lesson
An agent with a raw-SQL tool turns your database's worst case into
something a text generator can reach on every turn, and no prompt makes that
surface safe. Constrain the surface instead: expose a small catalog of
parameterized, result-capped tools against a read-only replica, and let the
role enforce timeouts and least privilege underneath. The model should be
choosing which question to ask — never writing the FROM
clause.
Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.
Comments
Post a Comment