Your AI Agent Ran a Full Table Scan: Governing What Agents Can Ask the Database

Database CPU sits at 95%. The app's request rate hasn't moved all week. The slow-query log, though, is full of statements no one on the team would ever write by hand:

SELECT * FROM events
WHERE payload::text LIKE '%refund%'
ORDER BY created_at DESC;

A leading-wildcard LIKE on a text-cast JSONB column, across a multi-terabyte table, selecting every column, sorting the entire result set, no LIMIT. It's a guaranteed sequential scan plus a disk sort. And it's running dozens of times an hour.

Then you remember: last week you wired an AI agent into the database through an MCP server, and you gave it a single, flexible tool called execute_sql. The agent isn't broken. It's doing exactly what a general-purpose SQL tool invites it to do — exploring.

The investigation

The first instinct is to blame a traffic spike, but the app's own request metrics are flat. So the load is coming from somewhere that isn't the app. pg_stat_statements settles it quickly — it ranks query fingerprints by total time, independent of who issued them:

SELECT
  calls,
  round(mean_exec_time)   AS mean_ms,
  round(total_exec_time)  AS total_ms,
  rows,
  query
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

The top few rows are the culprits: high mean_ms, modest calls, enormous rows scanned. Running EXPLAIN on one confirms the shape — Seq Scan on the big table, a Sort node spilling to disk, no index in sight. Checking pg_stat_activity during a spike shows the sessions all belong to the role the MCP server connects as. The queries aren't coming from a feature anyone shipped. They're coming from the agent reasoning its way through a question like "has this customer had any refunds?" — and deciding, reasonably enough for a text model, that a wildcard search of the whole events table is a fine way to find out.

This is a different failure mode from the usual "an agent exhausted the connection pool" story, where the queries are cheap but the connections are held across the model's thinking time. Here the connections are fine. The queries themselves are the load.

Root cause: a raw-SQL tool makes the planner's worst case reachable

Every database has a worst case. There is always some query — a full scan, a cartesian join, an unbounded sort — that will saturate CPU or I/O. In a normal application you never hit it, because a human wrote every query, a code reviewer read it, and an index was added before it shipped. The set of query shapes that can reach production is small, reviewed, and effectively frozen.

Handing an agent an execute_sql tool deletes that entire safety property. The set of possible query shapes is now "anything a language model might emit given a schema and a goal" — which is to say, the planner's full worst case is reachable on every turn, at machine speed, with no reviewer in the loop. The model isn't malicious or even bad at SQL. It's that an unbounded tool surface plus a probabilistic generator eventually produces the pathological query, and then produces it again.

You can feel the pull to fix this with the prompt — "only write efficient queries, always add a LIMIT, never use leading wildcards." Don't. A prompt is a suggestion to a sampler, not an invariant. The query surface is the thing to constrain, not the model's intentions.

BEFORE AFTER AI agent execute_sql(raw) any table, any shape PRIMARY seq scan + sort AI agent curated MCP tools typed params + LIMIT statement_timeout read-only role READ REPLICA index-backed The agent fills in blanks; it never writes the FROM clause.

The fix: curated, parameterized tools instead of raw SQL

The architectural move is to stop exposing SQL to the model at all. Instead of one execute_sql god-tool, you expose a small catalog of named tools, each backed by a single vetted statement with typed parameters. The agent chooses a tool and supplies the blanks; it never writes the query body, the FROM clause, or the join.

MCP Toolbox for Databases makes this a config file rather than a service you have to build. Each tool is a statement with bind parameters, and the source it runs against is yours to pick — so you point it at a read replica, not the primary:

sources:
  analytics-ro:
    kind: postgres
    host: 10.0.0.12          # read replica, never the primary
    port: 5432
    database: app
    user: agent_ro
    password: ${AGENT_DB_PASSWORD}

tools:
  recent_refund_events:
    kind: postgres-sql
    source: analytics-ro
    description: >
      Most recent refund events for one customer, newest first.
      Use this to answer questions about a customer's refunds.
    parameters:
      - name: customer_id
        type: string
        description: The customer's UUID.
      - name: max_rows
        type: integer
        description: Rows to return (1-100).
        default: 20
    statement: |
      SELECT id, customer_id, amount_cents, status, created_at
      FROM events
      WHERE type = 'refund'
        AND customer_id = $1
      ORDER BY created_at DESC
      LIMIT LEAST($2, 100);

Every property that made the original query dangerous is now fixed at config time, not left to the sampler. The predicate is an equality on an indexed column instead of a leading-wildcard scan. The column list is explicit. The LIMIT LEAST($2, 100) caps result size even if the agent asks for a million rows. And because the description tells the model what the tool is for, it reliably picks this over improvising — the curated tool is the path of least resistance.

Defense in depth at the role level

Curated tools constrain the common path, but you still want the database itself to refuse anything expensive, so that a future tool with a sloppy statement — or an escape-hatch tool that does allow some ad-hoc querying — can't pin CPU. Push the guardrails onto the role the agent connects as:

CREATE ROLE agent_ro LOGIN PASSWORD '...';
GRANT CONNECT ON DATABASE app TO agent_ro;
GRANT USAGE ON SCHEMA public TO agent_ro;
-- grant SELECT only on the tables the tools actually touch
GRANT SELECT ON events, customers TO agent_ro;

ALTER ROLE agent_ro SET statement_timeout = '5s';
ALTER ROLE agent_ro SET idle_in_transaction_session_timeout = '10s';
ALTER ROLE agent_ro SET default_transaction_read_only = on;

Now the worst a single query can do is run for five seconds before the server cancels it. The role can't write, can't touch tables outside its grants, and can't hold a transaction open across the model's thinking time. These are the same hygiene settings you'd reach for against any untrusted caller — the agent is just the newest one.

When you genuinely need ad-hoc SQL

Sometimes the whole point of the agent is open-ended data exploration, and a fixed catalog of tools defeats it. If you must expose an ad-hoc query tool, gate it rather than trust it: run the statement through EXPLAIN (FORMAT JSON) first, parse the top-level Total Cost, and reject the query before executing it if the estimate blows a threshold:

plan = run("EXPLAIN (FORMAT JSON) " + sql)        # cheap; doesn't execute
cost = plan[0]["Plan"]["Total Cost"]
if cost > MAX_COST:
    return f"Query rejected: estimated cost {cost:.0f} exceeds {MAX_COST}. " \
           "Add a more selective filter or a smaller time range."

Handing that rejection message back to the agent as a tool result is usually enough — it reads the feedback, adds a predicate, and tries again, which is exactly the loop you want it in. The expensive query never reaches the executor, and the statement_timeout is there as a backstop for anything the cost estimate underestimates.

Watching for the next one

Because the agent's query shapes evolve as it's used, the useful monitor is "a query fingerprint we've never seen before just entered the top of pg_stat_statements." That's a cheap thing to diff on a schedule, and it's the kind of pattern-change signal an AIOps or anomaly-detection layer flags well — a brand-new, high-cost fingerprint from the agent's role is worth an alert long before it shows up as a CPU page. The point isn't to catch the model being clever; it's to notice the query surface drifting and pull the newcomer into a curated tool before it becomes load.

Lesson

An agent with a raw-SQL tool turns your database's worst case into something a text generator can reach on every turn, and no prompt makes that surface safe. Constrain the surface instead: expose a small catalog of parameterized, result-capped tools against a read-only replica, and let the role enforce timeouts and least privilege underneath. The model should be choosing which question to ask — never writing the FROM clause.


Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.

Comments

Popular posts from this blog

Performance Testing 102: Little's Law and It's usage in Performance Testing

Performance Testing 104: Workload Modelling Designing & Process

Mastering the Art of Scaling in SaaS Applications