Your Agent Issued the Refund Twice: Durable Side Effects with the Transactional Outbox
Your support agent decides a customer is owed a refund. It calls the
issue_refund tool, the payment provider returns 200, and the
agent writes status = "refunded" to its own task row. Clean.
Except on about one run in a few thousand, the customer gets refunded
twice — or, on a different unlucky run, the agent swears it
refunded them and the money never moved.
Nothing in the model is broken. The prompt is fine, the tool schema is fine, the eval suite is green. What's broken is older and more boring than anything about LLMs: the agent is doing two writes to two different systems and pretending they happen together. They don't.
The symptom
You notice it from a reconciliation job, not from the agent. A finance export shows two refund transactions against one order ID, minutes apart, both tagged with the same agent run. Or a customer emails back asking where their money is, and the trace shows the tool call succeeded. When you pull the logs for the double case, you see something like this:
12:04:51 run=a91f tool=issue_refund order=5567 -> provider 200 (txn PR-8841)
12:04:52 run=a91f WRITE task status=refunded -> ERROR conn reset
12:05:30 run=a91f RETRY step: issue_refund order=5567 -> provider 200 (txn PR-8842)
12:05:31 run=a91f WRITE task status=refunded -> OK
The provider charged the refund, the follow-up write to mark the task
done failed, the orchestrator retried the step from the last durable
checkpoint — and the step included the side effect. Second refund issued.
The mirror image happens when the ordering is flipped: you record
refunded first, then crash before the provider call lands, and
now your state lies about an action that never happened.
Why agents walk into this so reliably
This is the dual-write problem, and distributed systems people have hit it for decades: when a single logical action has to touch two stores that don't share a transaction — here, the payment provider and your own database — there is no atomic way to do both. Any crash, timeout, or network blip in the gap between them leaves the two disagreeing.
Agent frameworks make it worse in two specific ways. First, their whole
execution model is resumable: durable orchestrators (Temporal,
Step Functions, LangGraph with a checkpointer, Spanner-backed agent state)
replay a workflow from the last saved step after a failure. Replay is
exactly what you want for a pure computation and exactly what you must not
do for a POST /refunds. Second, the model itself is
nondeterministic — re-running a step can produce a slightly
different tool call, so even naive dedup on "same arguments" misses it.
The failure window is small, which is why your eval suite never caught it. It only shows up under real concurrency and real partial failures — the exact conditions you don't reproduce in a notebook.
The fix: make the decision and the record of it atomic
The move is the transactional outbox. Instead of calling the side effect inline from the agent step, the step does one thing it can do atomically: it writes its state change and a row describing the intended action into the same database, in a single transaction. A separate relay process reads those outbox rows and performs the real side effect. The agent never calls the payment provider directly.
CREATE TABLE action_outbox (
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
run_id TEXT NOT NULL,
step_id TEXT NOT NULL, -- stable per logical step
action_type TEXT NOT NULL, -- 'issue_refund'
payload JSONB NOT NULL, -- {order_id, amount, currency}
idem_key TEXT NOT NULL, -- derived, see below
status TEXT NOT NULL DEFAULT 'pending', -- pending|sent
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (run_id, step_id) -- a step enqueues its action once
);
The agent step and the outbox insert share one transaction. Because of
the UNIQUE (run_id, step_id) constraint, a replay of the same
logical step is a no-op — the insert conflicts and is swallowed — even if
the model produced slightly different wording on the retry. The state write
and the "I intend to do X" record now live or die together:
def run_refund_step(conn, run_id, step_id, order):
idem_key = f"{run_id}:{step_id}:refund:{order['id']}"
with conn.transaction(): # one atomic unit
conn.execute(
"UPDATE agent_task SET status = 'refund_enqueued' "
"WHERE run_id = %s", (run_id,))
conn.execute(
"INSERT INTO action_outbox "
" (run_id, step_id, action_type, payload, idem_key) "
"VALUES (%s, %s, 'issue_refund', %s, %s) "
"ON CONFLICT (run_id, step_id) DO NOTHING",
(run_id, step_id, Json(order), idem_key))
# NOTE: no provider call here. The step is now pure + replay-safe.
A relay polls pending rows, performs the real call, and
marks them sent. This is deliberately at-least-once:
if the relay crashes after the provider call but before the status update,
it will retry the same row. That's fine, because the last piece closes the
loop — the provider must treat the idempotency key as a dedup token:
def relay_once(conn, provider):
rows = conn.execute(
"SELECT id, payload, idem_key FROM action_outbox "
"WHERE status = 'pending' ORDER BY id "
"LIMIT 100 FOR UPDATE SKIP LOCKED").fetchall()
for row in rows:
# Idempotency-Key makes a duplicate POST return the ORIGINAL
# result instead of charging a second refund.
provider.refund(row.payload, idempotency_key=row.idem_key)
conn.execute(
"UPDATE action_outbox SET status = 'sent' WHERE id = %s",
(row.id,))
FOR UPDATE SKIP LOCKED lets you run several relay workers
without them fighting over the same rows — standard queue-over-SQL
hygiene. Every serious payment API (Stripe, Adyen, PayPal) honors an
Idempotency-Key header for exactly this reason; internal
services should too. At-least-once delivery plus a dedup key on the
receiving end gives you effectively once, which is the strongest
honest guarantee available across two systems.
Where the newer infra fits
The reason this pattern is worth re-raising now is that the "two
systems" seam is showing up in every agentic stack: the agent keeps state
in an operational store and dispatches actions through a separate queue, and
teams wire those together with a naive dual write. Some databases are
starting to collapse the seam — Spanner's transactional queues, for
instance, let you enqueue a message in the same transaction that mutates
your rows, which is the outbox pattern turned into a native primitive and
removes the hand-rolled relay. Postgres users get close with
LISTEN/NOTIFY or logical decoding to drive the relay instead
of polling. The principle is identical regardless of the plumbing: the
state change and the intent-to-act commit together, and delivery is
idempotent.
Two guardrails worth adding while you're here. Keep a hard allowlist of
which action_type values the relay will execute, so a
prompt-injected or hallucinated action type fails closed rather than firing
an arbitrary side effect. And alert on outbox age — a pending
row older than a few minutes means the relay is wedged, and that is a far
cheaper page than a finance reconciliation finding it a day later.
Lesson
"The agent decided to issue a refund" and "a refund happened exactly once" are two different guarantees, and the gap between them is a classic dual-write race that resumable, nondeterministic agents hit more often than ordinary services. Don't try to make the model more careful about it — remove the dual write. Make the state change and the record of the intended action commit in one transaction, move the side effect behind an at-least-once relay, and make the downstream call idempotent. The fix predates LLMs by twenty years; the agents just gave it fresh ways to bite.
Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.
Comments
Post a Comment