Your Agent Issued the Refund Twice: Durable Side Effects with the Transactional Outbox

Your support agent decides a customer is owed a refund. It calls the issue_refund tool, the payment provider returns 200, and the agent writes status = "refunded" to its own task row. Clean. Except on about one run in a few thousand, the customer gets refunded twice — or, on a different unlucky run, the agent swears it refunded them and the money never moved.

Nothing in the model is broken. The prompt is fine, the tool schema is fine, the eval suite is green. What's broken is older and more boring than anything about LLMs: the agent is doing two writes to two different systems and pretending they happen together. They don't.

The symptom

You notice it from a reconciliation job, not from the agent. A finance export shows two refund transactions against one order ID, minutes apart, both tagged with the same agent run. Or a customer emails back asking where their money is, and the trace shows the tool call succeeded. When you pull the logs for the double case, you see something like this:

12:04:51  run=a91f  tool=issue_refund  order=5567  -> provider 200 (txn PR-8841)
12:04:52  run=a91f  WRITE task status=refunded             -> ERROR conn reset
12:05:30  run=a91f  RETRY step: issue_refund  order=5567   -> provider 200 (txn PR-8842)
12:05:31  run=a91f  WRITE task status=refunded             -> OK

The provider charged the refund, the follow-up write to mark the task done failed, the orchestrator retried the step from the last durable checkpoint — and the step included the side effect. Second refund issued. The mirror image happens when the ordering is flipped: you record refunded first, then crash before the provider call lands, and now your state lies about an action that never happened.

Why agents walk into this so reliably

This is the dual-write problem, and distributed systems people have hit it for decades: when a single logical action has to touch two stores that don't share a transaction — here, the payment provider and your own database — there is no atomic way to do both. Any crash, timeout, or network blip in the gap between them leaves the two disagreeing.

Agent frameworks make it worse in two specific ways. First, their whole execution model is resumable: durable orchestrators (Temporal, Step Functions, LangGraph with a checkpointer, Spanner-backed agent state) replay a workflow from the last saved step after a failure. Replay is exactly what you want for a pure computation and exactly what you must not do for a POST /refunds. Second, the model itself is nondeterministic — re-running a step can produce a slightly different tool call, so even naive dedup on "same arguments" misses it.

The failure window is small, which is why your eval suite never caught it. It only shows up under real concurrency and real partial failures — the exact conditions you don't reproduce in a notebook.

Dual write (racy) agent step payments task DB crash between the two = double or lost refund 1 2 Transactional outbox agent step task DB + outbox one transaction relay -> queue at-least-once consumer: idempotency key on provider at-least-once + dedup = effectively once

The fix: make the decision and the record of it atomic

The move is the transactional outbox. Instead of calling the side effect inline from the agent step, the step does one thing it can do atomically: it writes its state change and a row describing the intended action into the same database, in a single transaction. A separate relay process reads those outbox rows and performs the real side effect. The agent never calls the payment provider directly.

CREATE TABLE action_outbox (
  id           BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  run_id       TEXT        NOT NULL,
  step_id      TEXT        NOT NULL,     -- stable per logical step
  action_type  TEXT        NOT NULL,     -- 'issue_refund'
  payload      JSONB       NOT NULL,     -- {order_id, amount, currency}
  idem_key     TEXT        NOT NULL,     -- derived, see below
  status       TEXT        NOT NULL DEFAULT 'pending',  -- pending|sent
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE (run_id, step_id)               -- a step enqueues its action once
);

The agent step and the outbox insert share one transaction. Because of the UNIQUE (run_id, step_id) constraint, a replay of the same logical step is a no-op — the insert conflicts and is swallowed — even if the model produced slightly different wording on the retry. The state write and the "I intend to do X" record now live or die together:

def run_refund_step(conn, run_id, step_id, order):
    idem_key = f"{run_id}:{step_id}:refund:{order['id']}"
    with conn.transaction():                 # one atomic unit
        conn.execute(
            "UPDATE agent_task SET status = 'refund_enqueued' "
            "WHERE run_id = %s", (run_id,))
        conn.execute(
            "INSERT INTO action_outbox "
            "  (run_id, step_id, action_type, payload, idem_key) "
            "VALUES (%s, %s, 'issue_refund', %s, %s) "
            "ON CONFLICT (run_id, step_id) DO NOTHING",
            (run_id, step_id, Json(order), idem_key))
    # NOTE: no provider call here. The step is now pure + replay-safe.

A relay polls pending rows, performs the real call, and marks them sent. This is deliberately at-least-once: if the relay crashes after the provider call but before the status update, it will retry the same row. That's fine, because the last piece closes the loop — the provider must treat the idempotency key as a dedup token:

def relay_once(conn, provider):
    rows = conn.execute(
        "SELECT id, payload, idem_key FROM action_outbox "
        "WHERE status = 'pending' ORDER BY id "
        "LIMIT 100 FOR UPDATE SKIP LOCKED").fetchall()
    for row in rows:
        # Idempotency-Key makes a duplicate POST return the ORIGINAL
        # result instead of charging a second refund.
        provider.refund(row.payload, idempotency_key=row.idem_key)
        conn.execute(
            "UPDATE action_outbox SET status = 'sent' WHERE id = %s",
            (row.id,))

FOR UPDATE SKIP LOCKED lets you run several relay workers without them fighting over the same rows — standard queue-over-SQL hygiene. Every serious payment API (Stripe, Adyen, PayPal) honors an Idempotency-Key header for exactly this reason; internal services should too. At-least-once delivery plus a dedup key on the receiving end gives you effectively once, which is the strongest honest guarantee available across two systems.

Where the newer infra fits

The reason this pattern is worth re-raising now is that the "two systems" seam is showing up in every agentic stack: the agent keeps state in an operational store and dispatches actions through a separate queue, and teams wire those together with a naive dual write. Some databases are starting to collapse the seam — Spanner's transactional queues, for instance, let you enqueue a message in the same transaction that mutates your rows, which is the outbox pattern turned into a native primitive and removes the hand-rolled relay. Postgres users get close with LISTEN/NOTIFY or logical decoding to drive the relay instead of polling. The principle is identical regardless of the plumbing: the state change and the intent-to-act commit together, and delivery is idempotent.

Two guardrails worth adding while you're here. Keep a hard allowlist of which action_type values the relay will execute, so a prompt-injected or hallucinated action type fails closed rather than firing an arbitrary side effect. And alert on outbox age — a pending row older than a few minutes means the relay is wedged, and that is a far cheaper page than a finance reconciliation finding it a day later.

Lesson

"The agent decided to issue a refund" and "a refund happened exactly once" are two different guarantees, and the gap between them is a classic dual-write race that resumable, nondeterministic agents hit more often than ordinary services. Don't try to make the model more careful about it — remove the dual write. Make the state change and the record of the intended action commit in one transaction, move the side effect behind an at-least-once relay, and make the downstream call idempotent. The fix predates LLMs by twenty years; the agents just gave it fresh ways to bite.


Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.

Comments

Popular posts from this blog

Performance Testing 102: Little's Law and It's usage in Performance Testing

Performance Testing 104: Workload Modelling Designing & Process

Boosting Apache Web Server Performance: A Comprehensive Guide to Tuning and Optimizing