Your AI Agent Spends 8 Seconds Booting a Sandbox to Run 300ms of Code

An agent answers a question that requires it to run a three-line Python snippet — parse a CSV, sum a column, return the total. The code runs in about 300 milliseconds. The user waits eight seconds. Trace the request and the actual computation is a rounding error; everything else is the agent waiting for somewhere to run the code.

The bill tells the same story from the other side. You provisioned a generously sized pool of execution environments so agents never queue, and the utilization dashboard shows them sitting idle most of the day, warm and billable, waiting for the next tool call. You are paying full compute prices to keep empty rooms lit. Both symptoms — the latency and the cost — come from the same design decision: one fresh, full-sized sandbox per agent, cold on the way in and idle on the way out.

Where the eight seconds go

An agent that executes code or calls tools needs an isolated environment to do it in — you cannot run model-generated code in the same process as your orchestrator and hope for the best. The obvious implementation is one pod or one VM per session, created on demand. That obvious implementation is where the latency lives. Break down a cold start and it is almost never the workload:

# Time-to-usable-sandbox for a cold "one pod per session" design
# (illustrative breakdown, not measured telemetry)
scheduling        1.9s   # find a node with capacity, bind the pod
image pull        3.4s   # 1.2 GB runtime image, cold layer cache
runtime init      0.8s   # container start + language runtime boot
dependency import 1.6s   # first import of heavy libs (pandas, numpy, ...)
-----------------------
usable sandbox    7.7s
actual work       0.3s   # the code the agent actually wanted to run

Every line above the divider is fixed overhead the agent pays before it can do anything, and it pays it again on the next session because the environment was torn down in between. The single biggest contributor is usually the image pull on a cold node, followed by whatever the runtime does on first use — importing a scientific-Python stack alone can cost more than a second. None of it is the agent's fault, and none of it is fixable by making the model faster.

Root cause: a full machine per turn, held idle between turns

The naive design gets the worst of both ends. It is slow because each session starts from nothing, and it is expensive because the fix people reach for — keep the sandboxes running so they are warm — means paying for idle capacity between turns, which for a conversational agent is most of the wall-clock time. A human takes seconds to read a response and type the next message; the sandbox sits there the whole time, reserved and billed.

The way out is to stop treating "isolated," "fast to start," and "cheap when idle" as if they were one setting. They are three separate levers, and you tune them independently.

Lever 1: pick the isolation that matches the threat model

Running untrusted, model-generated code is the whole reason the sandbox exists, so the isolation boundary is not optional — but "boundary" is a spectrum, not a binary, and each point costs differently.

Plain containers share the host kernel. Fast to start, cheap, and the wrong choice for arbitrary untrusted code — a container escape is a host compromise.
gVisor (runsc) puts a user-space kernel between the workload and the host, intercepting syscalls. Much stronger isolation than a container, no full VM, but syscall-heavy workloads pay an interception tax.
microVMs (Firecracker, or containers-in-VMs via Kata) give each sandbox its own kernel behind a hardware virtualization boundary. Firecracker strips the device model down far enough that a microVM boots in around 125 ms — the property that lets it back AWS Lambda and Fargate — so you get VM-grade isolation without VM-grade startup.

In Kubernetes this is a RuntimeClass, chosen per workload rather than per cluster, so your trusted internal jobs keep running on the default runtime and only the untrusted agent sandboxes pay for stronger isolation:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc          # or "kata" for a microVM-backed runtime
---
apiVersion: v1
kind: Pod
metadata:
  name: agent-sandbox
spec:
  runtimeClassName: gvisor   # this pod runs sandboxed; others don't
  containers:
    - name: exec
      image: registry.internal/agent-runtime:pinned

Lever 2: stop cold-starting — snapshot and pre-warm

The image pull and the runtime init only have to happen the first time. The trick that makes agent sandboxes feel instant is to reach a ready-to-work state once — kernel booted, language runtime up, heavy libraries already imported — then snapshot the memory and device state of that microVM and restore from the snapshot per session. Restoring a snapshot is a memory-map, not a boot, so it lands in tens of milliseconds and the imported libraries are already resident. This is exactly how services like E2B and Modal hand an agent a warm Python environment in well under a second; Firecracker exposes the same snapshot/restore primitive directly.

Pair that with a small warm pool so a request never waits on even the snapshot restore. Keep N ready sandboxes, hand one out on demand, and replenish asynchronously:

# A warm pool: check out an already-ready sandbox, refill in the background.
async def acquire(pool):
    try:
        sb = pool.ready.get_nowait()          # instant: already restored & warm
    except asyncio.QueueEmpty:
        sb = await restore_from_snapshot()    # rare fallback under a burst
    asyncio.create_task(refill(pool))         # replace what we just took
    return sb

async def refill(pool):
    if pool.ready.qsize() < pool.target:
        pool.ready.put_nowait(await restore_from_snapshot())

The warm-pool size is the one real capacity-planning decision here: set it to your p95 concurrent-session count, not your peak, and let the snapshot-restore fallback absorb the rare burst above that. Sizing for peak is how you end up back at the idle-cost problem.

Lever 3: stop paying for idle — pause and scale to zero

A warm sandbox mid-conversation is not doing work; it is waiting for the human to type. That is the idle you are paying for. A microVM can be frozen: its vCPUs stop consuming host CPU and its state is held in (or snapshotted out of) memory, then resumed in milliseconds when the next tool call arrives. Fly Machines expose this as auto-stop/auto-start; a snapshot-based runtime does it by snapshotting on pause and restoring on resume. The session survives across turns without holding a running CPU between them.

At the fleet level the same logic drives the pool itself to zero when no agents are active — overnight, on weekends. With KEDA that is declarative, off a queue depth or a request-rate metric:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
spec:
  scaleTargetRef:
    name: agent-sandbox-pool
  minReplicaCount: 0          # no traffic -> no warm pool -> no bill
  cooldownPeriod: 300         # keep the pool warm 5 min after the last request
  triggers:
    - type: prometheus
      metadata:
        query: sum(agent_sessions_active)
        threshold: "1"

The order that pays off

Work these in sequence, because each changes the number the next one sees. First pick the isolation boundary — it constrains everything downstream, and retrofitting stronger isolation later means re-benchmarking. Then kill the cold start with a snapshot and a warm pool, which is what turns the eight seconds into sub-second and is the change your users feel. Only then tune pause and scale-to-zero, because idle cost is meaningless until sessions are cheap to resume; pausing a sandbox that takes eight seconds to wake up just moves the latency, it doesn't remove it. Done in that order, the same fleet serves more concurrent agents on less standing capacity, and the per-tool-call latency stops being dominated by furniture.

The lesson

"Isolated," "fast to start," and "cheap when idle" feel like one requirement and are three. Collapse them into a single "one pod per session" design and you lose on all three at once — slow starts, weak-or-expensive isolation, and a bill full of idle. Treat them as separate levers — a runtime class for the boundary, snapshots and a warm pool for startup, pause and scale-to-zero for idle — and each gets cheap on its own terms. The agent's real work was always 300 milliseconds; the engineering is making sure that's most of what the user waits for.

Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.

Search This Blog

Lets Architect the Performance Engineering