Your Temporal Workers Are at 20% CPU and the Agent Queue Is 10 Minutes Deep
A batch of agent jobs lands at the top of the hour. Workflows that normally finish in a few seconds sit there for ten minutes doing nothing. You open the worker dashboard braced for a fire — pegged CPU, OOM kills, thrashing — and instead everything is calm. CPU is at 20%. Memory is flat. The autoscaler is parked at its minimum replica count, perfectly content.
Nothing is broken. Everything is waiting. The tasks are sitting in Temporal's task queue before any worker picks them up, and that wait is invisible to every dashboard you just looked at.
The queue you can't see
When you run workflows on Temporal (or any durable-execution engine — the same shape shows up in Cadence, and conceptually in Step Functions), your code doesn't run on the Temporal service. The service is a task broker. It persists workflow state and hands out two kinds of tasks on named task queues: workflow tasks (advance the state machine) and activity tasks (do the actual side-effecting work — call an API, hit a database, invoke an LLM). Your workers are separate processes that long-poll those queues, execute tasks, and report results back.
That split is the whole point — it's what makes workflows survive a worker crash — but it means there's a hop that lives entirely inside the Temporal service: the gap between a task being scheduled (enqueued) and a worker starting it. Temporal names this metric directly: schedule-to-start latency. When it climbs, it almost never means your code is slow. It means tasks are waiting for a free worker, and you don't have enough capacity to drain the queue as fast as it fills.
Why CPU is exactly the wrong signal for agent workers
CPU-based autoscaling — the HPA default, the Cloud Run default — assumes that "busy" means "burning CPU." That assumption breaks for any I/O-bound worker, and agent orchestration is about as I/O-bound as work gets. A typical agent activity looks like this:
async def call_model(prompt: str) -> str:
# ~30s of awaiting a network response. ~0% CPU the whole time.
resp = await client.messages.create(
model="claude-sonnet-5",
max_tokens=2000,
messages=[{"role": "user", "content": prompt}],
)
return resp.content[0].textThat activity holds a concurrency slot on the worker for thirty seconds while using essentially no CPU. Run a few hundred of them concurrently and the worker is fully committed — every execution slot is occupied — while the CPU graph reads 20%. The autoscaler sees idle capacity that doesn't exist. Schedule-to-start climbs because there are no free slots to accept the next task, and the signal that would prove it is nowhere on the infra dashboard.
This is the same disease as scaling GPU inference on device utilization, but worse: a GPU at least looks busy. An agent worker waiting on an API looks asleep.
Size slots and pollers separately — they are two different limits
Before touching the autoscaler, make each worker carry its fair share. Temporal workers have two independent knobs that people routinely conflate:
- Execution slots — how many activities/workflow tasks this worker will run at once. Defaults are generous (1000 in the Go SDK).
- Pollers — how many goroutines/threads are long-polling the queue for new tasks. The default is 2. This is the one that bites.
Two pollers can only pull tasks so fast. If each poll round-trip delivers a task and you have 1000 free slots but only 2 pollers, you fill slots at the rate two long-polls can manage — and under a burst, slots sit empty while the queue backs up. For bursty, I/O-bound work you want many slots (the work is cheap to hold) and more pollers to feed them. Go SDK:
w := worker.New(c, "agent-activities", worker.Options{
// I/O-bound: holding a slot is nearly free, so allow lots.
MaxConcurrentActivityExecutionSize: 500,
// Default is 2. Raise it so slots actually get fed under burst.
MaxConcurrentActivityTaskPollers: 12,
MaxConcurrentWorkflowTaskPollers: 8,
})Python SDK, same idea:
worker = Worker(
client,
task_queue="agent-activities",
activities=[call_model],
max_concurrent_activities=500,
max_concurrent_activity_task_polls=12,
)Don't cargo-cult these numbers. Raise slots until a single worker's actual resource (here, outbound LLM concurrency / rate limits), not CPU, becomes the ceiling; raise pollers until schedule-to-start stops improving. More pollers on an idle queue just adds load to the Temporal service, so tune to the symptom.
Scale on schedule-to-start, not CPU
Once per-worker sizing is right, autoscale the fleet on the metric that actually tracks the backlog. The Temporal SDKs emit schedule-to-start as a histogram; scrape it with Prometheus and target the p95 wait rather than a replica count or a CPU percentage. A PromQL expression for the KEDA/HPA external metric:
histogram_quantile(0.95,
sum by (le, task_queue) (
rate(temporal_activity_schedule_to_start_latency_seconds_bucket{
task_queue="agent-activities"
}[2m])
)
)Scale up when p95 schedule-to-start crosses your budget — say 2 seconds for interactive agent work. Because the metric is a direct measure of queue wait, it rises the instant arrivals outrun worker capacity, regardless of what CPU is doing. That's Little's Law again: queue length (and the wait it produces) reflects when the arrival rate exceeds the service rate, and utilization simply doesn't carry that information for I/O-bound work.
The part reactive scaling can't fix: cold start
Here's the catch that the "just autoscale on the right metric" advice misses. Reactive scaling is a feedback loop, and the loop has latency: the metric has to breach, the scaler has to react, a new worker container has to cold-start, the SDK has to spin up its pollers, and only then do long-polls start draining the queue. On serverless platforms like Cloud Run, container cold start alone can be tens of seconds. By the time the new capacity is polling, the backlog is already minutes deep — which is exactly the ten-minute stall you started with.
For unpredictable load, you eat some of that; keep a sane minimum and a fast scale-up. But a lot of agent load is predictable: a nightly batch of document-processing agents, business-hours traffic, a scheduled re-index. For those, don't wait for the queue to breach — pre-warm workers ahead of demand. KEDA's cron scaler raises the floor on a schedule, so capacity is already polling when the tasks land:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: agent-workers
spec:
scaleTargetRef:
name: agent-worker
minReplicaCount: 2
maxReplicaCount: 40
triggers:
# Pre-warm before the 00:00 batch (times are UTC).
- type: cron
metadata:
timezone: UTC
start: "50 23 * * *" # 10 min early
end: "30 02 * * *"
desiredReplicas: "25"
# Reactive backstop for everything the schedule doesn't predict.
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring:9090
threshold: "2"
query: |
histogram_quantile(0.95, sum by (le) (rate(
temporal_activity_schedule_to_start_latency_seconds_bucket{
task_queue="agent-activities"}[2m])))On Cloud Run the same two-part strategy maps to a scheduled bump of the worker pool's minimum instances ahead of the window, with request/metric-based scaling as the backstop. The principle is identical: schedule the capacity you can predict, and only let the reactive loop handle the surprises.
Lesson
Schedule-to-start latency is the SLI for any task-queue worker fleet, and for I/O-bound agent workers it's the only one that tells the truth — CPU and even "worker looks idle" actively lie to you. Size execution slots and pollers as the two separate limits they are, autoscale on the queue wait rather than on utilization, and remember that reactive scaling is a loop with real latency: for the load you can predict, warm the workers before the tasks arrive instead of after the backlog does.
Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.
Comments
Post a Comment