Your Inference GPUs Are Starved, Not Slow: Finding the Idle Time in Your AI Bill

It usually shows up as a billing question, not an alert. The GPU line on the cloud bill has climbed quarter over quarter while throughput has stayed flat, and someone from finance wants to know why. You open the accelerator dashboard expecting pegged GPUs and instead find them hovering around 25–30% utilization. You are renting some of the most expensive compute available and using a third of it.

This is the most common failure mode in production AI infrastructure right now, and it is rarely a "buy fewer GPUs" problem. The GPUs are not slow. They are starved, fragmented, or idle — and every one of those is fixable without touching the model.

Step 1: distrust the utilization number you have

The first mistake is trusting nvidia-smi. Its headline GPU-Util field does not mean what most people assume. The NVIDIA docs define it as the percent of time over the sample window during which one or more kernels was executing. A single tiny kernel copying data counts the same as a fully saturated matrix multiply. It is entirely normal to see GPU-Util read 100% while the tensor cores — the part you are actually paying a premium for — sit almost idle.

To see the honest picture you need profiling counters, which means DCGM (NVIDIA Data Center GPU Manager) rather than the summary tool:

# "utilization.gpu" here only means a kernel was running --
# it is NOT a measure of how busy the SMs or tensor cores were
nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 1

# Real occupancy comes from profiling counters exposed by DCGM
dcgm-exporter &
curl -s localhost:9400/metrics | grep -E 'DCGM_FI_PROF_(SM|PIPE_TENSOR)_ACTIVE'

Two counters matter most. DCGM_FI_PROF_SM_ACTIVE is the fraction of time at least one warp was resident on a streaming multiprocessor, averaged across all SMs. DCGM_FI_PROF_PIPE_TENSOR_ACTIVE is the fraction of cycles the tensor pipes were actually doing work. For a transformer inference workload, tensor-pipe activity in the single digits while GPU-Util reads near 100% is the classic signature of an accelerator that is technically "busy" and economically wasted. Chart those two counters next to your GPU spend and the gap you have been paying for becomes visible.

Where the idle time actually hides

The GPU is starved by the input pipeline

The most common cause of low occupancy is that the GPU spends its time waiting for data. Decoding images, tokenizing text, or pulling batches from object storage happens on the CPU, and if the CPU cannot keep the GPU fed, the accelerator stalls between batches. The fix is to overlap host-side preparation with device-side compute so the next batch is ready before the current one finishes:

loader = DataLoader(
    dataset,
    batch_size=64,
    num_workers=8,          # parallel CPU preprocessing
    pin_memory=True,        # faster, async host->device copies
    prefetch_factor=4,      # keep batches queued ahead of the GPU
    persistent_workers=True # don't respawn workers every epoch
)

If tuning the loader does not close the gap, the preprocessing itself is too heavy for the CPU allocation. Move it off the hot path: precompute and cache tokenized or resized inputs, or push decode work onto the GPU with something like NVIDIA DALI. The rule of thumb is that the input pipeline should sustain a throughput comfortably higher than the model can consume, so the GPU is never the one waiting.

The batches are too small to fill the machine

A GPU is a throughput device. Feeding it one request at a time leaves most of its parallel units empty. Under real-time serving, requests arrive one by one, so a naive server processes them individually and never fills the hardware. The answer is server-side batching that collects requests arriving within a short window and runs them together. In Triton Inference Server that is a few lines of config:

# config.pbtxt
max_batch_size: 64
dynamic_batching {
  preferred_batch_size: [ 16, 32, 64 ]
  max_queue_delay_microseconds: 2000   # trade <=2ms latency for batching
}

For LLM serving the equivalent is continuous (in-flight) batching, which engines like vLLM and TensorRT-LLM do by default: instead of waiting for every sequence in a batch to finish, they admit new requests as soon as any sequence completes, keeping the GPU full even when generation lengths vary wildly. If you are running a hand-rolled generation loop that batches statically, this single change is often the largest throughput win available.

One small model is holding a whole GPU

Plenty of production models — embedding models, rerankers, classifiers, smaller distilled models — cannot come close to filling a modern data-center GPU. Give each one a dedicated device and you pay for 80GB of accelerator to run a workload that needs a fraction of it. Two mechanisms let you pack them:

MIG (Multi-Instance GPU) on A100/H100/Blackwell-class cards partitions one physical GPU into hardware-isolated instances, each with its own SMs and memory slice, so a noisy tenant cannot starve the others.
MPS (Multi-Process Service) lets multiple processes share a GPU's SMs concurrently with less isolation but finer packing — useful when the workloads trust each other.

# Split one GPU into isolated instances so small models share it
sudo nvidia-smi -mig 1                 # enable MIG mode
nvidia-smi mig -lgip                   # list available instance profiles
sudo nvidia-smi mig -cgi 19,19,19 -C   # create three instances (profile IDs
                                       # vary by GPU model)
nvidia-smi mig -lgi                    # confirm the instances exist

The GPU is idle between requests

The last bucket is the simplest and the most expensive: an accelerator that is allocated and running but serving no traffic — overnight, on weekends, or for a batch job that finished hours ago. On-demand serving should scale to zero when the request queue drains. With KEDA driving a Kubernetes deployment off a Prometheus query, that is declarative:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
spec:
  scaleTargetRef:
    name: inference-server
  minReplicaCount: 0        # scale the GPU pods to zero when idle
  cooldownPeriod: 120       # wait before scaling down, to avoid flapping
  triggers:
    - type: prometheus
      metadata:
        query: sum(rate(inference_requests_total[1m]))
        threshold: "1"

For batch and streaming pipelines the same idea shows up as pause/resume: managed runners such as Dataflow now let you pause a GPU pipeline and release the accelerators between bursts rather than holding them for the pipeline's whole lifetime. The prerequisite for any of this is checkpointing — if a long job cannot resume from a saved state, you cannot safely stop it, which also means you cannot run it on cheaper preemptible or spot GPUs. Checkpoint frequently enough that losing a node costs minutes, not hours, and a large slice of batch training and offline inference moves onto spot capacity at a fraction of on-demand cost.

The order that pays off

Work these in sequence, because each one changes the number the next one sees. Measure real occupancy with DCGM first — not nvidia-smi — so you are optimizing against the truth. Then fix starvation, because a GPU waiting on its input pipeline will not benefit from bigger batches. Then batch, to fill the device that is now being fed. Then pack idle-by-design small models with MIG or MPS. Finally, scale to zero and adopt preemptible capacity so the hours you are not serving stop appearing on the invoice. It is common for the first three steps alone to double effective throughput per GPU, which is the same thing as halving the cost per request without buying anything.

The lesson

"We need more GPUs" and "we are wasting the GPUs we have" produce identical dashboards if the only number you look at is nvidia-smi's GPU-Util. The expensive part of an accelerator is the tensor pipeline, so that is what you should be measuring and defending. Before anyone approves a bigger GPU budget, prove the ones you already own are actually computing — the reclaimed capacity is almost always cheaper than the next order.

Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.

Search This Blog

Lets Architect the Performance Engineering