One Dead GPU, 511 Sitting Idle: Surviving Hardware Failure in Long Training Runs

Here is a failure mode that doesn't show up in any single-machine benchmark: a multi-day training job across 512 GPUs stops making progress at 3am. The loss curve flatlines. No step completes. Every one of those 512 accelerators is allocated, powered, and billed — and all of them are doing nothing, parked at a collective-communication barrier, waiting on a peer that is never going to answer.

A few minutes later a watchdog fires and the whole job crashes with something like Watchdog caught collective operation timeout. If you were checkpointing every few hours, you've just lost everything computed since the last checkpoint across the entire cluster. The actual fault was one GPU throwing an uncorrectable memory error. One. The other 511 were fine.

Why one GPU can wedge the whole job

Synchronous data-parallel training — the default for anything large — runs a collective operation (an all-reduce of gradients) at the end of every step. Every rank contributes, and every rank blocks until all of them have. That barrier is the whole point: it's what keeps the model replicas in sync. It also means the job inherits the reliability of its worst member, not its average. A single rank that dies, hangs, or slows to a crawl doesn't degrade throughput gracefully. It stops the job.

At one or two GPUs this is a non-issue; hardware rarely fails over a weekend. The intuition breaks at scale. Treat each GPU as having some mean time between failures, and the job spans hundreds of them running flat-out for days. The probability that something fails during the run climbs toward a near-certainty. Published accounts of large training efforts describe hardware interruptions as a routine, recurring event over a long run — not a rare outage. Once failure during the run is the expected case, hand-fixing each one at 3am is not an operating model.

So the reliability problem splits into three jobs, and you need all three: detect the bad hardware quickly, remove it from the pool automatically, and make the job resumable so losing a node costs minutes instead of a day.

Detection: the signals a dying GPU actually emits

The most direct signal is the NVIDIA driver's XID errors, which land in the kernel ring buffer. These are worth alerting on directly rather than waiting for training to notice something is wrong:

# XID errors in dmesg — the ground truth for GPU hardware faults
$ dmesg -T | grep -i xid
[Tue Oct 14 03:11:52] NVRM: Xid (PCI:0000:8a:00): 48, pid=..., Double Bit ECC Error
[Tue Oct 14 03:11:52] NVRM: Xid (PCI:0000:8a:00): 79, GPU has fallen off the bus

A few of these are effectively terminal for the device until it is reset or the node is rebooted: XID 48 (double-bit ECC), XID 79 (GPU fell off the bus), and XID 63/64 (ECC row-remapping events) all mean "stop scheduling work here." Others are softer. The point is to classify them, not treat every XID the same.

For a proactive health check rather than waiting for a crash, NVIDIA's DCGM runs active diagnostics — memory tests, PCIe/NVLink bandwidth checks, a short compute stress. Run the quick level as a periodic liveness probe and the deeper levels when a node is suspect:

# Quick health check (seconds) — safe to run between jobs
$ dcgmi diag -r 1

# Deeper diagnostic (minutes) — memory + stress + interconnect
$ dcgmi diag -r 3

The nastier failures are the ones that don't announce themselves. A GPU with a degraded NVLink or a NIC that has quietly dropped from 400Gb/s to a fraction of that won't throw an XID — it just becomes a straggler. Because of the barrier, one slow rank sets the pace for all of them, and your step time creeps up with no error anywhere. Catch this by watching the spread of per-rank step times: if one rank is consistently the slowest to reach the all-reduce, that's your suspect. And always enable NCCL's async error handling so a hung collective surfaces as a fast, catchable error instead of an indefinite hang:

# Surface collective hangs/failures quickly instead of blocking forever
export NCCL_ASYNC_ERROR_HANDLING=1
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1   # newer PyTorch builds
export NCCL_TIMEOUT=1200                    # bound the watchdog, in seconds

Remediation: get the bad node out of the pool, automatically

Detection is useless if a human has to act on it. On Kubernetes, the clean pattern is node-problem-detector watching for the hardware signals and a remediation controller that cordons and drains on a confirmed fault, so the scheduler stops placing work there. A minimal custom monitor for XID errors looks like this:

# node-problem-detector: treat a fatal XID as a permanent node condition
{
  "plugin": "kmsg",
  "logPath": "/dev/kmsg",
  "rules": [
    {
      "type": "permanent",
      "condition": "GpuHardwareError",
      "reason": "XidFatal",
      "pattern": ".*Xid.*: (48|79|63|64),.*"
    }
  ]
}

Once that condition is set, a controller (or a simple reconcile loop) taints the node and drains it:

# Stop new scheduling, then evict so the job's rank can be rescheduled
kubectl taint nodes gpu-node-17 gpu-health=bad:NoSchedule
kubectl drain gpu-node-17 --ignore-daemonsets --delete-emptydir-data

The critical design choice is fail fast and quarantine, not retry-in-place. A GPU that threw a double-bit ECC error will very likely do it again; keeping it in rotation just means the next job trips over it too. Pull it, let an automated reset-or-reboot workflow try to recover it, and only return it to the pool after it passes a dcgmi diag -r 3.

Survival: checkpoint for MTBF, resume elastically

Detection and draining remove the bad hardware. They don't give you back the hours of compute between your last checkpoint and the crash. Two things do.

First, checkpoint often enough — but not so often you spend the run writing to disk. There's a classic result for this (the Young/Daly formula): the near-optimal checkpoint interval is roughly the square root of twice the checkpoint cost times the mean time between failures.

interval ≈ sqrt(2 × checkpoint_cost × MTBF)

# Illustrative: if a checkpoint costs 30s to write and the job's
# effective MTBF is ~2 hours (7200s):
#   interval ≈ sqrt(2 × 30 × 7200) ≈ 657s  → checkpoint ~every 10–11 min

The lower your MTBF (bigger fleet, more failures), the more frequently you should checkpoint — which only works if a checkpoint is cheap. That's the second piece: asynchronous checkpointing. Snapshot the model and optimizer state to host memory (or a local NVMe tier) quickly, then flush to durable storage in the background while training continues, so the GPUs aren't blocked on object-storage write latency. PyTorch's distributed checkpoint APIs support this directly:

import torch.distributed.checkpoint as dcp
from torch.distributed.checkpoint.state_dict_saver import async_save

# Non-blocking: returns a future; training continues while it flushes
future = async_save(state_dict, checkpoint_id=f"step-{step}")
# ... keep training ...
future.result()   # ensure the previous one landed before starting the next

Third, make the job itself elastic so it can resume on a smaller world size without a full manual relaunch. torchrun supports this with a min:max node range and a rendezvous backend: when a node dies, the remaining workers re-form the group, reload the last checkpoint, and continue.

torchrun \
  --nnodes=60:64 \            # tolerate losing up to 4 nodes
  --nproc-per-node=8 \
  --max-restarts=10 \
  --rdzv-backend=c10d \
  --rdzv-endpoint=$HEAD_NODE:29400 \
  train.py

Put together, the lifecycle of a single hardware failure changes from "job dies, page a human, lose hours" to a bounded, automatic loop:

The lesson

Past a certain fleet size, hardware failure stops being an incident and becomes a steady-state condition you design around — the same shift SRE made years ago when it stopped treating machine death as an emergency. The payoff isn't exotic: alert on XID and DCGM signals, taint and drain bad nodes without a human in the loop, checkpoint asynchronously at an interval tuned to your real MTBF, and run the job elastically so it re-forms around the loss. Do that and a dead GPU costs you a few minutes of rewound progress instead of a night's worth across the whole cluster. The expensive failures are almost never the ones you planned for.

---

Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.