From the source
Amazon's guide shows how to detect silent data corruption in PyTorch DDP training on SageMaker HyperPod, where a GPU returns a wrong but plausible number without raising an error.
- Existing HyperPod health monitoring, deep health checks and automatic node recovery cannot catch it, because the hardware never reports a fault, so detection has to come from the training job itself.
- The post adds three checks to the training loop: a non-finite gradient check for NaN, Inf and overflow, a ring replay audit that recomputes a micro-batch from another node and compares loss bit for bit and gradients by fingerprint, and a weight fingerprint comparison after the optimizer step.
- When a check fires, the suspect rank is mapped to its node, the job is stopped, the node is cordoned, and HyperPod replaces it before training resumes from a checkpoint saved before the suspect window.
- The checks were tested with software-injected faults on a cluster of 4 ml.p5.48xlarge instances, 32
NVIDIA H100 GPUs, and repeated on three instances, the minimum the checks need.
- The post cites
Meta's finding that during a 54-day period of Llama 3 405B pre-training on up to 16K H100 GPUs, 6 of 419 unexpected job interruptions were attributed to silent data corruption.






