# Amazon — Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

- Company: Amazon (amazon.com)
- Announced: 2026-11-11T23:30:03+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://aws.amazon.com/blogs/machine-learning/detecting-and-recovering-from-silent-data-corruption-on-amazon-sagemaker-hyperpod/
- Record: https://forck.live/items/34908-detecting-and-recovering-from-silent-data-corruption-on-amazon-sagemaker
- Subject: Bedrock / Nova
- Models affected: Llama 3 405B

Amazon's guide shows how to detect silent data corruption in PyTorch DDP training on SageMaker HyperPod, where a GPU returns a wrong but plausible number without raising an error. Existing HyperPod health monitoring, deep health checks and automatic node recovery cannot catch it, because the hardware never reports a fault, so detection has to come from the training job itself. The post adds three checks to the training loop: a non-finite gradient check for NaN, Inf and overflow, a ring replay audit that recomputes a micro-batch from another node and compares loss bit for bit and gradients by fingerprint, and a weight fingerprint comparison after the optimizer step. When a check fires, the suspect rank is mapped to its node, the job is stopped, the node is cordoned, and HyperPod replaces it before training resumes from a checkpoint saved before the suspect window. The checks were tested with software-injected faults on a cluster of 4 ml.p5.48xlarge instances, 32 NVIDIA H100 GPUs, and repeated on three instances, the minimum the checks need. The post cites Meta's finding that during a 54-day period of Llama 3 405B pre-training on up to 16K H100 GPUs, 6 of 419 unexpected job interruptions were attributed to silent data corruption.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/detecting-and-recovering-from-silent-data-corruption-on-amazon-sagemaker-hyperpod/:

> Silent data corruption (SDC) happens when hardware, such as a GPU, returns a wrong but plausible number and raises no error. In distributed training, the wrong number can flow into the gradients and weights of every replica. Loss can still look normal for hours, and by the time a team notices an unexplained divergence or a weaker model, the compute is spent. It happens at scale: during a 54-day period of Llama 3 405B pre-training on up to 16K H100 GPUs, Meta attributed 6 of 419 unexpected job interruptions to silent data corruption, and those were only the cases that interrupted the job and were traced to SDC.
> 
> Amazon SageMaker HyperPod already replaces nodes that report a fault: the health-monitoring agent watches for GPU errors, deep health checks stress-test nodes, and automatic node recovery swaps out bad hardware. SDC slips past all of these features, because the hardware never reports anything. Detection has to come from the training job itself.
> 
> In this post, we add three checks to a PyTorch DistributedDataParallel (DDP) training loop to detect SDC. When a check fires, we map the suspect rank to its node, contain the corruption, and have SageMaker HyperPod replace the node.

---

Record: https://forck.live/items/34908-detecting-and-recovering-from-silent-data-corruption-on-amazon-sagemaker
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
