From the source
Amazon added workload recovery and infrastructure checks to SageMaker HyperPod, aiming to catch training failures that node-level health checks miss and raise the goodput of training workloads.
- The existing HyperPod resiliency features check one node's hardware at a time, so they cannot tell whether GPUs compute correctly, whether a job is still making progress, or whether the Kubernetes components a job depends on are healthy.
- MultiNodeTrainingValidation is a new deep health check for EKS-orchestrated clusters that trains a small model on an instance and then across instances of the same type, failing the instance if the loss becomes NaN or infinite or training does not finish in time.
- The HyperPod Kubernetes Resource Monitoring Agent watches the Kubernetes layer for problems such as crash-looping device plugins, missing DaemonSets and tainted nodes, reporting the pods and nodes involved as Kubernetes events and CloudWatch logs when a job fails.
- Hung job detection finds Slurm jobs that have stopped making progress, using an agent on each node that watches the job's output and GPU activity or log patterns the customer defines, writes an event to CloudWatch and can cancel the job so Slurm requeues it.
- Amazon illustrates the cost of a missed hang with a pre-training job on 32 ml.p5.48xlarge instances, or 256
NVIDIA H100 GPUs, that stalls at 1:00 AM and goes unnoticed until 9:00 AM, wasting 256 instance-hours or 2,048 GPU-hours.
- At the SageMaker HyperPod on-demand price of approximately $66 per instance-hour in US East (N. Virginia), Amazon says the hang costs about $16,900, while a 15-minute hung job detection rule cuts that to about $530 and saves approximately $16,400.






