# Amazon — Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

- Company: Amazon (amazon.com)
- Announced: 2026-11-18T23:03:29+00:00
- Category: capability-change
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://aws.amazon.com/blogs/machine-learning/introducing-workload-recovery-and-expanded-infrastructure-checks-on-amazon-sagemaker-hyperpod/
- Record: https://forck.live/items/38321-introducing-workload-recovery-and-expanded-infrastructure-checks-on-amazon
- Subject: Bedrock / Nova

Amazon added workload recovery and infrastructure checks to SageMaker HyperPod, aiming to catch training failures that node-level health checks miss and raise the goodput of training workloads. The existing HyperPod resiliency features check one node's hardware at a time, so they cannot tell whether GPUs compute correctly, whether a job is still making progress, or whether the Kubernetes components a job depends on are healthy. MultiNodeTrainingValidation is a new deep health check for EKS-orchestrated clusters that trains a small model on an instance and then across instances of the same type, failing the instance if the loss becomes NaN or infinite or training does not finish in time. The HyperPod Kubernetes Resource Monitoring Agent watches the Kubernetes layer for problems such as crash-looping device plugins, missing DaemonSets and tainted nodes, reporting the pods and nodes involved as Kubernetes events and CloudWatch logs when a job fails. Hung job detection finds Slurm jobs that have stopped making progress, using an agent on each node that watches the job's output and GPU activity or log patterns the customer defines, writes an event to CloudWatch and can cancel the job so Slurm requeues it. Amazon illustrates the cost of a missed hang with a pre-training job on 32 ml.p5.48xlarge instances, or 256 NVIDIA H100 GPUs, that stalls at 1:00 AM and goes unnoticed until 9:00 AM, wasting 256 instance-hours or 2,048 GPU-hours. At the SageMaker HyperPod on-demand price of approximately $66 per instance-hour in US East (N. Virginia), Amazon says the hang costs about $16,900, while a 15-minute hung job detection rule cuts that to about $530 and saves approximately $16,400.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/introducing-workload-recovery-and-expanded-infrastructure-checks-on-amazon-sagemaker-hyperpod/:

> Large training jobs can fail even when every node looks healthy. A GPU with a subtle fault can pass its stress tests and still return wrong values. When a job stalls on an NCCL timeout, it keeps its GPUs, and Slurm still shows it as running. If a device plugin crashes on one node, the job fails with an error that never mentions the plugin. Teams usually find these problems only after the loss diverges or the job fails, and by then the compute is spent. Each of these failures lowers goodput, the share of your cluster’s time that goes to useful training. Today, we’re introducing workload recovery and infrastructure checks on Amazon SageMaker HyperPod that catch these failures sooner and raise the goodput of your training workloads.
> 
> SageMaker HyperPod already handles nodes that report hardware faults. The health monitoring agent (HMA) watches each node for GPU errors, deep health checks test nodes during provisioning, and automatic node recovery reboots or replaces faulty hardware. These features look at one node’s hardware at a time.

---

Record: https://forck.live/items/38321-introducing-workload-recovery-and-expanded-infrastructure-checks-on-amazon
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
