# Amazon — Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod

- Company: Amazon (amazon.com)
- Announced: 2026-11-19T16:41:35+00:00
- Category: capability-change
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://aws.amazon.com/blogs/machine-learning/introducing-enhanced-observability-in-amazon-sagemaker-hyperpod-to-diagnose-gpu-training-issues/
- Record: https://forck.live/items/38730-introducing-enhanced-infrastructure-observability-in-amazon-sagemaker-hyperpod
- Subject: Bedrock / Nova

Amazon added enhanced observability to SageMaker HyperPod, a capability that helps teams diagnose GPU training issues across EKS and Slurm clusters. The problem it answers is that a single unhealthy accelerator can stall a distributed job, and teams often spend hours or days sifting through logs and disconnected dashboards to find the cause. Diagnostic summary uses CloudWatch's DetectGroupOutliers API to compare each GPU in a job on utilization, memory, temperature, power and SM clock, flagging GPUs that deviate from their peers. Static rules surface known failure patterns including XID errors, ECC memory errors, EFA throughput drops, thermal throttling and overheating, and each detected issue names the affected job, cluster, node, accelerator and rank. Default metrics are enabled automatically and collected every 30 seconds with roughly 1–2% CPU overhead and 250–300 MB of memory, while advanced metrics covering the full node exporter suite, NVIDIA DCGM and AWS Neuron telemetry, and EFA metrics are opt-in because they raise telemetry volume and cost. CloudWatch is now the default metric store, which Amazon says offers lower pricing than Amazon Managed Service for Prometheus and tighter integration with the SageMaker console; Prometheus remains an option. The capability ships as an add-on for EKS clusters and an extension for Slurm clusters, with install and change permissions controlled through AWS IAM.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/introducing-enhanced-observability-in-amazon-sagemaker-hyperpod-to-diagnose-gpu-training-issues/:

> Large-scale model training is one of the most expensive workloads you can run on GPUs, and a single unhealthy accelerator can stall an entire distributed job. When training throughput drops, the hard part is rarely noticing that something is wrong. It is figuring out why and where. Teams often spend hours or days sifting through logs and disconnected dashboards to decide whether a slowdown came from a throttled GPU, a saturated network link, or a stalled data pipeline on the host.
> 
> Today, we are announcing enhanced observability in Amazon SageMaker HyperPod, a new capability that helps you diagnose GPU training issues faster. It brings together diagnostic and profiling capabilities for training workloads that run on both Amazon Elastic Kubernetes Service (Amazon EKS) and Slurm orchestrators:
> 
> Diagnostic summary uses Amazon CloudWatch’s DetectGroupOutliers API to compare each GPU in a training job on utilization, memory, temperature, power, and SM clock, and flags GPUs that deviate from their peers.

---

Record: https://forck.live/items/38730-introducing-enhanced-infrastructure-observability-in-amazon-sagemaker-hyperpod
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
