# Amazon — Fault tolerant distributed training on Amazon EKS using NVRx

- Company: Amazon (amazon.com)
- Announced: 2026-09-16T18:59:25+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://aws.amazon.com/blogs/machine-learning/fault-tolerant-distributed-training-on-amazon-eks-using-nvrx/
- Record: https://forck.live/items/11440-fault-tolerant-distributed-training-on-amazon-eks-using-nvrx
- Subject: Bedrock / Nova

Amazon's guide demonstrates how to integrate NVIDIA Resiliency Extension (NVRx) into PyTorch FSDP training on Amazon EKS to achieve fault-tolerant distributed training. It covers async checkpointing, in-process restart, and in-job restart with ft_launcher, and includes benchmark results on H100 GPUs at 2-node to 8-node scale.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/fault-tolerant-distributed-training-on-amazon-eks-using-nvrx/:

> In this post, we show how to integrate NVIDIA Resiliency Extension (NVRx) into PyTorch Fully Sharded Data Parallel (FSDP) training on Amazon Elastic Kubernetes Service (Amazon EKS) to solve both problems.

---

Record: https://forck.live/items/11440-fault-tolerant-distributed-training-on-amazon-eks-using-nvrx
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md