# Amazon — Distributed training with Ray Train DLCs and EFA on EKS

- Company: Amazon (amazon.com)
- Announced: 2026-10-28T18:19:54+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://aws.amazon.com/blogs/machine-learning/distributed-training-with-ray-train-dlcs-and-efa-on-eks/
- Record: https://forck.live/items/28902-distributed-training-with-ray-train-dlcs-and-efa-on-eks
- Subject: Bedrock / Nova
- Models affected: Qwen2.5-1.5B

AWS published a walkthrough for fine-tuning Qwen2.5-1.5B with PyTorch FSDP across eight NVIDIA L4 GPUs on two g6.12xlarge instances, using the Ray Train Deep Learning Container as published and building no image. The post argues the container, not the model code, is what usually stops PyTorch training from scaling to a cluster, since each image must carry a matching CUDA runtime, PyTorch build, NCCL, EFA userspace libraries, the AWS OFI NCCL plugin and an agreed Ray version. AWS says it assembles that stack, tests it as one unit, publishes it in Amazon ECR Public and keeps it patched, gating each release on multi-node EFA and multi-node KubeRay training tests. The sample pulls the tag public.ecr.aws/deep-learning-containers/ray:train-ml-cuda-v1.1, which runs as both Ray head and worker, so one tag serves every pod. AWS warns that a networking mismatch produces no error: NCCL drops to TCP sockets and the job runs to completion at a fraction of the speed. Job-specific differences are set outside the image, including packages installed at submit time through runtime_env and networking variables such as FI_PROVIDER=efa and NCCL_SOCKET_IFNAME. The vX.Y tags move with security rebuilds while fully qualified vX.Y.Z tags stay frozen, so AWS advises pinning a digest for a byte-identical run.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/distributed-training-with-ray-train-dlcs-and-efa-on-eks/:

> What usually stops PyTorch training from scaling out to a cluster is the container, not the model code. Every container needs the same CUDA runtime , a PyTorch build compiled against it, NCCL (NVIDIA Collective Communications Library), the Elastic Fabric Adapter (EFA) userspace libraries, the AWS OFI NCCL plugin that connects NCCL to EFA, and a Ray version the cluster operator agrees with. Building that image and rebuilding it each time one of those pieces moves is a project of its own, and it has nothing to do with the model you set out to train. The Ray Train Deep Learning Container (DLC) is that image, built by AWS. AWS assembles the stack, tests it as one unit, publishes it in Amazon Elastic Container Registry (Amazon ECR) Public and keeps it patched.
> 
> Figure 1: One EKS cluster with a control plane managed by AWS, a CPU system node (m7i.xlarge) running the KubeRay operator and Ray head, and two EFA-enabled g6.12xlarge nodes in one Availability Zone. KubeRay builds it from one RayCluster manifest, Ray Train places eight workers, one per GPU, and inter-node collectives cross over EFA.
> 
> Ray runs a job across machines as a single cluster with two roles.

---

Record: https://forck.live/items/28902-distributed-training-with-ray-train-dlcs-and-efa-on-eks
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md