Distributed training with Ray Train DLCs and EFA on EKS

AWS published a walkthrough for fine-tuning Qwen2.5-1.5B with PyTorch FSDP across eight NVIDIA L4 GPUs on two g6.12xlarge instances, using the Ray Train Deep Learning Container as published and building no image.

• The post argues the container, not the model code, is what usually stops PyTorch training from scaling to a cluster, since each image must carry a matching CUDA runtime, PyTorch build, NCCL, EFA userspace libraries, the AWS OFI NCCL plugin and an agreed Ray version.
• AWS says it assembles that stack, tests it as one unit, publishes it in Amazon ECR Public and keeps it patched, gating each release on multi-node EFA and multi-node KubeRay training tests.
• The sample pulls the tag public.ecr.aws/deep-learning-containers/ray:train-ml-cuda-v1.1, which runs as both Ray head and worker, so one tag serves every pod.
• AWS warns that a networking mismatch produces no error: NCCL drops to TCP sockets and the job runs to completion at a fraction of the speed.
• Job-specific differences are set outside the image, including packages installed at submit time through runtime_env and networking variables such as FI_PROVIDER=efa and NCCL_SOCKET_IFNAME.
• The vX.Y tags move with security rebuilds while fully qualified vX.Y.Z tags stay frozen, so AWS advises pinning a digest for a byte-identical run.