Run multi-node inference workloads using Ray LLM Deep Learning Containers

AWS announced the Ray LLM Deep Learning Container (DLC) for multi-node inference, enabling deployment of a single large language model across two GPU nodes on Amazon EKS with separate prefill and decode phases.

The container bundles vLLM, Ray Serve, NVIDIA NIXL for KV-cache transfer, and the EFA stack, exposing an OpenAI-compatible API without custom application code.

This setup addresses workloads such as RAG, document summarization, and agentic applications by preventing compute-heavy prefill from stalling token generation.