# Amazon — Run multi-node inference workloads using Ray LLM Deep Learning Containers

- Company: Amazon (amazon.com)
- Announced: 2026-10-12T15:50:14+00:00
- Category: capability-change
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://aws.amazon.com/blogs/machine-learning/run-multi-node-inference-workloads-using-ray-llm-deep-learning-containers/
- Record: https://forck.live/items/19907-run-multi-node-inference-workloads-using-ray-llm-deep-learning-containers
- Subject: Bedrock / Nova

AWS announced the Ray LLM Deep Learning Container (DLC) for multi-node inference, enabling deployment of a single large language model across two GPU nodes on Amazon EKS with separate prefill and decode phases. The container bundles vLLM, Ray Serve, NVIDIA NIXL for KV-cache transfer, and the EFA stack, exposing an OpenAI-compatible API without custom application code. This setup addresses workloads such as RAG, document summarization, and agentic applications by preventing compute-heavy prefill from stalling token generation.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/run-multi-node-inference-workloads-using-ray-llm-deep-learning-containers/:

> With the launch of the Ray LLM DLC for multi-node deployments, that same approach now extends to multi-node inference. You get a container purpose-built for serving models behind an HTTP endpoint, maintained and tested by AWS, with the full inference stack already assembled.

---

Record: https://forck.live/items/19907-run-multi-node-inference-workloads-using-ray-llm-deep-learning-containers
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md