Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine

Amazon SageMaker HyperPod introduces a tiered KV cache architecture using Curvine, a distributed cache filesystem, to extend KV cache across GPU, CPU, and shared NVMe, enabling cross-replica cache reuse and reducing time-to-first-token.