From the source
Together AI announces cache-aware prefill–decode disaggregation (CPD), a serving architecture that separates cold and warm workloads by cache hit rate to improve throughput and reduce time-to-first-token for long-context inference.
Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM serving
Infrastructure
From the source
At Together AI, we built cache-aware prefill–decode disaggregation (CPD), a serving architecture that purposely separates cold and warm workloads by cache hit rate, resulting in fast context reuse.
together.ai
