From the source
Together AI introduced Together Compiler (TKCC), a continual optimization loop that uses high-throughput agents to tune inference workloads for changing models, hardware and traffic.
- The company says inference workloads are too dynamic to optimize by hand, since architectures, workloads and hardware all vary widely.
- TKCC takes a traffic shape, model and hardware as input and has agents generate optimizations for an inference engine.
- Together says TKCC reaches up to 2.6x higher TPS across frontier open source models in less than 24 hours, and claims state-of-the-art results on SemiAnalysis' InferenceX low-latency benchmark.
- The company attributes the gains to 252x higher experiment throughput than baseline agentic optimization systems, and says Together Compiler is serving trillions of tokens today.
- TKCC uses persistent engines that hot-reload candidate forward passes instead of restarting engines, which Together says take tens of minutes to start.
