# Together AI — Together Compiler: Continual Compilation for AI inference

- Company: Together AI (together.ai)
- Announced: 2026-11-04
- Category: infrastructure-release
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://www.together.ai/blog/together-compiler-continual-compilation-for-ai-inference
- Record: https://forck.live/items/31556-together-compiler-continual-compilation-for-ai-inference
- Subject: Inference platform
- Models affected: Kimi K3, GLM 5.3, MiniMax M3.1-Flash, DeepSeek v4.1-Flash, Llama 1B

Together AI introduced Together Compiler (TKCC), a continual optimization loop that uses high-throughput agents to tune inference workloads for changing models, hardware and traffic. The company says inference workloads are too dynamic to optimize by hand, since architectures, workloads and hardware all vary widely. TKCC takes a traffic shape, model and hardware as input and has agents generate optimizations for an inference engine. Together says TKCC reaches up to 2.6x higher TPS across frontier open source models in less than 24 hours, and claims state-of-the-art results on SemiAnalysis' InferenceX low-latency benchmark. The company attributes the gains to 252x higher experiment throughput than baseline agentic optimization systems, and says Together Compiler is serving trillions of tokens today. TKCC uses persistent engines that hot-reload candidate forward passes instead of restarting engines, which Together says take tens of minutes to start.

## Evidence

Verbatim from https://www.together.ai/blog/together-compiler-continual-compilation-for-ai-inference:

> Reyna Abhyankar Dylan Lim Simran Arora
> 
> Table of contents
> 
> 40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...
> 
> Links in this article
> 
> AI inference workloads are dynamic. There are thousands of regimes spanning new models, hardware platforms, customers, use cases, and traffic shapes. It’s no longer scalable to optimize each of these regimes by hand.
> 
> Architectures are highly variable: it’s been ~11 years since the Transformer. We might have thought there would be more consolidation, but none of the top-10 open-source LLMs on OpenRouter use the same architecture!
> 
> Workloads are highly variable: Together serves many trillions of tokens per day across thousands of customers, each with a different traffic shape (batch size, sequence length, MoE expert imbalance, attention sparsity, session duration, domain).
> 
> Hardware is highly variable: H100s, B200s, B300s, NVL72, VR, Groq, and more each require distinct strategies.
> 
> Agents are a great fit for this problem, in theory.

## Follow-ups

- Unite.AI (reported, 4 Nov): Together AI Brings Continuous Compilation to Production Inference — https://forck.live/items/31581-together-ai-brings-continuous-compilation-to-production-inference

---

Record: https://forck.live/items/31556-together-compiler-continual-compilation-for-ai-inference
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md