# Together AI — Last year we built megakernels. This year we deleted them: The road to Together Compiler

- Company: Together AI (together.ai)
- Announced: 2026-11-16
- Category: developer-tool-release
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://www.together.ai/blog/road-to-together-compiler
- Record: https://forck.live/items/36946-last-year-we-built-megakernels-this-year-we-deleted-them-the-road-to-together
- Subject: Inference platform
- Models affected: Llama 3.2 1B, Llama 1B

Together AI has introduced Together Compiler, a framework that uses AI agents to rewrite and fuse the GPU kernels in a model's forward pass, which the company says beats hand-written megakernels by 28% on B200. The company says the compiler reached 2,118 tokens/s against 1,649 tokens/s for megakernels on Llama 3.2 1B. Together had previously helped release some of the first AI megakernels, which executed an entire model forward pass in a single fused GPU kernel, but says they were hard to put into production: after months of effort it had one model, one batch size and one customer. The company found that CUDA Graphs and PDL on top of default Torch Inductor kernels recovered 98% of megakernel performance without writing a single kernel, and that multi-kernels and agent loops beat the old megakernels by up to 1.7x, exceeding 1100 TPS for Llama 1B on H100. On the target Llama 1B workload, Inductor delivers 669 TPS, and Together says its heuristics cannot keep up on different architectures and newer hardware. The post is the first of three parts, covering Torch Compile's heuristics, why megakernels are not always the right optimization target, and the agent-driven production system Together ended up with.

## Evidence

Verbatim from https://www.together.ai/blog/road-to-together-compiler:

> Together Compiler uses AI agents to rewrite and fuse the GPU kernels in a model’s forward pass. It beats hand-written megakernels by 28% on B200 (2,118 vs 1,649 tokens/s on Llama 3.2 1B). This is how we got from one to the other.
> 
> Authors
> 
> Dylan Lim, Reyna Abhyankar, Simran Arora
> 
> Table of contents
> 
> 40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...
> 
> Introduction
> 
> AI inference is becoming increasingly heterogeneous. New model architectures are released every week, inference traffic shifts across different batch sizes and sequence lengths, and hardware platforms continue to proliferate. Achieving peak performance across all these regimes demands optimization frameworks that can keep up.
> 
> Last year, we helped release some of the first AI megakernels , where we executed an entire model forward pass within a single, highly fused GPU kernel. They reached record breaking speeds, but it was difficult to put them in production. After months of effort, we had a grand total of one model, one batch size, and one customer.
> 
> This year, we examined what we could simplify and discovered something surprising.

---

Record: https://forck.live/items/36946-last-year-we-built-megakernels-this-year-we-deleted-them-the-road-to-together
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
