From the source
Together AI has introduced Together Compiler, a framework that uses AI agents to rewrite and fuse the GPU kernels in a model's forward pass, which the company says beats hand-written megakernels by 28% on B200.
- The company says the compiler reached 2,118 tokens/s against 1,649 tokens/s for megakernels on Llama 3.2 1B.
- Together had previously helped release some of the first AI megakernels, which executed an entire model forward pass in a single fused GPU kernel, but says they were hard to put into production: after months of effort it had one model, one batch size and one customer.
- The company found that CUDA Graphs and PDL on top of default Torch Inductor kernels recovered 98% of megakernel performance without writing a single kernel, and that multi-kernels and agent loops beat the old megakernels by up to 1.7x, exceeding 1100 TPS for Llama 1B on H100.
- On the target Llama 1B workload, Inductor delivers 669 TPS, and Together says its heuristics cannot keep up on different architectures and newer hardware.
- The post is the first of three parts, covering Torch Compile's heuristics, why megakernels are not always the right optimization target, and the agent-driven production system Together ended up with.






