From the source
Together AI introduced Polymorphic Inference, with Together Compiler as its first implementation, an agent-driven system that continuously retunes inference as models, traffic and hardware change.
- The company says the compiler reached 1.3x to 3x the tokens per second of highly optimized open-source baselines within under 24 hours, and that it already runs across many top production inference workloads.
- Together AI argues fixed engine configurations go stale: prompts during one country's working hours reached about 97,000 tokens against 70,000 elsewhere, and prompt lengths rose 80% in two weeks after one model release.
- Agents generate and measure candidate optimizations while a persistent engine hot-reloads forward passes, and a startup comparison puts serving time at 9.0 seconds versus 467 seconds for the status-quo sequence, a figure covering engine startup rather than request latency.
- The paper reports 1.5x to 6.4x speedups on tested weather, biology, time-series and partial-differential-equation models with bit-exact outputs on the stated evaluations.
- Together AI also revised its earlier emphasis on megakernels, concluding fusion is not a universal goal for mixture-of-experts architectures and dynamic workloads.
- Customers can request early access through a limited beta.