How Together AI built the world’s fastest speech-to-text stack

Together AI announced optimizations to their speech-to-text (ASR) stack that reduce latency and improve throughput.

By using TensorRT with multi-profile engines for real audio shapes, moving the decoder loop entirely to the GPU with conditional CUDA graph nodes, and collapsing CPU process boundaries to eliminate redundant copies, they achieved the fastest ASR stack according to Artificial Analysis.

The stack serves NVIDIA Parakeet-TDT 0.6B v3 and OpenAI Whisper Large v3, with Parakeet transcribing 20 hours of speech in under 10 seconds.