How Deepgram optimized voice AI at scale with AWS Trainium

Summary by forck

Deepgram worked with AWS to port its Nova-3 speech-to-text model to AWS Trainium2 for offline batch workloads such as contact-center analytics and medical transcription.

• The engagement began with Nova-3 as standard PyTorch code, and over several months the two teams moved it to Trainium2 in PyTorch reconstruction and tuned it for the new hardware.
• The final configuration runs eight model replicas on a single Amazon EC2 trn2.3xlarge instance, and on Deepgram's offline, throughput-oriented batch workloads it processed 11–70 percent more audio per dollar than the compute options Deepgram had been running, measured using instance pricing at the time of the evaluation.
• Deepgram says the gains help it scale efficiently, keep shipping better models, and bring them to customers at lower prices, and that the benefits can extend to customers who self-host Deepgram models in their own VPC on Amazon SageMaker AI.
• The port used TorchNeuron, the native PyTorch integration of the Neuron Compiler, and the compiled path ran every model operation on Trainium2 with no CPU fallbacks detected.
• Deepgram replaced several layers with fused kernels from the public NKI Library and built a data-parallel Rust runtime so the model could run without Python or PyTorch in the serving path.