Reduce ASR inference costs by 75% with NVIDIA MPS on Amazon EC2

Amazon Web Services, NVIDIA, and Heidi Health announce a solution that uses NVIDIA CUDA Multi-Process Service and NVIDIA Triton Inference Server on Amazon EC2 GPU instances to reduce GPU infrastructure requirements for ASR inference by 75 percent, from 16 instances to 4, while maintaining sub-second latency at 92.1 requests per second per GPU, enabling Heidi Health to process over 2.4 million clinical consultations per week.

From the source
In this post, we focus on what comes after fine-tuning: serving that model efficiently. We demonstrate how NVIDIA CUDA Multi-Process Service (MPS), combined with NVIDIA Triton Inference Server™ on Amazon EC2 GPU instances, reduces GPU infrastructure requirements by 75 percent (from 16 instances to 4).
aws.amazon.com

Visit source ↗aws.amazon.com

Announced: 27 Aug 2026 16:05 UTC

Other stories

xAI · 29 AugGrok Bot now works with XCapability change

GitHub · 28 AugGitHub Copilot weekly releases — August 24Capability change

Amazon · 28 AugBatch write and discover records in Amazon SageMaker Feature StoreAPI release

Amazon · 28 AugHow Decathlon runs demand forecasting at scale with Chronos-2

Detected

27 Aug 2026 16:14 UTC (+9m after)

Amazon · 28 AugSpreading the load: How Salesforce met Multi-AZ HA with SageMaker Inference Components

Capability change

Hugging Face · 28 AugThe Open ASR Leaderboard Adds Its First Global South LanguageCapability change