From the source
Baseten has optimized Qwen3-ASR 1.7B for batch transcription, claiming the lowest cost per audio minute across all transcription providers.
- The Qwen team's open-weight speech recognition model is the top-ranking open-weight model on
Hugging Face's Open ASR Leaderboard, with a 5.4% average word error rate, and Baseten says its real-time deployment also has the lowest WER and latency on Coval's speech-to-text benchmark.
- Batch transcription is a throughput problem rather than a latency one, so Baseten profiled every stage of a request under load and changed four things: native audio decoding, KV cache quantization, host-to-device syncs and request handling.
- Replacing vLLM's PyAV loader with torchcodec hands whole files to FFmpeg in one native call, and long files are decoded in parallel across time ranges, so an hour of audio is preprocessed in well under a second.
- Storing the KV cache in FP8 halves the bytes read each step and doubles the sequences that fit in memory, with FlashInfer kernels used because vLLM's default FlashAttention-2 backend does not support FP8 cache on RTX PRO 6000; weights stay in BF16 because quantizing them raised word error rate.
