From the source
Hume added entity accuracy to its Real World VoiceEQ ASR leaderboard, arguing that word error rate hides the transcription failures that break voice-agent tasks.
- Named entities are rare in training data, so language-model priors pull transcriptions toward familiar words, turning a name like "Nguyen" into "new win." The held-out English evaluation covers 3,292 recordings with 3,257 tagged entities across 27 categories, 6.1 hours of speech from 711 speakers in 25 countries.
ElevenLabs Scribe v2 led entity accuracy at 81.7%, followed by MAI-Transcribe 2 at 80.9%, Google Chirp 3 at 80.3% and Azure Speech at 79.6%; only three systems passed 80%.
- Salesforce's 27-billion-parameter SOV-Transcribe-1 posted the lowest WER of 45 systems at 4.5% but ranked sixth on entity accuracy at 78.9%, while Chirp 3 and velma-2-english-vfast had nearly identical WER yet entity accuracy of 80.3% and 73.0%.
- Accuracy fell an average of 14 percentage points on foreign entities, and across the five leading systems averaged 44% for person names.
- Hume is licensing the ASR Entity Dataset and can collect custom data for other languages and domains.






