# Hume — When low word error rate is not enough: Measuring entity accuracy in ASR

- Company: Hume (hume.ai)
- Announced: 2026-11-20T16:00:00+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://www.hume.ai/blog/asr-entity-accuracy-benchmark
- Record: https://forck.live/items/39315-when-low-word-error-rate-is-not-enough-measuring-entity-accuracy-in-asr
- Subject: EVI / Octave
- Models affected: ElevenLabs Scribe v2, MAI-Transcribe 2, Google Chirp 3, Azure Speech, SOV-Transcribe-1, velma-2-english-vfast, Deepgram Nova-3

Hume added entity accuracy to its Real World VoiceEQ ASR leaderboard, arguing that word error rate hides the transcription failures that break voice-agent tasks. Named entities are rare in training data, so language-model priors pull transcriptions toward familiar words, turning a name like "Nguyen" into "new win." The held-out English evaluation covers 3,292 recordings with 3,257 tagged entities across 27 categories, 6.1 hours of speech from 711 speakers in 25 countries. ElevenLabs Scribe v2 led entity accuracy at 81.7%, followed by MAI-Transcribe 2 at 80.9%, Google Chirp 3 at 80.3% and Azure Speech at 79.6%; only three systems passed 80%. Salesforce's 27-billion-parameter SOV-Transcribe-1 posted the lowest WER of 45 systems at 4.5% but ranked sixth on entity accuracy at 78.9%, while Chirp 3 and velma-2-english-vfast had nearly identical WER yet entity accuracy of 80.3% and 73.0%. Accuracy fell an average of 14 percentage points on foreign entities, and across the five leading systems averaged 44% for person names. Hume is licensing the ASR Entity Dataset and can collect custom data for other languages and domains.

## Evidence

Verbatim from https://www.hume.ai/blog/asr-entity-accuracy-benchmark:

> People give voice agents the details that make a task work: their name, an email address to verify an account, a street address for a delivery, or a date and time to book an appointment. In each case, one wrong letter in a name or email address, or one wrong digit in an account number, can cause the entire task to fail.
> 
> These details are named entities. Because they make up only a small part of a sentence, overall word error rate can look strong even when the system gets this critical information wrong. In our new Real World VoiceEQ evaluation, two systems with almost the same WER were seven points apart in entity accuracy, and even the leading system missed nearly one in five entities.
> 
> To make these failures visible, we added entity accuracy to the Real World VoiceEQ ASR leaderboard. Alongside the benchmark, we are offering the Hume ASR Entity Dataset * for licensing to teams that want to train on these failure modes across languages, accents, and real recording conditions, and we can collect custom data for additional languages and domains.
> 
> Why named entities are difficult for ASR
> 
> Named entities are often rare in training data.

---

Record: https://forck.live/items/39315-when-low-word-error-rate-is-not-enough-measuring-entity-accuracy-in-asr
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
