From the source
A practical guide to layered Indic ASR evaluation: LLM-WER and LLM-CER, Intent and Entity scores, COMET, and open-source evaluation frameworks.
Introduction Measuring how well a speech recognition system performs in Indian languages is harder than it looks.
The standard metrics weren't built for them, and that mismatch quietly distorts how Indic ASR systems get evaluated.
Word Error Rate(WER), Character Error Rate(CER), and BLEU were developed primarily for English.
They work well when every word has a single accepted spelling, when languages don't mix mid-sentence, and when the gap between formal and colloquial usage is narrow.
Indian languages don't fit that description.
Colloquial and formal registers coexist and are equally understood by speakers.
English loanwords appear in both Indic and Latin script, sometimes within the same utterance.
Numbers have multiple valid written forms.
Applying these metrics without adjustment can make an Indic ASR system look significantly worse or better than it actually performs in practice.
This is a harder problem than it first appears.
It isn't just that the metrics are imperfect.
…
