Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Apple researchers evaluated LLMs for machine translation quality annotation using MQM and ESA schemes across 70 language pairs and WMT23/WMT25 data.

They found that LLM agreement with humans sometimes exceeds inter-human agreement, but both vary substantially and remain unreliable for most settings.

Humans struggle with fine-grained MQM annotations and low-resource pairs, while LLMs struggle with minor errors, wrong language variants, and error span annotation.