# Apple — Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

- Company: Apple (apple.com)
- Announced: 2026-10-12
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://machinelearning.apple.com/research/llm-translation-quality-annotation
- Record: https://forck.live/items/20051-large-language-models-for-machine-translation-quality-annotation-humans-and
- Subject: Machine Learning Research

Apple researchers evaluated LLMs for machine translation quality annotation using MQM and ESA schemes across 70 language pairs and WMT23/WMT25 data. They found that LLM agreement with humans sometimes exceeds inter-human agreement, but both vary substantially and remain unreliable for most settings. Humans struggle with fine-grained MQM annotations and low-resource pairs, while LLMs struggle with minor errors, wrong language variants, and error span annotation.

## Evidence

Verbatim from https://machinelearning.apple.com/research/llm-translation-quality-annotation:

> Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings.

---

Record: https://forck.live/items/20051-large-language-models-for-machine-translation-quality-annotation-humans-and
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md