# Apple — Evaluating the Evaluators: Investigating LLM Judges for Personalized Writing Style Assessment

- Company: Apple (apple.com)
- Announced: 2026-11-10
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://machinelearning.apple.com/research/evaluating-evaluators-llm-judges
- Record: https://forck.live/items/34760-evaluating-the-evaluators-investigating-llm-judges-for-personalized-writing
- Subject: Machine Learning Research

Apple researchers evaluated how well large language model judges assess personalized writing style, work accepted at the Can We Trust AI Evaluation? Workshop at NeurIPS 2026. The team generated text with four LLMs across three long-form writing tasks, collected human labels on which of two responses was better, and compared those with judge-LLM pairwise rankings. Judge quality correlated weakly with general LLM ability measured by MMLU, at a Pearson's rho of 0.2 or less, and varied by writing task and by whether the criteria were general helpfulness or style personalization. LLM evaluators were more consistent and performed better when comparing responses in pairs rather than scoring them independently. The researchers also found the strongest LLMs did not most reliably reproduce human evaluations, which they hypothesize relates to exaggerated sensitivity to subtle details that human annotators and weaker LLMs ignore.

## Evidence

Verbatim from https://machinelearning.apple.com/research/evaluating-evaluators-llm-judges:

> Authors Natalie Mackraz‡, Andrew Silva, Aini Putkonen, Brihi Joshi†**, Joelle Alcaidinho, Arnav Arora§**, Barry-John Theobald, Katherine Metcalf‡
> 
> This paper was accepted at the Can We Trust AI Evaluation? Workshop at NeurIPS 2026.
> 
> We evaluate how good large language model (LLM) judges are for personalized writing tasks. This is important when a user’s preferred writing style is known (e.g., “upbeat”) and an LLM judge is used to evaluate whether generated text adheres to this preference. We examine how the writing task, the combination of generator and judge LLM, the evaluation objective, and general commonsense and reasoning ability impact LLM judge performance. To this end, we generate text with four LLMs for three long-form writing tasks, collect human labels evaluating which of two responses is better, collect judge-LLM, pairwise-ranking annotations, and then compare the human and judge LLM annotations. We find that judge quality correlates weakly with general LLM ability measured using MMLU (Pearson’s ρ ≤ 0.2) and varies by both writing task and evaluation criteria (i.e., general helpfulness versus style personalization).

---

Record: https://forck.live/items/34760-evaluating-the-evaluators-investigating-llm-judges-for-personalized-writing
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
