From the source
Apple researchers evaluated how well large language model judges assess personalized writing style, work accepted at the Can We Trust AI Evaluation?
- Workshop at NeurIPS 2026.
- The team generated text with four LLMs across three long-form writing tasks, collected human labels on which of two responses was better, and compared those with judge-LLM pairwise rankings.
- Judge quality correlated weakly with general LLM ability measured by MMLU, at a Pearson's rho of 0.2 or less, and varied by writing task and by whether the criteria were general helpfulness or style personalization.
- LLM evaluators were more consistent and performed better when comparing responses in pairs rather than scoring them independently.
- The researchers also found the strongest LLMs did not most reliably reproduce human evaluations, which they hypothesize relates to exaggerated sensitivity to subtle details that human annotators and weaker LLMs ignore.
