# Hugging Face — What's going on with the Open LLM Leaderboard?

- Company: Hugging Face (huggingface.co)
- Announced: 2023-06-23
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://huggingface.co/blog/open-llm-leaderboard-mmlu
- Record: https://forck.live/items/2035-what-s-going-on-with-the-open-llm-leaderboard
- Subject: Platform

The blog post investigates why MMLU evaluation scores on the Open LLM Leaderboard differ from those reported in the LLaMA paper, finding that different implementations of the same benchmark yield varying results and even change model rankings.

## Evidence

Verbatim from https://huggingface.co/blog/open-llm-leaderboard-mmlu:

> The community was surprised that MMLU evaluation numbers of the current top model on the leaderboard, the LLaMA model 🦙, were significantly lower than the numbers in the published LLaMa paper.

---

Record: https://forck.live/items/2035-what-s-going-on-with-the-open-llm-leaderboard
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md