# Ai2 — BenchMIRT: What are LLM benchmarks actually measuring?

- Company: Ai2 (allenai.org)
- Announced: 2026-09-01
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://allenai.org/blog/benchmirt
- Record: https://forck.live/items/18534-benchmirt-what-are-llm-benchmarks-actually-measuring
- Subject: Olmo / Molmo / Asta

Ai2 introduced BenchMIRT, a method that uses multidimensional item response theory to audit LLM benchmarks at the individual prompt level. Applied to 100 models across 16 benchmarks, it independently recovered safety and general reasoning as two dominant dimensions. The analysis revealed that some benchmarks, such as BBQ, align more with general reasoning than with safety, and that BenchMIRT can identify the most informative questions in an evaluation.

## Evidence

Verbatim from https://allenai.org/blog/benchmirt:

> BenchMIRT helps researchers separate those signals and see what’s actually driving a benchmark’s score. It does this by analyzing how models perform on each question or task and estimating which underlying capabilities are most closely associated with getting it right.

---

Record: https://forck.live/items/18534-benchmirt-what-are-llm-benchmarks-actually-measuring
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md