BenchMIRT: What are LLM benchmarks actually measuring?

Ai2 introduced BenchMIRT, a method that uses multidimensional item response theory to audit LLM benchmarks at the individual prompt level.

• Applied to 100 models across 16 benchmarks, it independently recovered safety and general reasoning as two dominant dimensions.
• The analysis revealed that some benchmarks, such as BBQ, align more with general reasoning than with safety, and that BenchMIRT can identify the most informative questions in an evaluation.