Building better AI benchmarks: How many raters are enough?

Google Research introduces an evaluation framework for ML models that optimizes the trade-off between the number of items and raters per item to build reproducible AI benchmarks that capture human disagreement.

The research finds that the common practice of 3-5 raters is often insufficient and that more than 10 raters per item is often needed.

The simulator has been open-sourced on GitHub.