From the source
IBM Research introduces ScarfBench, an open benchmark for evaluating AI agents on enterprise Java framework migration tasks.
The benchmark includes 34 applications, 102 framework implementations, 204 migration tasks, about 151K lines of code, and 1,331 expert-written tests.
Evaluations show that even the strongest agents achieve less than 10% behavioral success, agents are overconfident in reporting build success (Claude Code reported 29/30 builds successful but only 22 actually built), migration is iterative, configuration dominates effort, and environment and tooling matter.
The leaderboard is available at scarfbench.info/leaderboard.
