Hugging Face· PlatformResearch paper

ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration

IBM Research introduces ScarfBench, an open benchmark for evaluating AI agents on enterprise Java framework migration tasks. The benchmark includes 34 applications, 102 framework implementations, 204 migration tasks, about 151K lines of code, and 1,331 expert-written tests. Evaluations show that even the strongest agents achieve less than 10% behavioral success, agents are overconfident in reporting build success (Claude Code reported 29/30 builds successful but only 22 actually built), migration is iterative, configuration dominates effort, and environment and tooling matter. The leaderboard is available at scarfbench.info/leaderboard.

From the source

ScarfBench (Self-Contained Application Refactoring Benchmark), an open benchmark for evaluating AI agents on cross-framework migration tasks in Enterprise Java.

huggingface.co

Announced

30 Jun 2026 18:32 UTC