From the source
The software that processes trillions in daily financial settlements, routes telephone calls across continents, and adjudicates insurance claims was written in COBOL, Fortran, and Java 7.
The engineers who understand it are retiring faster than they can be replaced.
Every major coding agent benchmark (SWE-bench, Terminal-Bench, SWE-Lancer) evaluates agents on modern Python and JavaScript.
None of them reflect the reality of working with some of the world's most critical infrastructure.
Today we're announcing Legacy-Bench : a new benchmark designed to measure frontier AI agent capabilities on legacy software engineering tasks.
What is Legacy-Bench Legacy-Bench consists of hundreds of tasks spanning six legacy language families and real enterprise domains.
The full benchmark is used for evaluation, with ten representative tasks publicly available as open samples.
Each task consists of a natural language instruction, a containerized environment with source files and data, a reference solution, and hidden verification tests the agent cannot see.
…
