ViBench 2.0: A Benchmark for Long-Horizon Software Building

From the source

Can agents build whole products one feature at a time? Published: Today we’re releasing ViBench 2.0 , a benchmark for long-horizon software development. It joins the Specialized Intelligence Index (SII) from Fireworks [1] , our partner for open-weight model inference and training.

The SII brings together practitioner-built benchmarks to compare open, closed, and specialized models on real work tasks across industries. Most coding benchmarks, like DeepSWE [2] , hand an agent a well-defined task inside an existing codebase.

In ViBench 2.0, the agent starts with an empty project instead. It then builds a whole product one feature at a time, modeled on products like Uber, GitHub, or Google Docs. ViBench 2.0 has seventeen products, 241 feature requests, and 544 test plans.

The best models, GPT-6.1 Sol, GPT-6 Astra, and Opus 5.5, pass all test plans in only 49 - 65% of their builds. Why we built ViBench 2.0 We modeled ViBench 1.0 [3] on what Replit users were building at the time, like collaborative whiteboards and resume builders. People now come to Replit to build entire businesses, which often include full-featured software products. …

replit.com