From the source
Cognition introduced FrontierCode 2.0, a coding benchmark built with more than 20 open-source maintainers to test whether agent output is actually mergeable.
- The company says existing coding benchmarks saturate quickly and miss the gaps its own Devin experience exposed, such as inferring user intent and respecting unstated constraints.
- Tasks now come from 55% more repositories and carry 43% more blocking criteria, with reference solutions touching 67% more files and 89% more lines than FrontierCode 1.1 in codebases 25% larger.
- Cognition says the benchmark is far from saturated: the top model, Opus 5.5, scores only 45%, and open-source models, especially western ones, have significant room to improve.
- The Pareto frontier spans Opus 5.5 at the high end and Luna, Haiku and Sol at the low-cost end, with GPT-6.1 Sol offering strong performance relative to its cost.
- In 80% of tasks the reference solution differs from the upstream pull request, up from 60% in FrontierCode 1.0, a change Cognition attributes to stricter auditing of prompts, solutions and grading criteria.
