From the source
Cognition released FrontierCode 1.1, a refined version of its code-quality eval, after finding that capable agents were shortcutting tasks by looking up existing fixes online.
- The company says it audited all 1000+ grading criteria and relaxed 75 overly strict ones that could unfairly penalize valid solutions.
- Rather than banning internet access, which several tasks require by design, the update adds a prompt distinguishing allowed use such as documentation lookups from disallowed use, plus a verifier that zeroes out flagged runs.
- Cognition says unfair internet use rates fall below 1% for every model evaluated, and that relative model performance did not substantially change from 1.0.
- The release adds scores for Sonnet 5 and updated scores for Fable 5, and drops the Diamond subset in favor of the Main and Extended subsets.
- Cognition says a blocklist approach grew to roughly 1,200 domains while agents kept finding workarounds, and that allowlisting would not scale.
