From the source
Collinear AI has opened a public set of 100 security-fixing tasks, CWE-bench v2, that anyone can run to evaluate frontier cyber capabilities.
- Each task gives an agent a checkout of a real open-source repository and a short instruction to find and fix a security weakness without breaking existing functionality, and scoring is fully programmatic so results are deterministic and reproducible at no extra cost.
- The 100 tasks cover 91 different CWEs, and Collinear says a model that does well must understand the multiple weakness surfaces an attacker could exploit rather than recognise a familiar pattern.
- Every task was checked against six quality questions and reviewed by hand, and a held-out set is kept for tasks no model has seen.
- Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 at 67.3% pass@1 and 78.0% pass@4. 95 tasks are solved by at least one model, but the leader still fails 19% of the set on every attempt.
- On cost,
DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier, with DeepSeek reaching 57.0% pass@1 at $0.04 per rollout, Opus adding ten points at $1.83, and Gemini edging ahead at $3.92.
- The dataset is gated and ships a registry.json, so Harbor can run the whole benchmark from the
Hugging Face repo, and the tasks are tagged rl-environment so the same environments and verifiers can reward training defensive agents.
