# Hugging Face — CWE-bench v2: More coverage, quality, and transparency

- Company: Hugging Face (huggingface.co)
- Announced: 2026-11-03
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://huggingface.co/blog/cwe-bench
- Record: https://forck.live/items/31119-cwe-bench-v2-more-coverage-quality-and-transparency
- Subject: Platform
- Models affected: Gemini 4 Argon, Claude Opus 5.5, DeepSeek V4.1 Flash (1017)
- Pricing: DeepSeek reaches 57.0% pass@1 at $0.04 per rollout, Opus adds ten points at $1.83, and Gemini edges ahead at $3.92

Collinear AI has opened a public set of 100 security-fixing tasks, CWE-bench v2, that anyone can run to evaluate frontier cyber capabilities. Each task gives an agent a checkout of a real open-source repository and a short instruction to find and fix a security weakness without breaking existing functionality, and scoring is fully programmatic so results are deterministic and reproducible at no extra cost. The 100 tasks cover 91 different CWEs, and Collinear says a model that does well must understand the multiple weakness surfaces an attacker could exploit rather than recognise a familiar pattern. Every task was checked against six quality questions and reviewed by hand, and a held-out set is kept for tasks no model has seen. Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 at 67.3% pass@1 and 78.0% pass@4. 95 tasks are solved by at least one model, but the leader still fails 19% of the set on every attempt. On cost, DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier, with DeepSeek reaching 57.0% pass@1 at $0.04 per rollout, Opus adding ten points at $1.83, and Gemini edging ahead at $3.92. The dataset is gated and ships a registry.json, so Harbor can run the whole benchmark from the Hugging Face repo, and the tasks are tagged rl-environment so the same environments and verifiers can reward training defensive agents.

## Evidence

Verbatim from https://huggingface.co/blog/cwe-bench:

> CWE-bench is designed to test the limits of frontier cyber capabilities. In our continued pursuit to push the horizon at Collinear AI , CWE-bench v2 opens up a public set of 100 tasks that anyone can run and evaluate their models on. The setup is the same as before. An agent gets a checkout of a real open-source repository and a short instruction, with the goal of finding and fixing a security weakness without breaking existing functionality. Scoring is fully programmatic, so results are deterministic and anyone can reproduce them without extra cost. All our programmatic checks have been curated with the help of LLM judge panels that shape them towards the direction of hardness and fairness. We also maintain a held-out set , so that we can keep tracking progress on tasks no model has seen. To evaluate your model on the held-out set, contact cyber@collinear.ai .
> 
> Our goal is to be exhaustive. A defender has to anticipate the ways an attacker could get in, and get it right every single time. These 100 tasks cover 91 different CWEs, spanning a wide range of vulnerability classes.

---

Record: https://forck.live/items/31119-cwe-bench-v2-more-coverage-quality-and-transparency
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md