# Microsoft — Iteratively improving public benchmarks

- Company: Microsoft (microsoft.com)
- Announced: 2026-11-20T17:18:37+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://commandline.microsoft.com/iteratively-improving-public-benchmarks/
- Record: https://forck.live/items/39313-iteratively-improving-public-benchmarks
- Subject: Foundry / Responsible AI

Microsoft says its audit of public coding benchmarks such as DeepSWE and Terminal-Bench 4.0 found that scores hide ambiguous tasks, graders that penalize valid solutions, and execution environments that introduce failures. The company argues that no benchmark is likely to be perfect and that perfection should not be a prerequisite for learning from one, so the value of auditing lies in continuous, rigorous judgment rather than flagging defects alone. Microsoft describes an iterative auditing process in which it submitted fixes where no existing pull requests were available and joined regular syncs with public benchmark maintainers to share findings and processes. It says the goal is to maintain reliable capability coverage and improve the benchmarks as a community effort, and it encourages anyone using public benchmarks to contribute.

## Evidence

Verbatim from https://commandline.microsoft.com/iteratively-improving-public-benchmarks/:

> Independent Review: Hanwen Xing; SWE Review and PR Submissions: Daphne Barretto, Jennifer Marsman, Nishant Manchanda
> 
> Public benchmarks have become a common yardstick for measuring model capabilities, with much attention focused on scores and rankings. Yet through our careful examination of coding benchmarks like DeepSWE and Terminal-Bench 4.0, we found that the story behind those numbers is far more nuanced. Tasks can be ambiguous, graders can penalize valid solutions, and execution environments can introduce failures.
> 
> However, in a rapidly evolving AI landscape, no benchmark is likely to be perfect, nor should perfection be a prerequisite for learning from it. The real value of auditing public benchmarks lies not just in flagging defects, but in bringing continuous, rigorous judgment to the evaluation process, maintaining reliable capability coverage, learning from the findings, and improving them together as community efforts.
> 
> Below, we share our iterative auditing and learning process.

---

Record: https://forck.live/items/39313-iteratively-improving-public-benchmarks
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
