From the source
Microsoft says its audit of public coding benchmarks such as DeepSWE and Terminal-Bench 4.0 found that scores hide ambiguous tasks, graders that penalize valid solutions, and execution environments that introduce failures.
- The company argues that no benchmark is likely to be perfect and that perfection should not be a prerequisite for learning from one, so the value of auditing lies in continuous, rigorous judgment rather than flagging defects alone.
- Microsoft describes an iterative auditing process in which it submitted fixes where no existing pull requests were available and joined regular syncs with public benchmark maintainers to share findings and processes.
- It says the goal is to maintain reliable capability coverage and improve the benchmarks as a community effort, and it encourages anyone using public benchmarks to contribute.






