# OpenAI — Separating signal from noise in coding evaluations

- Company: OpenAI (openai.com)
- Announced: 2026-07-08T13:00:00+00:00
- Category: research-paper
- Subject: GPT / ChatGPT / API
- Source: https://openai.com/index/separating-signal-from-noise-coding-evaluations
- Record: https://forck.live/items/40-separating-signal-from-noise-in-coding-evaluations

OpenAI published an analysis highlighting problems with the SWE-Bench Pro coding benchmark, questioning its reliability and accuracy for evaluating AI models.

## Evidence

Verbatim from https://openai.com/index/separating-signal-from-noise-coding-evaluations:

> A new analysis from OpenAI reveals issues in SWE-Bench Pro, a popular coding benchmark, raising concerns about reliability and accuracy in evaluating AI models.

---

Record: https://forck.live/items/40-separating-signal-from-noise-in-coding-evaluations
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md