# Ai2 — Can we trust LLMs to judge whether a research idea is novel?

- Company: Ai2 (allenai.org)
- Announced: 2026-10-27
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://allenai.org/blog/novelty-bench
- Record: https://forck.live/items/28347-can-we-trust-llms-to-judge-whether-a-research-idea-is-novel
- Subject: Olmo / Molmo / Asta
- Models affected: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, Claude Opus 4.6

Ai2 released Old Ideas, Novel Problems, a study finding that LLM novelty judges give unstable verdicts on research ideas, along with a benchmark, data and code. The work tests whether large language models can reliably tell a genuinely new research direction from one that repackages existing work, a judgment increasingly delegated to another LLM. Across six judges, GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, small changes to the evaluation setup produced dramatically different judgments of the same ideas. In one experiment, changing just a few words in a prompt shifted a model's accuracy at identifying which of two ideas is more novel by more than 50 percentage points. Retrieval and additional reasoning did not consistently improve judgments, and two purpose-built novelty evaluation systems performed worse than a simple prompted Claude Opus 4.6 baseline. Ai2 built the benchmark around ICLR 2026 peer reviews, mining reviewer passages that explicitly discussed originality rather than treating acceptance as a proxy for novelty. It assembled 154 high-novelty human ideas from accepted papers in the top-rated 10% of their ICLR research area and 145 lower-novelty ideas from rejected papers in the bottom 10%. A second setting paired those 154 human ideas with 154 ideas from a simple LLM generator, which Ai2 treats as weak labels; an expert preferred the human idea in all 30 pairs reviewed.

## Evidence

Verbatim from https://allenai.org/blog/novelty-bench:

> Today we’re releasing Old Ideas, Novel Problems , a new study of how reliably large language models can evaluate the novelty of scientific research ideas. Alongside the paper, we’re publishing a benchmark, data, and code for testing novelty judges, plus an automated pipeline researchers can use to build new evaluation sets.
> 
> At Ai2, we’ve been developing AI systems that support different parts of scientific discovery. DataVoyager helps researchers explore and analyze structured data, AutoDiscovery generates and tests hypotheses, and Theorizer synthesizes evidence across large bodies of literature into candidate scientific theories. Together with Asta , these projects point toward systems that can help researchers move from understanding prior work to proposing, testing, and refining new ideas.
> 
> But generating research ideas is only part of the challenge of automating scientific discovery. Those ideas also need to contribute something genuinely new, which means these systems need a way to distinguish a novel direction from one that mostly repackages existing work.
> 
> That judgment is often handed to another LLM—what we call a novelty judge .

---

Record: https://forck.live/items/28347-can-we-trust-llms-to-judge-whether-a-research-idea-is-novel
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md