From the source
Ai2 released Old Ideas, Novel Problems, a study finding that LLM novelty judges give unstable verdicts on research ideas, along with a benchmark, data and code.
- The work tests whether large language models can reliably tell a genuinely new research direction from one that repackages existing work, a judgment increasingly delegated to another LLM.
- Across six judges, GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6, small changes to the evaluation setup produced dramatically different judgments of the same ideas.
- In one experiment, changing just a few words in a prompt shifted a model's accuracy at identifying which of two ideas is more novel by more than 50 percentage points.
- Retrieval and additional reasoning did not consistently improve judgments, and two purpose-built novelty evaluation systems performed worse than a simple prompted Claude Opus 4.6 baseline.
- Ai2 built the benchmark around ICLR 2026 peer reviews, mining reviewer passages that explicitly discussed originality rather than treating acceptance as a proxy for novelty.
- It assembled 154 high-novelty human ideas from accepted papers in the top-rated 10% of their ICLR research area and 145 lower-novelty ideas from rejected papers in the bottom 10%.
- A second setting paired those 154 human ideas with 154 ideas from a simple LLM generator, which Ai2 treats as weak labels; an expert preferred the human idea in all 30 pairs reviewed.
