Can we trust LLMs to judge whether a research idea is novel?

Ai2 released "Old Ideas, Novel Problems," a study with a benchmark, data, code, and an automated pipeline for testing LLM novelty judges. Across six judges (GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, Claude Opus 4.6), small prompt changes shifted accuracy by more than 50 percentage points on AI-generated ideas, retrieval and extra reasoning did not consistently help, and two purpose-built novelty evaluators scored worse than a simple prompted Claude Opus 4.6 baseline while costing far more.