What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation

Ziyue Wang; Aomufei Yuan; Yiran Yao; Linli Yao; Hongyao Zuo; Ziwen Gong; Yuanxin Liu; Shicheng Li

cs.CLAug 24, 2026

What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation

Authors: Ziyue Wang, Aomufei Yuan, Yiran Yao, Linli Yao, Hongyao Zuo, Ziwen Gong, Yuanxin Liu, Shicheng Li,

Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Peking University · Tianjin University · Hainan University ·

Abstract

Large language models are increasingly used to propose research ideas, yet the prevailing ways of judging such ideas supply no shared decision rule: free-form judging sways with style and position, and scoring against a later paper rewards recovery of one realized trajectory. We introduce a benchmark that carries a proposal from Literature to Test: the Lit2Test benchmark centers on a six-field contract organized around a falsifying outcome, so that every proposal precommits the observation that would prove it wrong, making its quality decidable in the first place rather than merely arguable. Built prospectively from 200 real-paper neighborhoods, Lit2Test elicits proposals from four frontier models and compares them through 1,200 pairwise comparisons judged blind in both presentation orders. The protocol audits its own reliability through diagnostic controls and bounded human calibration, with three annotators corroborating the conclusions within explicitly stated reliability bounds. Lit2Test recovers a strict ranking of the four models in all 10,000 bootstrap replicates, and the separation comes from the quality of the proposed tests and metrics rather than from surface fluency. We release the benchmark, construction pipeline, and audit artifacts for public use.

Figures & tables

Figure 1: Three evaluation paradigms on the same input, a real ICLR neighborhood on long-context QA; every panel shows actual runs on this context, with idea I generated by GPT-5.2 and idea II by GLM-5. The two existing paradigms disagree: (a) free-form judging prefers idea I in both presentation orders, while (b) future-paper-as-answer scoring ( 25 ) prefers idea II and reverses with the choice of answer-key paper. (c) Our six-field contract route judges the same two models’ proposals written under the contract and yields a grounded verdict; “order-audited” means each pair is judged in both presentation orders.

Figure 2: The Lit2Test pipeline: (a) benchmark construction and (b) the shipped evaluation protocol, with specimen chips tracing neighborhood #034 (the Figure 1 neighborhood) through every stage. Each of the 1,200 canonical pairs is judged in both orders and finally folded into 950 order-stable plus 250 order-sensitive cases; only stable cases feed the ranking aggregation.

Model	BT [95% CI]	Primary	J2	vs. GPT	vs. CS	vs. GLM	vs. DS	Human WR
		Stable W–L–T		Folded head-to-head (W–L–T)
GPT-5.2	$\phantom{-}1.26$ [ $\phantom{-}1.14,\phantom{-}1.40$ ]	424–49–127	423–52–125	—	90–38–72	154–7–39	180–4–16	.805
Claude Sonnet 4.6	$\phantom{-}0.74$ [ $\phantom{-}0.62,\phantom{-}0.87$ ]	347–119–134	391–74–135	38–90–72	—	143–18–39	166–11–23	.775
GLM-5	$-0.73$ [ $-0.86,-0.61$ ]	118–343–139	97–405–98	7–154–39	18–143–39	—	93–46–61	.233
DeepSeek-V3.2	$-1.28$ [ $-1.42,-1.15$ ]	61–439–100	69–449–82	4–180–16	11–166–23	46–93–61	—	.214

Table 1: Main results on Lit2Test. BT : centered log-ability from the Bradley–Terry fit on order-stable cases (§3.3); 95% CIs from a 10,000-replicate case-level bootstrap. Primary / J2 Stable W–L–T : per-model wins–losses–ties over the 600 folded cases involving that model; W and L count order-stable cases, T counts order-sensitive cases. The primary judge is Gemini 3.1 Pro (Preview) (950 stable / 250 sensitive); J2 is an independent second judge (Doubao Seed 2.0 Pro) re-judging all 2,400 comparisons (980 stable / 220 sensitive). Folded head-to-head : per-pair W–L–T from the row model’s perspective over 200 folded cases per pair (primary judge). Human WR : majority win rate from the stratified human-calibration study (20 neighborhoods, 90 real pairs, 3 annotators; ties counted as 0.5). Abbreviations: GPT = GPT-5.2, CS = Claude Sonnet 4.6, GLM = GLM-5, DS = DeepSeek-V3.2.

Figure 3: Diagnostic controls and human calibration. (a) Target-score drops under clear and subtle single-field corruptions (0–4 rubric, 95% CIs;

n{=}20

cases per dimension); clear corruptions produce large drops while subtle ones produce smaller but nonzero drops. (b) Sham-adjusted preference for the clean proposal under subtle corruption, net of style-matched sham edits; all CIs exclude zero. (c) Judge stable-case BT strength vs. human majority win rate; dashed lines mark the BT midpoint and the chance baseline. Both instruments recover the same ranking and the same two-tier separation. Human–judge agreement: 87.2% on decisive stable cases (34/39), 11/12 naive-control detection, 88.3% bootstrap rankings within one inversion.

System	Output unit	Min. test	Metric	Sup./Fals.	Order	Human	Answer key
Si et al. ( 26 )	Full proposal	N	P	P	N	Y	Experts
AI Idea Bench ( 21 )	Motiv.+plan	N	N	N	N	N	Target paper
IdeaBench ( 5 )	Hypothesis text	N	N	N	N	N	Target paper
HARPA ( 29 )	Full proposal	P	Y	P	N	N	Exec. logs
ResearchBench ( 15 )	Hypothesis	N	N	N	Y	N	Target paper
LiveIdeaBench ( 23 )	Short idea	N	N	N	N	Y	Judge panel

Table 2: Positioning of Lit2Test against the closest research-ideation neighbors. Min. test : proposal must specify a minimal controlled experiment. Metric : an adjudicating measurement is required. Sup./Fals. : explicit bidirectional outcomes are required and evaluated. Order : same-pair forward/reverse reversal is measured. Human : bounded human calibration of the judge. Answer key : reference signal used for scoring. Y = yes, P = partial or generation-time only, N = no.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Batch	Internal name	Contexts
1	expansion40	40
2	next40	40
3	third40	40
4	fourth40	40
5	fifth40	40

Appendix

Table 6

Pair	W (row)	L (row)	T
GPT-5.2 vs Claude Sonnet 4.6	90	38	72
GPT-5.2 vs GLM-5	154	7	39
GPT-5.2 vs DeepSeek-V3.2	180	4	16
Claude Sonnet 4.6 vs GLM-5	143	18	39
Claude Sonnet 4.6 vs DeepSeek-V3.2	166	11	23
GLM-5 vs DeepSeek-V3.2	93	46	61

Appendix

Table 3: Complete six-pair folded outcomes (primary judge).

Model	Pair CI	Cluster CI	$\Delta$
GPT-5.2	0.268	0.299	+11.4%
Claude Sonnet 4.6	0.248	0.302	+21.9%
GLM-5	0.245	0.311	+27.0%
DeepSeek-V3.2	0.277	0.333	+20.2%

Appendix

Table 4: Confidence interval comparison: pair-level vs. context-cluster bootstrap (10,000 replicates each, centered log-ability).

Diagnostic	Result
Dimension audit valid judgments	188/188
Structured vs. holistic agreement	152/180 (84.4%)
Structured order-consistency	65/90 (72.2%)
Hidden real-vs-naive (auto)	8/8
Manipulation check valid judgments	120/120
clean-answer wins (per dimension)	40/40

Appendix

Table 5: Diagnostic controls and human calibration (moved from main text to preserve space).

Role	Model	Temp.
Participant	GPT-5.2	1.0
Participant	Claude-Sonnet-4.6-hq	1.0
Participant	GLM-5	1.0
Participant	DeepSeek-V3.2	1.0
Primary judge	Gemini-3.1-Pro-Preview	1.0
Second judge	Doubao-Seed-2.0-pro	1.0

Appendix

Table 6: Model identifiers and generation/judging settings.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?
May 28, 2026Sy-Tuyen Ho, Minghui Liu, Huy Nghiem +6AI Agent Benchmarks LLM Evaluation
University of Maryland · All Purpose AI
Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation
Apr 6, 2026Srujan P Mule, Aniketh Garikaparthi, Manasi PatwardhanScientific Idea Evaluation Large Language Model-Guided Scientific Discovery
IISER Pune · TCS Research
Can Large Language Models Forecast What Researchers Study Next?
Sep 1, 2026Fenghai Li, Zihan Tang, Haofei Yu +2LLM Evaluation Forecasting Benchmarks
University of Illinois Urbana-Champaign

May 28, 2026J/K move · Enter open · S save

SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?

Sy-Tuyen Ho, Minghui Liu, Huy Nghiem, Hieu Tran+5

University of Maryland · All Purpose AI

AI Agent Benchmarks LLM Evaluation LLM Reliability Automated Peer Review AI for Science Scientific Idea Evaluation+1

Open paper AskPDF arXiv

Autonomous AI research agents aim to speed up scientific discovery by automating the research pipeline from hypothesis generation to peer review. However, existing benchmarks rarely test a fundamental bottleneck whether LLMs can judge the methodological viability of a research idea

\textbf{before}

investing time and computational resources. Without a reliable ``first-gate'' filter, autonomous agents risk scaling flawed methodology rather than accelerating meaningful science. We introduce

\textbf{SoundnessBench}

, a curated benchmark of 1,099 real research proposals, with source verified extraction, reviewer-derived proxy labels, and proposal-only expert judgments on a subset of 327 proposals. Across 12 frontier LLMs, our evaluation reveals a pervasive

\textbf{optimism bias}

: standard prompting yields a

74.0\%

mean false-positive rate. Aggressive prompting reduces false approvals but substantially increases false rejections. Additional analyses show that optimism persists under controls for public corpus contamination, and surface features, as well as in literature-grounded agent evaluation. These results suggest a limitation in judging scientific soundness, indicating that current LLMs are not yet reliable as standalone gatekeepers for scientific rigor.

Abstract

LLM Evaluation

Explore similar work

SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?

Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation

Can Large Language Models Forecast What Researchers Study Next?