Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies

Shaolong Chen; Yanlin Fei; Nazhou Liu; Xinmiao Yu; Lei Li; Rahul Thapa; Madalina Ciobanu; Navan Preet Singh

cs.AIAug 17, 2026

Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies

Authors: Shaolong Chen, Yanlin Fei, Nazhou Liu, Xinmiao Yu, Lei Li, Rahul Thapa, Madalina Ciobanu, Navan Preet Singh,

Organizations: Prentis AI, San Francisco, CA · Stanford University, Stanford, CA · Titan Holdings, San Francisco, CA

Abstract

Can a language model recover the true research idea of a published paper when given only that paper's pre-publication bibliography? We introduce Reconstruction, a blind idea-recovery benchmark that withholds the seed paper and all contemporaneous or future literature, and asks models to propose hypotheses that an independent large language model judge matches against the held-out ground-truth idea. A strict anti-leakage protocol-temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies, which prevents prompt-time leakage of the seed idea. Across six scientific domains and 643 evaluated papers, seven frontier models achieve only modest Match rates (approx. 3-15%). We then evaluate a reference-only multi-agent (top 4) pipeline that combines cross-model review with a Swiss tournament over aligned hypothesis slots, without external web search. Cross-model review plus tournament selection raises Match rates to approx. 23-42% across all six domains, which is an observed approx. 2.4x lift over the best single-model baseline. This draft reports the protocol, anti-leakage design, and current results as an arXiv timestamp.

Figures & tables

Domain	CSV	Not launched	Unresolved	References $<3$	No date	Early stop	Default OK	MA $n_{s}{<}5$	Reported $n$
			Incomplete (by reason)
ML	168	0	24	18	1	0	125	5	120
Astronomy	91	4	0	0	0	0	87	2	85
Chemistry	115	4	1	1	0	0	109	4	105
Materials	131	9	0	1	0	0	121	3	$117^{a}$
Medicine	221	26	0	0	0	35	160	2	$78^{b}$

Table 1: Seed retention from CSV titles to the reported seed-paper set, with launched-but-incomplete failures broken out by reason. CSV : parsed titles; Not launched : excluded before Default run_started ; Incomplete (by reason) : launched but not completed; the four columns sum to

81{=}826{-}745

(CSV

-

Not launched

-

Default OK); Unresolved : title unresolved; References

<3

: fewer than three pre-

T_{0}

references; No date : no usable publication date; Early stop : launched papers left unfinished when the job hit an early-stop completion target chosen to limit time and cost (all 35 are Medicine: 195 launched, stop at 160); Default OK : completed blind-bibliography reconstruction; MA

n_{s}{<}5

: multi-agent–aligned papers dropped because at least one top 4 hypothesis-generation call returned fewer than five hypotheses (Swiss slot count

=\min_{m}|\mathrm{hyps}_{m}|{<}5

); Reported

n

: counts used in Table 2 (multi-agent–aligned with

n_{s}{=}5

). a Materials: 121 Default OK, minus one multi-agent content_filter refusal and three MA

n_{s}{<}5

drops

\Rightarrow

Reported

n{=}117

. b Medicine: Default completes 160; multi-agent evaluates the first 80 for cost, then drops two MA

n_{s}{<}5

papers

\Rightarrow

Reported

n{=}78

. Warning: Claude-Opus-4.8 can return empty hypotheses under OpenRouter content_filter ; in the Medicine Default audit this affected 4 completed papers (proposer marked unavailable; other models still scored). This is distinct from the Early-stop and MA

n_{s}{<}5

columns.

Model	ML 120	Astro 85	Chem 105	Mat 117	Med 78	Phys 138	Avg
Claude-Opus-4.8	$\mathbf{8.2}{\pm}2.4$	$\mathbf{14.7}{\pm}3.1$	$14.0{\pm}5.3$	$\mathbf{14.0}{\pm}3.7$	$14.2{\pm}3.0$	$\mathbf{15.0}{\pm}3.2$	$\mathbf{13.3}{\pm}2.3$
GPT-5.6-Sol-Pro	$7.6{\pm}1.3$	$12.9{\pm}2.6$	$\mathbf{15.0}{\pm}3.8$	$13.8{\pm}2.9$	$\mathbf{14.6}{\pm}2.7$	$12.7{\pm}2.1$	$12.8{\pm}2.4$
Kimi-K3	$7.9{\pm}1.9$	$10.7{\pm}2.6$	$9.8{\pm}2.6$	$10.1{\pm}2.2$	$11.1{\pm}1.9$	$10.7{\pm}2.4$	$10.0{\pm}1.0$
GLM-5.2	$7.2{\pm}1.2$	$9.5{\pm}2.8$	$9.8{\pm}3.0$	$9.2{\pm}2.3$	$10.1{\pm}2.0$	$10.2{\pm}2.5$	$9.3{\pm}1.0$
Gemini 3.1-Pro-Preview	$5.9{\pm}0.6$	$8.9{\pm}1.6$	$10.2{\pm}2.6$	$8.4{\pm}2.2$	$10.5{\pm}1.8$	$9.2{\pm}1.7$	$8.9{\pm}1.5$
DeepSeek-V4-Pro	$3.4{\pm}1.3$	$6.7{\pm}2.5$	$6.6{\pm}2.8$	$7.2{\pm}3.0$	$6.9{\pm}2.2$	$7.0{\pm}2.6$	$6.3{\pm}1.3$

Table 2: Reconstruction Match rates (%). For Default rows, cells show mean

\pm

std across leave-one-out judges (six judges, or three peer top models in the dagger block). For multi-agent, the center is the weighted judge-set aggregate

\mathrm{Match}_{d}(\mathrm{MA})

;

\pm

is the corresponding weighted

\sigma_{d}(\mathrm{MA})

. The Avg column reports the unweighted mean across the six domain columns, with

\pm

the standard deviation across domains (not across judges). Within each Default block (all seven models; dagger top 4), bold marks the column maximum. vs best single is multi-agent divided by the best dagger (top 4, other-top 3 judges) single-model score in that column.

Figure 1: Best top 4 single-model Match rate (dagger panel) vs. multi-agent (top 4) on six domains. Both bars include error bars showing the population standard deviation across their respective eligible judge scores. Annotations show the lift ratio.

Domain	$n$	MA Match [CI]	Best † [CI]	Lift [CI]	MA s@5	Best † s@5
ML	120	$22.9$ $[17.9,28.4]$	$9.3$ $[7.3,11.3]$	$2.5$ $[2.0,3.0]$	$40.1$	$38.9$
Astronomy	85	$36.5$ $[30.5,44.2]$	$15.5$ $[12.6,18.6]$	$2.4$ $[2.0,2.8]$	$58.9$	$56.9$
Chemistry	105	$38.4$ $[31.9,45.8]$	$16.8$ $[14.2,19.5]$	$2.3$ $[2.0,2.6]$	$59.0$	$60.6$
Materials	117	$40.1$ $[33.9,46.2]$	$15.4$ $[12.9,18.0]$	$2.6$ $[2.2,3.0]$	$63.3$	$54.4$
Medicine	78	$41.6$ $[34.9,49.6]$	$15.8$ $[13.3,18.5]$	$2.6$ $[2.3,3.1]$	$65.0$	$66.2$
Physics	138	$36.4$ $[31.1,41.7]$	$15.8$ $[13.6,18.2]$	$2.3$ $[2.0,2.6]$	$59.4$	$58.0$

Table 3: Paper-level bootstrap percentile 95% CIs for Match rate and lift vs. the best fixed top 4 dagger model per domain, plus success@5 (Appendix C ). Lift CI is for the ratio of bootstrap means within each domain (Overall uses the pooled

n{=}643

ratio). Paired wins use the stronger per-paper max among top 4 dagger models.

		Seed		Default		MA
Domain	$n$	mean	median	mean	median	mean	median
ML	120	182	178	45	40	93	77
Astronomy	85	204	210	67	58	135	112
Chemistry	105	166	163	53	48	110	90
Materials	117	175	172	56	50	113	95
Medicine	78	257	253	58	53	121	104

Table 4: Length in English words by domain (whitespace tokens). Seed : seed-paper title

+

abstract (one value per paper). Default / MA : hypothesis title

+

summary (Default: all top 4 hypotheses on the aligned papers; MA: five final hypotheses per paper).

Domain	$n$	Best †	A	B	C	D	MA
ML	120	9.3	7.1	8.5	22.0	31.3	22.9
Astronomy	85	15.5	14.6	13.0	31.6	46.9	36.5
Chemistry	105	16.8	12.2	14.2	33.7	47.7	38.4
Materials	117	15.4	13.2	13.2	32.9	45.8	40.1
Medicine	78	15.8	14.7	13.8	32.9	51.1	41.6
Physics	138	15.8	13.5	13.3	32.5	47.1	36.4

Table 5: Zero-compute bounds on candidate count from the Default top 4

4\times 5

grid (dagger peer judges). Best † : fixed domain-best top 4 model. A : one seeded realization of slot-wise random selection (one of four models chosen uniformly in each slot; seed 42). B : mean Match rate over all 20 hypotheses. By linearity,

\mathbb{E}[A]=B

for each paper (Appendix E ), but the reported realization of

A

need not equal

B

. C : slot oracle (

\max

over four models per slot; match-label peeking). D : unconstrained oracle (mean of the five highest among all 20; match-label peeking, no slot constraint). MA : multi-agent Match rate (Table 2 ). Entries are domain means of paper-level rates (%). This does not replace a matched best-of-20 control.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Match — ML (proposer: Kimi-K3)
seed	How much do language models memorize? We propose a new method for estimating how much a model knows about a datapoint and use it to measure the capacity of modern language models. Prior studies of language model memorization have struggled to disentangle memorization from generalization. We formally separate memorization into two components: unintended memorization, the information a model contains about a specific dataset, and generalization, the information a model contains about the true data-generation process. When we completely eliminate generalization, we can compute the total memorization, which provides an estimate of model capacity: our measurements estimate that GPT-style models have a capacity of approximately 3.6 bits per parameter. We train language models on datasets of increasing size and observe that models memorize until their capacity fills, at which point”grokking”begins, and unintended memorization decreases as models begin to generalize. We train hundreds of transformer language models ranging from 500K to 1.5B parameters and produce a series of scaling laws relating model capacity and data size to membership inference.
Default	How Many Bits Can a Language Model Memorize? Measuring Raw Memorization Capacity with Structureless Data Train transformer language models on uniformly random, structure-free token sequences so that generalization is impossible by construction, then use compression- and information-theoretic accounting to measure the total number of bits the model stores, yielding a clean bits-per-parameter estimate of memorization capacity and a principled separation of memorization from generalization.
MA	How Many Bits per Parameter? Measuring Stored, Retrievable, and Extractable Memorization of Language Models via Keyed Random-Data Pretraining Isolate pure memorization from generalization by pretraining decoder-only transformers on synthetic corpora of key-to-payload records built from uniformly random bitstrings, which are incompressible with high probability, so any predictability the model acquires is memorization by construction. Using a compression-based, description-length definition of memorization, measure three nested quantities—stored bits (code-length reduction given the model), retrievable bits (exact payload recovery given the key), and extractable bits (black-box recovery without keys, following)—as functions of parameter count, data load, epochs, and numerical precision, yielding empirical capacity curves rather than a presumed universal constant.
No-match — Physics (proposer: Claude-Opus-4.8)
seed	Emergence of phase coherence in a magnon Bose–Einstein condensate The spontaneous emergence of coherence is a defining feature of multibody quantum systems, underlying phenomena from superconductivity to quantum information processing. Although Bose–Einstein condensates provide a unique setting for studying this process, direct observation of how a condensate acquires a coherent global phase has remained challenging. Here we provide evidence of spontaneous phase formation in a magnon Bose–Einstein condensate. Using a phase-referenced detection technique, we track the phase of the condensate coherent state relative to an external reference, revealing how an initially incoherent magnon gas thermalizes and undergoes a spontaneous transition into a coherent quantum state with a well-defined macroscopic phase. This observation provides evidence of spontaneous symmetry breaking in a quasiparticle condensate, confirming a central prediction of Bose–Einstein condensation theory that extends across diverse quantum systems.
Default	Interference of Two Independently Prepared Room-Temperature Magnon Condensates Prepare two spatially separated magnon Bose-Einstein condensates in a YIG film via localized parametric pumping and let them expand and overlap, observing matter-wave-like interference fringes analogous to atomic BEC interference, but at room temperature.

Appendix

Table 6: Borderline Match / No-match examples illustrating the final judge rubric (Appendix A ). Within each example the Default and multi-agent (MA) hypotheses share the same seed paper and origin proposer. Seeds are shown as title plus abstract; hypotheses as title plus summary.

Domain	2023	2024	2025	2026	Total
ML	$0$ ( $0.0\%$ )	$2$ ( $1.7\%$ )	$33$ ( $27.5\%$ )	$85$ ( $70.8\%$ )	$120$
Astronomy	$0$ ( $0.0\%$ )	$3$ ( $3.5\%$ )	$17$ ( $20.0\%$ )	$65$ ( $76.5\%$ )	$85$
Chemistry	$0$ ( $0.0\%$ )	$0$ ( $0.0\%$ )	$3$ ( $2.9\%$ )	$102$ ( $97.1\%$ )	$105$
Materials	$1$ ( $0.9\%$ )	$6$ ( $5.1\%$ )	$11$ ( $9.4\%$ )	$99$ ( $84.6\%$ )	$117$
Medicine	$0$ ( $0.0\%$ )	$0$ ( $0.0\%$ )	$1$ ( $1.3\%$ )	$77$ ( $98.7\%$ )	$78$
Physics	$1$ ( $0.7\%$ )	$16$ ( $11.6\%$ )	$51$ ( $37.0\%$ )	$70$ ( $50.7\%$ )	$138$

Appendix

Table 7: Year

\times

domain counts for the aligned

n{=}643

seeds (earliest public date). Each cell shows count and share of that domain’s row total.

Model	KC	ML	Astro	Chem	Mat	Med	Phys	Possible %
Gemini 3.1-Pro-Preview	2025-01-31	$2$	$4$	$0$	$7$	$0$	$20$	$5.1\%$
DeepSeek-V4-Pro	2025-12-29 ∗	$35$	$20$	$3$	$18$	$1$	$68$	$22.6\%$
Qwen3.7-Max	2026-01-25 ∗	$40$	$33$	$21$	$32$	$29$	$76$	$35.9\%$
Claude-Opus-4.8	2026-01-31	$44$	$34$	$24$	$44$	$33$	$81$	$40.4\%$
GPT-5.6-Sol-Pro	2026-02-16	$67$	$39$	$28$	$47$	$42$	$87$	$48.2\%$
GLM-5.2	2026-02-20 ∗	$69$	$39$	$30$	$51$	$44$	$88$	$49.9\%$

Appendix

Table 8: Possible parametric-contamination candidates by domain: count of aligned seeds whose earliest public date is on or before the model knowledge cutoff (date

\leq

KC). Month-only KCs are filled to month-end. Starred KCs are inferred as release

{-}116

days (the minimum known release

-

KC gap among Claude / GPT / Gemini). The last column is the share of all

643

seeds.

Model / split	ML	Astro	Chem	Mat	Med	Phys	Avg
Claude-Opus-4.8 (all)	$8.2{\pm}2.4$	$14.7{\pm}3.1$	$14.0{\pm}5.3$	$14.0{\pm}3.7$	$14.2{\pm}3.0$	$15.0{\pm}3.2$	$13.3{\pm}2.3$
Claude-Opus-4.8 ( $\leq$ 2026-03-22)	$8.0{\pm}2.4$	$15.5{\pm}3.0$	$16.7{\pm}5.8$	$15.8{\pm}3.4$	$15.5{\pm}3.4$	$16.2{\pm}3.2$	$14.6{\pm}3.0$
Claude-Opus-4.8 ( $>$ 2026-03-22)	$8.6{\pm}2.7$	$13.4{\pm}3.5$	$12.0{\pm}5.0$	$11.0{\pm}4.3$	$10.5{\pm}2.1$	$12.3{\pm}3.6$	$11.3{\pm}1.5$
GPT-5.6-Sol-Pro (all)	$7.6{\pm}1.3$	$12.9{\pm}2.6$	$15.0{\pm}3.8$	$13.8{\pm}2.9$	$14.6{\pm}2.7$	$12.7{\pm}2.1$	$12.8{\pm}2.4$
GPT-5.6-Sol-Pro ( $\leq$ 2026-03-22)	$7.7{\pm}1.6$	$14.4{\pm}3.2$	$18.4{\pm}4.8$	$14.8{\pm}3.6$	$16.0{\pm}2.8$	$13.8{\pm}2.1$	$14.2{\pm}3.2$
GPT-5.6-Sol-Pro ( $>$ 2026-03-22)	$7.3{\pm}0.8$	$10.7{\pm}1.8$	$12.4{\pm}3.1$	$12.2{\pm}2.2$	$10.7{\pm}2.5$	$10.0{\pm}2.1$	$10.6{\pm}1.7$

Appendix

Table 9: Match rates (%) stratified at earliest public date 2026-03-22. Per model: all (

n{=}643

\leq

(

n{=}407

>

(

n{=}236

). Aggregation matches Table 2 . vs best single is multi-agent divided by the best dagger score in that column within the same date stratum.

Model	ML	Astro	Chem	Mat	Med	Phys	Avg
Claude-Opus-4.8	$-0.6$	$+2.1$	$+4.7$	$+4.9$	$+5.0$	$+3.9$	$+3.3$
GPT-5.6-Sol-Pro	$+0.4$	$+3.7$	$+5.9$	$+2.7$	$+5.3$	$+3.8$	$+3.6$
Kimi-K3	$+2.8$	$+0.6$	$+3.7$	$+4.1$	$+0.1$	$+1.8$	$+2.2$
GLM-5.2	$-3.0$	$0.0$	$+6.7$	$+3.8$	$+4.4$	$+3.8$	$+2.6$
Gemini 3.1-Pro-Preview	$-0.9$	$+1.1$	$+6.0$	$+3.4$	$+1.5$	$+0.7$	$+2.0$
DeepSeek-V4-Pro	$-0.8$	$+3.6$	$+1.8$	$+4.6$	$+2.1$	$+2.8$	$+2.4$

Appendix

Table 10: Per-domain

\Delta=\mathrm{Match}(\leq)-\mathrm{Match}(>)

(%) for the fixed cut 2026-03-22 (Table 9 ). Positive

\Delta

means the

\leq

side scores higher. Caution: the two sides are disjoint paper sets of unequal size (

n_{\leq}{=}407

vs.

n_{>}{=}236

) with different domain and year composition, so

\Delta

is a difference between two independent subset means, not a within-paper change; per-domain cells rest on even smaller subsets.

Model / split	ML	Astro	Chem	Mat	Med	Phys	Avg
Claude-Opus-4.8 (all)	$8.2{\pm}2.4$	$14.7{\pm}3.1$	$14.0{\pm}5.3$	$14.0{\pm}3.7$	$14.2{\pm}3.0$	$15.0{\pm}3.2$	$13.3{\pm}2.3$
Claude-Opus-4.8 ( $\leq$ 2026-01-31) [ $n$ =260]	$8.3{\pm}2.6$	$15.4{\pm}3.0$	$18.2{\pm}4.3$	$18.5{\pm}3.1$	$15.2{\pm}3.8$	$16.9{\pm}3.4$	$15.4{\pm}3.4$
Claude-Opus-4.8 ( $>$ 2026-01-31) [ $n$ =383]	$8.1{\pm}2.3$	$14.2{\pm}3.5$	$12.8{\pm}5.6$	$11.2{\pm}4.2$	$13.5{\pm}2.4$	$12.3{\pm}3.2$	$12.0{\pm}2.0$
GPT-5.6-Sol-Pro (all)	$7.6{\pm}1.3$	$12.9{\pm}2.6$	$15.0{\pm}3.8$	$13.8{\pm}2.9$	$14.6{\pm}2.7$	$12.7{\pm}2.1$	$12.8{\pm}2.4$
GPT-5.6-Sol-Pro ( $\leq$ 2026-02-16) [ $n$ =310]	$7.9{\pm}1.7$	$15.6{\pm}2.9$	$21.4{\pm}4.5$	$14.8{\pm}3.6$	$14.7{\pm}2.8$	$14.6{\pm}2.2$	$14.8{\pm}3.9$
GPT-5.6-Sol-Pro ( $>$ 2026-02-16) [ $n$ =333]	$7.2{\pm}0.9$	$10.6{\pm}2.5$	$12.6{\pm}3.6$	$13.1{\pm}2.6$	$14.5{\pm}2.7$	$9.4{\pm}2.1$	$11.2{\pm}2.5$

Appendix

Table 11: Match rates (%) stratified by each proposer’s knowledge cutoff (KC). Default/dagger

\leq

KC /

>

KC rows use that proposer’s KC at the paper level. Multi-agent assigns each paper uniquely using the latest KC among its five Swiss winners’ origin proposers (conservative

>

set):

n_{\leq}{=}406

n_{>}{=}237

. vs best single uses the best dagger score in the same stratum; the

\leq

KC/

>

KC lift rows are not a shared paper split.

Model	ML	Astro	Chem	Mat	Med	Phys	Avg
Claude-Opus-4.8	$+0.2$	$+1.2$	$+5.4$	$+7.2$	$+1.8$	$+4.6$	$+3.4$
GPT-5.6-Sol-Pro	$+0.7$	$+5.1$	$+8.8$	$+1.7$	$+0.1$	$+5.2$	$+3.6$
Kimi-K3	$+2.8$	$+0.6$	$+3.7$	$+4.1$	$+0.1$	$+1.8$	$+2.2$
GLM-5.2	$-1.3$	$+2.4$	$+6.5$	$+2.9$	$+3.2$	$+4.7$	$+3.1$
Gemini 3.1-Pro-Preview	$-6.0$	$+9.0$	—	$+1.2$	—	$+1.3$	$+0.6$
DeepSeek-V4-Pro	$-0.3$	$+4.6$	$+4.7$	$+2.1$	$-7.0$	$+1.3$	$+0.9$

Appendix

Table 12: Per-domain

\Delta=\mathrm{Match}(\leq\mathrm{KC})-\mathrm{Match}(>\mathrm{KC})

(%) under each model’s own KC (Table 11 ). For multi-agent, sides use the max origin-proposer KC; — marks empty strata (

n{=}0

). Caution: as in Table 10 the two sides are disjoint sets of unequal size, and here both their sizes and their membership change from row to row because each proposer has its own KC (from

n_{\leq}{=}33

for Gemini to

n_{\leq}{=}407

for Kimi; see Table 11 ). Rows are therefore not computed on a common split and

\Delta

values should not be compared across models as if they were; small-

n_{\leq}

rows (Gemini, DeepSeek) are especially unstable.

Scientific Reasoning in Language Models

Scientific reasoning in language models studies how AI systems use evidence, domain knowledge, and tools to solve problems and support discovery.

See all 132 papers

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research
Oct 1, 2026Sohyeon Kim, Yoonho Lee, Bo Liu +11AI Agents for Scientific Discovery Agentic Search
Stanford University · Seoul National University · University of Washington +3
Read, Grep, and Synthesize: Diagnosing Cross-Domain Seed Exposure for LLM Research Ideation
May 12, 2026Yunju Choi, Min SongResearch Idea Generation Large Language Model-Guided Scientific Discovery
1Yonsei University, Seoul, Republic of Korea · 2OnomaAI, Seoul, Republic of Korea.
ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes
Jul 5, 2026Qihao Zhao, Yangyu Huang, Yalun Dai +8Research Idea Generation Evidence-Grounded Generation
1Nanyang Technological University · 2Microsoft Research · 3National University of Singapore +1

Oct 1, 2026J/K move · Enter open · S save

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

Sohyeon Kim, Yoonho Lee, Bo Liu, Dayoon Ko+10

Stanford University · Seoul National University · University of Washington+3

AI Agents for Scientific Discovery Agentic Search AI Agent Benchmarks Information Retrieval IR Evaluation

Open paper AskPDF arXiv

What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.

Abstract

Scientific Reasoning in Language Models

Explore similar work

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

Read, Grep, and Synthesize: Diagnosing Cross-Domain Seed Exposure for LLM Research Ideation

ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes