Inspire: Benchmarking Scientific Literature Search for Open Research Problems

Jianrong Ding; Zhengyan Shi; Jianyuan Zhong; Kai Qiu; Qi Dai; Yifan Yang; Chong Luo; Qiang Xu

cs.IRSep 27, 2026

Inspire: Benchmarking Scientific Literature Search for Open Research Problems

Authors: Jianrong Ding, Zhengyan Shi, Jianyuan Zhong, Kai Qiu, Qi Dai, Yifan Yang, Chong Luo, Qiang Xu

Organizations: Department of Computer Science and Engineering, The Chinese University of Hong Kong · Microsoft Research Asia

Abstract

Scientific literature search often begins with an open research problem rather than a known target paper or a fixed candidate set. We introduce INSPIRE, a benchmark for evaluating agents that search prior literature to make progress on solution-redacted research problems. Each instance pairs a research brief with a target-specific cutoff three months before a later paper and evaluates ranked outputs against graded cited antecedents from that paper's realized research lineage. Search proceeds over an open corpus, while the identity of the target paper and membership of its cited antecedents remain hidden from the agent. Beyond end-to-end retrieval quality, INSPIRE uses logged search trajectories to distinguish three coupled stages: resource exposure, whether useful antecedents are surfaced during search; selection, whether exposed antecedents are retained; and ranking, how effectively retained papers are ordered. Across 476 computer-science targets under a shared search interface and budget, the strongest evaluated agent achieves 0.284 nDCG@10. Results show that current agents more readily recover an isolated antecedent than assemble a broader portfolio of relevant prior work. The stagewise analysis identifies resource exposure as the largest observed bottleneck, with further losses in selection and ranking. We additionally construct replay-valid hindsight demonstrations and show that they improve held-out search without changing test-time information, establishing that the benchmark provides an actionable learning signal. INSPIRE therefore enables both end-to-end comparison and stage-resolved diagnosis in a setting where the agent must construct its own working criterion of relevance.

Figures & tables

	Task setting		Research capability evaluated
Benchmark	Interactive open-corpus search	No supplied relevance cue	Resource exposure	Selection	Ranking
AutoResearchBench ( Xiong et al., 2026 )	$\bm{\checkmark}$	$\bm{\times}$	$\bm{\checkmark}$	$\bm{\times}$	$\bm{\times}$
PaperPilot ( Li et al., 2026 )	$\bm{\checkmark}$	$\bm{\times}$	$\bm{\checkmark}$	$\bm{\times}$	$\bm{\checkmark}$
ResearchBench ( Liu et al., 2025 )	$\bm{\times}$	$\bm{\times}$	$\bm{\times}$	$\bm{\checkmark}$	$\bm{\times}$
HyBIRD ( Yang et al., 2026 )	$\bm{\times}$	$\bm{\checkmark}$	$\bm{\checkmark}$	$\bm{\times}$	$\bm{\checkmark}$
PreScience ( Ajith et al., 2026 )	$\bm{\times}$	$\bm{\times}$	$\bm{\checkmark}$	$\bm{\times}$	$\bm{\checkmark}$

Table 1: Capability coverage of selected scientific-search benchmarks. A

\bm{\checkmark}

denotes coverage and a

\bm{\times}

denotes no coverage under our operational definitions. Resource exposure evaluates whether a system can surface the needed literature from a corpus. Selection requires evaluating which already exposed papers are retained or omitted, and ranking evaluates their submitted order. HyBIRD uses MIR ( Garikaparthi et al., 2025 ) . Only Inspire covers all five characteristics.

Figure 1: Source of the working relevance criterion. Left: prior tasks stabilize relevance through clues, constraints, anchors, feedback, or candidate pools. Right: Inspire supplies only a solution-redacted problem, so the agent must form provisional criteria that guide resource exposure, selection, and ranking. Retrieved evidence can revise those criteria and redirect subsequent search. The schematic contrasts information supplied during interaction; task difficulty can vary independently.

Figure 2: Backward construction and information boundary. Starting from a later target, the builder applies the three-month cutoff, reconstructs and screens a solution-redacted brief, and grades cited pre-cutoff antecedents. During evaluation, the policy receives only the brief and date-filtered search results; the target, its bibliography, and all grades remain evaluator-only. This separation permits hindsight scoring without exposing the realized solution lineage.

Model	nDCG@10	Recall@10	MRR	$\operatorname{AnyHit@10}$	$\geq 3$ hits	$\geq 5$ hits	Total calls
Claude Opus 5	0.284	0.215	0.476	0.723	0.321	0.067	21.1
Claude Sonnet 5	0.189	0.155	0.361	0.603	0.175	0.033	17.7
GPT-5.6-Sol	0.164	0.122	0.321	0.559	0.134	0.011	26.3
Gemini 3.5 Flash	0.151	0.120	0.299	0.521	0.104	0.011	34.7
Gemini 3.1 Pro Preview	0.110	0.082	0.258	0.403	0.047	0.000	20.9
Qwen3.6-35B-A3B Base	0.082	0.071	0.200	0.357	0.042	0.006	21.0

Table 2: Unified end-to-end comparison. All rows use the same targets, interface, search budget, failure policy, and scorer. nDCG@10, Recall@10, MRR, and AnyHit@10 include failed or empty episodes as zero. The multi-hit columns report exact recovery of at least one, three, or five graded antecedents among targets with enough labels; Total calls is the mean number of search and paper-access interactions.

Figure 3: Recovered antecedent counts across 476 targets. The horizontal axis is the number of graded antecedents recovered in the submitted top 10, and the vertical axis is the percentage of targets with exactly that count. Failures and non-submissions enter the zero bin. Concentration at zero and one shows that recovering an isolated antecedent is substantially more common than assembling a broader portfolio.

	Resource exposure		Selection	Ranking	Retention diagnostics
Model	Exposed gain	No relevant item exposed	Selected gain	Final nDCG@10	Gain kept	Rank quality
Claude Opus 5	0.480	17.4%	0.386	0.284	0.591	0.735
Claude Sonnet 5	0.323	31.1%	0.269	0.189	0.585	0.703
GPT-5.6-Sol	0.326	30.5%	0.233	0.164	0.504	0.706
Gemini 3.5 Flash	0.290	35.3%	0.218	0.151	0.520	0.691
Gemini 3.1 Pro Preview	0.184	52.5%	0.148	0.110	0.599	0.742

Table 3: Resource exposure, selection, and ranking funnel. Exposed gain oracle-sorts every graded antecedent encountered during search; Selected gain oracle-sorts the graded antecedents retained in the submission; final nDCG@10 additionally reflects their submitted order. Gain kept and Rank quality are the retention ratios in Equation 10 , while No relevant item exposed reports the zero-exposure rate. Empty submissions remain zero, and stage gaps describe the observed trajectories.

Figure 4: Observed funnel across resource exposure, selection, and ranking. For each model, the three points show mean Exposed gain, Selected gain, and final nDCG@10; horizontal bars are 95% target-bootstrap intervals. The right columns report zero-exposure rate, Gain kept, and Rank quality. Lower zero-exposure is better, whereas higher values are better for the other quantities. Horizontal separation localizes the stage of observed losses; causal mechanisms require intervention.

	Search and gather		Filter and construct answer			End-to-end
Condition	Exposed gain	No relevant item exposed	Selected gain	Gain kept	Rank quality	Final nDCG@10
Qwen3.6-35B-A3B Base	0.153	56.1%	0.119	0.532	0.683	0.082
Qwen3.6-35B-A3B + hindsight SFT	0.251	29.4%	0.219	0.661	0.759	0.166

Table 4: Controlled benchmark-usability test. The table reports changes in resource exposure, selection, and final ranking, allowing the end-to-end improvement to be localized, testing whether Inspire can produce useful supervision and measure held-out improvement.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Targets	Graded antecedents	Median/target	T3	T2	T1	Mean brief words	Target years
Train	1,549	10,268	7	430	2,905	6,933	183.4	2023 to 2024
Evaluation	476	3,574	10	179	1,244	2,151	179.3	2025 to 2026

Appendix

Table 5: Inspire dataset statistics. Train and evaluation partitions are reported separately. Antecedents are resolved pre-cutoff references cited by the later targets and automatically graded as T3 (most central), T2, or T1, operationalizing each target’s realized citation lineage. The final columns summarize brief length and target-release years.

Figure 5: Development example and automatic audit outcome. The agent receives a shortened CASSI brief and public-record cutoff; the later target and colored antecedent labels are evaluator-only. The target-identity audit did not name the paper but inferred a likely Mamba/state-space mechanism, illustrating the residual mechanism-level leakage captured by the audit. Tiers and roles come from arxiv:2501.01262 .

Figure 6: Held-out response to benchmark-derived SFT. The left panel compares boundary-reflected Gaussian kernel densities of matched Base and SFT failure-as-zero nDCG@10 values, including the probability mass at zero. The right panel traces aggregate Exposed gain, Selected gain, and final nDCG@10 under the same held-out protocol. Improvements at all three points indicate that the controlled intervention changes both candidate discovery and use of the discovered evidence.

Figure 7: Oracle nDCG@10 accumulated over normalized search progress. At each checkpoint, graded antecedents in the observed exposure prefix are ordered by their hindsight gains. The endpoint of each curve is therefore the best nDCG@10 supported by the complete exposed set, before losses from final selection and ordering.

Figure 8: Search-progress profile for the matched Base and hindsight-SFT conditions. The oracle is recomputed on each condition’s cumulative exposure prefix under the shared held-out protocol. Curve separation measures when the adapter makes additional graded gain available, independently of subsequent selection and submitted ordering.

Explore similar work

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research
Oct 1, 2026Sohyeon Kim, Yoonho Lee, Bo Liu +11AI Agents for Scientific Discovery Agentic Search
Stanford University · Seoul National University · University of Washington +3
ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments
Jun 18, 2026Tingyue Pan, Mingyue Cheng, Daoyu Wang +4Agentic Search Information Retrieval
State Key Lab of Cognitive Intelligence, University of Science and Technology of China
AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery
Apr 28, 2026Lei Xiong, Kun Luo, Ziyi Xia +15Deep Research Agents Scientific Discovery
AutoResearchBench Team*

Oct 1, 2026J/K move · Enter open · S save

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

Sohyeon Kim, Yoonho Lee, Bo Liu, Dayoon Ko+10

Stanford University · Seoul National University · University of Washington+3

AI Agents for Scientific Discovery Agentic Search AI Agent Benchmarks Information Retrieval IR Evaluation

Open paper AskPDF arXiv

What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.

Abstract

Explore similar work

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments

AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery