Scientific literature search often begins with an open research problem rather than a known target paper or a fixed candidate set. We introduce INSPIRE, a benchmark for evaluating agents that search prior literature to make progress on solution-redacted research problems. Each instance pairs a research brief with a target-specific cutoff three months before a later paper and evaluates ranked outputs against graded cited antecedents from that paper's realized research lineage. Search proceeds over an open corpus, while the identity of the target paper and membership of its cited antecedents remain hidden from the agent. Beyond end-to-end retrieval quality, INSPIRE uses logged search trajectories to distinguish three coupled stages: resource exposure, whether useful antecedents are surfaced during search; selection, whether exposed antecedents are retained; and ranking, how effectively retained papers are ordered. Across 476 computer-science targets under a shared search interface and budget, the strongest evaluated agent achieves 0.284 nDCG@10. Results show that current agents more readily recover an isolated antecedent than assemble a broader portfolio of relevant prior work. The stagewise analysis identifies resource exposure as the largest observed bottleneck, with further losses in selection and ranking. We additionally construct replay-valid hindsight demonstrations and show that they improve held-out search without changing test-time information, establishing that the benchmark provides an actionable learning signal. INSPIRE therefore enables both end-to-end comparison and stage-resolved diagnosis in a setting where the agent must construct its own working criterion of relevance.
Figures & tables
Task setting
Research capability evaluated
Benchmark
Interactive open-corpus search
No supplied relevance cue
Resource exposure
Selection
Ranking
AutoResearchBench ( Xiong et al., 2026 )
✓
×
✓
×
×
PaperPilot ( Li et al., 2026 )
✓
×
✓
×
✓
ResearchBench ( Liu et al., 2025 )
×
×
×
✓
×
HyBIRD ( Yang et al., 2026 )
×
✓
✓
×
✓
PreScience ( Ajith et al., 2026 )
×
×
✓
×
✓
Table 1: Capability coverage of selected scientific-search benchmarks. A ✓ denotes coverage and a × denotes no coverage under our operational definitions. Resource exposure evaluates whether a system can surface the needed literature from a corpus. Selection requires evaluating which already exposed papers are retained or omitted, and ranking evaluates their submitted order. HyBIRD uses MIR ( Garikaparthi et al., 2025 ) . Only Inspire covers all five characteristics.
Figure 1: Source of the working relevance criterion. Left: prior tasks stabilize relevance through clues, constraints, anchors, feedback, or candidate pools. Right: Inspire supplies only a solution-redacted problem, so the agent must form provisional criteria that guide resource exposure, selection, and ranking. Retrieved evidence can revise those criteria and redirect subsequent search. The schematic contrasts information supplied during interaction; task difficulty can vary independently.
Figure 2: Backward construction and information boundary. Starting from a later target, the builder applies the three-month cutoff, reconstructs and screens a solution-redacted brief, and grades cited pre-cutoff antecedents. During evaluation, the policy receives only the brief and date-filtered search results; the target, its bibliography, and all grades remain evaluator-only. This separation permits hindsight scoring without exposing the realized solution lineage.
Model
nDCG@10
Recall@10
MRR
AnyHit@10
≥3 hits
≥5 hits
Total calls
Claude Opus 5
0.284
0.215
0.476
0.723
0.321
0.067
21.1
Claude Sonnet 5
0.189
0.155
0.361
0.603
0.175
0.033
17.7
GPT-5.6-Sol
0.164
0.122
0.321
0.559
0.134
0.011
26.3
Gemini 3.5 Flash
0.151
0.120
0.299
0.521
0.104
0.011
34.7
Gemini 3.1 Pro Preview
0.110
0.082
0.258
0.403
0.047
0.000
20.9
Qwen3.6-35B-A3B Base
0.082
0.071
0.200
0.357
0.042
0.006
21.0
Table 2: Unified end-to-end comparison. All rows use the same targets, interface, search budget, failure policy, and scorer. nDCG@10, Recall@10, MRR, and AnyHit@10 include failed or empty episodes as zero. The multi-hit columns report exact recovery of at least one, three, or five graded antecedents among targets with enough labels; Total calls is the mean number of search and paper-access interactions.
Figure 3: Recovered antecedent counts across 476 targets. The horizontal axis is the number of graded antecedents recovered in the submitted top 10, and the vertical axis is the percentage of targets with exactly that count. Failures and non-submissions enter the zero bin. Concentration at zero and one shows that recovering an isolated antecedent is substantially more common than assembling a broader portfolio.
Resource exposure
Selection
Ranking
Retention diagnostics
Model
Exposed gain
No relevant item exposed
Selected gain
Final nDCG@10
Gain kept
Rank quality
Claude Opus 5
0.480
17.4%
0.386
0.284
0.591
0.735
Claude Sonnet 5
0.323
31.1%
0.269
0.189
0.585
0.703
GPT-5.6-Sol
0.326
30.5%
0.233
0.164
0.504
0.706
Gemini 3.5 Flash
0.290
35.3%
0.218
0.151
0.520
0.691
Gemini 3.1 Pro Preview
0.184
52.5%
0.148
0.110
0.599
0.742
Table 3: Resource exposure, selection, and ranking funnel. Exposed gain oracle-sorts every graded antecedent encountered during search; Selected gain oracle-sorts the graded antecedents retained in the submission; final nDCG@10 additionally reflects their submitted order. Gain kept and Rank quality are the retention ratios in Equation 10 , while No relevant item exposed reports the zero-exposure rate. Empty submissions remain zero, and stage gaps describe the observed trajectories.
Figure 4: Observed funnel across resource exposure, selection, and ranking. For each model, the three points show mean Exposed gain, Selected gain, and final nDCG@10; horizontal bars are 95% target-bootstrap intervals. The right columns report zero-exposure rate, Gain kept, and Rank quality. Lower zero-exposure is better, whereas higher values are better for the other quantities. Horizontal separation localizes the stage of observed losses; causal mechanisms require intervention.
Search and gather
Filter and construct answer
End-to-end
Condition
Exposed gain
No relevant item exposed
Selected gain
Gain kept
Rank quality
Final nDCG@10
Qwen3.6-35B-A3B Base
0.153
56.1%
0.119
0.532
0.683
0.082
Qwen3.6-35B-A3B + hindsight SFT
0.251
29.4%
0.219
0.661
0.759
0.166
Table 4: Controlled benchmark-usability test. The table reports changes in resource exposure, selection, and final ranking, allowing the end-to-end improvement to be localized, testing whether Inspire can produce useful supervision and measure held-out improvement.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Split
Targets
Graded antecedents
Median/target
T3
T2
T1
Mean brief words
Target years
Train
1,549
10,268
7
430
2,905
6,933
183.4
2023 to 2024
Evaluation
476
3,574
10
179
1,244
2,151
179.3
2025 to 2026
Appendix
Table 5: Inspire dataset statistics. Train and evaluation partitions are reported separately. Antecedents are resolved pre-cutoff references cited by the later targets and automatically graded as T3 (most central), T2, or T1, operationalizing each target’s realized citation lineage. The final columns summarize brief length and target-release years.
Figure 5: Development example and automatic audit outcome. The agent receives a shortened CASSI brief and public-record cutoff; the later target and colored antecedent labels are evaluator-only. The target-identity audit did not name the paper but inferred a likely Mamba/state-space mechanism, illustrating the residual mechanism-level leakage captured by the audit. Tiers and roles come from arxiv:2501.01262 .
Figure 6: Held-out response to benchmark-derived SFT. The left panel compares boundary-reflected Gaussian kernel densities of matched Base and SFT failure-as-zero nDCG@10 values, including the probability mass at zero. The right panel traces aggregate Exposed gain, Selected gain, and final nDCG@10 under the same held-out protocol. Improvements at all three points indicate that the controlled intervention changes both candidate discovery and use of the discovered evidence.
Figure 7: Oracle nDCG@10 accumulated over normalized search progress. At each checkpoint, graded antecedents in the observed exposure prefix are ordered by their hindsight gains. The endpoint of each curve is therefore the best nDCG@10 supported by the complete exposed set, before losses from final selection and ordering.
Figure 8: Search-progress profile for the matched Base and hindsight-SFT conditions. The oracle is recomputed on each condition’s cumulative exposure prefix under the shared held-out protocol. Curve separation measures when the adapter makes additional graded gain available, independently of subsequent selection and submitted ordering.
What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature available when the project began. Agentic search does no better than embedding retrieval (0.42 vs. 0.48 Recall@20) despite calling that same retriever as a tool. Even an agent built on Claude Fable 5.1, which may have seen the completed papers during training, reaches only 0.51 R@20. These results highlight the need for new training recipes that equip models with expert intuition for searching broad corpora. We envision ScholarCatalyst as a step toward scientific agents that can take a half-formed idea and point to the prior research it needs.
Sohyeon Kim, Yoonho Lee, Bo Liu +11
Stanford University · Seoul National University · University of Washington +3
Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, intent-driven literature exploration. However, existing benchmarks offer limited coverage of research intents and insufficient support for scalable answer construction and reproducible multi-turn evaluation. We propose ScholarQuest, a taxonomy-guided benchmark constructed from over 1,000 computer science topics and organized around four research intents. To construct answer sets at scale, we combine multi-source retrieval, citation expansion, and multi-stage relevance adjudication. We further build ScholarBase, a million-scale literature environment with unified retrieval and citation-expansion interfaces, together with a shared multi-turn search harness. We benchmark complete literature search systems and compare language models within this harness, evaluating retrieval coverage, search efficiency, and robustness across research intents. Results show that agentic methods outperform single-shot retrieval baselines, yet the best-performing search system achieves only 0.312 Recall@100 and 0.352 Recall@All. Comparisons under the shared harness reveal intent-dependent model strengths, with higher Planner token consumption not consistently yielding higher recall. We release the benchmark, retrieval backend, and evaluation harness for reproducible research
Tingyue Pan, Mingyue Cheng, Daoyu Wang +4
State Key Lab of Cognitive Intelligence, University of Science and Technology of China
Autonomous scientific research is significantly advanced thanks to the development of AI agents. One key step in this process is finding the right scientific literature, whether to explore existing knowledge for a research problem, or to acquire evidence for verifying assumptions and supporting claims. To assess AI agents' capability in driving this process, we present AutoResearchBench, a dedicated benchmark for autonomous scientific literature discovery. AutoResearchBench consists of two complementary task types: (1) Deep Research, which requires tracking down a specific target paper through a progressive, multi-step probing process, and (2) Wide Research, which requires comprehensively collecting a set of papers satisfying given conditions. Compared to previous benchmarks on agentic web browsing, AutoResearchBench is distinguished along three dimensions: it is research-oriented, calling for in-depth comprehension of scientific concepts; literature-focused, demanding fine-grained utilization of detailed information; and open-ended, involving an unknown number of qualified papers and thus requiring deliberate reasoning and search throughout. These properties make AutoResearchBench uniquely suited for evaluating autonomous research capabilities, and extraordinarily challenging. Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research, while many other strong baselines fall below 5%. We publicly release the dataset and evaluation pipeline to facilitate future research in this direction. We publicly release the dataset, evaluation pipeline, and code at https://github.com/CherYou/AutoResearchBench.