Deep research agents (DRAs) conduct long-horizon reasoning, search, and literature synthesis to provide long-form answers to complex, domain-specific questions. These agents are typically trained and evaluated on fully specified user queries. However, real-world user queries can often be underspecified. We introduce UNDERSPEC-DR, a framework for systematically evaluating how underspecified queries affect agents' final answer quality and intermediate search behavior. Across three DRAs we evaluated, underspecification substantially reduces how well DRA's answers fulfill users' information needs, even though they remain grounded in cited evidence. Our analysis of intermediate search behavior suggests a possible explanation: DRAs conduct broader searches that draw less on the original user query but rely more on their learned prior during training, which can be misaligned with users' intent. Our findings reveal the vulnerability of current DRAs to underspecified user queries and inform future efforts to build DRAs that proactively recognize when additional user input is needed to reliably complete deep research tasks.
Figures & tables
Figure 1: Overview of UnderSpec-DR . (a) From 176 initial–final query pairs in researchers’ conversations with DRAs, we identify three omitted information types: scope ( qs ), terminology ( qt ), and preference ( qp ). We then rewrite benchmark queries into matched variants with an LLM followed by human verification. (b) Each agent answers the complete query q or a variant qr by alternating search queries sk and retrieved documents Dk . Highlighted search-query tokens come from the user query (blue), earlier search queries (yellow), retrieved documents (green), or parametric knowledge (red). (c) We analyze answer quality (RQ1), whether agents recognize underspecification (RQ2), and search behavior (RQ3).
Rubric
Removed / Obscured Information
Example
# Query Variants
RQA
SQA
DRB
R1 Scope ( q\textscs )
Time, place, population, domain
Which regions and countries saw widespread acceptance and implementation of home delivery services for antiretroviral therapy (ART) during the COVID-19 pandemic ?
274
129
74
R2 Term ( q\textsct )
Precise technical terms (replaced by generic words)
How do non-aspirin NSAIDs pain relievers and selective COX-2 enzyme inhibitors differ in their associations with aneurysmal subarachnoid hemorrhage risk, and what is known about the heterogeneity across studies for these drug classes?
238
126
45
R3 Preference ( q\textscp )
Stated criteria, comparisons, or requirements on the answer
Which Obsidian plugins can effectively replicate Notion’s multi-view database functionality (including Table, Kanban, Calendar, and List views)? Please provide a detailed comparison of the strengths and weaknesses of these plugins.
27
65
58
Fully-Underspecified ( q\textscu )
R1–R3 jointly
How does the incidence of imminent fractures differ depending on the site of the (index) initial fracture according to pooled meta-analyses ?
499
287
100
Table 1: Underspecification rubrics, examples, and number of constructed variants in ResearchQA (RQA), ScholarQA-Bench (SQA), and DeepResearch Bench (DRB). Colors distinguish scope , terminology , and preference edits; struck text is removed and bold text denotes replacements.
Figure 2: Effects of query underspecification on answer quality across three benchmarks and DRAs. Top: Mean paired score changes (underspecified variant minus complete query). Filled and outlined bars indicate significant and nonsignificant differences, respectively, based on paired Wilcoxon signed-rank tests ( ∗p<.05 , ∗∗p<.01 , ∗∗∗p<.001 ). Bottom: Proportions of matched queries with lower scores (below zero) or higher scores (above zero), with colors indicating the magnitude of change. Segment labels are percentages; score changes within ±0.02 are omitted.
Model
Metric
Fully underspec.
w/o preference
w/o scope
w/o term
DR Tulu
Document Utilization
+2.85%∗∗∗
+0.01%
+1.51%∗
+0.07%
Citation Precision
+0.45%
+1.40%
−2.88%
+2.17%
GPT-5.4 w/ Tools
Document Utilization
−0.44%
+1.98%
+2.11%∗∗
−0.45%
Citation Precision
−2.14%
+4.68%
−0.48%
+0.83%
Table 2: Mean paired changes in document utilization in the final answer and citation precision (underspecified variant minus complete query), pooled across datasets within each model. Values are displayed as percentages and represent percentage-point differences. Significance is assessed using unadjusted paired Wilcoxon signed-rank tests. Stars denote ∗p<0.05 , ∗∗p<0.01 , and ∗∗∗p<0.001 .
Figure 5Figure 6
Figure 7: Changes in document novelty and context-driven term adoption under underspecification. Values show mean paired differences with 95% confidence intervals. Positive novelty changes indicate more newly retrieved documents; negative adoption changes indicate less use of retrieved content. Significance markers follow Fig. 6 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Agent
q
q\textscp
q\textscs
q\textsct
q\textscu
Rel. change q\textscu
ResearchQA
DR Tulu-8B
.791
.694 ns
.615 ***
.565 ***
.294 ***
− 62.8%
GPT-5.4 with tools
.610
.496 *
.466 ***
.449 ***
.215 ***
− 64.8%
Tongyi-DR
.840
.763 **
.681 ***
.624 ***
.334 ***
− 60.2%
ScholarQA-Bench
DR Tulu-8B
.477
.504 ns
.431 ns
.438 **
.308 ***
− 35.5%
GPT-5.4 with tools
.524
.495 ns
.489 ns
.496 ns
.377 ***
− 25.8%
Tongyi-DR
.438
.414 ns
.357 **
.395 ns
.207 ***
− 52.8%
Appendix
Table 3: Average answer quality for different types of query variants. Significance markers indicate differences from the fully specified condition ( q ). The last column reports the relative change from q to the fully underspecified condition ( q\textscu ), calculated as (q\textscu−q)/q .
ResearchQA
ScholarQA-Bench
DeepResearch Bench
Agent
Variant
Ask
Assume
Unaware
Ask
Assume
Unaware
Ask
Assume
Unaware
DR Tulu-8B
q\textscp
0.0
11.1
88.9
0.0
18.4
81.6
0.0
37.9
62.1
q\textscs
0.0
6.6
93.4
0.0
10.1
89.9
0.0
41.7
58.3
q\textsct
0.4
15.5
84.0
0.0
15.1
84.9
0.0
53.3
46.7
q\textscu
0.2
12.6
87.2
1.0
12.9
86.1
1.0
51.0
48.0
GPT-5.4 with tools
q\textscp
0.0
8.7
91.3
2.1
12.8
85.1
1.9
36.5
61.5
Appendix
Table 4: Clarification behavior (% of runs): asks for clarification (Ask), recognizes the ambiguity and states an assumption (Assume), or proceeds without recognizing the ambiguity (Unaware).
Figure 8: Paired changes in query drift between the first and final search queries (underspecified minus complete query), with 95% confidence intervals. Positive values indicate greater drift under underspecification. One, two, and three stars denote p<.05 , p<.01 , and p<.001 , respectively; open circles indicate nonsignificant differences.
q\textscp
q\textscs
q\textsct
q\textscu
Agent
q
q\textscp
q
q\textscs
q
q\textsct
q
q\textscu
DR Tulu-8B
56.39
56.40 ns
55.97
57.48 *
56.43
56.50 ns
56.59
59.44 ***
GPT-5.4 with tools
21.66
23.64 ns
20.82
22.93 **
21.29
20.85 ns
21.38
20.94 ns
Both agents
42.36
43.16 ns
41.12
42.88 ***
42.27
42.13 ns
44.23
45.92 ***
Appendix
Table 5: Mean document utilization (%, #cited/#retrieved ) for each underspecified variant and its paired complete query q , pooled across ResearchQA, ScholarQA-Bench, and DeepResearch Bench. Both agents pools the paired questions of DR Tulu-8B and GPT-5.4 with tools. Significance markers come from paired Wilcoxon signed-rank tests against q .
q\textscp
q\textscs
q\textsct
q\textscu
Agent
q
q\textscp
q
q\textscs
q
q\textsct
q
q\textscu
DR Tulu-8B
43.95
45.35 ns
45.43
42.54 ns
50.87
53.04 ns
50.45
50.90 ns
GPT-5.4 with tools
70.52
75.20 ns
70.59
70.11 ns
75.78
76.61 ns
70.93
68.79 ns
Appendix
Table 6: Mean citation precision (%, #supporting/#cited ) for each underspecified variant and its paired complete query q , pooled across ScholarQA-Bench and DeepResearch Bench. Only questions with a final answer and at least one cited document under both conditions are included. Significance markers come from paired Wilcoxon signed-rank tests against q .
Deep-research agents answer complex questions by interacting with search and browsing tools, yet they often search along a single evolving trajectory. Our trajectory-level analysis reveals a common failure mode in which the agent may encounter an early search state with several plausible directions, but follow one direction before collecting enough comparative evidence. Once this happens, subsequent tool calls tend to reinforce the same path, increasing the chance of failure when the initial direction is misleading. We further find that successful trajectories reduce this risk through two behaviors: grounding vague exploration in concrete candidates and shifting directions when the current path is weak or incomplete. Based on these findings, we propose HypoSearch, which generates lightweight hypotheses as soft search hints, explores them through bounded independent branches, and compares branch-level evidence before commitment. Across four deep-research benchmarks and three backbone models, HypoSearch consistently outperforms single-trajectory search and standard parallel baselines, improving Qwen3.5-122B from 46.7 to 60.0 on BC-small while using fewer tool calls than five independent trajectories. A pilot supervised fine-tuning study further shows that these behavioral signals can curate compact training trajectories and reduce degradation from unfiltered data.
Ruochen Zhou, Zhengyu Chen, Luan Zhang +3
City University of Hong Kong · Meituan · Independent +1
Deep Research Agents (DRA) aim to help users search the web, synthesize information, and deliver comprehensive investigative reports. Prior benchmarks often either under-evaluate a system's ability to produce meaningful insights and high-quality writing, or adopt coarse or LLM-defined criteria that are hard to verify and can diverge from human expert judgment. To address these issues, we introduce Deep Research Bench II, a new benchmark for evaluating DRAs. It contains 132 grounded research tasks across 22 domains; for each task, an agent must produce a research report that is evaluated by a set of 9,430 fine-grained binary rubrics in total, covering three dimensions: information recall, analysis, and presentation. All rubrics are derived from carefully selected expert-written investigative articles and are constructed through a four-stage LLM+human pipeline that combines automatic extraction with over 400 human-hours of expert review, ensuring that the criteria are verifiable and aligned with human expert judgment. We evaluate several state-of-the-art deep-research agents on Deep Research Bench II and find that even the strongest models satisfy fewer than 50% of the rubrics, revealing a substantial gap between current DRAs and human experts. We release the benchmark, evaluation scripts, and all rubrics at https://github.com/imlrz/DeepResearch-Bench-II to facilitate future research on deep-rearch agents.
Ruizhe Li, Mingxuan Du, Benfeng Xu +3
University of Science and Technology of China · Metastone Technology, Beijing, China
Scientific Deep Research (DR) agents answer user queries by synthesizing research papers into multi-section reports. User feedback can improve their utility, but existing protocols only score the final report, making it hard to study and learn which intermediate actions DR agents should take to improve reports. We collect DRACULA, the first dataset with user feedback on intermediate actions for DR. Over five weeks, nineteen expert CS researchers ask queries to a DR system that proposes actions (e.g., "Add a section on datasets"). Our users select actions they prefer, then judge whether an output report applied their selections successfully, yielding 8,103 action preferences and 5,230 execution judgments. After confirming a DR agent can execute DRACULA's actions, we study the predictability of user-preferred actions via simulation-how well LLMs predict the actions users select-a step toward learning to generate useful actions. We discover: (1) LLM judges initially struggle to predict action selections, but improve most when using a user's full selection history, rather than self-reported or extrapolated user context signals; (2) Users' selections for the same query differ based on unstated goals, bottlenecking simulation and motivating affordances that let users steer reports; and (3) Our simulation results inform an online intervention that generates new actions based on the user's past interactions, which users pick most often in follow-up studies. Overall, while work extensively studies execution, DRACULA reveals a key challenge is deciding which actions to execute in the first place. We open-source DRACULA's study design, user feedback, and simulation tasks to spur future work on action feedback for long-horizon agents.