Deep Research (DR) systems help researchers cope with ballooning publishing counts. Such tools synthesize scientific papers to answer research queries, but lack understanding of their users. We address this with MyScholarQA (MySQA), a personalized DR agent that: 1) infers a profile with a user's research interests; 2) proposes personalized actions for a user's input query; and 3) writes a multi-section report for the query that follows user-approved actions. We first test MySQA with NLP's standard protocol: we build a benchmark with synthetic users and LLM judges, where MySQA beats baselines in citation metrics and personalized action-following. However, we suspect this process does not cover all aspects of personalized DR users value, so we interview users in an online version of MySQA to unmask them. We reveal nine nuanced errors of personalized DR undetectable by our LLM judges, and we study qualitative feedback to form lessons for future DR design. In all, we argue for a pillar of personalization that easy-to-use LLM judges can lead NLP to overlook: real progress in personalization is only possible with real users.
Figures & tables
Figure 1: Overview of MySQA : our three-step personalized Deep Research system. (1) Researchers upload papers from Semantic Scholar, from which an LLM infers a profile that captures their interests. (2) When the researcher asks a query, MySQA proposes a list of actions that could alter the report, tailored to the researcher’s profile. (3) The system generates a report that answers the query and executes these actions through a multi-stage retrieval and LLM generation pipeline. To improve transparency and control, users can edit profiles, adjust actions, and view highlights in the report where MySQA personalizes content.
Figure 2: After a user selects research papers, MySQA infers an editable profile with inferences about the user, capturing their interests for personalizing reports.
Figure 3: Before answering a user’s query, MySQA proposes actions that change how the report could be created, which users can adjust to customize their report.
Figure 4: Reports in MySQA highlight personalized content, helping users navigate each action they select.
Model
Inf. Acc
Cit. Rel.
Cat. Acc.
Spec.
# Cite
# Words
G-Pro
97.1
97.4
99.4
3.73
2.45
23.2
Sonnet
92.5
97.4
99.1
4.12
1.91
21.6
OAI-o3
88.6
91.8
99.8
4.20
2.03
18.9
DS-r1
77.8
80.7
97.2
3.56
1.89
9.9
Table 1: Profile ( n1=25 ) inference accuracy, relevance, and specificity across LLM s. Highest scores are bold . Reasoning LLM s infer accurate, relevant user profiles.
Model
W.R.
pgen Coh.
pperson Coh.
pgen Uniq.
pperson Uniq.
G-Flash
91.3
93.2
84.0
42.9
60.7
Sonnet
93.5
91.8
82.0
50.4
68.2
GPT-4.1
94.7
93.5
84.1
49.1
66.3
DS-V3
94.3
89.2
72.3
54.5
72.2
Table 2: Personalized/generic action ( n2=16 ) win rate, coherence, and uniqueness. The highest score is bold . LLM judges that condition on synthetic users’ papers are more likely to select personalized actions, indicating that they can personalize actions based on user profiles.
Model
An. Cov.
An. Prec.
Cit. Prec.
Cit. Rec.
A Adh.
MySQA
91.4
89.9
91.8
81.4
83.2
SQA
88.9
89.1
90.5
76.9
81.3
OpenSc.
77.2
97.4
82.5
60.4
82.5
STORM
72.0
92.2
73.3
64.7
74.4
Sonar DR
81.0
82.9
64.3
46.3
75.0
o3 DR
89.1
90.2
79.2
56.7
93.8
Table 3: DR report quality and action adherence scores for query q and eight actions in Agen∪Aperson . The best score is bold . MySQA surpasses every DR baseline in three out of five metrics. We only evaluate o3 DR using ten examples due to latency and cost limitations.
Output
Aspect
Description
Freq.
Profile
DOMAIN
Uses terms, definitions, or details that do not capture the user’s domain of research.
27.6%
OVERCLAIM
Claims something applies to the user broadly, but only applies to some/parts of papers.
17.9%
CONVENTION
Infers a generic convention of the user’s field (e.g. "You enumerate contributions").
12.8%
CONTRAST
Has a contrast that misrepresents the user (e.g. "You are X, not Y", but the user is Y).
12.2%
Action
NARROW
The action is too specific and would overly constrain the information coverage.
43.8%
OFFTOPIC
The action deviates too far from the query, distracting from the user’s goal/intent.
23.6%
Table 4: The nine most common personalization errors in MySQA ’s outputs discovered from our interviews, missed in our offline evaluations. All twenty metrics we discover are in Table 9 .
Figure 5: User satisfaction on MySQA profiles, actions, and reports. Users are satisfied with 73% of them. 13 13 13 U10 spent the interview just viewing the profile ( n=20 ).
Figure 6: LLM judge accuracy for predicting users’ satisfaction of personalized profiles, actions, and reports. No model ever beats a majority class baseline ( Dror et al., 2018 , α=0.05 Binomial test with Bonferroni correction) .
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Split
# Queries
Avg # Papers / Query
Avg Query Length
Avg Paper Length
Total Instances
Dev
281
2.81
17.42
5855.51
281
Test
291
2.91
13.10
5742.30
291
Appendix
Table 5: Details of our collected synthetic dataset
Model
Ingredient Recall
Answer Precision
Citation Accuracy
Citation Recall
A Adherence
See All Actions
0.901
0.917
0.918
0.806
0.851
See Incremental Action
0.892
0.896
0.905
0.786
0.868
See Current Action
0.891
0.915
0.912
0.782
0.798
One-Shot Prompt
0.803
1.00
0.844
0.77
0.648
Appendix
Table 6: Ablation of MySQA across different strategies of where to include the actions in prompts. The dominant strategy is always giving MySQA the actions at each execution step.
Model
Ingredient Recall
Answer Precision
Citation Accuracy
Citation Recall
A Adherence
MySQA - Claude Sonnet
0.901
0.917
0.918
0.806
0.851
MySQA - Gemini Flash
0.87
0.989
0.953
0.846
0.823
MySQA - GPT-4.1
0.911
0.97
0.856
0.617
0.921
Appendix
Table 7: Ablation of different LLM s to power MySQA . LLM s have similar scores, so we choose Claude to match Singh et al. (2025) .
Model
Ingredient Recall
Answer Precision
Citation Accuracy
Citation Recall
A Adherence
4 actions
0.901
0.917
0.918
0.806
0.851
8 actions
0.914
0.899
0.918
0.814
0.832
12 actions
0.927
0.896
0.91
0.81
0.812
24 actions
0.938
0.885
0.915
0.816
0.788
30 actions
0.947
0.898
0.916
0.807
0.777
Appendix
Table 8: Ablation of MySQA across various number of actions. The model preserves report quality, but the proportion of actions followed drops as the number of actions increase.
Figure 7: Accuracy of LLM judges for predicting user satisfaction in personalization aspects over profiles, actions, and reports with one few-shot example . LLM judges are still worse than the majority class baseline.
Figure 8: Accuracy of LLM judges for predicting user satisfaction in personalization aspects over profiles, actions, and reports with no examples . LLM judges are still worse than the majority class baseline.
Figure 9: Accuracy of LLM judges for predicting user satisfaction in personalization aspects over profiles, actions, and reports with no metric definitions . LLM judges are still worse than the majority class baseline.
Output Type
Aspect
Description
Freq
Profile
DOMAIN
Uses terms, definitions, or details that do not capture the user’s domain of research
27.6%
OVERCLAIM
Claims something applies to the user broadly, but only applies to some/parts of papers
17.9%
CONVENTION
Infers a generic convention of the user’s field (e.g. "You enumerate contributions")
12.8%
CONTRAST
Has a contrast that misrepresents the user (e.g. "You are X, not Y", but the user is Y)
12.2%
UNIMPORT
The inference is true from their papers, but not a part of their papers they care about
8.3%
STRENGTH
States something too strongly about the user (e.g. “You are a deep expert in X”)
7.1%
Appendix
Table 9: Dimensions of personalization errors in MySQA ’s outputs uncovered in our interviews, missed by offline evaluation ( section 3.4 ). Bold text indicates aspects with sufficient data (50+ examples) for evaluating if LLM judges can simulate them ( section 4.3 ).
Figure 10: List of questions we ask participants in the formative. Sections marked with "[walkthrough]" indicates that participants also gave feedback on model outputs or high-level design in addition to answering the questions. During the study, we refer to the list of actions as “plans”, which users found clear.
Figure 11: Full example of one of the author’s profiles with two inferences per section in MySQA . We remove the paper citations after each profile inference for anonymity. Users can view, toggle, and edit their profile inferences.
Figure 12: Full example of actions for a query in MySQA . Users can view, toggle, and edit actions in the list. Clicking on the dropdown arrow gives a more elaborate description of each action.
Figure 13: Full example of a report for a query and actions in MySQA . Users can view each section of the report, collapse them for just a TL;DR, and view highlights for where MySQA personalized. Clicking on an action (left) enables/disables the highlights, revealing an action bar with: 1) the number of highlighted spans in the report; and 2) navigation arrows to jump between highlighted spans. Each plan action and section has buttons which we use to collect feedback.
Figure 14: We provide activities to entertain users while they wait for their profiles to be generated. Users can view jokes, fun facts, or play the Chrome T-Rex game. Adding engaging activities before annotation tasks can have positive outcomes Lewis et al. (2011) .
Table 10: Deep Research report quality and action instruction-following for query q and 8 actions in Agen∪Aperson . MySQA surpasses all DR tools in 3/5 metrics. We run OpenAI DR on 10 examples due to latency and cost issues.
Scientific Deep Research (DR) agents answer user queries by synthesizing research papers into multi-section reports. User feedback can improve their utility, but existing protocols only score the final report, making it hard to study and learn which intermediate actions DR agents should take to improve reports. We collect DRACULA, the first dataset with user feedback on intermediate actions for DR. Over five weeks, nineteen expert CS researchers ask queries to a DR system that proposes actions (e.g., "Add a section on datasets"). Our users select actions they prefer, then judge whether an output report applied their selections successfully, yielding 8,103 action preferences and 5,230 execution judgments. After confirming a DR agent can execute DRACULA's actions, we study the predictability of user-preferred actions via simulation-how well LLMs predict the actions users select-a step toward learning to generate useful actions. We discover: (1) LLM judges initially struggle to predict action selections, but improve most when using a user's full selection history, rather than self-reported or extrapolated user context signals; (2) Users' selections for the same query differ based on unstated goals, bottlenecking simulation and motivating affordances that let users steer reports; and (3) Our simulation results inform an online intervention that generates new actions based on the user's past interactions, which users pick most often in follow-up studies. Overall, while work extensively studies execution, DRACULA reveals a key challenge is deciding which actions to execute in the first place. We open-source DRACULA's study design, user feedback, and simulation tasks to spur future work on action feedback for long-horizon agents.
Deep Research Agents (DRA) aim to help users search the web, synthesize information, and deliver comprehensive investigative reports. Prior benchmarks often either under-evaluate a system's ability to produce meaningful insights and high-quality writing, or adopt coarse or LLM-defined criteria that are hard to verify and can diverge from human expert judgment. To address these issues, we introduce Deep Research Bench II, a new benchmark for evaluating DRAs. It contains 132 grounded research tasks across 22 domains; for each task, an agent must produce a research report that is evaluated by a set of 9,430 fine-grained binary rubrics in total, covering three dimensions: information recall, analysis, and presentation. All rubrics are derived from carefully selected expert-written investigative articles and are constructed through a four-stage LLM+human pipeline that combines automatic extraction with over 400 human-hours of expert review, ensuring that the criteria are verifiable and aligned with human expert judgment. We evaluate several state-of-the-art deep-research agents on Deep Research Bench II and find that even the strongest models satisfy fewer than 50% of the rubrics, revealing a substantial gap between current DRAs and human experts. We release the benchmark, evaluation scripts, and all rubrics at https://github.com/imlrz/DeepResearch-Bench-II to facilitate future research on deep-rearch agents.
Ruizhe Li, Mingxuan Du, Benfeng Xu +3
University of Science and Technology of China · Metastone Technology, Beijing, China
User requests serve as research specifications for deep research agents, shaping what evidence to seek and how to synthesize it. In personalized deep research, these specifications must additionally reflect user goals, constraints, preferences, and evaluation criteria. User context can be incorporated either within the deep research pipeline or into the research specification provided as its input. We focus on the latter, refining the user request into a personalized research specification before passing it to an unchanged deep research agent. This requires resolving three coupled decisions: which framing factors are relevant, whether the available user context sufficiently supports them, and whether to retrieve user memory, ask the user, or stop and refine the query. For training, G-STEER organizes framing factors as elicitation targets in an Intent Elicitation Graph that captures their dependencies. It learns a clarification policy from graph-scaffolded trajectories spanning diverse factor dependencies and evidence conditions. The policy produces a refined query while balancing target coverage against the costs of evidence acquisition. Experiments show that G-STEER achieves the strongest overall weighted target coverage and the highest downstream report personalization across both evaluated DRAs, while asking roughly one third as many user questions as a strong clarification baseline.
Soojin Yoon, Dongha Lee
Department of Artificial Intelligence, Yonsei University