Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users

Nishant Balepur; Malachi Hamada; Varsha Kishore; Sergey Feldman; Amanpreet Singh; Pao Siangliulue; Joseph Chee Chang; Eunsol Choi

cs.CLMar 17, 2026

Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users

Authors: Nishant Balepur, Malachi Hamada, Varsha Kishore, Sergey Feldman, Amanpreet Singh, Pao Siangliulue, Joseph Chee Chang, Eunsol Choi,

Organizations: University of Maryland · Allen Institute for Artificial Intelligence · New York University

Abstract

Deep Research (DR) systems help researchers cope with ballooning publishing counts. Such tools synthesize scientific papers to answer research queries, but lack understanding of their users. We address this with MyScholarQA (MySQA), a personalized DR agent that: 1) infers a profile with a user's research interests; 2) proposes personalized actions for a user's input query; and 3) writes a multi-section report for the query that follows user-approved actions. We first test MySQA with NLP's standard protocol: we build a benchmark with synthetic users and LLM judges, where MySQA beats baselines in citation metrics and personalized action-following. However, we suspect this process does not cover all aspects of personalized DR users value, so we interview users in an online version of MySQA to unmask them. We reveal nine nuanced errors of personalized DR undetectable by our LLM judges, and we study qualitative feedback to form lessons for future DR design. In all, we argue for a pillar of personalization that easy-to-use LLM judges can lead NLP to overlook: real progress in personalization is only possible with real users.

Figures & tables

Figure 1: Overview of MySQA : our three-step personalized Deep Research system. (1) Researchers upload papers from Semantic Scholar, from which an LLM infers a profile that captures their interests. (2) When the researcher asks a query, MySQA proposes a list of actions that could alter the report, tailored to the researcher’s profile. (3) The system generates a report that answers the query and executes these actions through a multi-stage retrieval and LLM generation pipeline. To improve transparency and control, users can edit profiles, adjust actions, and view highlights in the report where MySQA personalizes content.

Figure 2: After a user selects research papers, MySQA infers an editable profile with inferences about the user, capturing their interests for personalizing reports.

Figure 3: Before answering a user’s query, MySQA proposes actions that change how the report could be created, which users can adjust to customize their report.

Figure 4: Reports in MySQA highlight personalized content, helping users navigate each action they select.

Model	Inf. Acc	Cit. Rel.	Cat. Acc.	Spec.	# Cite	# Words
G-Pro	97.1	97.4	99.4	3.73	2.45	23.2
Sonnet	92.5	97.4	99.1	4.12	1.91	21.6
OAI-o3	88.6	91.8	99.8	4.20	2.03	18.9
DS-r1	77.8	80.7	97.2	3.56	1.89	9.9

Table 1: Profile (

n_{1}=25

) inference accuracy, relevance, and specificity across LLM s. Highest scores are bold . Reasoning LLM s infer accurate, relevant user profiles.

Model	W.R.	$p_{\text{gen}}$ Coh.	$p_{\text{person}}$ Coh.	$p_{\text{gen}}$ Uniq.	$p_{\text{person}}$ Uniq.
G-Flash	91.3	93.2	84.0	42.9	60.7
Sonnet	93.5	91.8	82.0	50.4	68.2
GPT-4.1	94.7	93.5	84.1	49.1	66.3
DS-V3	94.3	89.2	72.3	54.5	72.2

Table 2: Personalized/generic action (

n_{2}=16

) win rate, coherence, and uniqueness. The highest score is bold . LLM judges that condition on synthetic users’ papers are more likely to select personalized actions, indicating that they can personalize actions based on user profiles.

Model	An. Cov.	An. Prec.	Cit. Prec.	Cit. Rec.	$\mathcal{A}$ Adh.
MySQA	91.4	89.9	91.8	81.4	83.2
SQA	88.9	89.1	90.5	76.9	81.3
OpenSc.	77.2	97.4	82.5	60.4	82.5
STORM	72.0	92.2	73.3	64.7	74.4
Sonar DR	81.0	82.9	64.3	46.3	75.0
o3 DR	89.1	90.2	79.2	56.7	93.8

Table 3: DR report quality and action adherence scores for query

q

and eight actions in

\mathcal{A}_{\text{gen}}\cup\mathcal{A}_{\text{person}}

. The best score is bold . MySQA surpasses every DR baseline in three out of five metrics. We only evaluate o3 DR using ten examples due to latency and cost limitations.

Output	Aspect	Description	Freq.
Profile	DOMAIN	Uses terms, definitions, or details that do not capture the user’s domain of research.	27.6%
	OVERCLAIM	Claims something applies to the user broadly, but only applies to some/parts of papers.	17.9%
	CONVENTION	Infers a generic convention of the user’s field (e.g. "You enumerate contributions").	12.8%
	CONTRAST	Has a contrast that misrepresents the user (e.g. "You are X, not Y", but the user is Y).	12.2%
Action	NARROW	The action is too specific and would overly constrain the information coverage.	43.8%
Action	OFFTOPIC	The action deviates too far from the query, distracting from the user’s goal/intent.	23.6%

Table 4: The nine most common personalization errors in MySQA ’s outputs discovered from our interviews, missed in our offline evaluations. All twenty metrics we discover are in Table 9 .

Figure 5: User satisfaction on MySQA profiles, actions, and reports. Users are satisfied with

73\%

of them. 13 13 13 U10 spent the interview just viewing the profile (

n=20

Figure 6: LLM judge accuracy for predicting users’ satisfaction of personalized profiles, actions, and reports. No model ever beats a majority class baseline ( Dror et al., 2018 ,

\alpha=0.05

Binomial test with Bonferroni correction) .

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Split	# Queries	Avg # Papers / Query	Avg Query Length	Avg Paper Length	Total Instances
Dev	281	2.81	17.42	5855.51	281
Test	291	2.91	13.10	5742.30	291

Appendix

Table 5: Details of our collected synthetic dataset

Model	Ingredient Recall	Answer Precision	Citation Accuracy	Citation Recall	$\mathcal{A}$ Adherence
See All Actions	0.901	0.917	0.918	0.806	0.851
See Incremental Action	0.892	0.896	0.905	0.786	0.868
See Current Action	0.891	0.915	0.912	0.782	0.798
One-Shot Prompt	0.803	1.00	0.844	0.77	0.648

Appendix

Table 6: Ablation of MySQA across different strategies of where to include the actions in prompts. The dominant strategy is always giving MySQA the actions at each execution step.

Model	Ingredient Recall	Answer Precision	Citation Accuracy	Citation Recall	$\mathcal{A}$ Adherence
MySQA - Claude Sonnet	0.901	0.917	0.918	0.806	0.851
MySQA - Gemini Flash	0.87	0.989	0.953	0.846	0.823
MySQA - GPT-4.1	0.911	0.97	0.856	0.617	0.921

Appendix

Table 7: Ablation of different LLM s to power MySQA . LLM s have similar scores, so we choose Claude to match Singh et al. (2025) .

Model	Ingredient Recall	Answer Precision	Citation Accuracy	Citation Recall	$\mathcal{A}$ Adherence
4 actions	0.901	0.917	0.918	0.806	0.851
8 actions	0.914	0.899	0.918	0.814	0.832
12 actions	0.927	0.896	0.91	0.81	0.812
24 actions	0.938	0.885	0.915	0.816	0.788
30 actions	0.947	0.898	0.916	0.807	0.777

Appendix

Table 8: Ablation of MySQA across various number of actions. The model preserves report quality, but the proportion of actions followed drops as the number of actions increase.

Figure 7: Accuracy of LLM judges for predicting user satisfaction in personalization aspects over profiles, actions, and reports with one few-shot example . LLM judges are still worse than the majority class baseline.

Figure 8: Accuracy of LLM judges for predicting user satisfaction in personalization aspects over profiles, actions, and reports with no examples . LLM judges are still worse than the majority class baseline.

Figure 9: Accuracy of LLM judges for predicting user satisfaction in personalization aspects over profiles, actions, and reports with no metric definitions . LLM judges are still worse than the majority class baseline.

Output Type	Aspect	Description	Freq
Profile	DOMAIN	Uses terms, definitions, or details that do not capture the user’s domain of research	27.6%
	OVERCLAIM	Claims something applies to the user broadly, but only applies to some/parts of papers	17.9%
	CONVENTION	Infers a generic convention of the user’s field (e.g. "You enumerate contributions")	12.8%
	CONTRAST	Has a contrast that misrepresents the user (e.g. "You are X, not Y", but the user is Y)	12.2%
	UNIMPORT	The inference is true from their papers, but not a part of their papers they care about	8.3%
	STRENGTH	States something too strongly about the user (e.g. “You are a deep expert in X”)	7.1%

Appendix

Table 9: Dimensions of personalization errors in MySQA ’s outputs uncovered in our interviews, missed by offline evaluation ( section 3.4 ). Bold text indicates aspects with sufficient data (50+ examples) for evaluating if LLM judges can simulate them ( section 4.3 ).

Figure 10: List of questions we ask participants in the formative. Sections marked with "[walkthrough]" indicates that participants also gave feedback on model outputs or high-level design in addition to answering the questions. During the study, we refer to the list of actions as “plans”, which users found clear.

Figure 11: Full example of one of the author’s profiles with two inferences per section in MySQA . We remove the paper citations after each profile inference for anonymity. Users can view, toggle, and edit their profile inferences.

Figure 12: Full example of actions for a query in MySQA . Users can view, toggle, and edit actions in the list. Clicking on the dropdown arrow gives a more elaborate description of each action.

Figure 13: Full example of a report for a query and actions in MySQA . Users can view each section of the report, collapse them for just a TL;DR, and view highlights for where MySQA personalized. Clicking on an action (left) enables/disables the highlights, revealing an action bar with: 1) the number of highlighted spans in the report; and 2) navigation arrows to jump between highlighted spans. Each plan action and section has buttons which we use to collect feedback.

Figure 14: We provide activities to entertain users while they wait for their profiles to be generated. Users can view jokes, fun facts, or play the Chrome T-Rex game. Adding engaging activities before annotation tasks can have positive outcomes Lewis et al. (2011) .

Model	Answer Coverage	Answer Precision	Citation Precision	Citation Recall	Action Adherence
MyScholarQA	91.4	89.9	91.8	81.4	83.2
ScholarQA	88.9	89.1	90.5	76.9	81.3
OpenScholar	77.2	97.4	82.5	60.4	82.5
STORM	72.0	92.2	73.3	64.7	74.4
Perplexity Sonar DR	81.0	82.9	64.3	46.3	75.0
OpenAI DR (o3)	89.1	90.2	79.2	56.7	93.8

Appendix

Table 10: Deep Research report quality and action instruction-following for query

q

and

8

actions in

\mathcal{A}_{\text{gen}}\cup\mathcal{A}_{\text{person}}

. MySQA surpasses all DR tools in 3/5 metrics. We run OpenAI DR on 10 examples due to latency and cost issues.

Deep Research Agents

Deep research agents plan long-horizon searches, synthesize cited evidence, and produce reports for complex open-ended questions.

See all 115 papers

DRACULA: Hunting for the Actions Users Want Deep Research Agents to Execute
Apr 26, 2026Nishant Balepur, Malachi Hamada, Varsha Kishore +9Deep Research Agents LLM Agent Evaluation
University of Maryland · New York University · Allen Institute for Artificial Intelligence (Ai2)
DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
Jan 13, 2026Ruizhe Li, Mingxuan Du, Benfeng Xu +3Deep Research Agents LLM Agent Evaluation
University of Science and Technology of China · Metastone Technology, Beijing, China
Personalized Deep Research Query Refinement with Graph-Scaffolded Evidence Grounding
Aug 6, 2026Soojin Yoon, Dongha LeeDeep Research Agents Clarification Question Generation
Department of Artificial Intelligence, Yonsei University

Apr 26, 2026J/K move · Enter open · S save

DRACULA: Hunting for the Actions Users Want Deep Research Agents to Execute

Nishant Balepur, Malachi Hamada, Varsha Kishore, Sergey Feldman+8

University of Maryland · New York University · Allen Institute for Artificial Intelligence (Ai2)

Deep Research Agents LLM Agent Evaluation LLM Agents Human-AI Decision Making

Open paper AskPDF arXiv

Scientific Deep Research (DR) agents answer user queries by synthesizing research papers into multi-section reports. User feedback can improve their utility, but existing protocols only score the final report, making it hard to study and learn which intermediate actions DR agents should take to improve reports. We collect DRACULA, the first dataset with user feedback on intermediate actions for DR. Over five weeks, nineteen expert CS researchers ask queries to a DR system that proposes actions (e.g., "Add a section on datasets"). Our users select actions they prefer, then judge whether an output report applied their selections successfully, yielding 8,103 action preferences and 5,230 execution judgments. After confirming a DR agent can execute DRACULA's actions, we study the predictability of user-preferred actions via simulation-how well LLMs predict the actions users select-a step toward learning to generate useful actions. We discover: (1) LLM judges initially struggle to predict action selections, but improve most when using a user's full selection history, rather than self-reported or extrapolated user context signals; (2) Users' selections for the same query differ based on unstated goals, bottlenecking simulation and motivating affordances that let users steer reports; and (3) Our simulation results inform an online intervention that generates new actions based on the user's past interactions, which users pick most often in follow-up studies. Overall, while work extensively studies execution, DRACULA reveals a key challenge is deciding which actions to execute in the first place. We open-source DRACULA's study design, user feedback, and simulation tasks to spur future work on action feedback for long-horizon agents.

Abstract

Deep Research Agents

Explore similar work

DRACULA: Hunting for the Actions Users Want Deep Research Agents to Execute

DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports

Personalized Deep Research Query Refinement with Graph-Scaffolded Evidence Grounding