Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History

Serin Kim; Sangam Lee; Dongha Lee

cs.CLFeb 19, 2026

Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History

Authors: Serin Kim, Sangam Lee, Dongha Lee

Organizations: Department of Artificial Intelligence, Yonsei University, Seoul, Republic of Korea.

Abstract

Large language models have advanced web agents, yet current agents lack personalization capabilities. Since users rarely specify every detail of their intent, practical web agents must be able to interpret ambiguous queries by inferring user preferences and contexts. To address this challenge, we present Persona2Web, the first benchmark for evaluating personalized web agents on the real open web, built upon the clarify-to-personalize principle, which requires agents to resolve ambiguity based on user history rather than relying on explicit instructions. Persona2Web consists of: (1) user histories that reveal preferences implicitly over long time spans, (2) ambiguous queries that require agents to infer implicit user preferences, and (3) a reasoning-aware evaluation framework that enables fine-grained assessment of personalization. We conduct extensive experiments across various agent architectures, backbone models, history access schemes, and queries with varying ambiguity levels, revealing key challenges in personalized web agent behavior. For reproducibility, our codes and datasets are publicly available at https://serin-kimm.github.io/Persona2Web/.

Figures & tables

Figure 1: A personalized web agent generates user-specific responses by leveraging user history, whereas a general web agent often produces generic or random outputs that fail to align with user preference.

Benchmark	Domains	Environment	Personalization
WebLINX	–	Cached (155)	✗
Mind2Web	31	Cached (137)	✗
WebShop	1	Simulated	✗
WebArena	6	Simulated	✗
WebVoyager	5	Open web (15)	✗
AssistantBench	15+	Open web (258)	✗

Table 1: Comparison between Persona2Web and existing web agent benchmarks. Persona2Web offers extensive domain and website coverage, enables evaluation on the open web, and supports personalization assessment. Parenthesized numbers next to the environment indicate the number of websites used; simulated, function-call, and dialogue agent therefore omit website counts. Domain counts are recalculated based on our own criteria.

Figure 2: Overview of Persona2Web construction pipeline and reasoning-aware evaluation process.

Level	Web.	Pref.	Example Query
0	✓	✓	At zocdoc.com , search for doctors near Southside Jacksonville who is a female .
1	✗	✓	Search for doctors near Southside Jacksonville who is a female at my preferred website.
2	✗	✗	Search for doctors in my usual area that match my preferred provider gender at my preferred website.

Table 2: Examples of a Persona2Web query set. Each task provides 3 queries with varied ambiguity levels. Web. and Pref. denote website and preferences, respectively. Level 0 is a clear query that includes all explicit cues.

Statistic	Value	Statistic	Value
Users	50	Queries	150
History entries	102,568	Histories / user	2,051
Domains	7 / 21	Subdomains / user	17
Websites / user	35	Exact match ratio	3.56%

Table 3: Overall statistics of Persona2Web . Histories/user, subdomains/user, and websites/user denote averages across user. Domains are reported as domain/subdomain counts. Exact match ratio denotes the average proportion of history entries in which a preference value appears by exact string match.

Domain	User history		User profile
Domain	Mean	Total	Mean	Total
E-commerce	345	17,272	4	179
Travel/Transportation	659	32,943	4	201
Weather/Maps	271	13,546	1	50
Health/Medical	113	5,670	1	50
Entertainment/Media	261	13,069	3	150

Table 4: Domain coverage of user histories and profiles. Mean values indicate average of history entries and profiles across users.

Panel of Table 5 — Table 5: Performance comparison across two agent architectures (AgentOccam, Browser Use) with five backbone LLMs under three history access schemes (No-history, On-demand, Pre-execution). Metrics include $\mathcal{P}_{\text{web}}$ , $\mathcal{P}_{\text{pref}}$ , $\mathcal{P}_{\text{avg}}$ (average of $\mathcal{P}_{\text{web}}$ and $\mathcal{P}_{\text{pref}}$ ), Intent satisfaction (Intent), and Success Rate (SR). For No-history scheme, we report only $\mathcal{P}_{\text{avg}}$ for brevity.

Web Agents

Web agents use language and vision models to navigate websites, gather information, and complete tasks through interactive interfaces.

See all 167 papers

AdaptArena: Evaluating Test-Time Personalization of Web Agents
Sep 29, 2026Dongchan Shin, Xing Han Lù, Jiaqi Deng +7Web Agents LLM Personalization
Mila - Quebec AI Institute · McGill University · ServiceNow Research +2
APeB: Benchmarking Personalization Ability of Large Language Model Agents
Jul 3, 2026Garry Yang, Zizhe Chen, Xinru Chen +9LLM Personalization LLM Agent Evaluation
1The Chinese University of Hong Kong · 2ByteDance
AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web
Apr 13, 2026Shanshan Zhong, Kate Shen, Chenyan XiongWeb Agent Benchmarks Web Agents
Language Technologies Institute, School of Computer Science, Carnegie Mellon University · Anaxi Labs

Sep 29, 2026J/K move · Enter open · S save

AdaptArena: Evaluating Test-Time Personalization of Web Agents

Dongchan Shin, Xing Han Lù, Jiaqi Deng, Jay Gala+6

Mila - Quebec AI Institute · McGill University · ServiceNow Research+2

Web Agents LLM Personalization AI Agent Benchmarks User Preference Modeling

Open paper AskPDF arXiv

Large language model (LLM) agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-world settings where user intentions are underspecified and preferences are heterogeneous. In practice, users rarely provide explicit profiles, requiring agents to infer latent preferences from implicit signals. Despite its importance for deployment, this problem setting is largely underexplored in existing benchmarks. To address this gap, we introduce AdaptArena, a benchmark for evaluating test-time personalization of web agents via implicit preference inference. AdaptArena consists of 480 tasks, featuring both single-preference and double-preference scenarios. Each evaluation task must be solved by retrieving and leveraging the most relevant historical user trajectory that implicitly encodes the target preference. In addition, we introduce AdaptiveAgent, a retrieval-based framework for standardized evaluation of implicit preference inference. Experiments reveal a substantial performance gap: while oracle agents with access to ground-truth preferences achieve an 82.92% success rate, the evaluated LLM agents using our framework reach at most 15.62%. Furthermore, we find that correctly inferring user preferences is necessary but not sufficient for task success, as execution failures in downstream web interactions remain a significant bottleneck even when agents align with the target preference. These findings highlight implicit preference inference and robust action grounding as key challenges for deploying reliable, user-facing web agents. We release our code: https://github.com/McGill-NLP/web-agents-test-time-adaptations

Abstract

Web Agents

Explore similar work

AdaptArena: Evaluating Test-Time Personalization of Web Agents

APeB: Benchmarking Personalization Ability of Large Language Model Agents

AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web