Bridging Exploration and Persona: Information-Directed Dueling for Test-Time Agent Personalization

Jie Mao; Pingchen Lu; Xiangyi Wang; Hengquan Huang; Shuang Qiu; Min Zhang; Zhongxiang Dai

cs.AIOct 17, 2026

Bridging Exploration and Persona: Information-Directed Dueling for Test-Time Agent Personalization

Authors: Jie Mao, Pingchen Lu, Xiangyi Wang, Hengquan Huang, Shuang Qiu, Min Zhang, Zhongxiang Dai

Organizations: The Chinese University of Hong Kong, Shenzhen · University of Copenhagen · City University of Hong Kong · East China Normal University

Abstract

Effective test-time personalization requires LLM agents to adapt to query-dependent user preferences from lightweight interaction. We study persona selection with binary comparisons between candidate responses as the online signal. We present ID-TAP, a scalable, IDS-inspired neural dueling-bandit framework for large discrete persona spaces. ID-TAP combines a query-conditioned neural utility model with a lightweight Bradley-Terry explorer, and transfers comparison evidence across related queries through similarity-weighted warm starts. The neural model shares statistical strength across queries while preserving query-specific scores, and efficient BT updates score hypothetical challengers. Across seven user-conditioned generation benchmarks under reference- and model-based preference simulators, ID-TAP achieves the strongest aggregate performance among the evaluated baselines within a fixed ten-round comparison budget. These results support information-directed challenger selection as an effective design for test-time persona adaptation.

Figures & tables

Figure 1: Overview of ID-TAP. Historical context induces a shared persona space and each new query inherits similarity-weighted BT statistics from prior queries (1–2). At every interaction round, the neural utility model selects the champion and estimates utility gaps, while the query-local BT explorer computes a plug-in entropy-reduction proxy; their acquisition ratio selects the challenger (3). The agent generates the resulting response pair, obtains one binary preference, and updates both systems (4–5). After

T

rounds, the highest-scoring persona produces the personalized response, and the query-local BT increments are retained for later warm starts.

Dataset	PersonaAgent		POHF		PDO Sampling		ID-TAP
Dataset	R.L	W.R.	R.L	W.R.	R.L	W.R.	R.L	W.R.
LaMP-4	0.607	70.6 $\pm$ 8.6	0.659	68.3 $\pm$ 6.7	0.660	69.4 $\pm$ 7.5	0.781	73.0 $\pm$ 5.5
LaMP-5	0.723	59.7 $\pm$ 1.8	0.822	67.5 $\pm$ 1.0	0.817	68.7 $\pm$ 1.4	0.924	77.2 $\pm$ 0.4
LongLaMP-2	0.887	88.8 $\pm$ 1.3	0.880	79.5 $\pm$ 1.3	0.865	83.2 $\pm$ 0.7	0.939	84.7 $\pm$ 2.0
LongLaMP-3	0.882	82.3 $\pm$ 3.0	0.914	80.3 $\pm$ 2.8	0.912	77.2 $\pm$ 3.5	0.964	85.7 $\pm$ 2.1
LongLaMP-4	0.864	85.2 $\pm$ 1.3	0.868	84.9 $\pm$ 1.6	0.872	84.0 $\pm$ 0.7	0.962	87.9 $\pm$ 0.2

Table 1: Performance across seven benchmarks under two proxy-feedback mechanisms. R.L. is per-query normalized ROUGE-L. W.R. is the final-response win rate against a naive response and is reported as mean

\pm

SEM across users. Dataset results aggregate per-user means from three independent runs; the last row is the macro-average of dataset means. Bold marks the best result and underlining marks the second best.

Figure 2: Performance averaged over LaMP-4, LaMP-5, LongLaMP-2, LongLaMP-3, and LongLaMP-4. Left: query-wise normalized ROUGE-L. Right: optimization over ten simulated-preference rounds.

Method	LongLaMP-3	LongLaMP-4	PrefEval
Random Response	0.601	0.497	0.448
Random Acquisition + Neural Final	0.615	0.516	0.507
ID-TAP-NH	0.764	0.647	0.629
ID-TAP-NBT	0.774	0.657	0.629
Random Challenger	0.778	0.644	0.618
Full ID-TAP	0.783	0.664	0.629

Table 2: Normalized ROUGE-L for matched component ablations, aggregated from per-user means over three independent runs. NH removes neural and BT history; NBT retains neural history but removes BT transfer.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Symbol	Value
Small profile threshold	$T_{\text{small}}$	10
Large profile threshold	$T_{\text{large}}$	20
History context count (small)	$H_{\text{small}}$	5
History context count (large)	$H_{\text{large}}$	10
Total I/O pairs limit	$N_{\text{IO}}$	10

Appendix

Table 3: Data Splitting Parameters

Dataset	$N_{\mathrm{eval}}$	PersonaAgent	POHF	PDO Samp.	ID-TAP
LaMP-4	363	0.1700 $\pm$ 0.007846	0.1790 $\pm$ 0.007499	0.1768 $\pm$ 0.007085	0.1973 $\pm$ 0.006559
LaMP-5	400	0.3924 $\pm$ 0.009460	0.4385 $\pm$ 0.008671	0.4346 $\pm$ 0.008551	0.4927 $\pm$ 0.008513
LongLaMP-2	400	0.1729 $\pm$ 0.002141	0.1716 $\pm$ 0.002119	0.1678 $\pm$ 0.001918	0.1831 $\pm$ 0.002042
LongLaMP-3	230	0.1322 $\pm$ 0.001532	0.1391 $\pm$ 0.001712	0.1378 $\pm$ 0.001251	0.1456 $\pm$ 0.001562
LongLaMP-4	326	0.1217 $\pm$ 0.001576	0.1234 $\pm$ 0.001806	0.1234 $\pm$ 0.001547	0.1367 $\pm$ 0.001853
PrefEval	40	0.1700 $\pm$ 0.005887	0.2103 $\pm$ 0.006989	0.1943 $\pm$ 0.004623	0.2131 $\pm$ 0.007093

Appendix

Table 4: Unnormalized ROUGE-L performance before per-query normalization. Results are mean

\pm

SEM across users, where each user’s result averages three independent runs. Bold indicates the best result and underlined values indicate the second-best result in each dataset.

Dataset	$N_{\mathrm{user}}$	$\bar{\mu}$	$\bar{\sigma}$	$p_{10}$	$p_{90}$	Near-dup $\geq$ 0.95 (%)	Eff. Arm (%)	Silhouette ( $K$ =20)
LaMP-4	20	0.672	0.090	0.553	0.779	0.041	97.6	0.104
LaMP-5	20	0.694	0.081	0.590	0.790	0.094	95.3	0.125
LongLaMP-2	20	0.701	0.080	0.599	0.794	0.092	95.0	0.121
LongLaMP-3	20	0.657	0.093	0.537	0.769	0.018	98.2	0.107
LongLaMP-4	20	0.686	0.084	0.577	0.787	0.037	97.5	0.105
PrefEval	40	0.716	0.083	0.607	0.814	0.190	94.4	0.098

Appendix

Table 5: Persona space quality metrics across 7 benchmarks (180 users in total; 400 candidates per user).

N_{\mathrm{user}}

is the number of sampled users in each dataset.

\bar{\mu}

: mean pairwise cosine similarity;

\bar{\sigma}

: standard deviation;

p_{10}

p_{90}

: 10th/90th percentile of the pairwise similarity distribution; Near-dup: fraction of pairs with similarity

\geq 0.95

; Eff. Arm: percentage of candidates whose nearest-neighbor similarity

<0.95

; Silhouette: K-means silhouette coefficient at

K{=}20

Category	Parameter	Symbol	Value
Neural Network	Input dimension	$d_{\text{in}}$	$2\times d_{\text{emb}}$
	Hidden dimension	$d_{h}$	1024
	Number of hidden layers	$L$	1
	Activation function	$\sigma(\cdot)$	GELU
	Dropout rate	$p_{\text{drop}}$	0.05
Training	Learning rate	$\eta$	$5\times 10^{-4}$

Appendix

Table 6: Complete hyperparameter configuration for ID-TAP. The table is organized into four sections: neural network architecture, training settings, algorithm-specific parameters, and LLM configuration.

Dataset	$\bar{\mu}$ (IR)	$\bar{\mu}$ (KW)	$\Delta\bar{\mu}$
LaMP-4	0.672	0.720	$+0.048$
LongLaMP-4	0.686	0.695	$+0.009$
PrefEval	0.716	0.830	$+0.114$
UltraChat	0.715	0.834	$+0.119$

Appendix

Table 7: Mean pairwise cosine similarity under iterative rephrasing (IR) and keyword-conditioned construction (KW).

Dataset	POHF	PDO Samp.	ID-TAP
LaMP-4	24.1	25.7	73.5
LongLaMP-4	25.3	28.1	72.2
PrefEval	65.0	64.0	95.3
UltraChat	67.3	66.8	94.8

Appendix

Table 8: Arm-2 exploration rates under iterative rephrasing (IR).

Dataset	POHF	PDO Samp.	ID-TAP
LaMP-4	16.6 ( $-7.5$ )	17.6 ( $-8.1$ )	74.8 ( $+1.3$ )
LongLaMP-4	16.5 ( $-8.8$ )	20.3 ( $-7.8$ )	71.9 ( $-0.3$ )
PrefEval	61.0 ( $-4.0$ )	62.5 ( $-1.5$ )	94.4 ( $-0.9$ )
UltraChat	64.8 ( $-2.5$ )	60.5 ( $-6.3$ )	93.3 ( $-1.5$ )

Appendix

Table 9: Arm-2 exploration rates under keyword-conditioned construction (KW). Parentheses report the change from IR in percentage points.

Dataset group	Mean POHF exploration	Mean ID-TAP margin
LaMP-4 and LongLaMP-4	16.55%	$+0.0160$
PrefEval and UltraChat	62.90%	$+0.0065$

Appendix

Table 10: Association between conventional exploration under KW and ID-TAP’s ROUGE-L margin over the strongest baseline.

Dataset	ID-TAP	POHF	PDO Samp.	PersonaAgent
UltraChat	0.2430	0.2291	0.2213	0.1057
PrefEval	0.2587	0.2498	0.2430	0.1651
LaMP-4	0.2262	0.1900	0.2001	0.1529
LongLaMP-4	0.1392	0.1310	0.1320	0.1230

Appendix

Table 11: ROUGE-L with iterative-rephrasing personas (

K=100

n_{\mathrm{user}}=40

sampled users per dataset).

Dataset	ID-TAP	POHF	PDO Samp.	PersonaAgent
UltraChat	0.2565	0.2441	0.2496	0.1090
PrefEval	0.2614	0.2556	0.2527	0.1647
LaMP-4	0.2495	0.2201	0.2172	0.1554
LongLaMP-4	0.1388	0.1356	0.1350	0.1253

Appendix

Table 12: ROUGE-L with keyword-conditioned personas (

K=100

n_{\mathrm{user}}=40

sampled users per dataset).

Dataset	PersonaAgent	POHF	PDO Samp.	ID-TAP
LaMP-4	73.2	80.2	80.4	81.9
UltraChat	81.7	86.7	85.8	89.1

Appendix

Table 13: Win rate with GPT-4o used for both preference generation and evaluation.

Dataset	PersonaAgent	POHF	PDO Samp.	ID-TAP
LaMP-4	68.1	75.0	77.2	78.2
UltraChat	64.2	85.0	82.5	89.2

Appendix

Table 14: Win rate with GPT-4o preferences and DeepSeek-V3.2 evaluation.

Dataset	PersonaAgent	POHF	PDO Samp.	ID-TAP
UltraChat	0.838	0.890	0.889	0.892
PrefEval	0.847	0.885	0.887	0.888
LaMP-4	0.862	0.870	0.872	0.876
LongLaMP-4	0.812	0.814	0.812	0.818

Appendix

Table 15: BERTScore under a metric independent of the preference-optimization signal.

Dataset	Mean	Std.	SEM
UltraChat	1.1142	0.4211	0.0666
PrefEval	0.9513	0.1745	0.0276
LaMP-4	2.5674	0.9069	0.1434
LongLaMP-4	3.1499	0.9750	0.1542

Appendix

Table 16: ID-TAP wall-clock time with

K=100

and

n_{\mathrm{user}}=40

sampled users per dataset. Statistics are computed across per-user averages of three independent runs and reported in seconds per interaction round.

Dataset	Mean	Std.	SEM
UltraChat	0.0964	0.0189	0.0030
PrefEval	0.0651	0.0157	0.0025
LaMP-4	0.5709	0.1459	0.0231
LongLaMP-4	0.3172	0.0839	0.0133

Appendix

Table 17: ID-TAP algorithmic update time with

K=100

and

n_{\mathrm{user}}=40

sampled users per dataset. Statistics are computed across per-user averages of three independent runs and reported in seconds per interaction round.

Panel of Figure 5 — Figure 5: Query-level performance across all evaluated benchmarks.

Panel of Figure 6 — Figure 6: Iteration-level trajectories across all evaluated benchmarks.

LLM Personalization

LLM personalization adapts language models to users’ preferences, histories, expertise, and changing goals while balancing usefulness, privacy, and control.

See all 130 papers

AdaptArena: Evaluating Test-Time Personalization of Web Agents
Sep 29, 2026Dongchan Shin, Xing Han Lù, Jiaqi Deng +7Web Agents LLM Personalization
Mila - Quebec AI Institute · McGill University · ServiceNow Research +2
Online LLM Personalization from Pairwise Preferences Across Users
Oct 12, 2026Romain Puech, Georgy Salakhutdinov, Dimitris BertsimasLLM Personalization User Preference Modeling
Massachusetts Institute of Technology
APeB: Benchmarking Personalization Ability of Large Language Model Agents
Jul 3, 2026Garry Yang, Zizhe Chen, Xinru Chen +9LLM Personalization LLM Agent Evaluation
1The Chinese University of Hong Kong · 2ByteDance

Sep 29, 2026J/K move · Enter open · S save

AdaptArena: Evaluating Test-Time Personalization of Web Agents

Dongchan Shin, Xing Han Lù, Jiaqi Deng, Jay Gala+6

Mila - Quebec AI Institute · McGill University · ServiceNow Research+2

Web Agents LLM Personalization AI Agent Benchmarks User Preference Modeling

Open paper AskPDF arXiv

Large language model (LLM) agents have demonstrated strong performance on complex web navigation tasks, yet they remain brittle in real-world settings where user intentions are underspecified and preferences are heterogeneous. In practice, users rarely provide explicit profiles, requiring agents to infer latent preferences from implicit signals. Despite its importance for deployment, this problem setting is largely underexplored in existing benchmarks. To address this gap, we introduce AdaptArena, a benchmark for evaluating test-time personalization of web agents via implicit preference inference. AdaptArena consists of 480 tasks, featuring both single-preference and double-preference scenarios. Each evaluation task must be solved by retrieving and leveraging the most relevant historical user trajectory that implicitly encodes the target preference. In addition, we introduce AdaptiveAgent, a retrieval-based framework for standardized evaluation of implicit preference inference. Experiments reveal a substantial performance gap: while oracle agents with access to ground-truth preferences achieve an 82.92% success rate, the evaluated LLM agents using our framework reach at most 15.62%. Furthermore, we find that correctly inferring user preferences is necessary but not sufficient for task success, as execution failures in downstream web interactions remain a significant bottleneck even when agents align with the target preference. These findings highlight implicit preference inference and robust action grounding as key challenges for deploying reliable, user-facing web agents. We release our code: https://github.com/McGill-NLP/web-agents-test-time-adaptations

Abstract

LLM Personalization

Explore similar work

AdaptArena: Evaluating Test-Time Personalization of Web Agents

Online LLM Personalization from Pairwise Preferences Across Users

APeB: Benchmarking Personalization Ability of Large Language Model Agents