Learning What to Ask for Tomorrow: Active Preference Acquisition for Long-Term Personalized Web Agents

Yiqi Wang; Zizhi Zeng; Jiaqi Zhang; Yiqun Duan; Mingkai Zheng; Taotao Cai

cs.CLOct 22, 2026

Learning What to Ask for Tomorrow: Active Preference Acquisition for Long-Term Personalized Web Agents

Authors: Yiqi Wang, Zizhi Zeng, Jiaqi Zhang, Yiqun Duan, Mingkai Zheng, Taotao Cai

Abstract

Web agents increasingly serve the same user across repeated online tasks such as booking travel, shopping, and scheduling, so their long-term usefulness depends on how well they know the user's preferences. Passive memory and personalization methods cannot obtain preferences the user never reveals, and clarification methods that value a question by its benefit to the current task overlook the benefit of reusing the answer in later tasks. We study Active Preference Acquisition (APA): which missing preferences are worth the cost of asking? We define the Prospective Personalization Value (PPV) of a missing preference as its expected cumulative utility gain over future tasks minus the effort and disclosure-risk cost of asking, and select questions under a per-interaction budget. We characterize when cumulative reuse justifies acquisition despite insufficient single-task benefit, and evaluate this regime with an initial-profile diagnostic. APA computes the counterfactual gain of each candidate preference on public task prototypes with the agent's own task solver and aggregates it by the expected reuse of each task family. We evaluate it on ActiveWebPref, a controlled longitudinal benchmark with hidden but queryable user profiles and exact task utilities. At a horizon of 20 tasks, 34.4% of budget-feasible valuable missing preferences are amortization-dependent; APA acquires 80.3% of them, whereas current-task value of information acquires 5.1%. Under per-user asking-cost caps matched to APA's realized expenditure, APA achieves higher net utility than four heuristic selection baselines. In our cost setting, asking about every unknown preference yields lower net utility than not asking, while APA recovers 84.2% of its utility gain with 27.6% of its questions. The value of acquiring a user preference should account for its reuse across future tasks, not only its benefit to the task at hand.

Figures & tables

Figure 1. A single use does not justify asking, but cumulative reuse does. In session $t$ the user asks for a hotel, and the seat preference $m$ is unknown and useless for this task. Each later seat selection gains from knowing $m$ , but with illustrative numbers, a gain of 0.05 per use against an asking cost of 0.15, no single use justifies asking, so a criterion that values only the task at hand does not ask even when a flight is booked. Four uses gain 0.20, so acquiring $m$ once and reusing it pays off. Timeline from session t to t+3. In session t the user books a hotel; a missing seat preference has no immediate value. In sessions t+1 to t+3 flight, conference-travel and business-trip bookings each require selecting a seat; each use alone gains less than the cost of asking, but repeated reuse gains more.

Figure 2. Overview of APA. ① Gap discovery proposes preferences

\mathcal{G}_{t}

missing from the profile

P_{t}

. ② Future reuse modeling estimates how often each may be reused, from a coarse distribution over task families. ③ Prospective value estimation scores each by relevance and impact. ④ Cost-aware acquisition subtracts asking cost and disclosure risk and chooses Ask , Defer , or Skip . Here, the seat preference is asked; the dining preference, rarely reused because restaurant tasks are rare, is deferred for later re-scoring, as is the hotel budget once the seat question has used this interaction’s budget; and a previously refused political preference, which never enters the candidate set, is skipped. Answers update the profile for future tasks. Four-stage pipeline: gap discovery, future reuse modeling, a relevance-impact matrix for value estimation, and a cost-aware policy that asks about the seat preference, defers the rarely reused dining preference (and the hotel budget once the interaction budget is spent), and skips a previously refused political preference; answers update the persistent profile.

Statistic	Travel	Dining	Shopping	Scheduling	Work
# users (train/dev/test)	18/18/36	18/18/36	18/18/36	18/18/36	18/18/36
# task families	4	3	3	3	3
# attributes / user	27	27	27	27	27
% hidden at $t=0$	53.1	53.1	53.1	53.1	53.1
% high-sensitivity	11.1	11.1	11.1	11.1	11.1
mean uses / used field	1.687	2.000	1.862	1.780	2.178

Table 1. ActiveWebPref statistics by primary-user domain. Test combines 90 IID + 90 unseen users; each cell aggregates its 36 test users unless train/dev counts are shown. Attribute inventories are shared across domains; hidden fractions use all 27 fields. Structural use is counted over the first

T=10

future tasks.

Heuristics with per-user asking cost capped at APA’s realized expenditure
Policy	CTU $\uparrow$	AC $\downarrow$	RC $\downarrow$	NU $\uparrow$	#Q $\downarrow$	Unn. $\downarrow$	Regret $\downarrow$
Ask-Nothing	6.289	0.000	0.000	6.289	0.000	–	0.769
Ask-Everything	7.263	3.977	1.428	4.918	13.472	0.728	2.147
LLM-Importance	7.080	2.455	0.883	5.631	11.294	0.741	1.427
Random- $k$ (cost-matched)	6.585	0.654	0.096	6.234	3.696	0.698	0.824
Frequency (cost-matched)	6.748	0.659	0.072	6.400	3.783	0.592	0.658

Table 2. Main results on ActiveWebPref: Overall = 90 IID + 90 unseen users,

T=10

B=0.5

\lambda=0.5

\beta=0.25

K=10

. CTU: cumulative task utility; AC/RC: asking/disclosure risk cost; NU: net utility; #Q: questions; Unn.: unnecessary query rate. Cost-matched rows obey both the per-interaction budget and a per-user cap on total asking cost (Appendix B ); Random-

k

averages five seeds. Pointwise 95% CIs appear in Table 9 ; paired tests are in Table 10 .

Policy	NU $\uparrow$	$\Delta$ NU	$p_{\rm Holm}$	Acq.
Ask-Nothing	5.706	+0.679	0.004	0%
Ask-Everything	4.534	+1.852	0.004	–
LLM-Importance	5.347	+1.039	0.004	–
Random- $k$ (cost-matched)	5.815	+0.570	0.004	–
Frequency (cost-matched)	6.020	+0.365	0.004	–
Impact-only (cost-matched)	6.271	+0.114	0.109	–

Table 3. Amortization-enriched test set (60 users,

T=10

B=0.5

): NU, paired difference APA minus each policy with Holm

p

over the eight non-oracle baselines, and acquisition of its 202 amortization-dependent pairs.

	$T=10$ : 91 pairs, 71 users (36 IID + 35 unseen)			$T=20$ : 198 pairs, 120 users (63 IID + 57 unseen)
Policy	Acquired	Before use	Attr. gain	Acquired	Before use	Attr. gain
Ask-Nothing	0.0% (0)	0.0% (0)	0.000	0.0% (0)	0.0% (0)	0.000
Ask-Everything	100.0% (91)	100.0% (91)	0.254	100.0% (198)	100.0% (198)	0.576
LLM-Importance	100.0% (91)	35.2% (32)	0.204	100.0% (198)	49.0% (97)	0.431
Current-task VoI (public)	3.3% (3)	0.0% (0)	-0.001	5.1% (10)	0.0% (0)	0.017
APA	70.3% (64)	33.0% (30)	0.227	80.3% (159)	46.5% (92)	0.307

Table 4. Initial-profile amortization diagnostic: feasible pairs with

c_{m}\leq B=0.5

\lambda=0.5

\beta=0.25

K=10

. Overall cohorts contain at least one such pair. Rates divide by all designated pairs, including unacquired pairs; counts appear in parentheses.

	$T=10$ : 71 users (36 IID + 35 unseen)			$T=20$ : 120 users (63 IID + 57 unseen)
Policy	MVMR $\downarrow$	$\Delta$ CTU $\uparrow$	NU $\uparrow$	MVMR $\downarrow$	$\Delta$ CTU $\uparrow$	NU $\uparrow$
Ask-Nothing	1.000	0.000	6.291	1.000	0.000	12.437
Ask-Everything	0.000	1.044	4.988	0.000	2.098	12.153
LLM-Importance	0.027	0.903	5.726	0.041	1.864	12.817
Current-task VoI (public)	0.459	0.516	6.639	0.463	1.231	13.462
APA	0.188	0.868	6.787	0.150	1.862	13.725

Table 5. Whole-sequence outcomes on the budget-feasible initial-profile amortization cohorts: each user has at least one designated pair.

B=0.5

\lambda=0.5

\beta=0.25

K=10

. MVMR is the missed valuable rate (Appendix B );

\Delta

CTU is the paired-cohort mean gain over Ask-Nothing.

Figure 3. Acquisition cost versus CTU: full range (a) and detail (b). Overall = 90 IID + 90 unseen users,

T=10

K=10

\beta=0.25

; all

B\in\{0.25,0.5,0.75,1\}

and selection

\lambda\in\{0,0.1,0.25,0.5,1,2\}

settings. Points retain the saved mean coordinates; the APA step line is its observed configuration envelope. Two side-by-side panels show acquisition cost versus CTU, with the full range at left and a magnified central region at right.

Figure 4. Acquisition cost versus NU: full range (a) and detail (b). Overall = 90 IID + 90 unseen users,

T=10

K=10

\beta=0.25

; all

B\in\{0.25,0.5,0.75,1\}

and selection

\lambda\in\{0,0.1,0.25,0.5,1,2\}

settings. Points retain the saved mean coordinates; the APA step line is its observed configuration envelope. NU uses common evaluation prices

\lambda=0.5

\beta=0.25

. Two side-by-side panels show acquisition cost versus NU, with the full range at left and a magnified central region at right.

Figure 5. Horizon scaling: paired NU(APA) minus NU(Current-task VoI (public)) on the IID + unseen test set (180 users) and the amortization-enriched test set (60 users);

B=0.5

\lambda=0.5

\beta=0.25

K=10

, nested saved prefixes. Bands and error bars are pointwise 95% paired-user bootstrap CIs. Paired net-utility differences over five horizons.

Variant	NU $\uparrow$	Unn. $\downarrow$
APA (full)	6.755	0.387
No FutureDist	6.760	0.342
Next-Task Only	6.683	0.315
No Cost ( $\lambda=0$ )	6.498	0.557
No Risk ( $\beta=0$ )	6.727	0.397
Gold Candidate Schema	6.753	0.378

Table 6. Ablations: Overall 180 users (IID + unseen),

T=10

B=0.5

K=10

. NU always uses evaluation

\lambda=0.5

\beta=0.25

; No Cost/No Risk change only the selection coefficient. No FutureDist uses uniform task-family weights. Next-Task Only values the actual next task instance, a privileged lookahead without hidden answers. Gold Candidate Schema bypasses

K

with the complete publicly eligible missing-field pool.

Figure 6. What each policy chooses to learn: successful acquisitions divided by initially unknown fields in each class, pooled over 90 IID + 90 unseen users,

T=10

B=0.5

\lambda=0.5

\beta=0.25

K=10

. Classes overlap; no ephemeral fields occur (N/A). High-sensitivity fields are refused. Acquisition fractions for stable, semi-stable, ephemeral, distractor and high-sensitivity fields.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Policy	CTU $\uparrow$	AC $\downarrow$	RC $\downarrow$	NU $\uparrow$	#Q $\downarrow$	Unn. $\downarrow$	Regret $\downarrow$
Ask-Nothing	6.289	0.000	0.000	6.289	0.000	–	0.776
Ask-Everything	7.263	3.977	1.428	4.918	13.472	0.728	2.147
Random- $k$	6.605	0.684	0.086	6.241	3.981	0.700	0.824
Frequency	7.248	3.977	1.428	4.903	13.472	0.736	2.162
Impact-only	7.250	3.977	1.428	4.904	13.472	0.742	2.161
LLM-Importance	7.206	3.977	1.428	4.860	13.472	0.739	2.204

Appendix

Table 7. Full-policy comparison at

B=1

: Overall = 90 IID + 90 unseen users,

B=1

\lambda=0.5

\beta=0.25

T=10

K=10

. Ask-Everything and Oracle Future-VoI (unlimited) have no budget cap; Random-

k

uses five seeds. These measurements use

B=1

, unlike Table 2 (

B=0.5

Policy	CTU $\uparrow$	AC $\downarrow$	RC $\downarrow$	NU $\uparrow$	#Q $\downarrow$	Unn. $\downarrow$	Regret $\downarrow$
Random- $k$ (cost-matched)	6.585	0.654	0.096	6.234	3.696	0.698	0.824
Frequency (cost-matched)	6.748	0.659	0.072	6.400	3.783	0.592	0.658
Impact-only (cost-matched)	7.037	0.666	0.083	6.683	3.744	0.462	0.376
LLM-Importance (cost-matched)	6.603	0.660	0.339	6.188	3.011	0.704	0.870
APA (ours)	7.109	0.680	0.058	6.755	3.717	0.387	0.304

Appendix

Table 8. Cost-matched comparison: Overall = 90 IID + 90 unseen users;

B=0.5

\lambda=0.5

\beta=0.25

T=10

K=10

. Each control pays every attempted asking cost and obeys both the per-arrival cap and its user's realized APA total-cost cap. Random-

k

uses five seeds.

Metric	Ask- Nothing	Ask- Everything	LLM- Importance	Current-task VoI (public)	APA (ours)	Oracle Future-VoI	Random- $k$ (cost-matched)	Frequency (cost-matched)	Impact-only (cost-matched)	LLM- Importance (cost-matched)
CTU $\uparrow$	6.289 [6.112, 6.455]	7.263 [7.142, 7.382]	7.080 [6.947, 7.205]	6.853 [6.717, 6.984]	7.109 [6.985, 7.233]	7.265 [7.148, 7.380]	6.585 [6.426, 6.739]	6.748 [6.592, 6.895]	7.037 [6.901, 7.170]	6.603 [6.448, 6.751]
AC $\downarrow$	0.000 [0.000, 0.000]	3.977 [3.851, 4.101]	2.455 [2.382, 2.528]	0.379 [0.351, 0.410]	0.680 [0.642, 0.720]	0.400 [0.366, 0.436]	0.654 [0.614, 0.692]	0.659 [0.622, 0.698]	0.666 [0.629, 0.706]	0.660 [0.622, 0.698]
RC $\downarrow$	0.000 [0.000, 0.000]	1.428 [1.339, 1.514]	0.883 [0.808, 0.956]	0.000 [0.000, 0.000]	0.058 [0.036, 0.081]	0.028 [0.011, 0.047]	0.096 [0.057, 0.137]	0.072 [0.047, 0.103]	0.083 [0.058, 0.111]	0.339 [0.294, 0.386]
NU $\uparrow$	6.289 [6.112, 6.455]	4.918 [4.772, 5.067]	5.631 [5.491, 5.776]	6.664 [6.524, 6.793]	6.755 [6.626, 6.880]	7.058 [6.939, 7.175]	6.234 [6.071, 6.394]	6.400 [6.240, 6.549]	6.683 [6.547, 6.819]	6.188 [6.029, 6.341]
#Q $\downarrow$	0.000 [0.000, 0.000]	13.472 [13.183, 13.750]	11.294 [11.044, 11.550]	2.044 [1.878, 2.206]	3.717 [3.522, 3.917]	2.122 [1.950, 2.311]	3.696 [3.451, 3.936]	3.783 [3.572, 4.000]	3.744 [3.544, 3.961]	3.011 [2.822, 3.206]
Unn. $\downarrow$	–	0.728 [0.713, 0.743]	0.741 [0.722, 0.758]	0.314 [0.263, 0.366]	0.387 [0.350, 0.426]	0.029 [0.015, 0.044]	0.698 [0.667, 0.728]	0.592 [0.552, 0.633]	0.462 [0.420, 0.503]	0.704 [0.664, 0.743]

Appendix

Table 9. Overall main results with pointwise 95% user-bootstrap CIs: 180 pooled IID/unseen users,

T=10

B=0.5

\lambda=0.5

\beta=0.25

K=10

; 2,000 replicates, seed 20261003. No-question Unn. is undefined. Cost-matched controls follow Table 8 ; Random-

k

CIs resample five seeds, and Unn. uses 179 defined users.

Experiment	Comparator	$\Delta$ NU	95% CI	$p$	$p_{\rm Holm}$
Main comparison	Ask-Nothing	0.466	[0.349, 0.583]	0.0005	0.0075
Main comparison	Ask-Everything	1.837	[1.750, 1.922]	0.0005	0.0075
Main comparison	LLM-Importance	1.123	[1.049, 1.197]	0.0005	0.0075
Main comparison	Current-task VoI (public)	0.091	[0.019, 0.174]	0.0205	0.0615
Main comparison	Oracle Future-VoI	-0.304	[-0.342, -0.265]	0.0005	0.0075
Main comparison	Random- $k$ (cost-matched)	0.521	[0.425, 0.635]	0.0005	0.0020

Appendix

Table 10. Paired Overall NU contrasts (APA minus comparator). Holm families comprise 15 main and 15 ablation contrasts across IID, unseen and Overall (IID + unseen). The amortization family has two Overall contrasts:

T=10

(71 paired users) and

T=20

(120 paired users), each using 2,000 paired resamples (seed 20261003). Settings match the corresponding main, amortization subset or ablation experiment. The cost-matched rows form a separate four-test Overall Holm family (180 paired users; 2,000 replicates, seed 20261003).

T	$\Delta$ NU	95% CI	$p_{\rm Holm}$
1	-0.011	[-0.021, -0.003]	0.2664
3	-0.021	[-0.049, 0.007]	1.0000
5	-0.004	[-0.042, 0.036]	1.0000
10	0.091	[0.019, 0.174]	0.2664
20	0.175	[0.047, 0.327]	0.2474

Appendix

Table 11. Overall horizon contrasts, APA minus Current-task VoI (public);

B=0.5

\lambda=0.5

\beta=0.25

, nested prefixes, 180 users. Pointwise CIs; Holm adjustment across 15 horizon contrasts.

	Main comparison ( $n=180$ )		Amortization ( $T=10$ , $n=71$ )		Amortization ( $T=20$ , $n=120$ )
Policy	NDCG	Spearman	NDCG	Spearman	NDCG	Spearman
Ask-Nothing	–	–	–	–	–	–
Ask-Everything	–	–	–	–	–	–
LLM-Importance	0.608	0.020	0.625	0.032	0.657	0.023
Current-task VoI (public)	0.550	0.072	0.538	0.057	0.584	0.164
APA (ours)	0.681	0.215	0.669	0.176	0.691	0.296

Appendix

Table 12. Ranking diagnostics for the main comparison and the initial-profile amortization cohorts: main comparison, Overall 180 IID/unseen users;

T=10

B=0.5

\lambda=0.5

\beta=0.25

K=10

. Amortization cohorts contain 71 users at

T=10

and 120 at

T=20

, with the same

B

\lambda

\beta

and

K

. Means are over users with defined diagnostics; – indicates no recorded ranking diagnostic.

$T$	$\Delta$ NU	95% CI	$p_{\rm Holm}$
1	$-0.017$	[ $-0.040$ , 0.002]	0.364
3	$-0.015$	[ $-0.066$ , 0.034]	1.000
5	0.030	[ $-0.057$ , 0.129]	1.000
10	0.354	[0.194, 0.533]	0.0025
20	1.296	[0.786, 1.834]	0.0025

Appendix

Table 13. Horizon contrasts on the amortization-enriched test set: APA minus Current-task VoI (public), 60 paired users, pointwise 95% CIs, Holm over five horizons.

Variant, test set	$T{=}1$	3	5	10	20
APA-v2, IID+unseen	0.034	0.038	0.043	$-0.007$	$-0.013$
APA-v2 (2 $\times$ cost), IID+unseen	0.005	0.031	0.001	$-0.003$	0.068
APA-v2, amortization	0.023	0.021	0.026	0.035	$-0.013$
APA-v2 (2 $\times$ cost), amortization	0.009	$-0.024$	$-0.004$	$-0.088$	0.102

Appendix

Table 14. Mean paired NU difference of APA-v2 variants to APA by horizon (180 IID + unseen or 60 amortization-enriched users; main settings). Means only.

Policy	CTU $\uparrow$	AC $\downarrow$	RC $\downarrow$	NU $\uparrow$	#Q $\downarrow$	Unn. $\downarrow$	Regret $\downarrow$
Ask-Nothing	6.430	0.000	0.000	6.430	0.000	–	0.691
Ask-Everything	7.360	3.966	1.411	5.024	13.522	0.726	2.098
LLM-Importance	7.202	2.463	0.883	5.750	11.378	0.744	1.371
Current-task VoI (public)	6.959	0.383	0.000	6.768	2.067	0.332	0.353
APA (ours)	7.191	0.694	0.056	6.830	3.800	0.387	0.291
Oracle Future-VoI	7.328	0.399	0.028	7.121	2.133	0.028	0.000

Appendix

Table 15. IID: 90 users,

T=10

B=0.5

\lambda=0.5

\beta=0.25

K=10

. Definitions and regret references match Table 2 . Cost-matched rows use the per-user caps of Table 8 ; Random-

k

averages five seeds.

Policy	CTU $\uparrow$	AC $\downarrow$	RC $\downarrow$	NU $\uparrow$	#Q $\downarrow$	Unn. $\downarrow$	Regret $\downarrow$
Ask-Nothing	6.149	0.000	0.000	6.149	0.000	–	0.847
Ask-Everything	7.166	3.988	1.444	4.811	13.422	0.731	2.196
LLM-Importance	6.957	2.447	0.883	5.513	11.211	0.737	1.483
Current-task VoI (public)	6.747	0.376	0.000	6.560	2.022	0.295	0.436
APA (ours)	7.027	0.665	0.061	6.680	3.633	0.387	0.316
Oracle Future-VoI	7.203	0.401	0.028	6.996	2.111	0.029	0.000

Appendix

Table 16. Unseen users: 90 users,

T=10

B=0.5

\lambda=0.5

\beta=0.25

K=10

. Definitions and regret references match Table 2 . Cost-matched rows use the per-user caps of Table 8 ; Random-

k

averages five seeds. Cost-matched Unn. excludes one no-question user (89 defined users).

Figure 7. Overall sensitivity (90 IID + 90 unseen users): performance (top: CTU, NU, regret) and behavior (bottom: AC, RC, questions). Each grid has

\beta\in\{0,0.25,0.5,1\}

by column and

B\in\{0.25,0.5,0.75,1\}

by line style;

\lambda\in\{0,0.1,0.25,0.5,1,2\}

on each horizontal axis.

T=10

K=10

. Shading shows pointwise 95% CIs. NU uses each cell's own

\lambda

and

\beta

. All saved means and CI endpoints are retained. Two compact Overall grids show all budget and coefficient combinations for APA and Oracle Future-VoI.

Active Preference Learning

See all 19 papers

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents
May 27, 2026Bin Wu, Guanyun Zou, Bingbing Wang +2Proactive Assistance LLM Agent Memory
Beijing University of Posts and Telecommunications · Nanjing University of Aeronautics and Astronautics · Harbin Institute of Technology (Shenzhen) +1
Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History
Feb 19, 2026Serin Kim, Sangam Lee, Dongha LeeWeb Agents Web Agent Benchmarks
Department of Artificial Intelligence, Yonsei University, Seoul, Republic of Korea.
AdaptArena: Evaluating Test-Time Personalization of Web Agents
Sep 29, 2026Dongchan Shin, Xing Han Lù, Jiaqi Deng +7Web Agents LLM Personalization
Mila - Quebec AI Institute · McGill University · ServiceNow Research +2

May 27, 2026J/K move · Enter open · S save

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents

Bin Wu, Guanyun Zou, Bingbing Wang, Huan Zhao+1

Beijing University of Posts and Telecommunications · Nanjing University of Aeronautics and Astronautics · Harbin Institute of Technology (Shenzhen)+1

Proactive Assistance LLM Agent Memory LLM Agent Evaluation Agent Benchmarks Long-Horizon LLM Agents Preference Elicitation+1

Open paper AskPDF arXiv

A long-lived LLM agent, such as OpenClaw, earns its value by acting on a user's preferences and constraints across sessions, not just the current request. Yet today's agents keep what a user volunteers but rarely ask for what stays unspoken, leaving a proactivity gap in long-lived LLM agents: an agent cannot act on a preference it never obtained. As users delegate more of their affairs to agents, the impact of this gap grows. We isolate one concrete, controllable slice of this gap as Ask-to-Remember (ATR): the agent decides whether to ask now for a reusable user preference that the current task does not need but a later session with the same user will. ATR is hard even to evaluate: the right question is underdetermined and its payoff deferred to tasks that may never arise. ATRBench, to the best of our knowledge the first ATR benchmark, makes it measurable by fixing each user's preferences as hidden ground truth, so success demands asking, not recall. Across eight frontier LLM agents, defaults fall at least 62 points below an oracle handed the relevant preference, and prompting closes little of it. Diagnostics identify acquisition as the bottleneck. ATRBench surfaces this proactivity gap in current agents and offers a diagnostic testbed for closing it.

Abstract

Active Preference Learning

Explore similar work

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents

Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History

AdaptArena: Evaluating Test-Time Personalization of Web Agents