Ask Less, Ask Better: Allocating User Attention in Proactive LLM Agents with Interruption Budgets

Xuanming Zhang; Amir H. Rezaeian; Miguel Ballesteros; Weiyi Sun; Werner Geyer; Lydia B. Chilton; Zhou Yu; Yassine Benajiba

cs.CLOct 17, 2026

Ask Less, Ask Better: Allocating User Attention in Proactive LLM Agents with Interruption Budgets

Authors: Xuanming Zhang, Amir H. Rezaeian, Miguel Ballesteros, Weiyi Sun, Werner Geyer, Lydia B. Chilton, Zhou Yu, Yassine Benajiba,

Organizations: Columbia University · Oracle AI

Abstract

LLM agents can ask clarifying questions to complete tasks, but answering each question demands users' time and effort. Existing evaluations typically measure this burden by counting questions, treating necessary and avoidable questions equally. We propose an evaluation protocol that distinguishes how often agents ask from how well they ask. An interruption budget limits the number of questions, while a user-attention cost charges extra for questions that are hard to read or not worth asking, as rated by a human-validated LLM judge. Beyond question counts, this cost captures perceived effort: on conversation pairs with the same number of questions, human annotators agreed with the cost on which conversation required more effort 75% of the time. Across eight LLMs and four customer-service domains, limiting agents to one to three questions reduces user-attention cost by 36-73% while keeping 86-94% of the task reward they earn with unlimited questions. At the same budget, reviewing each question against the memory of users' earlier answers cuts attention cost by another 13% with little reward loss and yields more valuable questions. Models also differ sharply in how well they ask: at the same user-attention cost, the most efficient models gain more than twice as much reward from their questions as the others. These results show that asking less and asking better are distinct goals, and that evaluations of proactive agents should measure both.

Figures & tables

Figure 1: Question count alone misses what each question costs the user. Three agents complete the same request (adapted from real runs); B and C each ask once under a two-question interruption budget, but C incurs the lowest user-attention cost and is preferred. We define: Avoidability

=1-{}

(judged question value)

/10

. Reading effort and question value are scored by a human-validated LLM judge and defined in Section 3 .

Panel of Table 1 — Table 1: Evaluation features addressed by related work. Limit : caps actions, turns, or queries; Content : cost depends on question content; Matched : policies compared at equal cost or question count; Judge : evaluator validated against humans; Burden : humans compare conversation burden. Details: Appendix C.3 .

Reading effort : five dimensions, each rated 1–5
1. Goal clarity	Is it clear what the agent needs and why?
2. Single ask	One answerable thing, not several bundled?
3. Answer effort	Answerable without decoding jargon?
4. Actionability	If it asks for an action, is it concrete?
5. Conciseness	As short as possible while still clear?
Question value : high only if all four criteria hold

Table 2: What the LLM judge scores. Reading effort uses readability ( Säuberli et al., 2026 ) , with an overall score

r(q)\in[0,20]

; question value follows Dong et al. (2026) , with

v(q)\in[0,10]

. See full prompts in Appendix I , and notation in Table 9 .

Figure 2: The MGR agent. The backbone LLM proposes a step 1 . Tool calls go straight to the tools. An LLM classifier 2 sends answers and policy-required requests to the user directly; these are not interruptions and spend no budget. While budget remains, a discretionary question goes to self-review 3 , a second backbone LLM call that checks it against memory

M

: approved or substituted questions are delivered and spend one budget; blocked ones are not asked.

	Customer service			Interactive research
Method	$R\uparrow$	$\tilde{A}\downarrow$	$\tilde{V}\uparrow$	$R\uparrow$	$\tilde{A}\downarrow$	$\tilde{V}\uparrow$
Unrestricted	49.4	29.5	63.5	32.7	19.0	64.3
Frugal prompt	47.0	24.9	65.7	31.7	15.8	67.0
VoI gate	47.4	30.7	58.6	32.3	18.4	67.3
Budgeted	44.7	13.7	69.4	29.6	16.0	68.0
Budgeted + MGR	43.9	12.0	71.7	28.7	15.1	69.8

Table 3: Reward, user-attention cost, and judged question value by method. Customer service: four domains, eight models; interactive research: Ambiguous BrowseComp-Plus, five models; three trials each. Budgeted rows average

N{=}1

–3 and

N{=}1

–8, respectively.

\tilde{A}

\tilde{V}

: 0–100 scales (Appendix E ). Per-domain and utility results: Tables 21 and 26 .

	Annotator judgment
Pairs (60 each)	Question burden	Overall burden
Different counts	$90.8\ [84.9,95.8]$	$62.5\ [52.5,72.5]$
Equal counts	${71.7}\ [59.1,82.9]$	${75.0}\ [64.5,84.9]$
value alone	$71.1\ [58.5,82.8]$	$73.6\ [62.7,83.7]$
effort alone	$62.7\ [48.1,76.4]$	$57.3\ [44.0,70.0]$
question words	$49.6\ [35.7,63.4]$	$56.7\ [44.2,69.6]$

Table 4: Human burden study: agreement with the user-attention cost ranking. Share (%) of non-tie answers choosing the conversation ranked higher by the full cost or, in indented rows, by one score alone (question words: word count of the agent’s discretionary questions); brackets: 95% pair-bootstrap intervals. Annotators rarely answered “same” (at most 7% per setting). Details: Appendix G .

Figure 3: Budgets trade reward for user attention, at rates that differ by model. Each path is one model, averaged over four domains, from No-ask (square, zero cost) through Budgeted at

N=1

–3 (dots) to Unrestricted (diamond); triangles and circles mark the Frugal prompt and VoI gate. The slope from the square is asking efficiency, the reward gained per unit of attention cost; upper-left is better. Per-domain panels: Figure 5 .

Policy	$A\downarrow$	$V\uparrow$	$\Delta R\uparrow$	$\Delta\tilde{V}\uparrow$
	Observed		Equal cost
Budgeted	13.7	69.2	–	–
Budgeted + Review	12.8	68.6	$-0.7^{*}$	$-0.9^{*}$
Budgeted + Review + Summary	12.3	70.6	$+0.1\phantom{{}^{*}}$	$+0.9^{*}$
Budgeted + MGR	12.0	71.5	$+0.2\phantom{{}^{*}}$	$+1.7^{*}$

Table 5: Memory ablation for self-review. Means over

N=1

–3, four domains, and eight models, on the scales of Table 3 .

V

averages domains within model, so it differs slightly from that table. Deltas: differences from Budgeted at equal expected cost; ∗ : unadjusted paired 95% interval excludes zero. Holm-adjusted tests: Table 28 .

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Customer service
Dataset	Total	Dev	Test
Retail	114	23	91
Airline	50	10	40
Telecom	114	23	91
Ambiguous WebShop	200	40	160
Interactive research

Appendix

Table 6: Dataset sizes and development/test splits. WebShop and BrowseComp-Plus use the ambiguous variants.

	Requests	User-attention		$\Delta R$ at	Reviews	Vetoed	Fixed message
Policy	per session	cost	Reward	equal cost	per session	(%)	(%, $\tau$ -bench)
Unrestricted	3.12	31.0	58.9	–	–	–	0.0
VoI gate as run ( $c=0.27$ )	3.40	34.6	57.3	–	–	–	0.0
VoI gate, enforced veto ( $c=0.50$ )	1.73	17.2	51.0	$-4.2$ ∗	6.4	71	6.9
VoI gate, enforced veto ( $c=0.65$ )	0.44	4.1	44.8	$-3.0$ ∗	8.1	94	12.0
Budgeted ( $N=1$ )	0.94	8.5	50.9	0	–	–	6.2

Appendix

Table 7: Enforced VoI gate. Three GPT models, four domains, one trial, 1,146 sessions per threshold. Every discretionary draft is reviewed and vetoes enforced. Unrestricted, VoI gate as run, and Budgeted rows: main evaluation, three trials, same tasks. The original row uses a different gate; it is not another point in the same threshold sweep.

R,A

: main display scales;

\Delta R

: VoI gate minus cost-matched Budgeted. Fixed message:

\tau

fallback rate. ∗ : unadjusted paired 95% interval excludes zero. Maximum reviews: 52/86 at

c=0.50/0.65

		Requests	User-attention		Question	$\Delta R$ at	Reviews	Vetoed	Fixed message
$N$	Policy	per session	cost	Reward	value	equal cost	per session	(%)	(%, $\tau$ -bench)
1	Budgeted	0.94	8.5	50.9	78.1	0	–	–	6.2
	Budgeted + VoI gate	0.91	8.8	46.5	67.8	$-4.6$ ∗	1.2	20	5.4
	Budgeted + MGR	0.83	7.4	49.3	79.3	$-0.3$	–	–	6.9
2	Budgeted	1.63	15.4	54.5	73.1	0	–	–	3.8
	Budgeted + VoI gate	1.61	16.1	50.3	64.6	$-4.6$ ∗	2.1	21	4.5

Appendix

Table 8: VoI gate under the same budget. Budgeted plus the enforced VoI gate at

c=0.27

; three GPT models, four domains, three trials, 3,438 sessions per budget. Budgeted and Budgeted + MGR rows: main evaluation, three trials, same tasks.

R,A

: main display scales; value: 0–100;

\Delta R

: policy minus cost-matched Budgeted. Vetoed: share of reviews rejecting the draft. Fixed message:

\tau

fallback rate. ∗ : unadjusted paired 95% interval excludes zero. Maximum reviews: 8/10/14 at

N=1/2/3

Symbol	Meaning
$\tau$ , $t$	Interaction session; step index
$\pi$	Policy: ask, call a tool, or respond
$R(\tau)$	Task reward in $[0,1]$
$q$	Delivered discretionary request message
$Q(\tau)$	All such requests in $\tau$
$N$	Interruption budget, $\|Q(\tau)\|\leq N$

Appendix

Table 9: Notation for Sections 3 and 4 .

Judge–annotator agreement by benchmark
Comparison	Readability	Question value
Human–human	92.5 [86.1, 97.6]	87.5 [79.0, 94.9]
Judge–annotator (mean)	90.0 [84.0, 95.6]	90.0 [84.2, 95.1]
Second judge (mean)	71.2 [61.3, 80.6]	76.9 [68.5, 85.0]
$\tau$ -bench ( $n=40$ )	90.0	93.8
WebShop ( $n=40$ )	90.0	86.2

Appendix

Table 10: Human validation on the main customer-service benchmarks. Agreement (%) on 80 held-out request pairs per dimension, with task-clustered 95% intervals. Labels are A/B/tie; score differences of at most one raw point imply a judge tie. Judge–annotator agreement averages agreement with each of the two annotators. The second-judge row is Grok-4.6’s agreement with the annotators on the same pairs (Appendix E.3 ).

Three labels
Comparison	Agreement (%)	Cohen’s $\kappa$
Human–human	90.0 [84.0, 95.0]	0.84 [0.73, 0.92]
Classifier–A1	90.0 [84.0, 95.0]	0.84 [0.74, 0.92]
Classifier–A2	92.0 [87.0, 97.0]	0.87 [0.77, 0.95]
Classifier–human	91.0 [86.5, 95.5]	0.85 [0.77, 0.92]
Discretionary versus non-discretionary

Appendix

Table 11: Human validation of request classification. Two annotators label the same 100 cases. Classifier–human rows average the two annotator comparisons, including their separate

\kappa

values. Brackets are 95% case-bootstrap intervals (2,000 draws), retaining both annotators’ labels in each sampled case.

Agreement on the same questions
	Question value	Reading effort
Questions	1989	1989
Mean, first judge	$67.0$	$16.2$
Mean, second judge	$69.1$	$27.1$
Rank correlation	$0.69$	$0.85$
Mean abs. difference	$14.1$	$11.3$

Appendix

Table 12: Cross-family judge sensitivity. Fixed sample of 1,989 questions scored by GPT-5.6-Sol and Grok-4.6 with identical prompts; scores 0–100. Policy effects weight models/domains equally; brackets: 95% sampled-question bootstrap intervals. Own-vendor contrasts measure relative preference for same-vendor backbones against the other judge.

Cost definition	Budgeted, $N=1$	Budgeted, $N=2$	Budgeted, $N=3$	MGR, mean $N=1$ –3
Messages	70.4 [68.9, 71.7]	49.0 [46.7, 51.1]	33.2 [30.6, 35.7]	10.9 [9.9, 12.0]
Atomic questions	67.8 [66.1, 69.3]	47.1 [44.8, 49.4]	31.7 [29.2, 34.0]	11.4 [10.4, 12.5]
Count + reading effort	71.0 [69.5, 72.4]	49.9 [47.6, 52.1]	34.1 [31.5, 36.5]	11.3 [10.3, 12.4]
Full discretionary cost	73.0 [71.6, 74.3]	51.9 [49.7, 54.0]	35.7 [33.1, 38.2]	12.7 [11.6, 13.8]
Full cost + exempt requests	37.7 [35.7, 39.8]	26.5 [24.5, 28.5]	17.3 [15.3, 19.2]	4.5 [3.7, 5.4]

Appendix

Table 13: Accounting sensitivity: cost saved (%). Eight models, four domains, three trials. Budgeted is compared with Unrestricted; the MGR column compares Budgeted + MGR with Budgeted at the same allowance. Brackets: exploratory paired parent-task bootstrap 95% intervals (10,000 draws). Savings are ratios of equally weighted model/domain/budget means. These rescore fixed trajectories; the enforced message budgets are unchanged.

Policy	Messages	Atomic questions	Exempt requests	Bundled (%)
Unrestricted	2.84 [2.68, 2.99]	5.39 [5.06, 5.72]	3.44 [3.20, 3.67]	52.5 [51.0, 54.0]
Frugal prompt	2.43 [2.30, 2.56]	4.20 [3.95, 4.46]	3.66 [3.42, 3.90]	47.6 [46.1, 49.1]
VoI gate (original)	2.91 [2.74, 3.09]	4.56 [4.29, 4.84]	3.67 [3.43, 3.90]	35.0 [33.7, 36.2]
Budgeted, $N=1$	0.84 [0.83, 0.85]	1.74 [1.69, 1.79]	3.70 [3.46, 3.94]	64.1 [62.4, 65.8]
Budgeted, $N=2$	1.45 [1.42, 1.48]	2.85 [2.75, 2.95]	3.65 [3.41, 3.89]	57.7 [56.1, 59.2]
Budgeted, $N=3$	1.90 [1.85, 1.94]	3.68 [3.53, 3.83]	3.66 [3.42, 3.90]	55.2 [53.5, 56.8]

Appendix

Table 14: Delivered requests per session and bundling. Same panel and exploratory intervals as Table 13 . Atomic counts use the existing classifier; a bundled message contains more than one estimated component question. The bundled share divides aggregate bundled-message count by aggregate discretionary-message count. Required-only exchanges are exempt in the primary protocol.

Figure 4: Attention savings and question-value gains are widespread; reward gains at equal attention cost vary by model. Budgeted + MGR versus Budgeted, across

N=1

–3 and four domains, with paired 95% task-bootstrap intervals. (a) Attention cost saved and (b) change in judged question value at the same interruption budget. (c) Reward difference from a Budgeted comparator matched on expected attention cost. Positive values favor the MGR agent; diamonds summarize all eight models. Means: Table 27 ; by domain: Table 15 .

Reward at equal user-attention cost
	Wide window minus
	Budgeted	Budgeted
	+ Review	+ MGR
User-attention cost (%)	$-1.0$	$+4.8$ ∗
Question value	$+2.5$ ∗	$-3.3$ ∗
Reward	$+1.4$	$+0.1$

Appendix

Table 17: Budgeted + Review with a wider evidence window. Evidence expands to 9,072 tokens versus 6,000 plus 3,072 memory tokens. Three GPT models, three

\tau

domains,

N=1

–3, one trial. Cost changes are relative percent; reward/value changes are display points. Wide review reads 12.1k versus 8.8k tokens and blocks 14% versus 20% for Budgeted + MGR. ∗ : unadjusted paired 95% interval excludes zero.

Elicitation recall (%), eight models
	$N=1$	$N=2$	$N=3$	$N=1$ –3
Budgeted	$25.8$	$29.0$	$30.1$	$28.3$
Budgeted + Review	$23.3$	$26.0$	$27.0$	$25.4$
Budgeted + Review + Summary	$21.9$	$24.4$	$26.2$	$24.2$
Budgeted + MGR	$20.0$	$23.0$	$24.7$	$22.6$
Differences in recall (points)

Appendix

Table 19: Elicitation recall on WebShop. Share (%) of a goal’s withheld items that the shopper discloses in any reply, by the simulator’s literal matcher. Eight models, three trials, 160 test tasks; Unrestricted reaches 31.1%. The matched comparison covers the 23 of 24 model–budget cells inside the range of the Budgeted mixture (seven models at

N=3

). The rule comparison uses the review-revision runs (two trials). Brackets: 95% task-bootstrap intervals.

Figure 5: Reward–user-attention tradeoffs by domain. Eight models, three trials. Curves connect Budgeted

N=1

–3; markers show No-ask, Unrestricted, Frugal prompt, and VoI gate. Upper-left is better.

Method	Main				BrowseComp
	$R\uparrow$	$\tilde{A}\downarrow$	$\tilde{U}\uparrow$	$\tilde{V}\uparrow$	$R\uparrow$	$\tilde{A}\downarrow$	$\tilde{U}\uparrow$	$\tilde{V}\uparrow$
Unrestricted	50.8	33.3	55.7	63.2	32.7	19.0	47.7	64.3
Frugal prompt	48.2	28.4	55.5	65.9	31.7	15.8	48.0	67.0
VoI gate	49.0	35.2	53.9	58.5	32.3	18.4	47.6	67.3
Budgeted	46.0	14.6	58.2	68.4	29.6	16.0	46.5	68.0
Budgeted + MGR	44.5	12.9	57.7	70.8	28.7	15.1	46.1	69.8

Appendix

Table 21: Main domains and Ambiguous BrowseComp-Plus. Five matched models, three trials each. Models: Section 5.3 . Main is the equal four-domain mean. Budgeted rows average

N=1

–3 on Main and

N=1

–8 on BrowseComp-Plus, allowing additional research clarifications about hidden identifying constraints. Budgets and models have equal weights. The ranges differ; Table 26 uses eight models. Scales and value aggregation follow that table (

\lambda=0.03

). Bold marks the best displayed value.

Figure 6: A three-question budget buys more reward per unit of attention overall, significantly so for three of eight models. Reward gained over No-ask per unit of user-attention cost, averaged over four domains, with 95% task-bootstrap intervals. Budgeted (

N=3

) is more cost-effective for three models, less for Llama-3.3-70B, and statistically tied for four. Hatched: unrestricted asking has no clear gain over No-ask for this model, so its comparison says little. Numbers: Table 22 .

Figure 7: Attention valuation changes the preferred budget. GPT-6-Astra utility, averaged equally over four domains and three trials, on the utility scale of Appendix E . The logarithmic axis varies

\lambda

; the dashed line marks 0.03 and the inset enlarges

\lambda\leq 0.1

. VoI gate denotes the original (as-run) gate (Appendix C ).

Figure 8: Asking frequency and measured reward return vary across models and domains. Unrestricted means over three trials. Top: discretionary messages per session. Bottom: reward gain over the evaluated No-ask baseline, in points. No-ask retains required exchanges and is not an optimized no-question policy. Message frequency alone does not diagnose excessive or insufficient interaction.

GPT-6-Astra
	Guarded				Original
Policy	$R\uparrow$	$A\downarrow$	$\bar{U}\uparrow$	$V\uparrow$	$R\uparrow$	$A\downarrow$	$\bar{U}\uparrow$	$V\uparrow$
Unrestricted	0.783	3.135	-0.059	8.374	0.883	1.676	0.433	8.416
Budgeted ( $N=1$ )	0.583	1.240	0.250	8.114	0.858	1.026	0.583	8.362
Budgeted + MGR ( $N=1$ )	0.700	1.106	0.403	8.569	0.833	0.997	0.566	8.801
Budgeted ( $N=2$ )	0.750	2.066	0.195	8.572	0.867	1.520	0.458	8.387

Appendix

Table 25: Interruption allocation under two simulator regimes. Each entry equally averages Retail, Airline, and Telecom, with 20 shared test tasks per domain and two trials.

A=A_{\mathrm{full}}

;

V

is mean judged question value. Both simulator regimes are rerun on the same fixed panel.

Method	$N$	Retail				Airline				Telecom				WebShop
		$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$
No-ask	0	0.58	–	–	–	0.70	–	–	–	0.49	–	–	–	0.29	–	–	–
Unrestricted	$\infty$	0.73	3.36	0.62	8.86	0.88	3.88	0.77	8.19	0.81	2.39	0.74	7.31	0.56	2.17	0.49	6.38
Frugal prompt	$\infty$	0.68	1.93	0.62	9.60	0.81	2.61	0.73	9.10	0.84	2.01	0.78	8.03	0.55	1.66	0.50	7.25
VoI gate	$\infty$	0.69	3.92	0.57	8.28	0.88	4.81	0.74	7.52	0.81	2.79	0.72	6.81	0.56	2.71	0.48	6.29
Budgeted	1	0.65	1.27	0.61	8.57	0.80	1.15	0.77	8.75	0.65	1.20	0.62	7.81	0.56	1.50	0.52	6.84

Appendix

Table 29: GPT-6-Astra: three-trial means.

Method	$N$	Retail				Airline				Telecom				WebShop
		$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$
No-ask	0	0.52	–	–	–	0.57	–	–	–	0.30	–	–	–	0.23	–	–	–
Unrestricted	$\infty$	0.66	5.68	0.49	6.70	0.68	5.46	0.52	6.61	0.36	3.61	0.25	6.42	0.44	2.42	0.37	6.41
Frugal prompt	$\infty$	0.64	4.24	0.51	7.22	0.64	4.16	0.52	6.93	0.27	2.26	0.21	5.92	0.39	1.84	0.33	7.16
VoI gate	$\infty$	0.66	7.15	0.45	5.51	0.60	6.86	0.39	5.73	0.31	2.69	0.23	6.50	0.42	3.11	0.32	6.01
Budgeted	1	0.57	1.31	0.53	7.58	0.64	1.24	0.60	7.74	0.26	1.21	0.22	7.82	0.38	1.32	0.34	7.33

Appendix

Table 30: GPT-5.4: three-trial means.

Method	$N$	Retail				Airline				Telecom				WebShop
		$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$
No-ask	0	0.52	–	–	–	0.49	–	–	–	0.24	–	–	–	0.26	–	–	–
Unrestricted	$\infty$	0.71	7.42	0.48	5.76	0.53	8.71	0.27	5.71	0.29	8.62	0.03	6.73	0.42	2.12	0.36	7.15
Frugal prompt	$\infty$	0.66	7.07	0.45	5.75	0.53	7.67	0.30	6.00	0.26	6.42	0.07	6.87	0.36	1.30	0.32	7.44
VoI gate	$\infty$	0.69	8.05	0.45	5.50	0.57	9.73	0.27	5.11	0.30	7.60	0.08	5.78	0.39	2.94	0.30	6.20
Budgeted	1	0.51	1.31	0.47	8.05	0.43	1.43	0.39	7.28	0.25	1.30	0.21	8.12	0.39	1.13	0.36	7.79

Appendix

Table 31: GPT-4.1: three-trial means.

Method	$N$	Retail				Airline				Telecom				WebShop
		$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$
No-ask	0	0.05	–	–	–	0.35	–	–	–	0.05	–	–	–	0.25	–	–	–
Unrestricted	$\infty$	0.08	4.67	-0.06	4.43	0.42	5.61	0.26	5.51	0.07	2.64	-0.01	4.08	0.24	0.31	0.23	6.24
Frugal prompt	$\infty$	0.08	4.51	-0.06	4.26	0.42	5.55	0.25	5.03	0.08	3.57	-0.03	3.73	0.22	0.11	0.21	7.38
VoI gate	$\infty$	0.11	3.96	-0.01	4.52	0.40	5.41	0.24	4.70	0.05	1.57	0.00	4.28	0.24	0.13	0.23	5.92
Budgeted	1	0.11	1.28	0.08	4.96	0.38	1.33	0.34	6.86	0.06	1.14	0.03	4.66	0.26	0.55	0.24	6.76

Appendix

Table 32: Llama-4-Mav: three-trial means.

Method	$N$	Retail				Airline				Telecom				WebShop
		$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$
No-ask	0	0.38	–	–	–	0.38	–	–	–	0.68	–	–	–	0.25	–	–	–
Unrestricted	$\infty$	0.76	12.73	0.38	5.25	0.48	12.21	0.12	5.27	0.65	4.91	0.50	5.76	0.39	1.12	0.36	7.72
Frugal prompt	$\infty$	0.74	11.50	0.40	5.31	0.47	11.10	0.13	5.47	0.66	4.87	0.51	5.55	0.35	0.86	0.32	7.79
VoI gate	$\infty$	0.67	14.66	0.23	4.77	0.44	12.24	0.07	4.91	0.64	4.25	0.52	5.28	0.37	1.10	0.33	7.34
Budgeted	1	0.46	1.69	0.41	4.23	0.28	1.42	0.24	7.72	0.67	1.56	0.63	7.01	0.40	1.08	0.37	7.73

Appendix

Table 33: Grok-4.3: three-trial means.

Method	$N$	Retail				Airline				Telecom				WebShop
		$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$
No-ask	0	0.05	–	–	–	0.53	–	–	–	0.18	–	–	–	0.25	–	–	–
Unrestricted	$\infty$	0.10	2.28	0.03	6.23	0.51	0.12	0.50	–	0.18	2.19	0.11	5.44	0.38	2.68	0.30	6.12
Frugal prompt	$\infty$	0.10	2.26	0.03	6.74	0.49	0.38	0.48	–	0.18	0.82	0.15	4.91	0.36	1.79	0.30	6.66
VoI gate	$\infty$	0.07	1.93	0.02	6.23	0.51	0.17	0.50	–	0.18	0.95	0.15	5.43	0.36	3.10	0.27	5.38
Budgeted	1	0.07	0.48	0.05	7.64	0.51	0.02	0.51	–	0.18	0.72	0.15	6.41	0.31	1.45	0.27	6.01

Appendix

Table 34: Llama-3.3-70B: three-trial means.

Method	$N$	Retail				Airline				Telecom				WebShop
		$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$
No-ask	0	0.35	–	–	–	0.62	–	–	–	0.41	–	–	–	0.07	–	–	–
Unrestricted	$\infty$	0.80	5.15	0.65	6.92	0.90	6.43	0.70	6.72	0.56	3.19	0.47	6.60	0.38	1.55	0.33	7.20
Frugal prompt	$\infty$	0.83	4.80	0.68	7.18	0.87	5.25	0.71	7.13	0.54	2.74	0.46	6.08	0.33	1.40	0.29	7.46
VoI gate	$\infty$	0.77	5.69	0.60	6.56	0.84	6.44	0.64	6.31	0.52	2.57	0.45	6.41	0.39	1.61	0.34	7.18
Budgeted	1	0.59	1.23	0.55	9.08	0.72	1.28	0.68	7.92	0.45	1.10	0.42	8.11	0.41	1.43	0.37	7.39

Appendix

Table 35: Grok-4.6: three-trial means.

Method	$N$	Retail				Airline				Telecom				WebShop
		$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$	$R$	$A$	$U$	$V$
No-ask	0	0.25	–	–	–	0.54	–	–	–	0.23	–	–	–	0.24	–	–	–
Unrestricted	$\infty$	0.60	5.64	0.43	5.81	0.62	4.22	0.49	6.19	0.28	5.78	0.11	6.59	0.35	2.31	0.28	6.63
Frugal prompt	$\infty$	0.54	5.64	0.37	5.49	0.56	3.38	0.46	6.35	0.27	4.44	0.14	6.73	0.34	1.34	0.30	7.29
VoI gate	$\infty$	0.51	6.91	0.30	4.90	0.62	4.57	0.48	5.31	0.25	5.01	0.10	5.10	0.34	2.83	0.25	6.01
Budgeted	1	0.36	1.34	0.32	7.26	0.59	1.03	0.56	8.20	0.26	1.37	0.22	7.27	0.33	1.38	0.29	6.80

Appendix

Table 36: GPT-OSS-120B: three-trial means.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 711 papers

May 27, 2026

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents

A long-lived LLM agent, such as OpenClaw, earns its value by acting on a user's preferences and constraints across sessions, not just the current request. Yet today's agents keep what a user volunteers but rarely ask for what stays unspoken, leaving a proactivity gap in long-lived LLM agents: an agent cannot act on a preference it never obtained. As users delegate more of their affairs to agents, the impact of this gap grows. We isolate one concrete, controllable slice of this gap as Ask-to-Remember (ATR): the agent decides whether to ask now for a reusable user preference that the current task does not need but a later session with the same user will. ATR is hard even to evaluate: the right question is underdetermined and its payoff deferred to tasks that may never arise. ATRBench, to the best of our knowledge the first ATR benchmark, makes it measurable by fixing each user's preferences as hidden ground truth, so success demands asking, not recall. Across eight frontier LLM agents, defaults fall at least 62 points below an oracle handed the relevant preference, and prompting closes little of it. Diagnostics identify acquisition as the bottleneck. ATRBench surfaces this proactivity gap in current agents and offers a diagnostic testbed for closing it.

Bin Wu, Guanyun Zou, Bingbing Wang +2

Beijing University of Posts and Telecommunications · Nanjing University of Aeronautics and Astronautics · Harbin Institute of Technology (Shenzhen) +1

Proactive Assistance LLM Agent Memory LLM Agent Evaluation Agent Benchmarks Long-Horizon LLM Agents Preference Elicitation+1

Ask PDF

Sep 29, 2026

Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gym

Proactive LLM agents can turn idle compute into useful support before users ask. Yet even correct work can misread user context, impose review costs, or undermine trust. This work proposes foundations for designing, realizing, and evaluating proactive LLM agents around three joint principles (3T): Task Capability, anticipating relevant needs and correctly performing useful work; Temporal Allocation, allocating compute according to resource availability and when results are needed; and Trust, sustaining users' confidence and appropriate reliance on the agent. We connect these objectives to a design space organized around five dimensions: task scope, anticipation horizon, activation trigger, processing timing, and intervention depth, and specify the situation and system modeling needed to support its choices, including user and environment representations, backbone LLMs, and agent harnesses. Lastly, we propose PROACTIVITY-GYM, a simulation-based evaluation testbed including multi-day scenarios, stateful environments, and persona-conditioned simulated users that can evaluate the consequences of proactive assistance across interactions. Evaluations across 23 model-harness configurations uncover substantial performance gaps across 3T and reveal that LLM judges often conflate task capability and trust. A human study with 30 participants demonstrates the importance of the joint 3T optimization: participants show sharp trust declines after intervention misalignment despite correct outcomes, and prefer sleep-time assistance, even when imperfect, to preserve ongoing focus. Together, these findings support designing and evaluating proactive agents through the joint consideration of useful work, compute allocation, and evolving user trust.

Jio Oh, Seunghyun Do, Young-Jun Lee +2

KAIST · University of Minnesota

LLM Agent Evaluation Long-Horizon Agent Evaluation Proactive Assistance LLM Agents Trust in AI Proactive Agents

Ask PDF

Oct 22, 2025

Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents

LLM-based agents are increasingly moving towards proactivity: rather than awaiting instruction, they exercise agency to anticipate user needs and solve them autonomously. However, evaluating proactivity is challenging; current benchmarks are constrained to localized context, limiting their ability to test reasoning across sources and longer time horizons. To address this gap, we present PROBE (Proactive Resolution Of BottlEnecks). PROBE decomposes proactivity as a pipeline of three core capabilities: (1) searching for unspecified issues, (2) identifying specific bottlenecks, and (3) executing appropriate resolutions. We apply PROBE to evaluate leading LLMs and popular agentic frameworks, showing that even state-of-the-art models struggle to solve this benchmark. Computing our consistent measurements across frontier LLMs and agents, we find that the best end-to-end performance of 40% is achieved by both GPT-5 and Claude Opus-4.1. Additionally, we demonstrate the relative capabilities of each model and analyze mutual failure modes. Our results highlight the current limitations of autonomous action in agentic systems, and expose promising future research directions.

Gil Pasternak, Dheeraj Rajagopal, Julia White +4

LLM Agent Evaluation AI Agent Benchmarks Proactive Assistance Proactive Agents

Ask PDF

Abstract

LLM Agents

Explore similar work

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents

Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gym

Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents