Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents

Ido Levy; Asaf Yehudai; Segev Shlomov; Asaf Adi; Leshem Choshen

cs.AISep 29, 2026

Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents

Authors: Ido Levy, Asaf Yehudai, Segev Shlomov, Asaf Adi, Leshem Choshen

Organizations: IBM · Weizmann Institute of Science

Abstract

An agent that uses tools typically responds to what the user explicitly asks, yet completing the task may require information the user never requested. Work on proactive agents mainly studies whether and when an agent should act on its own, not what information it should pursue. We study a distinct axis of proactivity: its content. Horizontal proactivity pursues unstated information that the current context already identifies, and vertical proactivity pursues needs that only earlier evidence reveals. A need graph, recovered from a benchmark's own decomposition, records which needs depend on which, so both forms, and whether the agent stops at the right time, can be scored from a transcript without a model judge. To learn this behavior, we propose Q&D (questioner and drafter), which trains a questioner to prefer the question whose continuation retrieves more of the required evidence, with no reward model or judge. On held-out splits of three multi-hop question-answering benchmarks, at equal retrieval spend, the trained questioner improves both forms of proactivity over the same model, prompted, and outperforms a prompted model $15\times$ larger in the same role on two of the three, and the gain persists after controlling for question volume and length. Without further training, we place the questioner in an interactive customer-service agent with a simulated customer, where it completes more tasks while asking fewer questions, and in retail it outperforms the $15\times$ larger model with fewer follow-up turns from the customer. These results show that proactivity depends not only on whether an agent acts without being asked, but also on what it chooses to pursue and when it stops.

Figures & tables

quantity	definition
required-evidence coverage	share of the required needs recovered, $\|V\cap S\|/\|V\|$
breadth	independent lines in which the run resolves a need at depth $\geq 1$
depth-weighted recall	$\sum_{d}w_{d}C_{d}/\sum_{d}w_{d}$ with $w_{d}=0.5,1,1.5,2$ for $d=0,1,2$ and 3 to 5
deepest need resolved	$\max\{d_{v}:v\in V\cap S\}$ , the level the run reached
out-of-order rate	share of resolved needs taken before a prerequisite (lower is better)
stop-when-done	$\Pr[\text{stop}\mid V\subseteq S_{t}]$ over the decision points the policy reaches

Table 1: What is read from a run against its need graph.

V

is the set of required needs,

S_{t}

the evidence held after

t

questions (

S

at the end),

d_{v}

a need’s depth, and

C_{d}

the coverage at depth

d

. Appendix A gives each estimator and its edge cases.

Equal spend
	MuSiQue		StrategyQA		2WikiMultiHopQA
	Baseline	Q&D	Baseline	Q&D	Baseline	Q&D
Required-evidence coverage (%)	78.3	89.5	78.4	85.5	88.2	93.1
Depth-weighted recall (%)	77.9	90.5	51.2	56.6	84.8	90.3
Deepest need resolved (depth)	1.53	1.75	0.54	0.64	0.57	0.63
Breadth (lines)	0.87	0.99	0.47	0.52	0.77	0.84

Table 2: At equal spend, the trained questioner recovers more of the required evidence than the same model, prompted, on every suite, and more than GPT-OSS-120B on two of three. Each column pair gives a baseline’s level and Q&D’s over

200

held-out tasks per suite, the baseline being the same model, prompted, except on rows that name GPT-OSS-120B. At equal spend a pair is read at its lower question count, so Q&D’s level varies by comparator. Bold marks the better of a pair where the difference is decided (Section 4 ), and

\downarrow

a measure where lower is better. Differences in the text are computed before rounding (Table 10 ).

	MuSiQue				StrategyQA
supervision	gain	stop $\|$ done	ask $\|$ not done	asks	gain	stop $\|$ done	ask $\|$ not done	asks
same model, prompted	—	15.3	96.4	6.29	—	14.6	96.3	5.59
imitation (first stage)	$+4.9$	95.1	86.8	3.05	$+8.0$	90.7	83.0	1.34
question pairs	$\boldsymbol{+16.7}$	98.7	86.0	2.47	$\boldsymbol{+8.4}$	98.0	83.4	1.28
stop contrasts alone	$-0.5$	10.1	99.6	6.95	$-1.7$	9.7	98.1	5.75
final stage, seed 1	$+15.7$	42.7	94.2	4.23	$+8.2$	54.6	91.2	2.13

Table 3: What each training signal teaches, on development tasks (

132

MuSiQue and

333

StrategyQA), each row training the same model on one signal (Section 3 ), the final stage on both. Gain is coverage over the same model, prompted, in points at a matched spend (Appendix C ). Stop

|

done and ask

|

not done are the percent of decision points at which a policy stops once the required evidence is in and asks while it is not, and asks counts questions per task. Bold marks each column’s largest value, not a tested difference, and no row leads on all three (intervals in Appendix G ).

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

suite	graphs	nodes	edges	depth histogram	behind an edge
MuSiQue	800	2,660	1,860	0:1,196 1:800 2:530 3:134	$55.0\%$
StrategyQA	2,290	6,720	4,483	0:3,721 1:2,320 2:615 3:60 4:4	$44.6\%$
2WikiMultiHopQA	12,576	31,120	11,956	0:19,164 1:11,956	$38.4\%$

Appendix

Table 4: The three need-graph sets, every depth recomputed from the stored edges.

quantity	definition	reads
required-evidence coverage	$\|V\cap S\|\,/\,\|V\|$	evidence recovered
breadth	$\|\{f\in\mathcal{F}:f\cap S\neq\emptyset\}\|$	independent lines touched
depth-weighted recall	$\sum_{d}w_{d}\,C_{d}\,/\,\sum_{d}w_{d}$ , over depths present	coverage per level, weighted deep
coverage at depth $k$	$\|\{v:d_{v}=k,\,v\in S\}\|\,/\,\|\{v:d_{v}=k\}\|$	one level on its own
deepest need resolved	$\max\{d_{v}:v\in V\cap S\}$	the level reached at all
out-of-order rate	$\|\{v\in S:\exists u\in\mathrm{pa}(v),\;t_{u}>t_{v}\}\|\,/\,\|V\cap S\|$	taken out of order

Appendix

Table 5: Everything this paper measures, read from a finished run against its need graph. Breadth is a count, since tasks differ in how many lines they have, and a line’s facet holds only its needs below the surface, so breadth counts the lines in which a run resolves a need beyond the line’s first. Depth-weighted recall averages coverage at each depth the task has,

C_{d}

, with weights

w_{d}

0.5

1.0

and

1.5

at depths

0

1

and

2

and

2.0

at depths

3

5

, set in the scorer. A depth beyond five carries no weight, which matters only on the two deeper constructed corpora, whose chains run to six and eight. The stopping pair is read at every decision point, excluding states the budget forced.

\tau

is the tokens spent.

level	name	the needs it pursues	read by
0	stated	named in the request	coverage of stated needs
1	horizontal	at depth zero, unstated	breadth, not separable
2	vertical, one step	after one prerequisite	depth-weighted recall
3	vertical, deep	behind two or more prerequisites	deepest need resolved
4	open	in no annotated graph	not scored by a graph
	order	taken before its prerequisite	out-of-order rate (lower better)

Appendix

Table 6: The content axis of proactivity as a ladder, each level defined on the need graph. Order and stopping complete the axis: a policy that reaches deep needs by luck takes them out of prerequisite order, and knowing that the required evidence is in hand is the content-axis counterpart of staying silent. Levels 1 to 3 sit one above the depth of the needs they pursue, since level 0 is the request.

suite	unit	allowance	comparator	comparator	$\Delta$ coverage	95% interval
		granted	spend	questions	under budget
MuSiQue	question tokens	91.8	65.5	4.261	$+0.1487$	$[+0.1130,+0.1949]$
MuSiQue	question words	74.3	53.3	4.250	$+0.1534$	$[+0.1174,+0.2019]$
StrategyQA	question tokens	62.2	42.0	2.974	$+0.0466$	$[+0.0255,+0.0692]$
StrategyQA	question words	48.2	33.2	2.881	$+0.0515$	$[+0.0295,+0.0766]$

Appendix

Table 7: Coverage when the comparator is charged the questioner’s own question budget, on development. The trained questioner averages seeds 1 and 2 within each task. Intervals are the task-clustered bootstrap at

1{,}000

resamples.

arm	fitted from	training records	what it varies
imitation reference	base model	$14{,}302$ decisions	nothing (the comparison point)
stop-weighted imitation	base model	$14{,}302$ decisions	stop rows down-weighted to $0.1$
question pairs	imitation reference	question pairs of a $17{,}430$ -pair export	preference on question pairs
stop contrasts only	imitation reference	$18{,}919$ pairs, one kind	preference on stop contrasts only
both kinds, from reference	imitation reference	$31{,}473$ pairs, three kinds	preference on both kinds at once
both kinds, from pairs	question pairs	$31{,}473$ pairs, three kinds	the same set, from a trained start

Appendix

Table 8: The arms. The reference is an imitation adapter fitted to the

14{,}302

recorded decisions of the same model, prompted, in the fixed-cohort export, two epochs at learning rate

10^{-4}

, every other arm varies one thing against it, and the trained questioner is the last row. Its combined set holds

11{,}305

question pairs,

1{,}249

recorded stop contrasts and

18{,}919

synthesized ones,

31{,}473

of the

36{,}349

pairs in its preference file, the

4{,}876

decided by a finished state being left out of training. By source they are

19{,}448

MuSiQue,

9{,}786

StrategyQA and

2{,}232

2WikiMultiHopQA pairs and

7

synthesized stop contrasts from a small synthetic fixture. The stop-contrast-only set is those

18{,}919

synthesized contrasts alone, and the question-pair arm’s

17{,}430

-pair export is the fixed-cohort preference export. Every adapter uses rank

32

, scaling

64

, dropout

0.05

and no bias term, over the same seven projection modules. The preference arms train for one epoch at learning rate

5\times 10^{-6}

with a divergence weight of

0.1

supervision	MuSiQue	StrategyQA	2WikiMultiHopQA
imitation reference	$+0.0858$	$+0.0271$	$+0.0587$
filtered demonstrations	$[+0.0492,+0.1231]$	$[-0.0070,+0.0631]$	$[+0.0299,+0.0938]$
stop-weighted imitation	$+0.0746$	$+0.0260$	$+0.0594$
the same, stop rows down-weighted	$[+0.0410,+0.1100]$	$[-0.0106,+0.0638]$	$[+0.0325,+0.0925]$
question pairs	$+0.1177$	$+0.0693$	$+0.0719$
question pairs, from the reference	$[+0.0848,+0.1531]$	$[+0.0337,+0.1067]$	$[+0.0456,+0.1025]$

Appendix

Table 9: The same arms on the held-out split, each against the same model, prompted, at gate-matched spend (Appendix C ),

200

tasks per suite under one contrast function. The last row, both pair kinds fitted to the question-pair arm, has the largest point estimate on every suite, about a hundredth above the question-pair arm it starts from, a point ordering and not a tested difference. The two question-pair rows start from different imitation checkpoints, the first from the reference row above and the second from the one fitted to the pooled export, the question-pair row of Table 3 . Intervals are bias-corrected bootstraps at

10{,}000

resamples, with any bound within

0.01

of zero re-read at

50{,}000

across three reseeds.

Equal spend: both arms at the lower of their two question counts
	MuSiQue	StrategyQA	2WikiMultiHopQA
required-evidence coverage	$+11.2$ $[+8.7,+14.3]$	$+7.0$ $[+4.0,+10.3]$	$+5.0$ $[+1.7,+8.4]$
breadth	$+0.120$ $[+0.084,+0.165]$	$+0.050$ $[+0.008,+0.098]$	$+0.070$ $[+0.015,+0.128]$
depth-weighted recall	$+12.5$ $[+9.5,+16.1]$	$+5.4$ $[+2.6,+8.4]$	$+5.5$ $[+2.0,+9.3]$
deepest need resolved	$+0.218$ $[+0.151,+0.306]$	$+0.098$ $[+0.035,+0.176]$	$+0.056$ $[+0.001,+0.118]$ ∘
out-of-order rate †	$+5.7$ $[+0.7,+10.6]$	$+0.5$ $[-4.9,+5.3]$ ∘	$-1.2$ $[-5.3,+1.9]$ ∘

Appendix

Table 10: Table 2 as differences: trained minus the named comparator over

200

held-out tasks per suite (

178

83

and

113

on the out-of-order row, where it is defined), with bias-corrected intervals over tasks at

10{,}000

resamples. ∘ marks a cell not decided (Section 4 ) and

\dagger

a row where lower is better. Differences are in percentage points, except breadth (in lines) and the deepest need resolved (in levels).

	suite	seed 1	seed 2	seeds 1+2
required-evidence coverage	MuSiQue	$+0.108$	$+0.116$	$+0.112$
	StrategyQA	$+0.064$	$+0.076$	$+0.070$
	2WikiMultiHopQA	$+0.048$	$+0.051$	$+0.050$
breadth	MuSiQue	$+0.120$	$+0.120$	$+0.120$
	StrategyQA	$+0.043$	$+0.058$	$+0.050$
	2WikiMultiHopQA	$+0.065$	$+0.075$	$+0.070$

Appendix

Table 11: Table 2 ’s cells per training seed, trained minus the same model, prompted, at equal spend,

n=200

tasks per suite on every row but the out-of-order one. The last column averages the two seeds within each task, and its intervals and verdicts are in Table 2 . The out-of-order rate is read only on the pairs where both arms define it, so each seed’s cell is over its own tasks and the averaged cell over every task where either seed defines it, on MuSiQue

175

and

174

tasks against

178

. The averaged cell is therefore not the midpoint of the two seeds’ cells. † Lower is better.

suite	split	$\Delta$	95% interval	$n$ tasks	own-stop $\Delta$
MuSiQue	development	$+0.156$	$[+0.121,+0.204]$	$132$	$+0.103$
MuSiQue	held out	$+0.128$	$[+0.102,+0.159]$	$200$	$+0.096$
StrategyQA	development	$+0.080$	$[+0.059,+0.103]$	$333$	$-0.009$
StrategyQA	held out	$+0.080$	$[+0.051,+0.112]$	$200$	$-0.012$
2WikiMultiHopQA	held out	$+0.079$	$[+0.052,+0.111]$	$200$	$+0.015$

Appendix

Table 12: Required-evidence coverage under the selection gate’s rule, the only rule with a development reading: the trained questioner (two training seeds averaged within the task) at its own stopping point, the same model, prompted, at its own prefix of that length. Table 2 reads the same held-out runs at equal spend. Both suites with a development reading agree with it in sign and in excluding zero, and the MuSiQue gain shrinks from development to held out, as selection on the development split predicts. 2WikiMultiHopQA has no development reading, so its cell is a first reading rather than a replication. The splits were scored under different scoring hashes, so across splits this is an agreement in sign and interval, never a difference of differences. Development intervals are at

1{,}000

resamples, held-out ones at

10{,}000

. The last column is the difference at each policy’s own stop under the ceiling of eight.

criterion	gated	MuSiQue		StrategyQA		2WikiMultiHopQA
		dev	held out	dev	held out	held out
required-evidence coverage	yes	$+0.156$	$+0.128$	$+0.080$	$+0.080$	$+0.079$
question diversity, seed 1	yes	$0.827$	$0.826$	$0.888$	$0.849$	$0.805$
question diversity, seed 2	yes	$0.841$	$0.833$	$0.884$	$0.830$	$0.808$
question length	yes	$17.53$ fail	$17.76$ fail	$22.57$ fail	$22.67$ fail	$12.36$ fail
malformed output	yes	$0.000$	$0.001$	$0.003$	$0.001$	$0.005$

Appendix

Table 13: Every gated criterion for the trained questioner’s two training seeds, development against held out, under one instrument and estimator. Question diversity is printed per training seed, since pooling two policies’ questions measures something else, and read per task and rollout seed. Question length fails on every suite and split and makes every overall verdict a fail. Ungated rows are read at the shared cap of eight, not at equal spend (Appendix C ). Coverage at depth two or more is absent, not zero, on 2WikiMultiHopQA, which has no required need below depth one. Promptness on MuSiQue falls from

+0.142

on development, whose interval at

1{,}000

resamples excludes zero, to

+0.015

held out, whose interval spans zero, so held-out promptness there is not decided and the drop between the splits is not a tested difference.

control	at most the trained questioner’s spend †	shared cap of eight	$n$
answer length	$+0.8848$ $[+0.8660,+0.9020]$	$+0.8848$ $[+0.8660,+0.9020]$	375
compute budget	$+0.8841$ $[+0.8654,+0.9013]$	$+0.8841$ $[+0.8654,+0.9013]$	375
random questions	$+0.5049$ $[+0.4723,+0.5408]$	$+0.3648$ $[+0.3333,+0.4021]$	376
state-blind template	$+0.2439$ $[+0.2177,+0.2718]$	$+0.1892$ $[+0.1614,+0.2171]$	400
depth one ∗	$+0.1448$ $[+0.1242,+0.1668]$	$+0.1380$ $[+0.1176,+0.1596]$	400
evidence blind ∗	$+0.0701$ $[+0.0529,+0.0861]$	$+0.0369$ $[+0.0200,+0.0525]$	400

Appendix

Table 14: The controls against the trained questioner, its two training seeds averaged within each task, pooled over the held-out splits of MuSiQue and StrategyQA, as required-evidence coverage of the trained questioner minus the control. The first column charges each control at most the trained questioner’s spend, and the second reads both arms at the shared cap of eight retrieval calls. The three ablations of the questioner’s own weights are read for each seed against its own ablation. Each control takes its own overlap with the trained questioner as its population, which is why the counts differ. † The control is read at the longest prefix of its own run that stays within the trained questioner’s retrieval calls on the same task and rollout seed, and the trained questioner is read whole. On a pair where the control asks at least as many questions this is equal spend, both arms at the lower count. On a pair where the control asks fewer, the trained questioner keeps its extra questions, the gate-matched spend of Appendix C . ∗ The depth-one and evidence-blind ablations ask fewer questions than the trained questioner, so their first column is not an equal-spend reading, and Figure 7 reads them with both arms at the lower question count.

arm	words	comparator	relative 95% CI	StrategyQA	MuSiQue
imitation reference	13.80	12.99	$[+0.031,+0.093]$	pass	pass
stop contrasts alone	16.59	12.99	$[+0.256,+0.297]$	fail	pass
both kinds	14.31	12.99	$[+0.072,+0.129]$	fail	pass
both kinds, from the question-pair arm	22.57	12.99	$[+0.705,+0.768]$	fail	fail
stop rows down-weighted to $0.1$	14.36	12.99	$[+0.079,+0.134]$	fail	pass

Appendix

Table 15: The gated one-sided length non-inferiority test on StrategyQA, with the MuSiQue verdict beside it: mean question words against the same model, prompted, where the upper bound of the relative interval may not exceed a margin of

0.10

. All four trained arms fail on StrategyQA and the one from the question-pair arm fails MuSiQue as well. The reference passes by

0.007

of margin, so the criterion decides more than its precision can bear.

composition	pairs	persist.	stop	$\Delta$ cov.	95% CI	div.	asks
MuSiQue , 132 tasks
imitation reference	none	0.868	0.951	$+0.049$	$[+0.002,+0.097]$	0.830	3.05
stop contrasts alone	18,919	0.996	0.101	$-0.005$	$[-0.056,+0.046]$	0.447	6.95
both kinds	31,473	0.890	0.926	$+0.068$	$[+0.021,+0.117]$	0.801	3.36
both kinds, from the question-pair arm	31,473	0.944	0.427	$+0.156$	$[+0.121,+0.204]$	0.834	4.24
StrategyQA , 333 tasks

Appendix

Table 16: Training composition against behavior on the development split. Persistence is the rate of asking when the task is not done and stop the rate of stopping when it is, per decision point the policy itself reaches. Coverage is against the same model, prompted, at gate-matched spend, with a task-clustered interval. Diversity is the within-task distinct-trigram share, against a gate floor of

0.65

and untrained values of

0.380

and

0.458

. Asks is questions per episode. Every arm but the last of each block is one training run, and the last averages two, scored in a separate pass against the same comparator runs.

suite	node	implied	observed	detectable	stronger reader
MuSiQue	$+0.097$	$+0.043$	$+0.026$ $[-0.006,+0.060]$	$0.047$	$+0.026$ $[-0.015,+0.067]$
StrategyQA	$-0.005$	$-0.001$	$-0.003$	$0.056$	$-0.001$ $[-0.053,+0.049]$
2WikiMultiHopQA	$+0.001$	$+0.001$	$+0.031$	$0.053$	$+0.007$ $[-0.018,+0.035]$

Appendix

Table 17: The answer gain the trained questioner’s evidence would imply, the gain observed, and a stronger reader. The node column is the difference in covering the answer-bearing need. Answer token overlap, the trained questioner minus the same model, prompted,

200

held-out tasks per suite,

10{,}000

-resample intervals. The minimum detectable effect is at

80\%

power and two-sided

0.05

from the observed paired spread. The last column is the declared primary, the difference between that difference read under a Claude Opus 5 evidence-only reader and read under the frozen answerer, at each arm’s own stop. Intervals here cluster on the MuSiQue question template, and the point moves with them: the estimate is the mean over clusters, so here each template weighs the same whatever its number of tasks, where Table 2 , clustered on the task, weighs each task the same. That is why the observed MuSiQue cell reads

+0.026

here and

+0.024

there for the same runs.

		Task success			Follow-up turns	Hit rate	Recall
domain	prompt	Seed 1	Seed 2	Pooled	Pooled	Pooled	Pooled
retail	base	$+0.213$	$+0.193$	$+0.203$	$-0.660$	$+0.254$	$+0.512$
				$[+0.100,+0.357]$	$[-1.853,+0.387]$
retail	stop	$+0.233$	$+0.167$	$+0.200$	$-0.737$	$+0.265$	$+0.504$
				$[+0.100,+0.340]$	$[-2.343,+0.870]$
airline	base	$+0.059$	$+0.078$	$+0.069$	$-0.059$	$+0.185$	$+0.368$

Appendix

Table 18: The trained questioner against the same model, prompted, on the tool-use benchmark, at equal, separately enforced budgets. Differences are trained minus prompted, over tasks, with bias-corrected intervals at

10{,}000

resamples.

domain	prompt	retrieved a record	exact repeat	near-exact repeat
retail	base	$+0.582$	$-0.613$	$-0.205$
		$[+0.447,+0.693]$	$[-0.675,-0.527]$	$[-0.288,-0.126]$
retail	stop	$+0.587$	$-0.609$	$-0.237$
		$[+0.450,+0.701]$	$[-0.671,-0.522]$	$[-0.313,-0.163]$
airline	base	$+0.382$	$-0.516$	$+0.076$
		$[+0.258,+0.512]$	$[-0.605,-0.427]$	$[-0.065,+0.249]$

Appendix

Table 19: Exploratory question measures on the tool-use benchmark, declared after one interim look and so exploratory whatever they show. Differences are trained minus prompted, over tasks (

25

retail,

17

airline), with the two training seeds pooled within each task and bias-corrected intervals at

10{,}000

resamples, not read at

50{,}000

. A near-exact repeat shares at least four content words with an earlier question in the same dialogue.

domain	prompt	Follow-up turns	Task success	Hit rate	Questions
retail	base	$\mathbf{-1.60}$ $[-3.05,-0.19]$	$\mathbf{+0.170}$ $[+0.070,+0.310]$	$+0.099$	$-0.87$
retail	stop	$\mathbf{-1.75}$ $[-3.18,-0.33]$	$\mathbf{+0.127}$ $[+0.043,+0.240]$	$+0.130$	$+0.19$
airline	base	$+0.75$ $[-0.39,+1.75]$	$0.000$ $[-0.039,+0.049]$	$+0.052$	$+1.09$
airline	stop	$-0.07$ $[-1.42,+1.21]$	$+0.034$ $[-0.005,+0.147]$	$+0.067$	$+1.40$

Appendix

Table 20: The trained questioner against GPT-OSS-120B, prompted, on the tool-use benchmark at equal, separately enforced budgets. Differences are trained minus prompted, over tasks, with the two training seeds pooled within each task, and 95% intervals. A negative difference in follow-up turns means fewer turns from the simulated user. Bold: decided, every interval from

50{,}000

resamples at three bootstrap seeds excludes zero. Hit rate and questions are secondary point differences.

comparison	raw	Cohen’s $\kappa$ [95% clustered]	$n$ / clusters
the two retained members, both panels	77.8%	0.555 [0.481, 0.624]	2,508 / 201
first retained with replaced	65.2%	0.304 [0.244, 0.364]	3,156 / 211
second retained with replaced	58.2%	0.163 [0.053, 0.259]	2,648 / 202
first retained with replacement	79.0%	0.581 [0.522, 0.638]	2,596 / 202
second retained with replacement	84.1%	0.682 [0.629, 0.734]	2,223 / 197
rule with first retained	61.5%	0.230 [0.167, 0.294]	3,480 / 215

Appendix

Table 21: chance-corrected agreement on the axis that decides a pair, slot coding, sided votes

question asked of the rater	items	raters agree	chance-corrected
does candidate A reach an unstated need	$98$	$87.8\%$	$+0.756$
does candidate B reach an unstated need	$92$	$88.0\%$	$+0.761$
which candidate is the better move	$56$	$67.9\%$	$+0.241$

Appendix

Table 22: Agreement between the two people on the human anchor’s items, per question asked of them, with chance-corrected agreement.

	MuSiQue	StrategyQA	2WikiMultiHopQA
prerequisite edges	$466$	$383$	$191$
bare task question	$0.35$	$0.72$	$0.48$
another task’s question (chance)	$0.28$	$0.24$	$0.60$
queries without the parent’s answer	$0.70$	$0.72$	$0.71$
queries with the parent’s answer	$0.83$	$0.72$	$0.86$
queries with a random entity instead	$0.70$	$0.74$	$0.69$

Appendix

Table 23: Whether a prerequisite edge gates retrieval: per-query rates at which a query retrieves the child’s evidence in the top five, over testable edges, the better of two prompts. StrategyQA parents carry no answer, so their evidence title stands in for it.

base	training seeds	MuSiQue ( $n=132$ )	StrategyQA ( $n=333$ )
$1.7$ B	$3$	$+0.1098$ $(+0.1016,+0.1187)$	$+0.0309$ $(+0.0273,+0.0349)$
$4$ B	$3$	$+0.0434$ $(+0.0215,+0.0549)$	$+0.0640$ $(+0.0611,+0.0691)$
$8$ B	$1$	$+0.0492$	$+0.0804$

Appendix

Table 24: The matched-recipe ladder: each cell a paired coverage difference against the same model, prompted, at gate-matched spend, on the development tasks. Three-seed rows give the mean across training seeds with the observed range in parentheses, and the largest size is one training run.

run	all pairs	length matched
32B, final checkpoint	$+0.031$ $[+0.001,+0.063]$	$+0.009$ $[-0.029,+0.044]$
32B, lowest imitation loss	$+0.016$ $[0.000,+0.034]$	$+0.003$ $[-0.023,+0.029]$
8B, same recipe	$+0.043$ $[+0.012,+0.074]$	$+0.059$ $[+0.017,+0.096]$

Appendix

Table 25: Preference training at two scales, read offline on the development question pairs as the gain in ranking accuracy over each run’s own starting checkpoint, with 95% intervals over tasks. The 32-billion-parameter final checkpoint is reported first by a rule fixed before the reading, because imitation loss rose at every checkpoint after the first. Length matched keeps the pairs whose two questions are within ten tokens of each other in length.

	seed 1	seed 2	seed 3	seed 0, retrained	four runs pooled
MuSiQue	$+0.108$	$+0.116$	$+0.118$	$+0.109$	$+0.113$
	$[+0.081,+0.141]$	$[+0.090,+0.148]$	$[+0.092,+0.148]$	$[+0.082,+0.139]$	$[+0.088,+0.142]$
StrategyQA	$+0.064$	$+0.076$	$+0.077$	$+0.064$	$+0.070$
	$[+0.032,+0.099]$	$[+0.045,+0.110]$	$[+0.046,+0.111]$	$[+0.031,+0.099]$	$[+0.041,+0.102]$
2WikiMultiHopQA	$+0.048$	$+0.051$	$+0.057$	$+0.055$	$+0.053$
	$[+0.015,+0.084]$	$[+0.019,+0.085]$	$[+0.026,+0.091]$	$[+0.021,+0.093]$	$[+0.021,+0.087]$

Appendix

Table 26: Coverage at equal spend for the four complete runs of the final stage, each against the same model, prompted, with both arms at the lower of their two question counts,

200

held-out tasks per suite. Seeds 1 and 2 are the runs Table 2 averages, and seed 3 and the retrained seed 0 were trained from the start afterwards. Beneath each point is its 95% interval at

10{,}000

resamples. The pooled column averages the four runs within each task. The runs differ by a between-run standard deviation of

0.005

0.007

and

0.004

on the three suites, against a mean within-run bootstrap standard error of

0.015

0.017

and

0.018

Explore similar work

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents
May 27, 2026Bin Wu, Guanyun Zou, Bingbing Wang +2Large Language Model Agents Proactive
Beijing University of Posts and Telecommunications · Nanjing University of Aeronautics and Astronautics · Harbin Institute of Technology (Shenzhen) +1
Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluation
Sep 3, 2026Yan Tang, Tingyu Cao, Yuanbo Tang +2Proactive Large Language Model Agents
UniCone Team
Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gym
Sep 29, 2026Jio Oh, Seunghyun Do, Young-Jun Lee +2Large Language Model Agents User Simulation
KAIST · University of Minnesota

May 27, 2026, cs.CLJ/K move · Enter open · S save

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents

Bin Wu, Guanyun Zou, Bingbing Wang, Huan Zhao+1

Beijing University of Posts and Telecommunications · Nanjing University of Aeronautics and Astronautics · Harbin Institute of Technology (Shenzhen)+1

Large Language Model Agents Proactive Factual Recall Openclaw

Open paper AskPDF arXiv

A long-lived LLM agent, such as OpenClaw, earns its value by acting on a user's preferences and constraints across sessions, not just the current request. Yet today's agents keep what a user volunteers but rarely ask for what stays unspoken, leaving a proactivity gap in long-lived LLM agents: an agent cannot act on a preference it never obtained. As users delegate more of their affairs to agents, the impact of this gap grows. We isolate one concrete, controllable slice of this gap as Ask-to-Remember (ATR): the agent decides whether to ask now for a reusable user preference that the current task does not need but a later session with the same user will. ATR is hard even to evaluate: the right question is underdetermined and its payoff deferred to tasks that may never arise. ATRBench, to the best of our knowledge the first ATR benchmark, makes it measurable by fixing each user's preferences as hidden ground truth, so success demands asking, not recall. Across eight frontier LLM agents, defaults fall at least 62 points below an oracle handed the relevant preference, and prompting closes little of it. Diagnostics identify acquisition as the bottleneck. ATRBench surfaces this proactivity gap in current agents and offers a diagnostic testbed for closing it.

Abstract

Explore similar work

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents

Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluation

Foundations of Proactive Agents: Principles, Technical Layers, and Proactivity-Gym