When Should AI Agents Ask? Clarification Requests Under Limited Human Attention

Hanzhao Wang; Shengjie Liu; Zhengchao Wang

cs.AIOct 30, 2026

When Should AI Agents Ask? Clarification Requests Under Limited Human Attention

Authors: Hanzhao Wang, Shengjie Liu, Zhengchao Wang

Organizations: The University of Sydney Business School, The University of Sydney · Amazon

Abstract

AI agents increasingly carry out delegated tasks in parallel, with one person overseeing several of them at once. In practice, an agent often lacks information its task requires, such as an unstated preference, constraint, or specification. It can ask the human to clarify, or it can search for the information itself, for example in past records, which takes time and effort and may fail. In this paper, we study how to coordinate these decisions across several agents that share one human with limited attention. We build a stylized model in which each agent can search, ask, or continue the parts of its task that do not depend on the missing information, and we formulate the problem as a dynamic program. We first show that whether an agent should ask depends on more than its own uncertainty: an agent with other work to do can be better off asking earlier, and which question the human should answer first depends on all waiting questions, so no ranking computed agent by agent is optimal. We then propose the Pairwise Coordination Policy (PCP), which uses exact one- and two-task problems to estimate how pairs of tasks delay each other when they share the human. Theoretically, PCP is exactly optimal when at most two requirements remain unresolved and costs at most twice the optimum when every unresolved requirement must be resolved by the human. We evaluate PCP on 864 synthetic cases and in a case study with LLM agents. On the synthetic benchmark, PCP's mean gap to optimal cost is 0.07%, and it reduces cost by an average of 18-31% relative to three standard rules for when to ask. In the LLM case study, agents completed every task correctly, and PCP achieved the lowest mean cost in all tested settings.

Figures & tables

Panel of Figure 1 — Figure 1: Two Codex coding agents that need input from the same person at the same time.

Panel of Figure 2 — Figure 2: Status $z_{i}$ of task $i$ ’s requirement. Each search succeeds with probability $q_{i}(k_{i})$ .

Panel of Figure 3 — Figure 3: Regions of Theorem 2 and Proposition 1 for $w=\kappa=\eta=1$ and $c=0.1$ . In both, an agent with one work unit and withdrawable questions asks immediately, whereas the same agent without work (above $\theta_{0}$ ) or without withdrawal (above $\theta_{\rm nw}$ ) would search first. Both regions also require $p>c/\eta=0.1$ , which binds only for the withdrawal region when $a<0.1$ . The withdrawal region closes at $a=1$ , where $\theta_{\rm nw}=\theta_{1}$ .

Policy	Mean	P95	Worst	Opt.	Saving
PCP	0.07	0.32	1.87	56
PCP, coordinate search	0.07	0.35	1.87	56
PCP, no-submission ties	0.52	4.37	15.2	47	0.43
Singleton	1.24	5.78	21.0	38	1.10
Always Ask	30.9	119.2	236.2	19	18.2
Random-order Ask	58.8	188.2	330.9	9	30.6

Table 1: Performance on 864 portfolios. Mean, 95th percentile (P95), and largest (Worst) gap to the optimal cost,

100(J-V^{*})/V^{*}

(%). Opt. is the share of portfolios in which the policy’s cost equals the optimal cost up to a relative difference of

10^{-8}

(%). Saving is PCP’s mean saving relative to the policy,

100(J-J^{\rm PCP})/J

, averaged over portfolios (%).

Figure 4: Mean saving (%) of PCP relative to the singleton policy for portfolios of four tasks, by the amount of independent work and (a) the search cost or (b) the availability of the human. Each cell averages 36 portfolios.

Figure 5: Larger portfolios (

\alpha=0.8

c=0.5

, 200 common sample paths per instance). (a) PCP’s mean saving relative to the singleton policy for each instance (thin lines, colored by the amount of work) and the mean over the eight instances of each size (thick line). (b) PCP’s saving relative to each policy, averaged over the instances of each size.

		Mean cost				PCP’s saving (%) relative to
$a$	$c$	PCP	Singleton	Always Ask	Search Once	Singleton	Always Ask	Search Once
1	0.1	12.6 (1.1)	12.9 (0.9)	13.4 (0.0)	20.1 (4.6)	2.0 (7.2)	5.8 (7.9)	34.8 (12.1)
1	1.5	13.4 (0.0)	13.4 (0.0)	13.4 (0.0)	25.3 (4.2)	0.0 (0.0)	0.0 (0.0)	45.6 (9.7)
0.25	0.1	31.5 (20.4)	33.1 (20.8)	42.1 (16.8)	44.5 (22.5)	2.5 (14.7)	27.2 (24.5)	23.8 (29.0)
0.25	1.5	40.3 (25.3)	46.6 (23.3)	46.6 (23.3)	45.4 (22.1)	11.2 (27.2)	11.2 (27.2)	13.9 (20.7)

Table 2: Case study with LLM agents (four tasks, ten portfolios per setting). Mean cost, with the lowest in each setting in bold, and PCP’s mean saving (%) relative to each benchmark, with standard deviations across portfolios in parentheses. Search Once denotes Search Once Then Ask.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Section	Instances	$n$	$b$	$a$	$p$	$\alpha$	$c$
Benchmark	6.1	864	3, 4	0, 1, 2, 4	0.25, 0.5, 1	0.3, 0.7	0.8, 1	0.1, 0.5, 1.5
Human-resolution regime	6.1.2	180	3, 4, 5	0, 1, 2, 3, 4, 6	0.5, 1	0	–	–
Homogeneous family a	6.1.2	all $(n,b)$	2–30	$0,\ldots,1.5n$	1	0	–	–
Larger portfolios	6.1.3	56	4–48 (e)	0, 2	0.5, 1	0.3, 0.7	0.8	0.5
Misestimated inputs	6.1.4	144	four-task benchmark portfolios with the first weight draw
Mechanisms, one task b	C.6	1,296	1	0, 1, 2, 3	$0.1,\ldots,0.9$	$0.1,\ldots,0.9$	0.5, 0.65, 0.8, 1	0.1

Appendix

Table C.1: Instances of the synthetic experiments. Unless noted otherwise, all tasks in a portfolio share

b

p

\alpha

, and

c

\kappa_{i}=0.1

\eta=0.5

, the weights are drawn independently and uniformly from

\{0.5,1,2,4\}

, and all tasks start unsubmitted with no failed searches. The benchmark has three weight draws for each combination of the factors, and the human-resolution regime and the mechanism portfolios have five. (a)

w_{i}=1

and

\kappa_{i}=\eta=0

. (b)

w=\kappa=\eta=1

, with 81 combinations of

a

and

p

for each

\alpha

and

b

. (c) The weights are fixed across

b

. (d) Each task draws its own

w_{i}

p_{i}

c_{i}

, and

\kappa_{i}\in\{0.05,0.1\}

, and

\eta\in\{0.05,0.1\}

is common. (e)

n\in\{4,8,12,16,24,32,48\}

Purchase	Revision	Status	Issuer	Text
other	3	approved	buyer	Approved requirement: the maximum permitted lead time is 7 calendar days.
this	3	informational	operations	The team would prefer delivery near the end of the planning week. No numeric calendar-day limit is authorized in this note.
this	2	approved	buyer	Superseded requirement: offers must arrive within 5 calendar days. See revision 3 for the current decision.
this	3	draft	buyer	Proposal under discussion: set a ceiling of 18 calendar days. This proposal is not approved.
this	3	approved	supplier	Supplier recommendation: allow 13 calendar days. This is our delivery recommendation, not buyer authorization.
this	3	approved	buyer	For this purchase, accept lead times of no more than 10 calendar days. An internal 11-day planning estimate is not the binding requirement.

Appendix

Table C.2: A search packet from the run in which the search succeeds. The binding document is the last one.

$a$	$c$	$V^{*}$	PCP	Singleton	Always Ask	Search Once Then Ask	Saving
1	0.1	12.89	12.89	12.89	13.40	20.73	0.0
1	1.5	13.40	13.40	13.40	13.40	26.33	0.0
0.25	0.1	30.63	30.68	30.97	42.65	40.25	1.0
0.25	1.5	37.12	37.25	42.65	42.65	45.85	12.7

Appendix

Table C.3: Expected cost under the model in the LLM case study, and PCP’s saving (%) relative to the singleton policy computed from these expected costs.

$a$	$c$	PCP	Singleton	Always Ask	Search Once Then Ask
1	0.1	0.000	0.000	0.128	1.351
1	1.5	0.000	0.000	0.000	2.268
0.25	0.1	0.004	0.060	2.626	0.921
0.25	1.5	0.011	1.496	1.693	0.598

Appendix

Table C.4: Average expected excess cost per decision in the LLM case study.

Figure C.1 : The mechanisms of Section 4 beyond the settings of the theorems. Single tasks have

w=\kappa=\eta=1

c=0.1

, and 81 combinations of

a

and

p

. (a) Share of settings in which one more unit of work raises the optimal expected number of human answers. The curve for

\alpha=0.5

coincides with that for

\alpha=0.65

. (b) Share of settings in which withdrawal lowers the optimal cost and raises the expected number of answers. The dashed curves show the 90 portfolios of four tasks that share one human, with

\alpha=1

Level	Saving	PCP gap	Singleton gap	$\Delta$ answers	$\Delta$ searches
$n=3$	0.83	0.039	0.91	-0.09	0.26
$n=4$	1.38	0.091	1.57	-0.19	0.58
$b=0$	1.23	0.002	1.35	-0.15	0.45
$b=1$	1.34	0.046	1.45	-0.17	0.48
$b=2$	1.49	0.100	1.69	-0.13	0.43
$b=4$	0.35	0.113	0.47	-0.11	0.33

Appendix

Table C.5: Benchmark results by factor: PCP’s mean saving relative to the singleton policy (%), mean gaps to the optimal cost (%), and the mean difference between PCP and the singleton policy in the expected numbers of answers and searches.

	PCP		Singleton
Inputs used by the policy	Mean	Worst	Mean	Worst
Correct inputs	0.10	1.76	1.51	14.19
$p$ halved	0.64	5.79	4.02	33.04
$p$ multiplied by 1.5	0.67	9.29	0.99	12.82
$\alpha$ multiplied by 0.75	0.70	5.57	2.44	19.08
$a$ decreased by 0.2	0.74	10.26	0.53	5.79

Appendix

Table C.6: Policies built with misestimated inputs and evaluated under the true parameters: gap to the true optimal cost (%), 144 portfolios.

Human-AI Interaction

Human-AI interaction studies how people and AI systems communicate, collaborate, make decisions, and shape one another over time.

See all 343 papers

Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?
May 8, 2026Anmol Gulati, Hariom Gupta, Elias Lumer +2LLM Agent Evaluation Long-Horizon Agent Evaluation
PricewaterhouseCoopers U.S.
Ask Less, Ask Better: Allocating User Attention in Proactive LLM Agents with Interruption Budgets
Oct 17, 2026Xuanming Zhang, Amir H. Rezaeian, Miguel Ballesteros +6LLM Agents LLM Agent Evaluation
Columbia University · Oracle AI
Rational Clarification by Assistive Agents via Value-of-Information Reasoning
Sep 29, 2026T. Duy Nguyen-Hien, Yee Whye Teh, Wee Sun Lee +1Value of Information Clarification Question Generation
Department of Computer Science, National University of Singapore · Department of Statistics, University of Oxford · Agency for Science, Technology and Research (A*STAR)

May 8, 2026J/K move · Enter open · S save

Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?

Anmol Gulati, Hariom Gupta, Elias Lumer, Sahil Sen+1

PricewaterhouseCoopers U.S.

LLM Agent Evaluation Long-Horizon Agent Evaluation LLM Agents

Open paper AskPDF arXiv

Long-horizon AI agents execute complex workflows spanning hundreds of sequential actions, yet a single wrong assumption early on can cascade into irreversible errors. When instructions are incomplete, the agent must decide not only whether to ask for clarification but when, and no prior work measures how clarification value changes over the course of execution. We introduce a forced-injection framework that provides ground-truth clarifications at controlled points in the agent's trajectory across four information dimensions (goal, input, constraint, context), three agent benchmarks, and four frontier models (three per benchmark; one on a single benchmark only; 84 task variants; 6,000+ runs). Counter to the common intuition that "earlier is always better," we find that the value of clarification depends sharply on what information is missing: goal clarification loses nearly all value after 10% of execution (pass@3 drops from 0.78 to baseline), while input clarification retains value through roughly 50%. Deferring any clarification type past mid-trajectory degrades performance below never asking at all. Cross-model Kendall tau correlations (0.78-0.87 among models sharing identical task coverage; 0.34-0.67 across the full 4-model panel) confirm these timing profiles are substantially task-intrinsic. A complementary study of 300 unscripted sessions reveals that no current frontier model asks within the empirically optimal window, with strategies ranging from over-asking (52% of sessions) to never asking at all. These empirical demand curves provide the quantitative foundation that existing theoretical frameworks require but have lacked, and establish concrete design targets for timing-aware clarification policies. Code and data will be publicly released.

Abstract

Human-AI Interaction

Explore similar work

Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?

Ask Less, Ask Better: Allocating User Attention in Proactive LLM Agents with Interruption Budgets

Rational Clarification by Assistive Agents via Value-of-Information Reasoning