From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings
Authors: Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Xuan Jiang, Haris N. Koutsopoulos, Jinhua Zhao, Yuan Zhou
Organizations: Massachusetts Institute of Technology · Tsinghua University · University of California, San Diego · Tongji University · Peking University · McGill University · Northeastern University
LLM agents are increasingly asked to make plans for people, such as scheduling drivers or assigning nurses. However, people's requests are often incomplete, and the supporting information for decision-making may be outdated, requiring agents to ask for clarification. Existing benchmarks typically provide fully specified problems or rely on uncalibrated LLM-simulated users, leaving this ability insufficiently tested. We introduce Ask2Decide, a benchmark of 43 planning tasks that evaluates whether agents ask for missing information that matters for their decisions. An operator holds facts absent from the agent's request and records, and optimization solvers determine which facts change the optimal plan. We also include similarly worded facts that leave the optimal plan unchanged to test whether agents distinguish useful information from irrelevant details. We collect 79 interaction sessions from 19 participants and find that, after an incorrect plan, human replies omit the correcting fact far more often than replies from an uncalibrated simulated user (43% versus 7%). Using these interactions, we calibrate a simulated operator that more closely matches human reply patterns. Evaluating four frontier LLMs, we find that their initial plans are infeasible on 58-98% of tasks when information is missing, compared with 5-28% when all facts are provided upfront. Agents uncover facts that change the optimal plan only slightly more often than facts that leave it unchanged (39-52% versus 32-41%). These results show that obtaining decision-relevant information through interaction remains a major bottleneck for LLM planning. This is ongoing work and will be updated. The date of this version is shown on the first page.
Figures & tables
Default LLM operator
Human operators
Initial
Final
Initial
Final
Interactive
48.5%
6.1%
32.9%
7.6%
Full brief
3.0%
0.0%
6.3%
3.8%
Table 1: Share of infeasible plans with a default LLM operator and with human operators (22 tasks, planning agent: DeepSeek V4.1 Flash).
Figure 1: Overview of Ask2Decide on hard1_dialaride . (a) The agent gets a brief and the records, and the operator holds today’s notes, each labelled by the solver with its regret Δ . The twin H5 has the same form as H2 but leaves the best plan unchanged. (b) The operator answers only questions about a specific object, and the agent’s initial plan is scored on the true problem. Replies are constructed for illustration.
Figure 2: How a task is built (dial-a-ride example). Steps 1–2 are written by the task author, or by a script for the frontier splits. Steps 3–4 are run by the solver. Below each step: the design criterion it secures.
Category
Example from the task suite
Basic information
A standing rule staff know but the records omit: “our vans hold 60 crates at most” (S#; constraint).
Sudden accident
“One truck broke down this morning, so we can run 3 truck trips at most today” (value); “the road from warehouse 1 to warehouse 4 is closed today” (availability).
New information
“The hotel chain ordered 600 kg of Value candy” (constraint); “CA1502 picked up a tailwind, now expected at 08:15” (objective).
System update
“The grid company has new ramping rules: at most 20 MW an hour” (constraint); a price change this week (objective).
Human error
A record nobody updated: “C07’s certificate for routine nursing checks has lapsed” while the records still list C07 for them (availability).
Table 2: The five categories of hidden information. Parentheses give each example’s effect on the model.
Split
Tasks
Items
Rel.
Twins
Max Δcond
Easy
8
32
16
16
0.235
Medium
8
37
21
16
0.667
Hard
8
55
29
17
0.484
Frontier-A
17
101
59
42
0.836
Frontier-B
2
10
6
4
0.058 †
All
43
235
131
95
0.836
Table 3: The 43 tasks by split. Rel.: hidden items with Δcond>0 . † Upper end of a certified interval.
Figure 3: Study design. (a) People play the operator against a solver-equipped planning agent; the agent asks and plans, the person answers and accepts or rejects plans, and every exchange is logged. (b) The logs calibrate CalOp , which takes the person’s place in the same loop, so that any LLM can be evaluated without new human sessions. Orange: LLM components; teal: people and data; blue: solver.
Humans
DefaultOp
CalOp
(a) Does it answer like a person? (321 replies)
Distance to humans ↓
–
0.395
0.213
Characters per reply
24.2
35.1
25.3
Notes disclosed per reply
1.36
1.24
1.41
Withholds fact after wrong plan
0.43
0.07
0.48
(b) Does it answer correctly?
Table 4: Can a simulated operator stand in for people? Each block answers one question. A good operator is close to the Humans column (79 sessions).
(a) Initial plan
(b) Final plan
(c) All-info
Model
#Q
Loss ↓
Infeas. ↓
Opt. ↑
Loss ↓
Infeas. ↓
Opt. ↑
All-info loss ↓
Asking gap
GPT-6.1 Sol
8.2
0.603
58%
28%
0.424
42%
49%
0.029
0.574
Gemini 3.8 Flash
3.9
0.797
79%
16%
0.451
44%
42%
0.058
0.739
DeepSeek V4.1 Flash
5.9
0.819
81%
16%
0.447
44%
51%
0.258
0.561
GLM-5.3 Flash
0.6
0.979
98%
0%
0.623
58%
19%
0.173
0.807
Table 5: Main results on all 43 tasks with CalOp . (a) The benchmark score: the initial plan in the interactive condition. (b) The final plan after up to two revisions. (c) The same model’s loss when told every hidden item, and the asking gap, (a) minus (c). #Q: questions per episode; Opt.: share of optimal plans.
Model
Easy
Medium
Hard
Front.-A
Front.-B
GPT-6.1 Sol
.18/.00
.27/.00
.67/.00
.88/.07
1.0/.01
Gemini 3.8 Flash
.53/.00
.63/.00
.88/.13
.94/.09
1.0/.01
DeepSeek V4.1 Flash
.40/.00
.63/.00
1.0/.13
1.0/.59
1.0/.01
GLM-5.3 Flash
.89/.00
1.0/.01
1.0/.02
1.0/.42
1.0/.01
Final-plan loss, after feedback
GPT-6.1 Sol
.01
.13
.14
.82
1.0
Table 6: By split. Top: initial-plan loss, interactive / all-info. Middle: final-plan loss after feedback. Bottom: questions per episode.
Disclosed
Asked about
Model
Relevant
Twins
Relevant
Twins
GPT-6.1 Sol
52%
41%
39%
27%
Gemini 3.8 Flash
39%
32%
36%
31%
DeepSeek V4.1 Flash
43%
38%
36%
29%
GLM-5.3 Flash
3%
0%
2%
0%
Frontier-A only (17 tasks)
Table 7: Relevant recall and twin rate before the initial plan. Disclosed: the operator stated the item; asked about: a question targeted it.
Initial plan
Final plan
Condition
Operator
#Q
Infeasible
Loss
Infeasible
Optimal
Loss
Interactive
DefaultOp
4.67
14/24
0.594
3/24
17/24
0.136
Interactive
CalOp
4.33
16/24
0.676
4/24
20/24
0.167
Full brief + feedback
DefaultOp
0.08
2/24
0.083
2/24
22/24
0.083
Full brief + feedback
CalOp
0.04
1/24
0.042
1/24
23/24
0.042
Table 8: Interactive and full-brief conditions with each operator (DeepSeek V4.1 Flash, 24 tasks).
Benchmark
Hidden
Human
Plan
Value of
Zero-value
info
data
scored
an item
controls
NL4Opt, OptMATH, OPT-Engine
×
×
✓
–
×
HeuriGym
×
×
✓
–
×
QuestBench
✓
×
×
required var.
×
OR-Clarify
✓
×
×
annotated
×
τ -bench
✓
×
✓
–
×
Table 9: Comparison with related benchmarks.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Split
Items
Cond. rel.
Controls
easy1_dispatch
easy
4
2
2
easy2_laundromat
easy
4
2
2
easy3_family_trip
easy
4
2
2
easy4_calf_feed
easy
4
2
2
easy5_store_delivery
easy
4
2
2
easy6_ad_buy
easy
4
2
2
Appendix
Table 10: Task inventory by split, with counts of hidden items, conditionally relevant items and controls.
Source
Metric
Mean [95% task interval]
Humans
Optimal
.815 [.695, .918]
Feasible
.915 [.832, .980]
Loss
.117 [.047, .201]
DefaultOp
Optimal
.833 [.697, .955]
Feasible
.939 [.864, 1.000]
Loss
.085 [.016, .173]
Appendix
Table 11: Task-balanced historical operator outcomes. Simulated runs use seeds 0–2; the additional default seed 3 is excluded to keep the comparison matched. Intervals are descriptive, not equivalence tests.