Frontier coding agents solve complex tasks when given complete context but collapse when specifications are incomplete or ambiguous. The bottleneck is not raw capability, but judgment: knowing when to act autonomously and when to ask for help. Current benchmarks are blind to this failure mode. They supply unambiguous detailed instructions and solely reward execution correctness, so an agent that makes a lucky guess for a missing requirement will score identically to one that would have asked to be certain. We present HiL-Bench (Human-in-the-Loop Benchmark) to measure this selective escalation skill. Each task contains human-validated blockers (missing information, ambiguous requests, contradictory information) that surface only through progressive exploration, not upfront inspection. Our core metric, Ask-F1, the harmonic mean of question precision and blocker recall, captures the tension between over-asking and silent guessing; its structure architecturally prevents gaming through question spam. Evaluation across SWE and text-to-SQL domains reveals a large universal judgment gap: no frontier model recovers more than a fraction of its full-information performance when deciding whether to ask. Failure analysis identifies three key help-seeking patterns: overconfident wrong beliefs with no gap detection; high uncertainty detection yet persistent errors; broad, imprecise escalation without self-correction. These consistent patterns confirm poor help-seeking is a model-level flaw, not task-specific. RL training on shaped Ask-F1 reward shows judgment is trainable: a 32B model improves both help-seeking quality and task pass rate, with gains that transfer across domains. The model does not learn domain-specific heuristics for when to ask; it learns to detect unresolvable uncertainty and act on it.
Figures & tables
Figure 1: (A) Models achieve 75–89% pass@3 with complete information but only 4–24% when they must judge when to ask, despite access to ask_human() . Near-zero “No Tool” bars confirm tasks require clarification; the bottleneck is judgment, not capability. (B) Ask-F1’s precision–recall decomposition reveals distinct failure profiles across model families, with no model in the well-judged quadrant. RLVR on shaped Ask-F1 reward shifts a 32B model toward calibrated help-seeking (arrow), confirming judgment is trainable.
Figure 2: Example agent evaluation workflow. Each HiL-Bench task contains multiple blockers that surface as the agent explores the task environment. At each point, the agent must judge whether to ask or proceed. Ask-F1 scores both detection (recall) and targeting (precision).
Task Completion (Pass@3)
Selective Escalation
Model
Full Info
w/ ask_human()
Gap ( Δ )
Recall
Precision
Ask-F1
Text-to-SQL
GPT-5.3-Codex
86.0%
5.3%
-81 pp
14.0%
55.3%
22.3%
GPT-5.4
86.0%
17.3%
-69 pp
22.7%
54.3%
32.1%
Gemini 3.1 Pro
88.7%
35.3%
-53 pp
59.2%
35.6%
44.5%
Claude Opus 4.6
90.7%
39.3%
-51 pp
61.2%
54.3%
57.5%
Table 1: The Judgment Gap: SQL vs. SWE. While models easily solve fully specified tasks, their performance collapses when they must actively seek help ( ask_human() ) to resolve ambiguities. This massive gap is driven by poor selective escalation. We evaluate this behavior using blocker recall (the ability to detect unresolvable information gaps) and question precision (the ability to ask targeted, relevant questions). The Ask-F1 score is the harmonic mean of these two metrics. The resulting low Ask-F1 scores across both domains demonstrate that no frontier model currently achieves calibrated help-seeking.
Figure 3: Failure fingerprints reveal distinct judgment signatures across model families in SQL. (A) Baseline failure modes (within-dimension percentages). GPT models are accuracy-dominant. Claude is completion- and self-assessment-dominant. Gemini shows high logic self-assessment failures. (B) Failure distribution shifts with ask_human() . Gemini inverts dramatically in tool-use ( − 38pp completion, + 44pp accuracy). Claude sees a similar shift to a lesser degree ( − 12pp completion, + 15pp accuracy). GPT rows remain largely unchanged. See Figure 7 for a corresponding SWE analysis and more details in Appendix F .
Figure 4: RLVR closes the judgment gap and transfers across domains. (A) Precision–recall space. Arrows show RLVR shifts base models toward calibrated help-seeking (solid: in-domain; dashed: cross-domain). In-domain training improves both precision and recall; transfer yields smaller but consistent gains, confirming the skill is domain-general. (B) Ask-F1 and Pass@3 improve in lockstep. Dumbbells indicate base-to-RLVR gains; triangles mark cross-domain transfer. Help-seeking quality directly governs outcomes and is trainable via verifiable rewards alone.
Figure 5: The Judgment Matrix. Most agents are in the bottom left box.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Blocker Type
Definition
Examples
Missing Information (42% of blockers)
Required or important values not present in the task context. Agent cannot determine the correct value without external input.
SWE: Deprecation warning required on legacy API access but version string, API call, and message shape are all unspecified; wildcard listeners must materialize as concrete IP keys but the address families to enumerate are unstated. SQL: “Overcrowded” required as a school filter but neither defined nor encoded in the schema; seven similarly labeled rating columns exist but which one represents “overall” rating for filtering is unspecified.
Ambiguous Requests (36% of blockers)
Multiple valid interpretations or implementations exist. Each one leads to a different solution.
SWE: One requirement says defaultCountry updates must sync the UI; another says non-empty updates after mount should be ignored—both are stated. Post-scan metrics must be recorded but it is unclear whether DB model counts or only scan counters are in scope. SQL: “Dominance rating” could refer to several numeric fields (overall score vs. attribute composites); “early days of the platform” could refer to almost any timeframe.
Contradictory Information (22% of blockers)
Conflicting or misleading specifications that cannot both be satisfied. Agent must identify the contradiction and seek authoritative resolution.
SWE: One spec line pushes preserving attribution in referral URLs; another pushes sanitization—both cannot hold without a precedence rule. Expected behavior says show only the strongest CPE confidence per vulnerability; another requirement insists every applicable confidence be preserved. SQL: The request mixes “hydrocarbon” with metals and heteroatoms, but by definition hydrocarbons contain only C and H—the constraints are definitionally contradictory. Patients must be simultaneously middle-aged and geriatric, normally disjoint age bands with no reconciling rule provided.
Appendix
Table 2: Blocker taxonomy: three types of information gaps with definitions and representative examples drawn from the dataset.
SWE
SQL
Total
Tasks
150
150
300
Avg. blockers / task
3.55
3.99
3.77
Total blockers
533
598
1131
Missing information
205 (38.5%)
271 (45.3%)
476 (42.1%)
Ambiguous requests
161 (30.2%)
247 (41.3%)
408 (36.1%)
Contradictory information
167 (31.3%)
80 (13.4%)
247 (21.8%)
Appendix
Table 3: Dataset statistics for HiL-Bench .
Model
Mode
N
Completion
Accuracy
Strategy
Claude Opus 4.6
baseline
40
82.5
15.0
2.5
Claude Opus 4.6
ask_human
30
70.0
30.0
0.0
Gemini 3.1 Pro
baseline
50
56.0
38.0
6.0
Gemini 3.1 Pro
ask_human
17
17.6
82.4
0.0
GPT 5.3 Codex
baseline
19
5.3
73.7
21.1
GPT 5.3 Codex
ask_human
18
16.7
72.2
11.1
Appendix
Table 4: Tool use failure mode distribution in SQL (%). N = number of tool-use failure traces.
Model
Mode
N
Accuracy
Strategy
Self-Assessment
Completion
Claude Opus 4.6
baseline
20
5.0
50.0
45.0
0.0
Claude Opus 4.6
ask_human
20
5.0
55.0
40.0
0.0
Gemini 3.1 Pro
baseline
22
9.1
68.2
18.2
4.5
Gemini 3.1 Pro
ask_human
35
5.7
60.0
31.4
2.9
GPT 5.3 Codex
baseline
25
12.0
60.0
20.0
8.0
GPT 5.3 Codex
ask_human
34
2.9
58.8
35.3
2.9
Appendix
Table 5: Alignment failure mode distribution in SQL (%). N = number of alignment failure traces.
Model
Mode
N
Accuracy
Strategy
Self-Assessment
Completion
Claude Opus 4.6
baseline
97
41.2
20.6
16.5
21.6
Claude Opus 4.6
ask_human
86
51.2
11.6
12.8
24.4
Gemini 3.1 Pro
baseline
99
42.4
11.1
24.2
22.2
Gemini 3.1 Pro
ask_human
83
56.6
13.3
22.9
7.2
GPT 5.3 Codex
baseline
96
88.5
8.3
2.1
1.0
GPT 5.3 Codex
ask_human
98
78.6
12.2
0.0
9.2
Appendix
Table 6: Logic failure mode distribution in SQL (%). N = number of logic failure traces.
Figure 6: Domain shifts in failure attribution. Each cell shows the difference in percentage points between SWE and SQL as a proportion of failures within a capability dimension. For example, 33pp more of GPT 5.4’s tool-use failures are Completion in SWE compared to in SQL. Higher numbers indicate a higher percentage of failures in SWE.
Figure 7: Failure modes in SWE. Compared to SQL, distributions congregate towards the same failure subtypes in all models
Figure 8: Average tokens used per task, per model. Model behaviors described above are also reflected here; GPT models execute quickly and overconfidently, using the least amount of tokens. Gemini sits more in the middle, while Claude explores, reflects, and explores some more, using up to five times as many tokens as other models.
Model
SWE
SQL
GPT 5.3 Codex
1.5
1.0
GPT 5.4 Pro
1.8
1.7
Gemini 3.1 Pro
2.7
6.6
Claude Opus 4.6
4.7
4.5
Appendix
Table 7: Questions asked per task. GPT models ask the lowest amount of questions in either domain. Gemini tends to over-ask on SQL. Claude sits more in the middle, but this doesn’t necessarily translate to better integration of blocker resolutions into its workflow to solve tasks.
Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these information gaps remain poorly characterized across models and execution harnesses. We present AM-Bench, a benchmark built from 48 tasks adapted from Terminal-Bench 2.0 and Harness-Bench. Each task includes a fully specified reference and six variants spanning ambiguous or missing goals, context, and constraints. We evaluate every pairing of five models and four harnesses over 20,160 runs. Our results reveal distinct execution deficits: goal gaps hinder completion, context gaps increase token use, and constraint gaps weaken compliance without necessarily preventing completion. Missingness lowers success more than ambiguity in all three domains. These effects also reshape system comparisons: advantages under complete information need not persist, and rankings depend on model × harness pairings and the evaluation criterion -- completion, success, or repeat consistency. Estimates based on observed clarification requests and matched fully specified runs further suggest that clarification could recover unfinished work while leaving most constraint violations unresolved. Together, these findings motivate joint model-harness evaluation across information conditions and targeted clarification of unresolved requirements before dependent actions. To support future research on agents under task underspecification, we provide the AM-Bench project repository at https://github.com/s4-lab-cuhksz/AM-Bench.git.
Kaiyu Huang, Yiting Li, Angyang Gu +3
Tongji University · The Chinese University of Hong Kong, Shenzhen
AI agents that send emails, edit files, and make purchases must decide when to act on their own and when to check with the user first. This decision is usually evaluated by showing a model a proposed action, asking whether it should proceed, and scoring agreement with human labels. We introduce DelegationBench to test whether such scores can be trusted. It has 156 scenarios with four possible responses (act, ask for permission, ask for missing information, refuse), and most scenarios come in matched pairs that change a single feature: whether the action was requested, what is at stake, whether it can be undone, or who will see it. Across ten models from five families, agreement scores mislead in three ways. A simple keyword rule, which we wrote after seeing the benchmark, agrees with our annotators more often than eight of the models, yet its decision changes in only 9 of 48 matched pairs. Equivalent ways of asking the same question change how often a model acts by up to 52.5 percentage points. And every model stops to ask the user less often when it must carry out the task with tools than when it judges a proposed action. When rules are stated explicitly, the same models follow them almost perfectly, so the gaps are not explained by a general inability to follow rules. We release the benchmark and evaluation tools and recommend reporting these properties separately rather than as one score.
In hierarchical reasoning, failures often originate at intermediate decision points where the agent commits to a wrong branch without recognizing that it lacks critical information. Rather than treating clarification as an external uncertainty trigger, we propose ACTION-RATING, a formulation that places it inside the agent's action space on a shared ordinal scale with navigation, so that asking competes directly with acting at every decision point and help-seeking becomes observable at intermediate states. Two structurally distinct information-seeking modes emerge from the agent's own ratings: mandatory (no viable branch) and opportunistic (residual uncertainty despite a leading candidate). On Harmonized Tariff Schedule classification (30,000-node taxonomy, three benchmarks, 9~LLMs across 4 families), we observe a regime shift from mandatory to opportunistic clarification, with Information-Seeking Effectiveness (ISE), a local diagnostic defined as the fraction of help interactions followed by a correct next navigation step (not a final-task metric), rising from 50% to 74%. Three diagnostic contrasts fail to reproduce this structure. A separability test shows that the information-seeking pattern (mode split, ISE ranking) persists when answer quality is degraded (-18.8% accuracy), supporting an empirical separation between where an agent seeks help and the quality of the help it receives. Under the controlled answer channel, accuracy gains reach +16.2% at 10-digit; we read this as an upper bound on what better localization could unlock, not a deployment estimate.