Frontier coding agents solve complex tasks when given complete context but collapse when specifications are incomplete or ambiguous. The bottleneck is not raw capability, but judgment: knowing when to act autonomously and when to ask for help. Current benchmarks are blind to this failure mode. They supply unambiguous detailed instructions and solely reward execution correctness, so an agent that makes a lucky guess for a missing requirement will score identically to one that would have asked to be certain. We present HiL-Bench (Human-in-the-Loop Benchmark) to measure this selective escalation skill. Each task contains human-validated blockers (missing information, ambiguous requests, contradictory information) that surface only through progressive exploration, not upfront inspection. Our core metric, Ask-F1, the harmonic mean of question precision and blocker recall, captures the tension between over-asking and silent guessing; its structure architecturally prevents gaming through question spam. Evaluation across SWE and text-to-SQL domains reveals a large universal judgment gap: no frontier model recovers more than a fraction of its full-information performance when deciding whether to ask. Failure analysis identifies three key help-seeking patterns: overconfident wrong beliefs with no gap detection; high uncertainty detection yet persistent errors; broad, imprecise escalation without self-correction. These consistent patterns confirm poor help-seeking is a model-level flaw, not task-specific. RL training on shaped Ask-F1 reward shows judgment is trainable: a 32B model improves both help-seeking quality and task pass rate, with gains that transfer across domains. The model does not learn domain-specific heuristics for when to ask; it learns to detect unresolvable uncertainty and act on it.
Figures & tables
Figure 1: (A) Models achieve 75–89% pass@3 with complete information but only 4–24% when they must judge when to ask, despite access to ask_human() . Near-zero “No Tool” bars confirm tasks require clarification; the bottleneck is judgment, not capability. (B) Ask-F1’s precision–recall decomposition reveals distinct failure profiles across model families, with no model in the well-judged quadrant. RLVR on shaped Ask-F1 reward shifts a 32B model toward calibrated help-seeking (arrow), confirming judgment is trainable.
Figure 2: Example agent evaluation workflow. Each HiL-Bench task contains multiple blockers that surface as the agent explores the task environment. At each point, the agent must judge whether to ask or proceed. Ask-F1 scores both detection (recall) and targeting (precision).
Task Completion (Pass@3)
Selective Escalation
Model
Full Info
w/ ask_human()
Gap ( Δ )
Recall
Precision
Ask-F1
Text-to-SQL
GPT-5.3-Codex
86.0%
5.3%
-81 pp
14.0%
55.3%
22.3%
GPT-5.4
86.0%
17.3%
-69 pp
22.7%
54.3%
32.1%
Gemini 3.1 Pro
88.7%
35.3%
-53 pp
59.2%
35.6%
44.5%
Claude Opus 4.6
90.7%
39.3%
-51 pp
61.2%
54.3%
57.5%
Table 1: The Judgment Gap: SQL vs. SWE. While models easily solve fully specified tasks, their performance collapses when they must actively seek help ( ask_human() ) to resolve ambiguities. This massive gap is driven by poor selective escalation. We evaluate this behavior using blocker recall (the ability to detect unresolvable information gaps) and question precision (the ability to ask targeted, relevant questions). The Ask-F1 score is the harmonic mean of these two metrics. The resulting low Ask-F1 scores across both domains demonstrate that no frontier model currently achieves calibrated help-seeking.
Figure 3: Failure fingerprints reveal distinct judgment signatures across model families in SQL. (A) Baseline failure modes (within-dimension percentages). GPT models are accuracy-dominant. Claude is completion- and self-assessment-dominant. Gemini shows high logic self-assessment failures. (B) Failure distribution shifts with ask_human() . Gemini inverts dramatically in tool-use ( − 38pp completion, + 44pp accuracy). Claude sees a similar shift to a lesser degree ( − 12pp completion, + 15pp accuracy). GPT rows remain largely unchanged. See Figure 7 for a corresponding SWE analysis and more details in Appendix F .
Figure 4: RLVR closes the judgment gap and transfers across domains. (A) Precision–recall space. Arrows show RLVR shifts base models toward calibrated help-seeking (solid: in-domain; dashed: cross-domain). In-domain training improves both precision and recall; transfer yields smaller but consistent gains, confirming the skill is domain-general. (B) Ask-F1 and Pass@3 improve in lockstep. Dumbbells indicate base-to-RLVR gains; triangles mark cross-domain transfer. Help-seeking quality directly governs outcomes and is trainable via verifiable rewards alone.
Figure 5: The Judgment Matrix. Most agents are in the bottom left box.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Blocker Type
Definition
Examples
Missing Information (42% of blockers)
Required or important values not present in the task context. Agent cannot determine the correct value without external input.
SWE: Deprecation warning required on legacy API access but version string, API call, and message shape are all unspecified; wildcard listeners must materialize as concrete IP keys but the address families to enumerate are unstated. SQL: “Overcrowded” required as a school filter but neither defined nor encoded in the schema; seven similarly labeled rating columns exist but which one represents “overall” rating for filtering is unspecified.
Ambiguous Requests (36% of blockers)
Multiple valid interpretations or implementations exist. Each one leads to a different solution.
SWE: One requirement says defaultCountry updates must sync the UI; another says non-empty updates after mount should be ignored—both are stated. Post-scan metrics must be recorded but it is unclear whether DB model counts or only scan counters are in scope. SQL: “Dominance rating” could refer to several numeric fields (overall score vs. attribute composites); “early days of the platform” could refer to almost any timeframe.
Contradictory Information (22% of blockers)
Conflicting or misleading specifications that cannot both be satisfied. Agent must identify the contradiction and seek authoritative resolution.
SWE: One spec line pushes preserving attribution in referral URLs; another pushes sanitization—both cannot hold without a precedence rule. Expected behavior says show only the strongest CPE confidence per vulnerability; another requirement insists every applicable confidence be preserved. SQL: The request mixes “hydrocarbon” with metals and heteroatoms, but by definition hydrocarbons contain only C and H—the constraints are definitionally contradictory. Patients must be simultaneously middle-aged and geriatric, normally disjoint age bands with no reconciling rule provided.
Appendix
Table 2: Blocker taxonomy: three types of information gaps with definitions and representative examples drawn from the dataset.
SWE
SQL
Total
Tasks
150
150
300
Avg. blockers / task
3.55
3.99
3.77
Total blockers
533
598
1131
Missing information
205 (38.5%)
271 (45.3%)
476 (42.1%)
Ambiguous requests
161 (30.2%)
247 (41.3%)
408 (36.1%)
Contradictory information
167 (31.3%)
80 (13.4%)
247 (21.8%)
Appendix
Table 3: Dataset statistics for HiL-Bench .
Model
Mode
N
Completion
Accuracy
Strategy
Claude Opus 4.6
baseline
40
82.5
15.0
2.5
Claude Opus 4.6
ask_human
30
70.0
30.0
0.0
Gemini 3.1 Pro
baseline
50
56.0
38.0
6.0
Gemini 3.1 Pro
ask_human
17
17.6
82.4
0.0
GPT 5.3 Codex
baseline
19
5.3
73.7
21.1
GPT 5.3 Codex
ask_human
18
16.7
72.2
11.1
Appendix
Table 4: Tool use failure mode distribution in SQL (%). N = number of tool-use failure traces.
Model
Mode
N
Accuracy
Strategy
Self-Assessment
Completion
Claude Opus 4.6
baseline
20
5.0
50.0
45.0
0.0
Claude Opus 4.6
ask_human
20
5.0
55.0
40.0
0.0
Gemini 3.1 Pro
baseline
22
9.1
68.2
18.2
4.5
Gemini 3.1 Pro
ask_human
35
5.7
60.0
31.4
2.9
GPT 5.3 Codex
baseline
25
12.0
60.0
20.0
8.0
GPT 5.3 Codex
ask_human
34
2.9
58.8
35.3
2.9
Appendix
Table 5: Alignment failure mode distribution in SQL (%). N = number of alignment failure traces.
Model
Mode
N
Accuracy
Strategy
Self-Assessment
Completion
Claude Opus 4.6
baseline
97
41.2
20.6
16.5
21.6
Claude Opus 4.6
ask_human
86
51.2
11.6
12.8
24.4
Gemini 3.1 Pro
baseline
99
42.4
11.1
24.2
22.2
Gemini 3.1 Pro
ask_human
83
56.6
13.3
22.9
7.2
GPT 5.3 Codex
baseline
96
88.5
8.3
2.1
1.0
GPT 5.3 Codex
ask_human
98
78.6
12.2
0.0
9.2
Appendix
Table 6: Logic failure mode distribution in SQL (%). N = number of logic failure traces.
Figure 6: Domain shifts in failure attribution. Each cell shows the difference in percentage points between SWE and SQL as a proportion of failures within a capability dimension. For example, 33pp more of GPT 5.4’s tool-use failures are Completion in SWE compared to in SQL. Higher numbers indicate a higher percentage of failures in SWE.
Figure 7: Failure modes in SWE. Compared to SQL, distributions congregate towards the same failure subtypes in all models
Figure 8: Average tokens used per task, per model. Model behaviors described above are also reflected here; GPT models execute quickly and overconfidently, using the least amount of tokens. Gemini sits more in the middle, while Claude explores, reflects, and explores some more, using up to five times as many tokens as other models.
Model
SWE
SQL
GPT 5.3 Codex
1.5
1.0
GPT 5.4 Pro
1.8
1.7
Gemini 3.1 Pro
2.7
6.6
Claude Opus 4.6
4.7
4.5
Appendix
Table 7: Questions asked per task. GPT models ask the lowest amount of questions in either domain. Gemini tends to over-ask on SQL. Claude sits more in the middle, but this doesn’t necessarily translate to better integration of blocker resolutions into its workflow to solve tasks.