Large language models are increasingly used to translate natural language questions about structured data into executable programs. A fundamental challenge is recognizing when a question admits multiple computational interpretations and identifying the decisions needed to resolve them. We investigate whether language models recognize these unresolved decisions before generating code, or silently adopt assumptions that may lead to incorrect results. We refer to the latter behavior as Underspecified Decision Completion (UDC). To distinguish different causes of UDC, we introduce Grounded Code Generation (GCG), a diagnostic protocol that separates the identification of unresolved decisions, their resolution using evidence, and their translation into executable code. By examining each stage separately, we can determine where the decisions responsible for incorrect results were first missed or mishandled. We evaluate this framework on BIRD-Pandas with twelve models. The results show that answering clarification questions consistently improves execution accuracy, yet substantial errors remain. Our findings suggest that the main difficulty lies not in answering clarification questions, but in recognizing which questions need to be asked in the first place.
Figures & tables
Figure 1: Three examples of UDC, in which a model asks several clarification questions and receives answers, yet does not ask which quantity to rank by, who counts as an account holder, or whether a summary row is a valid entity, so its program returns a result that differs from the reference.
Primary query object
Count
Share
No clue ( N )
Phase 1 dimension
COMPUTATION
647
23.7%
38.8%
EXECUTION_METHOD
SELECTION_RULE
479
17.5%
30.7%
BUSINESS_RULE
DATA_REFERENCE
398
14.6%
32.7%
DATA_REFERENCE
RESULT_CONTRACT
374
13.7%
55.1%
RESULT_CONTRACT
REPRESENTATION
331
12.1%
23.3%
DATA_REFERENCE or BUSINESS_RULE
RELATIONSHIP
330
12.1%
35.8%
EXECUTION_METHOD
Table 1: Pooled distribution of the preliminary discrepancies and the Phase 1 dimension each object maps to.
Figure 2: Overview of the Grounded Code Generation (GCG) framework.
Direct Gen.
Grounded Code Generation
Model
sim.
mod.
chall.
Over.
sim.
mod.
chall.
Over.
Llama-3.2-3B
9.41
3.23
3.45
6.98
14.38
11.21
13.10
13.30
Meta-Llama-3-8B
18.81
4.09
6.21
13.17
22.05
6.68
8.97
16.17
Llama-3.2-11B-Vision
21.30
15.30
8.97
18.32
28.86
22.20
20.69
26.08
Qwen3.5-4B
11.78
8.62
8.97
10.56
40.86
30.82
28.97
36.70
Qwen3.5-9B
24.86
25.00
16.55
24.12
44.65
38.15
33.10
41.58
Table 2: Execution accuracy (%) of twelve models on BIRD-Pandas under direct generation and under the GCG diagnostic protocol.
Model
Ndiag
P1 [95% CI]
P2
P3-N
P3-T
Qwen3-Max
404
38.6 [34.0, 43.4]
27.2
16.8
17.3
DeepSeek-V4-Pro
384
67.2 [62.3, 71.7]
9.6
6.0
17.2
Qwen3.6-27B
340
62.4 [57.1, 67.3]
12.6
9.7
15.3
Gemini-3.1-Pro
281
72.2 [66.7, 77.2]
13.5
2.8
11.4
Table 3: Share of GCG failures by error attribution category. Ndiag is the number of GCG failures on which the diagnostic pipeline produced a label.
Figure 3: Elicitation profile on BIRD-Pandas. (a) Percentage of input prompts by number of Phase 1 raised clarification questions. (b) Share (%) of each model’s Phase 2 sub-questions by evidence type.
Phase 1 coverage (%)
Recall by Evidence Source (%)
Recall by Level (%)
Model
Decisions
Recall
Prec.
F1
Q
S
K
N
NO_CLUE
Other
Qwen3-Max
1,128
82.4
81.6
82.0
84.0
79.5
86.8
40.9
56.4
86.0
DeepSeek-V4-Pro
1,050
71.0
84.7
77.3
78.5
68.1
68.7
47.5
51.3
75.5
Qwen3.6-27B
1,022
76.5
84.4
80.3
85.8
72.0
75.2
40.0
53.9
81.3
Gemini-3.1-Pro
642
66.5
88.8
76.0
76.2
64.9
57.8
41.4
47.5
70.8
Pooled
3,842
75.1
84.2
79.4
81.5
71.9
74.8
43.0
52.5
79.5
Table 4: Phase 1 coverage of reference execution decisions on the failing input prompts.
Model
NP1
NO_CLUE (%)
IMPLICIT (%)
PARTIAL (%)
EXPLICIT (%)
UDC (%)
Qwen3-Max
156
32.3
26.3
28.7
12.6
87.4
DeepSeek-V4-Pro
258
33.3
22.3
31.1
13.3
86.7
Qwen3.6-27B
212
36.1
24.9
32.2
6.8
93.2
Gemini-3.1-Pro
203
32.4
19.2
36.3
12.1
87.9
Table 5: Distribution of identified root-cause gaps by explicit level, for input prompts attributed to Phase1_Missed_Gap .
Figure 4: Phase1_Missed_Gap rate shows a positive association with the reference-program complexity proxy in all four analyzed models.
Model
w/o Phase 1
+ Free-form Phase 1
+ GCG
GPT-5.4-mini
37.94
48.36
66.36
Qwen3.6-27B
59.91
63.87
77.84
DeepSeek-V4-Pro
62.06
65.24
74.97
Table 6: Execution accuracy (%) under Phase 1 prompt settings.
Oracle Model
chall.
mod.
sim.
Over.
Qwen3-Max
63.45
62.28
67.35
65.45
DeepSeek-V4-Pro
63.21
61.35
66.20
64.45
Qwen3.5-27B
61.66
62.52
62.40
62.37
Qwen3.5-35B-A3B
63.35
58.61
63.98
62.30
Table 7: Phase 2 oracle strength ablation.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Query object
QID
Question (excerpt)
Label; level/src
Discrepancy
COMPUTATION
590
Which user ID has the lowest view?
OMIT_TIE_HANDLING ; IMPLICIT / N
Model returns first ID ( idxmin ); reference keeps all tied users.
RESULT_CONTRACT
71
What is the district code for the School […] in Fresno?
LIMIT_TO_FIRST_ROW ; PARTIAL / Q
Model returns first code ( iloc[0] ); reference returns all.
BOUNDARY_GUARD
22
Which school in Contra Costa has the highest […] test takers?
OMIT_SNAME_NOTNULL ; PARTIAL / S
Model omits sname.notna() filter; may return null.
SELECTION_RULE
70
How many active and closed […] Schools in Alpine?
OMIT_STATUS_FILTER ; EXPLICIT / Q
Model does not filter by StatusType .
Appendix
Table 8: Representative discrepancies from Qwen3-Max.
Query object
Qwen3-Max
DeepSeek-V4-Pro
Gemini-3.1-Pro
No clue ( N )
IMPLICIT or NO_CLUE
COMPUTATION
227
244
176
38.8%
52.1%
SELECTION_RULE
165
181
133
30.7%
50.9%
DATA_REFERENCE
145
155
98
32.7%
53.5%
RESULT_CONTRACT
128
131
115
55.1%
66.0%
REPRESENTATION
137
130
64
23.3%
46.5%
RELATIONSHIP
132
141
57
35.8%
67.3%
Appendix
Table 9: Query-object distribution by model.
Explicit Level
Evidence Source
Model
NO_CLUE
IMPL.
PART.
EXPL.
N
Q
S
K
Qwen3-Max
34.7
23.8
28.9
12.6
39.3
23.0
26.9
10.8
DeepSeek-V4-Pro
39.0
20.1
28.5
12.4
39.8
25.8
23.9
10.5
Gemini-3.1-Pro
35.7
17.7
28.4
18.3
37.9
32.6
20.5
9.1
Appendix
Table 10: Explicit Level and Evidence Source distributions (%) of the discrepancies of each model (1,000, 1,046, and 684 discrepancies).
Model
C → C
W → W
W → C
C → W
Net
Qwen3-Max
865
306
265
98
+167
DeepSeek-V4-Pro
885
317
265
67
+198
Qwen3.6-27B
843
264
351
76
+275
Gemini-3.1-Pro
1,043
214
210
67
+143
Appendix
Table 11: Paired question counts ( N=1,534 ) from Direct Generation to GCG. C and W mark a correct or wrong answer, so W → C questions are recovered, C → W questions regress, and Net = W → C − C → W.
Component
Observed content
Question
Which school in Contra Costa has the highest number of test takers?
The model asks whether to filter by satscores.cname or join schools ; how to resolve ties; which school-name column to display; and how to handle missing NumTstTakr . It does not ask whether the selected row must have a non-null sname .
Phase 2 evidence
The raised clarification questions receive reference-consistent answers: use cname , return one row, use sname , and exclude missing NumTstTakr .
Phase 3 model program
The model applies the answered conditions and uses nlargest(1, ’NumTstTakr’)[[’sname’]] , but does not apply sname.notna() .
Outputs
Reference program code: Dougherty Valley High . Model: null .
Appendix
Table 12: Q22 worked example. Both programs use nlargest ; they differ only in the sname.notna() filter.
Model
Human Precision
Pipeline Precision
Human Recall
Pipeline Recall
Qwen3-Max
86
82
84
82
DeepSeek-V4-Pro
88
85
75
71
Qwen3.6-27B
85
84
80
77
Gemini-3.1-Pro
85
89
70
67
Pooled (avg)
86
85
77
74
Appendix
Table 13: Human-confirmed and pipeline Phase 1 gap coverage analysis (%) on the same 120-prompt sample drawn from each model’s Phase1_Missed_Gap set. Pipeline values are recomputed on this sample.
Failure action
Primary object
Category
Annotator model
Human
Annotator model
Human
Omission / COMPUTATION
44.2
45.0
23.7
24.5
Wrong substitution / SEL. RULE
32.2
31.5
17.5
18.0
Unwarranted addition / DATA REF
12.5
12.0
14.6
15.0
Misapplication / RESULT CONTR.
5.5
6.0
13.7
13.0
Incorrect computation / REPRES.
3.2
3.5
12.1
12.5
Appendix
Table 14: Pooled annotator-model and human-consensus distributions (% of discrepancies) for failure action (left) and primary object (right) on 150 stratified questions.
Alignment level
Criterion
Count
Rate (%)
Fully aligned
Evidence precisely answers the semantic need of the question
714/973
73.4
Partially aligned
Evidence gives the key information but with ambiguity or omission
187/973
19.2
Not aligned
Evidence is off-target or contradicts the question’s direction
72/973
7.4
Fully or partially aligned (LLM judge)
901/973
92.6
Fully or partially aligned (human, n=120 )
110/120
91.7
Appendix
Table 15: Phase 2 evidence alignment.
Model
Reas. tokens
P1
P2
P3
GCG
Direct
Qwen3-Max
4,560
127.9
13.1
120.1
261.1
121.2
DeepSeek-V4-Pro
3,200
54.3
12.4
58.7
125.4
59.1
Qwen3.6-27B
3,360
50.0
12.2
52.4
114.6
52.4
Gemini-3.1-Pro
0
8.9
11.2
5.1
25.2
5.1
Appendix
Table 16: Per-question cost of GCG. Every model receives 7,264 input tokens. Reasoning tokens are approximate means; times are in seconds.
Code language models are now trusted collaborators in production workflows for debugging, refactoring, and iterative repair, and every benchmark that evaluates them assumes the instructions they act on are correct. We study what happens when that assumption breaks. We evaluate code language models across four experiments designed to assess whether models resist or obey incorrect instructions in single-pass and iterative repair settings, using the RunBugRun dataset of algorithmic Python problems with deterministic test cases. Our findings reveal a striking behavioral pattern: models correctly identify an incorrect instruction as wrong, then follow it anyway. This compliance unknowingly introduces errors beyond the original bug, and the corrupted code state cannot be recovered through subsequent self-guided iterative repair, which fails to converge across passes. We term this Blind Obedience, characterize the Ghost (Unknown) Errors it introduces, quantify the proportion of cases where semantic corruption proves irrecoverable, and show that extended reasoning cannot reverse it. These findings surface behavioral properties invisible to pass-rate evaluation, with direct consequences for code language models deployed in production settings.
Raj Jaiswal, Anany Singh Divy, Savar Bhasin +3
IIIT Delhi · Microsoft Research India · IIT Kanpur
Coding agents can generate code that appears correct but implements behavior the user never intended. This mismatch can arise when an agent silently resolves underspecified requirements through its own assumptions. As subsequent development builds on these assumptions, correcting the resulting behavior can become increasingly costly. Early clarification can help prevent such mismatches, but unnecessary questions can interrupt developers and slow down development. Existing methods struggle to identify key clarification questions while avoiding unnecessary ones. Therefore, we propose CONTRA, a training-free method that combines broad question discovery with semantic and execution-based question qualification. CONTRA first generates candidate questions and filters out those unrelated to required behavior or already resolved by the requirement. For each remaining question, it generates programs conditioned on two plausible answers and checks for stable behavioral differences on shared inputs. It then uses the interaction history to select among qualified questions or stop asking. Experiments on ClarifyCodeBench show that CONTRA achieves the highest F1 with all four coding agents, exceeding the best baseline macro-average F1 by 13.88 percentage points. With the same LLM and evaluation protocol, CONTRA also achieves higher clarification recall and F1 than the coding harnesses Claude Code and OpenHands. To support practical use, we also implement CONTRA as a Claude Code plugin that integrates selective clarification into everyday development.
Large language models (LLMs) often achieve strong performance on reasoning benchmarks, but final-answer accuracy alone does not show whether they faithfully execute the procedure specified in a prompt. We introduce a controlled diagnostic benchmark for procedural execution, where models are given a step-wise arithmetic procedure and two numeric inputs, and must return the final computed value. Complexity is varied through procedure length and look-back dependencies over intermediate variables. Average first-answer accuracy drops from 63% on 5-step procedures to 20% on 95-step procedures. Generation-level analysis shows that failures often involve missing answers, premature answers, self-correction after an initial error and under-executed traces. These findings suggest that apparent reasoning ability can mask substantial weaknesses in faithful long-horizon procedural execution.