Asking but Missing: Decision Coverage in Natural-Language Data Querying
Organizations: Zhejiang University of Technology (zjut.edu.cn)
Abstract
Large language models are increasingly used to translate natural language questions about structured data into executable programs. A fundamental challenge is recognizing when a question admits multiple computational interpretations and identifying the decisions needed to resolve them. We investigate whether language models recognize these unresolved decisions before generating code, or silently adopt assumptions that may lead to incorrect results. We refer to the latter behavior as Underspecified Decision Completion (UDC). To distinguish different causes of UDC, we introduce Grounded Code Generation (GCG), a diagnostic protocol that separates the identification of unresolved decisions, their resolution using evidence, and their translation into executable code. By examining each stage separately, we can determine where the decisions responsible for incorrect results were first missed or mishandled. We evaluate this framework on BIRD-Pandas with twelve models. The results show that answering clarification questions consistently improves execution accuracy, yet substantial errors remain. Our findings suggest that the main difficulty lies not in answering clarification questions, but in recognizing which questions need to be asked in the first place.
Figures & tables
| Primary query object | Count | Share | No clue ( N ) | Phase 1 dimension |
| COMPUTATION | 647 | 23.7% | 38.8% | EXECUTION_METHOD |
| SELECTION_RULE | 479 | 17.5% | 30.7% | BUSINESS_RULE |
| DATA_REFERENCE | 398 | 14.6% | 32.7% | DATA_REFERENCE |
| RESULT_CONTRACT | 374 | 13.7% | 55.1% | RESULT_CONTRACT |
| REPRESENTATION | 331 | 12.1% | 23.3% | DATA_REFERENCE or BUSINESS_RULE |
| RELATIONSHIP | 330 | 12.1% | 35.8% | EXECUTION_METHOD |
| Direct Gen. | Grounded Code Generation | |||||||
| Model | sim. | mod. | chall. | Over. | sim. | mod. | chall. | Over. |
| Llama-3.2-3B | 9.41 | 3.23 | 3.45 | 6.98 | 14.38 | 11.21 | 13.10 | 13.30 |
| Meta-Llama-3-8B | 18.81 | 4.09 | 6.21 | 13.17 | 22.05 | 6.68 | 8.97 | 16.17 |
| Llama-3.2-11B-Vision | 21.30 | 15.30 | 8.97 | 18.32 | 28.86 | 22.20 | 20.69 | 26.08 |
| Qwen3.5-4B | 11.78 | 8.62 | 8.97 | 10.56 | 40.86 | 30.82 | 28.97 | 36.70 |
| Qwen3.5-9B | 24.86 | 25.00 | 16.55 | 24.12 | 44.65 | 38.15 | 33.10 | 41.58 |
| Model | P1 [95% CI] | P2 | P3-N | P3-T | |
| Qwen3-Max | 404 | 38.6 [34.0, 43.4] | 27.2 | 16.8 | 17.3 |
| DeepSeek-V4-Pro | 384 | 67.2 [62.3, 71.7] | 9.6 | 6.0 | 17.2 |
| Qwen3.6-27B | 340 | 62.4 [57.1, 67.3] | 12.6 | 9.7 | 15.3 |
| Gemini-3.1-Pro | 281 | 72.2 [66.7, 77.2] | 13.5 | 2.8 | 11.4 |
| Phase 1 coverage (%) | Recall by Evidence Source (%) | Recall by Level (%) | ||||||||
| Model | Decisions | Recall | Prec. | F1 | Q | S | K | N | NO_CLUE | Other |
| Qwen3-Max | 1,128 | 82.4 | 81.6 | 82.0 | 84.0 | 79.5 | 86.8 | 40.9 | 56.4 | 86.0 |
| DeepSeek-V4-Pro | 1,050 | 71.0 | 84.7 | 77.3 | 78.5 | 68.1 | 68.7 | 47.5 | 51.3 | 75.5 |
| Qwen3.6-27B | 1,022 | 76.5 | 84.4 | 80.3 | 85.8 | 72.0 | 75.2 | 40.0 | 53.9 | 81.3 |
| Gemini-3.1-Pro | 642 | 66.5 | 88.8 | 76.0 | 76.2 | 64.9 | 57.8 | 41.4 | 47.5 | 70.8 |
| Pooled | 3,842 | 75.1 | 84.2 | 79.4 | 81.5 | 71.9 | 74.8 | 43.0 | 52.5 | 79.5 |
| Model | NO_CLUE (%) | IMPLICIT (%) | PARTIAL (%) | EXPLICIT (%) | UDC (%) | |
| Qwen3-Max | 156 | 32.3 | 26.3 | 28.7 | 12.6 | 87.4 |
| DeepSeek-V4-Pro | 258 | 33.3 | 22.3 | 31.1 | 13.3 | 86.7 |
| Qwen3.6-27B | 212 | 36.1 | 24.9 | 32.2 | 6.8 | 93.2 |
| Gemini-3.1-Pro | 203 | 32.4 | 19.2 | 36.3 | 12.1 | 87.9 |
| Model | w/o Phase 1 | + Free-form Phase 1 | + GCG |
| GPT-5.4-mini | 37.94 | 48.36 | 66.36 |
| Qwen3.6-27B | 59.91 | 63.87 | 77.84 |
| DeepSeek-V4-Pro | 62.06 | 65.24 | 74.97 |
| Oracle Model | chall. | mod. | sim. | Over. |
| Qwen3-Max | 63.45 | 62.28 | 67.35 | 65.45 |
| DeepSeek-V4-Pro | 63.21 | 61.35 | 66.20 | 64.45 |
| Qwen3.5-27B | 61.66 | 62.52 | 62.40 | 62.37 |
| Qwen3.5-35B-A3B | 63.35 | 58.61 | 63.98 | 62.30 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Query object | QID | Question (excerpt) | Label; level/src | Discrepancy |
| COMPUTATION | 590 | Which user ID has the lowest view? | OMIT_TIE_HANDLING ; IMPLICIT / N | Model returns first ID ( idxmin ); reference keeps all tied users. |
| RESULT_CONTRACT | 71 | What is the district code for the School […] in Fresno? | LIMIT_TO_FIRST_ROW ; PARTIAL / Q | Model returns first code ( iloc[0] ); reference returns all. |
| BOUNDARY_GUARD | 22 | Which school in Contra Costa has the highest […] test takers? | OMIT_SNAME_NOTNULL ; PARTIAL / S | Model omits sname.notna() filter; may return null. |
| SELECTION_RULE | 70 | How many active and closed […] Schools in Alpine? | OMIT_STATUS_FILTER ; EXPLICIT / Q | Model does not filter by StatusType . |
| Query object | Qwen3-Max | DeepSeek-V4-Pro | Gemini-3.1-Pro | No clue ( N ) | IMPLICIT or NO_CLUE |
| COMPUTATION | 227 | 244 | 176 | 38.8% | 52.1% |
| SELECTION_RULE | 165 | 181 | 133 | 30.7% | 50.9% |
| DATA_REFERENCE | 145 | 155 | 98 | 32.7% | 53.5% |
| RESULT_CONTRACT | 128 | 131 | 115 | 55.1% | 66.0% |
| REPRESENTATION | 137 | 130 | 64 | 23.3% | 46.5% |
| RELATIONSHIP | 132 | 141 | 57 | 35.8% | 67.3% |
| Explicit Level | Evidence Source | |||||||
| Model | NO_CLUE | IMPL. | PART. | EXPL. | N | Q | S | K |
| Qwen3-Max | 34.7 | 23.8 | 28.9 | 12.6 | 39.3 | 23.0 | 26.9 | 10.8 |
| DeepSeek-V4-Pro | 39.0 | 20.1 | 28.5 | 12.4 | 39.8 | 25.8 | 23.9 | 10.5 |
| Gemini-3.1-Pro | 35.7 | 17.7 | 28.4 | 18.3 | 37.9 | 32.6 | 20.5 | 9.1 |
| Model | C C | W W | W C | C W | Net |
| Qwen3-Max | 865 | 306 | 265 | 98 | +167 |
| DeepSeek-V4-Pro | 885 | 317 | 265 | 67 | +198 |
| Qwen3.6-27B | 843 | 264 | 351 | 76 | +275 |
| Gemini-3.1-Pro | 1,043 | 214 | 210 | 67 | +143 |
| Component | Observed content |
| Question | Which school in Contra Costa has the highest number of test takers? |
| Reference program code | satscores[(cname == ’Contra Costa’) & (sname.notna())].nlargest(1, ’NumTstTakr’)[[’sname’]] |
| Phase 1 raised clarification questions | The model asks whether to filter by satscores.cname or join schools ; how to resolve ties; which school-name column to display; and how to handle missing NumTstTakr . It does not ask whether the selected row must have a non-null sname . |
| Phase 2 evidence | The raised clarification questions receive reference-consistent answers: use cname , return one row, use sname , and exclude missing NumTstTakr . |
| Phase 3 model program | The model applies the answered conditions and uses nlargest(1, ’NumTstTakr’)[[’sname’]] , but does not apply sname.notna() . |
| Outputs | Reference program code: Dougherty Valley High . Model: null . |
| Model | Human Precision | Pipeline Precision | Human Recall | Pipeline Recall |
| Qwen3-Max | 86 | 82 | 84 | 82 |
| DeepSeek-V4-Pro | 88 | 85 | 75 | 71 |
| Qwen3.6-27B | 85 | 84 | 80 | 77 |
| Gemini-3.1-Pro | 85 | 89 | 70 | 67 |
| Pooled (avg) | 86 | 85 | 77 | 74 |
| Failure action | Primary object | |||
| Category | Annotator model | Human | Annotator model | Human |
| Omission / COMPUTATION | 44.2 | 45.0 | 23.7 | 24.5 |
| Wrong substitution / SEL. RULE | 32.2 | 31.5 | 17.5 | 18.0 |
| Unwarranted addition / DATA REF | 12.5 | 12.0 | 14.6 | 15.0 |
| Misapplication / RESULT CONTR. | 5.5 | 6.0 | 13.7 | 13.0 |
| Incorrect computation / REPRES. | 3.2 | 3.5 | 12.1 | 12.5 |
| Alignment level | Criterion | Count | Rate (%) |
| Fully aligned | Evidence precisely answers the semantic need of the question | 714/973 | 73.4 |
| Partially aligned | Evidence gives the key information but with ambiguity or omission | 187/973 | 19.2 |
| Not aligned | Evidence is off-target or contradicts the question’s direction | 72/973 | 7.4 |
| Fully or partially aligned (LLM judge) | 901/973 | 92.6 | |
| Fully or partially aligned (human, ) | 110/120 | 91.7 | |
| Model | Reas. tokens | P1 | P2 | P3 | GCG | Direct |
| Qwen3-Max | 4,560 | 127.9 | 13.1 | 120.1 | 261.1 | 121.2 |
| DeepSeek-V4-Pro | 3,200 | 54.3 | 12.4 | 58.7 | 125.4 | 59.1 |
| Qwen3.6-27B | 3,360 | 50.0 | 12.2 | 52.4 | 114.6 | 52.4 |
| Gemini-3.1-Pro | 0 | 8.9 | 11.2 | 5.1 | 25.2 | 5.1 |
LLM Code Generation
Large language model code generation studies how models translate instructions and context into executable programs and how to make them reliable.
See all 336 papers