Conversational Task Disambiguation over Tabular Data: Leakage-Aware Formulation, Benchmark Suite, and Training

Nafiseh Ghoroghchian; Luis Scoccola; Tina Sedaghat; Omid Vaheb; Hannah Chen; Dino D'Agostino; Keyvan Golestan

cs.LGOct 7, 2026

Conversational Task Disambiguation over Tabular Data: Leakage-Aware Formulation, Benchmark Suite, and Training

Authors: Nafiseh Ghoroghchian, Luis Scoccola, Tina Sedaghat, Omid Vaheb, Hannah Chen, Dino D'Agostino, Keyvan Golestan

Organizations: Layer 6 AI

Abstract

Conversational task disambiguation over tabular data uses dialogue to resolve missing information about a user's intended task before producing a solution over tables or databases. Existing evaluation and training lack a leakage-aware foundation. Task success mixes the agent's disambiguation and solution-generation capabilities and can also reflect oracle leakage, that is, information that a user simulator reveals beyond what a real user would. Existing datasets also lack a shared representation of ambiguities and access boundaries. We introduce the notion of an ambiguous verifiable task, which formalizes ambiguities and resolutions, decomposing the agent into an asking policy and a solution policy, and the environment into an oracle and verifier. This framework provides baselines and metrics for evaluating task disambiguation separately from solution generation, formal definitions of oracle leakage, judge-free leakage diagnostics, and a training objective for the asking policy. We instantiate the framework in text-to-SQL with AmbiTab, a benchmark suite that unifies six ambiguous datasets under a common representation specifying what the agent, oracle, and verifier may access. We evaluate clarification strategies and oracle leakage, and train an asking policy with reinforcement learning. The trained asker improves our disambiguation metrics on all six datasets and task success on five, and our leakage diagnostics measure how training affects oracle leakage.

Figures & tables

Figure 1: Agent and environment decompositions, task record, and example of a text-to-SQL AVT with two ambiguities and four interpretations.

Panel of Figure 2 — Figure 2: The ask-then-solve protocol ( Section A.2 ). Oracle leakage can arise from undeserved matcher credit or responses revealing more than the credited resolutions.

Dataset	Missing information and construction	Records
AmbiTab-Bird	Rewrite BIRD tasks with alternative readings; filter by SQL probes and a model judgment.	664
AmbiTab-EW-Bird	Withhold the BIRD evidence field; retain rows where providing it improves performance for at least one reference model.	2,537
AmbiTab-BIRD-Interact	Normalize source ambiguity annotations; freeze audited, model-authored responses.	600
AmbiTab-Ambrosia	Use each annotated reading and its native intent as a separate record.	2,670
AmbiTab-AmbiQT	Build executable databases for paired SQL readings and assign each reading as a hidden intent.	1,722
AmbiTab-TACO	Verbalize source SQL while withholding table or value choices.	187

Table 1: The six AmbiTab datasets. AmbiTab-Bird and AmbiTab-EW-Bird counts combine their train and development splits.

Figure 3: The values of Table 8 ( Section C.4 ) plus each oracle’s own no-disambiguation SR, under the deterministic (navy), Qwen3-14B (yellow), and Gemma-3-27B (red) oracles. (a) SR without disambiguation (pale), with the evaluated asker (solid), and with full disambiguation (black line); hatching is the gap the asker leaves. (b-c) Asker recall and precision; hatching is the shortfall from 100%. (d-g) Leakage diagnostics in percentage points (Bird and EW-Bird dev, Ambrosia); the deterministic oracle’s

\Delta J_{\mathrm{resp}}

\Delta J_{\mathrm{full}}

, and

\Delta\mathrm{Rec}_{\mathrm{none}}

are zero by definition (open circles), and its

\Delta\mathrm{Rec}_{\mathrm{spec}}^{Q}

is the reference recall. Error bars: seed SDs over five seeds in (a-c) and three in (d-g). Dataset labels omit the AmbiTab prefix.

Figure 4: Change in leakage when the LLM responder also reads the full reference SQL, with the matcher unchanged, shown minus hidden. (a)

\Delta J_{\mathrm{resp}}

; (b)

\Delta J_{\mathrm{full}}

. Above the line, the default responder reads the SQL and the new runs hide it; below it, the new runs show it. Error bars are paired 95% AVT-cluster bootstrap intervals ( Table 15 ). AmbiTab-BIRD-Interact was not run, and dataset labels omit the AmbiTab prefix.

		SR			Precision			Recall
Dist.	Corpus	Base	RL	$\Delta$	Base	RL	$\Delta$	Base	RL	$\Delta$
ID	AT-Bird dev	59.5	71.9	+12.4	31.0	37.0	+6.0	67.7	83.4	+15.7
	AT-EW-Bird dev	69.0	72.2	+3.2	33.6	38.0	+4.4	69.8	79.3	+9.5
OOD	AT-BIRD-Interact	9.7	9.7	+0.0	32.1	36.4	+4.3	57.7	65.1	+7.4
	AT-Ambrosia	35.0	37.3	+2.2	24.0	27.3	+3.3	71.7	81.8	+10.1
	AT-AmbiQT	56.3	59.4	+3.1	19.5	21.7	+2.2	58.3	65.0	+6.7

Table 2: Question-F1 reward results with the deterministic oracle and fixed solver: SR, precision, and recall (%) of the untrained (Base) and trained (RL) askers. Dist.: in-distribution (ID) development splits or out-of-distribution (OOD) corpora; AT abbreviates AmbiTab.

\Delta

: RL minus Base (pp). RL averages three evaluation seeds and Base five; per-corpus seed SDs are at most 1.5 pp ( Table 13 ). SR ties at displayed precision show zero

\Delta

. The mean weights corpora equally. Full comparison in Table 13 .

Figure 5: Change from the untrained asker ( Tables 2 , 13 and 16 ); gray bands span

\pm 1.96

SE of the seed-mean difference. Bird and EW-Bird are dev splits.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Name	Reading
Ambiguous verifiable task
$\mathrm{Text}$	texts	Specifications, ambiguities, questions, responses, resolutions, and solutions
$x$	task specification	The public request, visible to the agent
$\Theta$ , $\theta$	interpretations	$\Theta$ is their distribution; $\theta\sim\Theta$ is the hidden reading of $x$ , drawn once per episode
$\mathrm{Amb}$ , $\alpha$	ambiguities	The finite set of registered ambiguities of $x$ , each named by a text
$\rho$	resolution map	$\rho(\theta,\alpha)$ is what $\theta$ decides for ambiguity $\alpha$

Appendix

Table 3: Notation.

Corpus	Pool	Records	DBs	With ambiguities	Ambiguities
AmbiTab-Bird	train	537	42	496	798
	dev	127	11	119	198
AmbiTab-EW-Bird	train	2,091	67	2,091	4,417
	dev	446	11	446	911
AmbiTab-BIRD-Interact	all	600	22	593	1,755
AmbiTab-Ambrosia	all	2,670	763	2,670	2,670

Appendix

Table 4: Evaluated pools and registered ambiguities. Counts include 49 zero-ambiguity AmbiTab-Bird controls and 7 AmbiTab-BIRD-Interact records without a registered ambiguity.

Corpus	Construction	Selection and scope
AmbiTab-Bird	Construct alternative readings and executable alternative SQL for BIRD tasks, then use Qwen to rewrite the task specification and supply fixed resolutions ( Li et al., 2023 ) .	Require a Qwen3-8B reference-result miss without clarification and a match with all resolutions, followed by a blind Gemma-3-27B necessity judgment; retain zero-ambiguity controls.
AmbiTab-EW-Bird	Split BIRD evidence on semicolons, turn each hint into a masked knowledge ambiguity, and withhold it from the visible prompt ( Li et al., 2023 ) .	Keep a row if Qwen3-Coder-30B-A3B or Gemma-3-27B fails without evidence and succeeds with it; evaluate only retained ambiguity-bearing rows, with source train and validation pooled as train and source test as dev.
AmbiTab-BIRD-Interact	Retain source task specifications, SQL, PostgreSQL databases, and annotations ( Huo et al., 2026 ) ; normalize duplicate terms, repair one reviewed missing fragment, and freeze an audited catalog of LLM-authored natural-language responses.	Response authoring uses the request, schema, reference SQL, and annotations; the full 600-record pool is evaluated, with local database-disjoint slices also available.
AmbiTab-Ambrosia	Convert each reading in the official test partition to a record with its native intent and SQL ( Saparina & Lapata, 2024 ) ; derive terms from scope, attachment, or vagueness annotations and copy the SQLite databases.	Exclude few-shot examples; local database-disjoint train and test slices are selected within each ambiguity type toward a 20% test target, and both derive from the upstream test partition.
AmbiTab-AmbiQT	For the column-synonym subset, build a SQLite database containing both alternatives, copying original values to one column and resampling the other with replacement ( Bhaskar et al., 2023 ) .	Reject protected key columns and pairs whose SQL readings fail or return equal results; emit both hidden intents for each of 861 retained source items, with the copied column assigned by a per-unit draw.
AmbiTab-TACO	Deterministically verbalize US single-table SELECT queries while withholding table or literal-value choices ( Deng et al., 2026 ) .	Keep executable, nonempty queries with a result-distinct table alternative; 187 of 5,159 source files survive, while value alternatives are not separately required to change the result.

Appendix

Table 5: Source-to-record conversions. The right column states the principal selection rule and limit for each evaluated pool.

Variant	Corpora	Matcher and responder
Deterministic intent	AmbiTab-BIRD-Interact, AmbiTab-Ambrosia, AmbiTab-AmbiQT, AmbiTab-TACO	Match terms and aliases, preferring quoted targets and then earliest mentions; return the selected frozen response verbatim, with refusals for unmatched questions or ties.
Deterministic knowledge	AmbiTab-Bird, AmbiTab-EW-Bird	Rank terms and evidence-derived concepts, prefer unresolved matches, and return the stored resolution; repeats can restate evidence.
Scoped LLM	AmbiTab-BIRD-Interact, AmbiTab-Ambrosia, AmbiTab-AmbiQT, AmbiTab-TACO	Classify from routing fields and respond from retained annotations; the responder sees all retained annotations and receives reference SQL when SQL grounding is used.
Unfiltered LLM	AmbiTab-Bird, AmbiTab-EW-Bird	Classify and respond with the task context, withheld evidence, and reference SQL available.

Appendix

Table 6: Evaluated oracle variants and their runtime access.

Setting	Value
Asker	Qwen3-8B
Solver	Qwen3-Coder-30B-A3B-Instruct, fixed
Oracle	Deterministic, Qwen3-14B, or Gemma-3-27B
Default asking prompt	Checklist (ledger)
Asker and solver decoding	Temperature 0.7; top- $p=0.8$ (trained askers in Section D.5 : top- $p=0.95$ )
Qwen3-14B oracle decoding	Temperature 0.7; top- $p=0.8$ ; top- $k=20$

Appendix

Table 7: Inference settings shared across corpora unless explicitly varied.

Figure 6: Prompt tradeoffs under the deterministic oracle: (a) precision versus recall; (b) asking gain versus recall. Colors identify datasets and shapes identify prompts; error bars show seed standard deviations. Lines fit each dataset’s three prompt means; correlations remove dataset means ( Section C.3 ).

Strategy	Instructions and history
Direct	Ask about unclear parts of the request without an explicit checklist; stop when the request is clear or no different ambiguity remains to be raised.
Checklist (chat)	Enumerate potentially ambiguous terms; read prior turns as native chat messages; stop when all terms are resolved or no distinct unasked term remains.
Checklist (ledger)	Enumerate potentially ambiguous terms; read the original request and explicit lists of questions with resolved and unavailable responses; continue while an ambiguity remains open, subject to the budget.

Appendix

Table 9: Clarification strategies. All prompts request at most one clarification question per turn.

Panel of Figure 7 — Figure 7: Asking-policy training with a multi-turn GRPO variant with DAPO-style asymmetric clipping and zero-advantage group filtering. The main question-F1 reward $R_{\mathrm{F1}}$ balances productive questions and ambiguity coverage; every retained asking decision receives its trajectory’s normalized advantage. Three trajectories illustrate the update; training uses $K=16$ .

Setting	Value
Initialization / backend	Qwen3-8B / FSDP
Optimizer	AdamW, $(\beta_{1},\beta_{2})=(0.9,0.98)$ , $\epsilon=10^{-8}$
Learning rate / schedule	Peak $5\times 10^{-5}$ ; cosine decay to 0, 5% linear warmup
Schedule horizon	136 optimizer updates, estimated from four decisions per trajectory
Weight decay / gradient clipping	0 / 1.0
LoRA rank / alpha / dropout	32 / 64 / 0 (adapter scaling $\alpha/r=2$ )

Appendix

Table 11: Asker-training hyperparameters. Oracle and reward choices vary as specified in Section D.3 .

Figure 8: Learning curves through iteration 48 for the main deterministic-oracle configuration with question-F1 reward

R_{\mathrm{F1}}

: (a) training reward, (b) native AmbiTab-Bird development recall, and (c) SR. Faint and solid lines show mean trajectory rewards per rollout batch and their smoothed trend; rings mark the selected checkpoint at iteration 40. The blue band in (a) extends from the maximum smoothed reward down by two sample standard deviations of the per-batch mean rewards. Dotted lines show the untrained development-only inference mean across five seeds, using the deterministic-oracle runs underlying Table 8 . Native and reference evaluations use different protocols and solvers.

Figure 9: The main policy trained with question-F1 reward and the oracle/reward ablations through iteration 48: native AmbiTab-Bird development checkpoints relative to the five-seed untrained inference reference. (a-c) Recall, precision, and SR differences in percentage points. Rings mark the designated evaluation checkpoints, including iteration 16 for outcome reward with the deterministic oracle. Gray shading shows

\pm

one sample standard deviation across the five development-only reference seeds from the deterministic-oracle runs underlying Table 8 . Colored row bands span

[\max-2\,\mathrm{SD},\max]

, using each row’s maximum trained-checkpoint value and the corresponding reference SD. Trained checkpoints use native validation with the Qwen3-8B solver; the reference uses the common inference protocol with the fixed Coder-30B solver. (d) Fraction of logged request groups with nonconstant trajectory rewards before group filtering, pooling counts over the same 48 rollout batches for each configuration.

Figure 10: Outcome-reward training dynamics on AmbiTab-BIRD-Interact through iteration 48. Training recall and precision decline over the window, and binary rewards provide limited within-request contrast.

Policy	SR	Precision	Recall	F1
Question-F1 reward (untrained)	54.3	32.2	70.4	43.7
Question-F1 reward	60.6	36.8	83.3	50.6
Outcome reward (deterministic oracle) (untrained)	55.9	31.0	68.1	42.2
Outcome reward (deterministic oracle)	57.5	33.3	73.0	45.3
Outcome reward (LLM oracle) (untrained)	52.8	30.6	68.1	41.9
Outcome reward (LLM oracle)	57.5	31.8	70.2	43.4

Appendix

Table 12: Native validation before training and at the selected checkpoint, using the adapted Qwen3-8B solver. Rates are percentages. All rows use AmbiTab-Bird dev (

n=127

). Question-F1 reward is the main configuration, trained with the deterministic oracle; the remaining trained policies are ablations. Selection and oracle settings are given in Sections D.3 and D.5 .

SQL success
Policy	AmbiTab- Bird	AmbiTab- EW-Bird	AmbiTab- BIRD-Interact	AmbiTab- Ambrosia	AmbiTab- AmbiQT	AmbiTab- TACO	Mean
Untrained	59.5 $\pm$ 1.4	69.0 $\pm$ 0.8	9.7 $\pm$ 0.3	35.0 $\pm$ 0.2	56.3 $\pm$ 0.7	72.8 $\pm$ 1.5	50.4 $\pm$ 0.4
Question-F1 reward	71.9 $\pm$ 0.5	72.2 $\pm$ 1.0	9.7 $\pm$ 0.3	37.3 $\pm$ 0.2	59.4 $\pm$ 0.3	77.0 $\pm$ 1.4	54.6 $\pm$ 0.2
Outcome reward (deterministic oracle)	62.7 $\pm$ 1.2	71.1 $\pm$ 1.6	9.8 $\pm$ 0.1	35.5 $\pm$ 0.7	56.8 $\pm$ 0.2	74.3 $\pm$ 1.9	51.7 $\pm$ 0.4
Outcome reward (LLM oracle)	64.3 $\pm$ 1.2	71.4 $\pm$ 1.6	9.7 $\pm$ 0.6	35.9 $\pm$ 0.6	56.8 $\pm$ 0.5	73.4 $\pm$ 0.8	51.9 $\pm$ 0.6
Precision

Appendix

Table 13: Training and transfer with a fixed Coder-30B solver and deterministic oracle. Question-F1 reward is the main configuration, trained with the deterministic oracle; the remaining trained policies are ablations. Question-F1 reward uses iteration 40; Outcome reward (deterministic oracle) uses iteration 16; Outcome reward (LLM oracle) uses iteration 40. Selected trained policies use all three evaluation seeds; the untrained baseline retains five seeds. Rates are percentages, reported as means with sample SD in subscripts. Mean equally weights all six families within each seed, using only seeds complete across all six. Bold values mark the highest mean in each column within each metric, including ties. AmbiTab-Bird and AmbiTab-EW-Bird use dev; AmbiTab-AmbiQT uses column synonyms. Dashes denote unavailable results. Protocol details appear in Sections D.3 and D.5 .

Oracle	Dataset	$\Delta J_{\mathrm{resp}}$	$\Delta\mathrm{Rec}_{\mathrm{spec}}^{Q}$	$\Delta J_{\mathrm{full}}$	$\Delta\mathrm{Rec}_{\mathrm{none}}$
Qwen3-14B	AmbiTab-Bird dev	$3.94\pm 1.36$	$0.00\pm 0.00$	$2.36\pm 4.09$	$7.84\pm 0.97$
	AmbiTab-EW-Bird dev	$\mathbf{3.36\pm 0.22}$	$0.22\pm 0.00$	$2.77\pm 2.15$	$0.82\pm 0.26$
	AmbiTab-BIRD-Interact	$-0.44\pm 0.67$	$0.14\pm 0.05$	$-0.11\pm 0.95$	$1.88\pm 0.10$
	AmbiTab-Ambrosia	$4.00\pm 1.68$	$7.24\pm 0.57$	$2.93\pm 0.94$	$54.87\pm 0.56$
	AmbiTab-AmbiQT	$1.61\pm 0.47$	$9.85\pm 0.29$	$-0.31\pm 0.23$	$0.35\pm 0.10$
	AmbiTab-TACO	$\mathbf{6.95\pm 1.93}$	$0.71\pm 0.56$	$\mathbf{1.07\pm 1.07}$	$20.50\pm 1.11$

Appendix

Table 14: Leakage diagnostics of the untrained asker, mean

\pm

SD over three seeds. AmbiTab-Bird and AmbiTab-EW-Bird are development splits, and AmbiTab-Ambrosia has 534 records. Bold marks the larger of the two LLM-oracle values. For the deterministic oracle, which is the reference,

\Delta J_{\mathrm{resp}}

\Delta J_{\mathrm{full}}

, and

\Delta\mathrm{Rec}_{\mathrm{none}}

are 0 by definition, and

\Delta\mathrm{Rec}_{\mathrm{spec}}^{Q}

is the reference recall of the specification-randomized policy.

			Difference, reference SQL shown minus hidden
Oracle	Dataset	New run	$\Delta J_{\mathrm{resp}}$	$\Delta J_{\mathrm{full}}$	$\Delta\mathrm{Rec}_{\mathrm{none}}$
Qwen3-14B	AmbiTab-Bird dev	SQL hidden	3.15 [ $-0.26$ , 6.56]	0.52 [ $-2.36$ , 3.67]	0.28 [0.00, 1.12]
	AmbiTab-EW-Bird dev	SQL hidden	3.96 [1.42, 6.35]	2.84 [0.22, 5.61]	0.00 [ $-0.22$ , 0.22]
	AmbiTab-Ambrosia	SQL shown	2.87 [0.88, 4.95]	4.99 [3.00, 7.02]	0.25 [ $-0.96$ , 1.52]
	AmbiTab-AmbiQT	SQL shown	0.81 [ $-0.08$ , 1.72]	1.24 [0.43, 1.96]	$-0.14$ [ $-0.29$ , $-0.02$ ]
	AmbiTab-TACO	SQL shown	$-0.18$ [ $-3.03$ , 2.50]	$-0.36$ [ $-2.50$ , 1.96]	$-2.50$ [ $-5.35$ , 0.00]

Appendix

Table 15: Change in the leakage diagnostics when the LLM responder reads the full reference SQL, shown minus hidden, three seeds pooled, with paired 95% AVT-cluster bootstrap intervals. AmbiTab-Bird and AmbiTab-EW-Bird are development splits, and AmbiTab-Ambrosia is a 534-record. New run: the condition added for this study.

		AmbiTab-Bird dev		AmbiTab-EW-Bird dev		AmbiTab-Ambrosia		AmbiTab-AmbiQT		AmbiTab-TACO
Oracle	Asking policy	$\Delta J_{\mathrm{resp}}$	$\Delta\mathrm{Rec}_{\mathrm{spec}}^{Q}$	$\Delta J_{\mathrm{resp}}$	$\Delta\mathrm{Rec}_{\mathrm{spec}}^{Q}$	$\Delta J_{\mathrm{resp}}$	$\Delta\mathrm{Rec}_{\mathrm{spec}}^{Q}$	$\Delta J_{\mathrm{resp}}$	$\Delta\mathrm{Rec}_{\mathrm{spec}}^{Q}$	$\Delta J_{\mathrm{resp}}$	$\Delta\mathrm{Rec}_{\mathrm{spec}}^{Q}$
Deterministic	Untrained	0	$3.50\pm 0.24$	0	$2.91\pm 0.29$	0	$0.50\pm 0.29$	0	$13.09\pm 0.32$	0	$0.98\pm 0.62$
	Question-F1 reward	0	$4.76\pm 0.24$	0	$4.11\pm 0.23$	0	$0.94\pm 0.00$	0	$16.80\pm 0.15$	0	$0.98\pm 0.31$
	Outcome reward (Qwen3-14B oracle)	0	$3.64\pm 0.24$	0	$3.20\pm 0.43$	0	$0.62\pm 0.11$	0	$12.54\pm 0.21$	0	$0.62\pm 0.31$
Qwen3-14B	Untrained	$3.94\pm 1.36$	$0.00\pm 0.00$	$3.36\pm 0.22$	$0.22\pm 0.00$	$4.00\pm 1.68$	$7.24\pm 0.57$	$1.61\pm 0.47$	$9.85\pm 0.29$	$6.95\pm 1.93$	$0.71\pm 0.56$
	Question-F1 reward	$3.41\pm 3.28$	$0.14\pm 0.24$	$1.94\pm 3.19$	$0.22\pm 0.00$	$2.87\pm 0.84$	$7.68\pm 0.32$	$0.79\pm 0.82$	$9.66\pm 0.54$	$4.99\pm 1.11$	$1.52\pm 0.15$

Appendix

Table 16: Leakage diagnostics of two trained asking policies and the untrained asker, mean

\pm

SD over three seeds. Under the deterministic oracle,

\Delta J_{\mathrm{resp}}

is 0 by definition and

\Delta\mathrm{Rec}_{\mathrm{spec}}^{Q}

is the reference recall.

Oracle	Diagnostic	AmbiTab-Bird dev	AmbiTab-EW-Bird dev	AmbiTab-Ambrosia	AmbiTab-AmbiQT	AmbiTab-TACO
Qwen3-14B	$\Delta J_{\mathrm{resp}}$	3.94 [ $-2.36$ , 10.24]	$\mathbf{1.72}$ [ $-1.87$ , 5.23]	2.18 [ $-0.86$ , 5.08]	0.87 [ $-0.48$ , 2.19]	0.71 [ $-2.85$ , 4.10]
	$\Delta\mathrm{Rec}_{\mathrm{spec}}^{Q}$	$-0.14$ [ $-0.60$ , 0.00]	0.00 [0.00, 0.00]	$-0.12$ [ $-0.72$ , $-0.07$ ]	1.18 [ $-2.05$ , 3.51]	$-0.18$ [ $-1.85$ , 0.22]
Gemma-3-27B	$\Delta J_{\mathrm{resp}}$	1.31 [ $-4.99$ , 6.82]	2.77 [ $-0.82$ , 6.21]	0.25 [ $-2.75$ , 3.17]	2.65 [1.32, 4.03]	$-0.36$ [ $-3.92$ , 3.21]
	$\Delta\mathrm{Rec}_{\mathrm{spec}}^{Q}$	1.82 [0.44, 3.81]	0.07 [ $-0.52$ , 0.25]	$-5.43$ [ $-6.42$ , 5.07]	$-8.87$ [ $-12.49$ , $-7.27$ ]	$-3.92$ [ $-16.67$ , $-0.88$ ]

Appendix

Table 17: Outcome reward (Qwen3-14B oracle) minus question-F1 reward, paired by task and seed, three seeds pooled, with 95% AVT-cluster bootstrap intervals. Bold marks the primary comparison, fixed before the runs.

Explore similar work

May 10, 2026cs.AI

Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents

As data-science agents shift from co-pilots to auto-pilots, silent misframing becomes a critical failure mode. Agents quietly commit to plausible but unintended task framings, producing clean, executable artifacts that hide their incorrect assessment of the task. Existing benchmarks score whether the pipeline runs, ignoring whether the agent recognized the task was underspecified. We introduce Ambig-DS, two diagnostic suites: one for prediction-target ambiguity (Ambig-DS-Target, 51 tasks built on DSBench, a tabular modeling benchmark) and one for evaluation-objective ambiguity (Ambig-DS-Objective, 61 tasks built on MLE-bench, a Kaggle-style ML competition benchmark), constructed so that scoring uses each source benchmark's original evaluator. For every task we pair the original, fully specified version with an ambiguous variant produced by controlled edits; a human-and-LLM verification pipeline confirms each variant admits multiple plausible interpretations with decision-relevant consequences. The suites are analyzed independently and ambiguity lowers performance in both. Across five agents spanning efficient to frontier-class models, we find in our controlled diagnostic setting: (i) failures are silent commitments: wrong-target submissions on Target, wrong-metric or non-committal baseline submissions on Objective, rather than execution errors; (ii) allowing the agent to ask one clarifying question recovers much of the loss under idealized conditions, suggesting missing framing information drives a substantial part of the observed degradation; but (iii) agents cannot reliably tell when to use it: permissive prompts induce over-asking on clear tasks, while conservative prompts induce silent defaulting on ambiguous ones. Recognizing target and objective underspecification, not pipeline execution, is the bottleneck missing from standard DS-agent evaluations.

Josefa Lia Stoisser, Marc Boubnovski Martell, Sidsel Boldsen +2

Novo Nordisk London, UK

LLM Agent Evaluation AI Agent Benchmarks LLM Agents Data Analysis Agents

Ask PDF

Oct 7, 2026cs.CL

ARCS: Towards Precise Text-to-SQL via Structured Disambiguation

As text-to-SQL systems move beyond demonstrations toward real-world deployment, ambiguity in user questions becomes a primary source of errors. Such ambiguities are often subtle, domain- or data-specific, and can silently cause system outputs to deviate from the user's true intent. Ambiguity is traditionally addressed through conversational clarification, which is often inefficient, cognitively demanding, and poorly aligned with real-world user workflows. We propose structured disambiguation, a new paradigm in which ambiguity is resolved through explicit, constrained interactions rather than free-form dialogue. We construct ARCS (Ambiguity Resolution Corpus for SQL), the first text-to-SQL benchmark featuring naturally occurring, unconstrained ambiguities over real-world databases, with complete annotations of all valid ambiguity points, interpretations, and SQL queries. Experimental results show that text-to-SQL remains challenging in the presence of ambiguity: gpt-6-sol achieves only 51% end-to-end execution accuracy, and no open-source model exceeds 27%.

Yihao Hu, Yanlin Feng, Naoki Otani +1

Duke University · Megagon Labs

Text-to-SQL

Ask PDF

Apr 24, 2026cs.CL

CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems

NL2SQL systems deployed in industry settings often encounter ambiguous or unanswerable queries, particularly in interactive scenarios with incomplete user clarification. Existing benchmarks typically assume a single source of ambiguity and rely on user interaction for resolution, overlooking realistic failure modes. We introduce Clarity, a framework for automatically generating an NL2SQL benchmark with multi-faceted ambiguities and diverse user behaviors across both single- and multi-turn settings. Using a constraint-driven pipeline, Clarity transforms executable SQL into ambiguous queries, augmented with grounded conversational continuations and schema-level metadata. Empirical evaluation on Spider and BIRD shows that leading NL2SQL systems, including those based on strong LLMs, suffer significant performance degradation under multi-faceted ambiguity. While these systems often detect ambiguity, they struggle to accurately localize and resolve the underlying schema-level sources. Our results highlight the need for more robust ambiguity detection and resolution in industry-grade NL2SQL systems.

Tabinda Sarwar, Farhad Moghimifar, Cong Duy Vu Hoang +6

Oracle Corporation

Question Answering Text-to-SQL

Ask PDF