TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification

Neelesh Kumar Shukla; Debasmita Panda; Srutanik Bhaduri; Aditya Banerjee; Vasu Rangarajan; Viji Krishnamurthy

cs.CLAug 18, 2026

TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification

Authors: Neelesh Kumar Shukla, Debasmita Panda, Srutanik Bhaduri, Aditya Banerjee, Vasu Rangarajan, Viji Krishnamurthy

Organizations: Oracle Corporation India · Oracle Corporation United States

Abstract

Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at inference time in real-world deployments. This creates a critical verification problem: given only a user question, database context, and generated SQL, can a system estimate whether the generated query is likely to correctly answer the question? Recent approaches use LLMs as judge or specialized agents to inspect generated SQL, but their decisions can be difficult to trace. Outcome Reward Models (ORMs) address this by learning from execution-labeled candidate SQLs and assigning correctness scores to unseen queries, yet they still provide limited visibility into the signals behind each verification. To address this limitation, we propose TraceSQL, a lightweight and traceable verification model built on explicit diagnostic features. TraceSQL combines 67 features capturing question ambiguity, question requirements, question-schema-SQL consistency, SQL structure, and intent alignment. These signals remain available for examining which factors influence each prediction and for tracing decisions back to diagnostic evidence. On BIRD development databases, TraceSQL achieves 66.47% F1 and 64.48% ROC-AUC, compared with 61.87% F1 and 58.26% ROC-AUC for the GradeSQL-7B ORM baseline on the same generated-SQL evaluation. Feature attribution further shows that the model relies on both semantic grounding and deterministic SQL-structure signals. These results show that SQL verification can be performed with a lightweight learned model while retaining feature-level evidence for inspecting and diagnosing its predictions.

Figures & tables

Figure 1. TraceSQL feature-construction pipeline. Diagnostic evidence is transformed into five feature families through LLM-based probe assessment, direct projection of Pair Analyzer rule statuses, and deterministic SQL-AST extraction using SQLGlot. The resulting $10+5+32+10+10=67$ features are aligned at the candidate level and merged into the unified model representation, while richer diagnostic evidence is retained separately for traceability.

Feature Family	#	Representative Features	Captured Evidence
Ambiguity	10	Ambiguity Overall Status; Metric Clarity; Time-Scope Clarity	Whether the question is sufficiently specified with respect to the requested metric, entities, temporal scope, constraints, references, and analytical intent.
Question Planning	5	Question Planner: Analytical Operation; Question Planner: Output Structure; Question Planner: Grouping	Whether the candidate SQL reflects the analytical operation, expected output structure, grouping, ordering, and SQL operations required by the question.
Pair Analysis	32	Join-Key Support; Data-Validity Rule Support; Additional Grounding Check	Whether tables, columns, joins, identifiers, data types, temporal semantics, metrics, validity rules, and business concepts are grounded in the available database context.
SQL Structure	10	SQL Aggregation Usage; SQL LIMIT Usage; SQL DISTINCT Usage	Deterministic structural properties of the candidate SQL extracted from its SQLGlot abstract syntax tree.
Intent Alignment	10	Intent Alignment: Evaluation Score; Intent Alignment: Grouping; Intent Alignment: Business Term	Whether the candidate SQL aligns with the requested metric, filters, scope, grouping, ranking, and business terminology.
Total	67

Table 1. Overview of the five TraceSQL diagnostic feature families and representative signals captured by each family.

Feature	Perm.	Mean $\|\mathrm{SHAP}\|$
SQL DISTINCT Usage	0.0344	0.0074
SQL LIMIT Usage	0.0335	0.0147
SQL Aggregation Usage	0.0327	0.0073
Additional Grounding Check	0.0317	0.0199
Pattern-Based Business Meaning Support	0.0219	0.0117
Data-Validity Rule Support	0.0165	0.0070

Table 2. Leading features by permutation importance in the selected TraceSQL model. Permutation values are ROC-AUC decreases after shuffling; SHAP values are mean absolute contributions.

Model	Acc.	Prec.	Rec.	F1	AUC
GradeSQL-7B	57.46	63.64	60.21	61.87	58.26
TraceSQL	62.46	68.11	64.91	66.47	64.48

Table 3. Overall generated-SQL performance on 1,521 candidates from 11 held-out BIRD development databases. Bold indicates the better result for each metric.

Database	GradeSQL-7B					TraceSQL
	Acc.	Prec.	Rec.	F1	AUC	Acc.	Prec.	Rec.	F1	AUC
California Schools	45.45	64.86	40.68	50.00	58.33	51.14	69.05	49.15	57.43	56.25
Card Games	55.26	50.47	62.79	55.96	53.98	61.05	56.82	58.14	57.47	64.73
Codebase Community	61.83	73.45	66.94	70.04	59.31	65.59	75.00	72.58	73.77	66.03
Debit Card Specializing	48.33	44.19	73.08	55.07	46.27	60.00	55.00	42.31	47.83	60.35
European Football 2	57.14	71.43	55.56	62.50	59.48	61.90	75.38	60.49	67.12	69.36

Table 4. Database-level generated-SQL performance. Bold indicates the better result for each metric within a database.

Model	Acc./Rec.	Prec.	F1	AUC
GradeSQL-7B	62.84	100.00	77.18	N/A
TraceSQL	60.17	100.00	75.13	N/A

Table 5. Verification performance on 1,534 ground-truth SQL queries from the BIRD development databases. Since all instances belong to the positive class, Accuracy equals Recall, Precision is 100%, and ROC-AUC is undefined.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Feature

Versioning Rule Support

Name and Alias Support

Effective Date Meaning

User Context Mapping Support

User Context Resolution Support

Appendix

Table 6. Pair Analysis input features not retained after FLAML preprocessing and feature transformation.

Feature family	Count	Extraction
Ambiguity	10	Question-level ambiguity status, probability, and eight probe verdicts.
Question Planning	5	Probe verdicts comparing planned question requirements with candidate SQL support.
Pair Analysis	32	Direct projection of canonical Pair Analyzer rule statuses.
SQL Structure	10	Deterministic binary indicators extracted from the SQL AST.
Intent Alignment	10	Four evaluator fields and six probe-based alignment verdicts.
Total	67	Fixed candidate-level representation supplied to FLAML.

Appendix

Table 7. TraceSQL feature families.

Probe ID	Feature Name	Meaning
–	Ambiguity_Overall_Status	Overall Boolean assessment indicating whether the user question is considered ambiguous.
–	Ambiguity_Assessment_Probability	Estimated degree of ambiguity assigned by the Ambiguity Detector on a 0–100 scale.
AQ001	Ambiguity_Metric_Clarity	Captures uncertainty concerning the metric, measure, quantitative value, or derived calculation requested by the question.
AQ002	Ambiguity_Entity_Clarity	Captures uncertainty concerning the entity, subject, record, or target referenced by the question.
AQ003	Ambiguity_Time_Scope_Clarity	Captures uncertainty concerning the time period or temporal scope required by the question.
AQ004	Ambiguity_Constraint_Clarity	Captures uncertainty concerning filters, thresholds, limits, or other logical constraints expressed in the question.

Appendix

Table 8. Ambiguity features used in TraceSQL.

Probe ID	Feature Name	Meaning
QP001	Question_Planner_Analytical_Operation_Support	Whether the candidate SQL supports the analytical operation required by the question, such as lookup, aggregation, comparison, ranking, or trend analysis.
QP002	Question_Planner_Output_Structure_Support	Whether the candidate SQL supports the expected output columns, answer structure, and result granularity required by the question.
QP003	Question_Planner_Grouping_Support	Whether the candidate SQL supports the grouping requirements and aggregation dimensions required by the question.
QP004	Question_Planner_Ordering_or_Ranking_Support	Whether the candidate SQL supports ordering, ranking, top- $k$ , sort direction, or extremum requirements expressed by the question.
QP005	Question_Planner_SQL_Operations_Support	Whether the candidate SQL contains the operations required to answer the question, such as joins, filters, aggregation, window functions, DISTINCT , or set operations.

Appendix

Table 9. Question-planning features used in TraceSQL.

Rule ID	Feature Name	Meaning
schema_reference _verification	Pair_Analyzer_Schema_Reference_Support	Whether referenced schemas are present and supported by the catalog.
table_reference _verification	Pair_Analyzer_Table_Reference_Support	Whether referenced tables are present and supported by the catalog.
column_reference _verification	Pair_Analyzer_Column_Reference_Support	Whether referenced columns are present and supported by the catalog.
schema_scope_description _verification	Pair_Analyzer_Schema_Scope_Support	Whether schema-level descriptions support the question and SQL intent.
table_scope_description _verification	Pair_Analyzer_Table_Scope_Support	Whether table descriptions support the question and SQL intent.
column_scope_description _verification	Pair_Analyzer_Column_Scope_Support	Whether column descriptions support the question and SQL intent.

Appendix

Table 10. Pair-analysis features used in TraceSQL (Part I).

Rule ID	Feature Name	Meaning
temporal_fields _verification	Pair_Analyzer_Date_Time_Field_Distinction	Whether similar date/time fields are clearly distinguished so that the intended field can be identified.
temporal_queries _verification	Pair_Analyzer_Temporal_Analysis_Support	Whether temporal metadata supports historical, period-based, fiscal, or as-of analysis requested by the question.
timezone_semantics _verification	Pair_Analyzer_Timezone_Handling_Support	Whether timezone handling is documented sufficiently to interpret operational timestamps correctly.
effective_date_semantics _verification	Pair_Analyzer_Effective_Date_Meaning	Whether effective dates and validity periods clearly indicate when records become active, expire, or apply.
metric_definition _verification	Pair_Analyzer_Metric_Definition_Support	Whether a metric is documented sufficiently to validate the metric requested by the question.
metric_aggregation_logic _verification	Pair_Analyzer_Metric_Calculation_Support	Whether aggregations, ratios, and rollups follow the documented meaning of the metric.

Appendix

Table 11. Pair-analysis features used in TraceSQL (Part II).

Feature Name	Meaning
SQL Aggregation Usage	Whether the candidate contains an aggregate operation.
SQL Grouping Usage	Whether the candidate contains a GROUP BY operation.
SQL Join Usage	Whether the candidate joins multiple relations.
SQL Filtering Usage	Whether the candidate contains WHERE or HAVING filtering.
SQL Temporal Filter Usage	Whether the candidate contains a temporal/date-related filter.
SQL ORDER BY Usage	Whether the candidate contains an ORDER BY clause.

Appendix

Table 12. Deterministic SQL-structure features used in TraceSQL.

Feature Name	Meaning
Intent Alignment Overall Score	Reference-free evaluator score for overall alignment between the question and candidate SQL.
Intent Alignment Confidence	Confidence associated with the reference-free alignment evaluation.
Intent Alignment Gate Decision	Evaluator gate decision, such as accepting or requesting intervention.
Intent Alignment Intervention Reason	Categorized reason for intervention when the evaluator does not accept the candidate.
Intent Alignment: Requested Metric	Whether the metric or measure requested by the question is aligned with the candidate SQL.
Intent Alignment: Filter Conditions	Whether filtering constraints in the question are represented by the candidate SQL.

Appendix

Table 13. Intent-alignment features used in TraceSQL.

Figure 2. Mean absolute SHAP importance for the fifteen most influential TraceSQL features. Larger values indicate greater average contribution to the fitted model output. Horizontal bar chart showing the fifteen TraceSQL features with the largest mean absolute SHAP values.

Feature	SHAP	Perm.	Native
Additional Grounding Check	1	4	2
SQL LIMIT Usage	2	2	5
Pattern-Based Business Meaning Support	3	5	3
SQL DISTINCT Usage	4	1	14
SQL Aggregation Usage	5	3	1
Data-Validity Rule Support	6	6	4

Appendix

Table 14. Cross-method agreement among the leading TraceSQL features. Lower rank indicates greater importance within each method.

Feature Family	SHAP	Perm.	Native
Ambiguity	0	0	0
Question Planning	1	1	1
Pair Analysis	3	3	4
SQL Structure	4	5	3
Intent Alignment	2	1	2

Appendix

Table 15. Feature-family representation among the top ten features under each importance method.

Figure 3. Pairwise correlation matrix for the numeric TraceSQL features. Correlation heatmap for ambiguity probability, SQL-structure indicators, and intent-alignment score and confidence.

Figure 4. Local SHAP explanation for the representative false-positive verification decision. Local feature-attribution plot for a false-positive TraceSQL prediction on a Formula 1 question about the 2017 Chinese Grand Prix.

Database	GradeSQL-7B	TraceSQL
California Schools	41.57	67.42
Card Games	59.16	43.98
Codebase Community	65.59	66.13
Debit Card Specializing	68.75	29.69
European Football 2	65.12	62.79
Financial	60.38	75.47

Appendix

Table 16. Database-level acceptance rate (%) on the 1,534 ground-truth SQL queries. Because all instances belong to the positive class, acceptance rate is equivalent to accuracy and recall. Bold indicates the higher result.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 169 papers

Test-Time Verification for Text-to-SQL via Outcome Reward Models
Jun 29, 2026Mattia Tritto, Giuseppe Farano, Dario Di Palma +4Reward Modeling LLM Answer Verification
Polytechnic University of Bari, Bari, Italy · IBM T.J. Watson Research Center, Yorktown Heights, NY, USA
What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study
Jul 7, 2026Robert RichardsonLLM Answer Verification Selective Prediction
Affiliation
PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agents
Apr 19, 2026Yuan Tian, Tianyi ZhangText-to-SQL
Purdue University

Jun 29, 2026J/K move · Enter open · S save

Test-Time Verification for Text-to-SQL via Outcome Reward Models

Mattia Tritto, Giuseppe Farano, Dario Di Palma, Gaetano Rossiello+3

Polytechnic University of Bari, Bari, Italy · IBM T.J. Watson Research Center, Yorktown Heights, NY, USA

Reward Modeling LLM Answer Verification Test-Time Scaling Text-to-SQL

Open paper AskPDF arXiv

Improving the reliability of large language models (LLMs) at inference time is a central challenge in structured reasoning tasks such as Text-to-SQL. Common test-time inference strategies, including Best-of-N sampling and Majority Voting, rely on heuristic signals such as execution success or output frequency, which provide limited semantic discrimination across candidate outputs. In this work, we study Outcome Reward Models (ORMs) as learned semantic scoring functions for test-time verification in Text-to-SQL. While ORMs have been previously explored for test-time scaling and alignment, their application to structured query generation remains underexplored. We introduce GradeSQL, a scalable framework for training task-specific ORMs via automated candidate generation and execution-based labeling, enabling verifier training without manual annotation. We integrate ORMs into a verification-driven Best-of-N pipeline and evaluate our approach on the BIRD and Spider benchmarks across multiple open-source LLM families. ORM-based selection consistently outperforms execution-based Best-of-N and Majority Voting, with gains of up to +4.33% on BIRD and +2.10% on Spider. We further show that ORMs scale effectively with larger candidate sets and yield stronger improvements on complex queries. Overall, our results demonstrate that ORM-based verification provides a simple, effective, and scalable alternative to heuristic test-time selection strategies for Text-to-SQL. Code datasets and models are publicly available.

Abstract

Text-to-SQL

Explore similar work

Test-Time Verification for Text-to-SQL via Outcome Reward Models

What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agents