ROSE: An Intent-Centered Evaluation Metric for NL2SQL

Wenqi Pei; Shizheng Hou; Boyan Li; Han Chen; Zhichao Shi; Yuyu Luo

cs.DBApr 14, 2026

ROSE: An Intent-Centered Evaluation Metric for NL2SQL

Authors: Wenqi Pei, Shizheng Hou, Boyan Li, Han Chen, Zhichao Shi, Yuyu Luo

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · National University of Singapore

Abstract

Execution Accuracy (EX), the widely used metric for evaluating the effectiveness of Natural Language to SQL (NL2SQL) solutions, is becoming increasingly unreliable. It is sensitive to syntactic variation, ignores that questions may admit multiple interpretations, and is easily misled by erroneous ground-truth SQL. To address this, we introduce ROSE, an intent-centered metric that focuses on whether the predicted SQL answers the question, rather than consistency with the ground-truth SQL under the reference-dependent paradigm. ROSE employs an adversarial Prover-Refuter cascade: SQL Prover assesses the semantic correctness of a predicted SQL against the user's intent independently, while Adversarial Refuter uses the ground-truth SQL as evidence to challenge and refine this judgment. On our expert-aligned validation set ROSE-VEC, ROSE achieves the best agreement with human experts, outperforming the next-best metric by nearly 24% in Cohen's Kappa. We also conduct a largescale re-evaluation of 19 NL2SQL methods, revealing four valuable insights. We release ROSE and ROSE-VEC to facilitate more reliable NL2SQL research.

Figures & tables

Figure 1: ROSE Scoring Cascade. The red path indicates the workflow when execution results do not match, while the blue path is followed when they do.

Backbone	Metric	$\boldsymbol{\kappa}$ (%)	Acc (%)	MCC (%)	F1 (%)
Deterministic	EM	0.51	27.86	5.07	1.86
	ETM	6.60	35.56	18.47	20.63
	EX	25.56	55.90	37.23	57.00
OpenAI o3	LLM-SQL-Solver	12.24	42.05	25.54	33.92
	FLEX	56.70	78.97	62.01	83.31
	ROSE w/o Refuter	60.74	85.47	61.46	90.40

Table 1: Performance of NL2SQL metrics on ROSE-VEC. Within each block, Bold indicates the best metric, Underline indicates the second best. Open-source models are represented with , while closed-source models are represented with . Separate results on the BIRD and Spider splits are in Appendix J.1 .

Label	Model	Count	Precision (%)
GoldX	OpenAI o3	51	84.32
	Gemini-2.5 Pro	76	68.42
	DeepSeek-R1	76	61.84
AmbQ	OpenAI o3	57	91.23
	Gemini-2.5 Pro	63	73.02
	DeepSeek-R1	108	51.85

Table 2: Precision of ROSE diagnostic labelling for GoldX and AmbQ on ROSE-VEC. Separate results on the BIRD and Spider splits are in Appendix J.2 .

Method	Date	Model	Simple		Moderate		Challenge		Overall
Method	Date	Model	EX	ROSE	EX	ROSE	EX	ROSE	EX	ROSE
Prompting Methods
GPT-5	Aug 2025	–	69.86	91.10	51.03	87.24	46.46	89.90	55.74	88.93
Alpha-SQL-32B	Feb 2025	Qwen2.5-Coder	81.02	91.24	69.47	79.65	52.58	70.10	69.35	81.09
OpenSearch-SQL	Sep 2024	DeepSeek-Chat	77.94	91.18	69.78	80.44	56.25	67.71	69.37	80.96
RSL-SQL	Oct 2024	DeepSeek-Chat	74.29	86.43	59.83	71.79	52.04	62.24	62.50	74.15

Table 3: Benchmarking results of NL2SQL methods on BIRD mini-Dev with EX and ROSE

{}_{\textit{o3-2504}}

. Underlined methods denote base models. The score gap is illustrated in Appendix L .

Figure 2: ROSE vs. EX scores for NL2SQL methods grouped by base model.

Figure 3: The growing divergence between ROSE and EX over time for prompting-based systems.

Method	GoldX	AmbQ	GoldX $\cup$ AmbQ	Avg.
OpenSearch-SQL	86.00	58.33	77.14	20.35
Alpha-SQL-32B	90.00	47.62	76.12	19.13
RSL-SQL	84.62	68.18	77.94	18.08
OmniSQL-32B	82.61	60.00	74.60	18.18

Table 4: Discordance rate (%) between EX and ROSE on GoldX, AmbQ.

Figure 4: Share of discordant cases by question type.

Figure 5: Average gap between ROSE and EX across difficulty levels for prompting and fine-tuned methods.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Aspect	EX	LLM-SQL-Solver	FLEX	ROSE
Basis for Judgment	Match	Match	Comparison	Independent and Comparison
Role of Ground-truth SQL	Reference	Reference	Reference	Challenge Evidence
Robust to Ambiguity	No	No	Partially	Yes
Robust to Ground-truth Error	No	No	Partially	Yes
Diagnostic Output	No	No	No	Yes ( AmbQ , GoldX )

Appendix

Table 5: Key differences between our proposed metric, ROSE, and prior evaluation metrics.

Backbone	Metric	$\boldsymbol{\kappa}$ (%)	Acc (%)	MCC (%)	F1 (%)
Deterministic	EX	43.56	69.57	51.36	71.18
Qwen3-235B -A22B-Thinking	FLEX	53.59	76.71	54.89	74.40
Qwen3-235B -A22B-Thinking	ROSE	65.11	83.85	65.71	87.38
Qwen3-30B -A3B-Thinking	FLEX	47.74	72.98	52.78	75.90
Qwen3-30B -A3B-Thinking	ROSE	62.63	81.99	64.59	85.28
DeepSeek-R1-Distill -Qwen-32B	FLEX	43.01	72.36	43.33	66.16

Appendix

Table 6: Performance on ROSE-VEC-BIRD with other open-source backbones.

Backbone	Metric	$\boldsymbol{\kappa}$ (%)	Acc (%)	MCC (%)	F1 (%)
OpenAI o3	Unified w/o GT	53.35	80.43	54.00	86.09
	Unified	66.35	83.85	68.22	86.87
	ROSE w/o GT	71.01	86.34	72.25	89.11
	ROSE	80.68	90.99	81.64	92.91
Gemini-2.5 Pro	Unified w/o GT	45.06	78.88	50.07	86.01
	Unified	59.90	81.06	61.02	84.86

Appendix

Table 7: Ablation studies of ROSE on ROSE-VEC-BIRD. “w/o GT” indicates removing ground-truth supervision.

Threads	Wall-clock Time
1	120min37s
2	64min59s
4	42min41s
8	18min

Appendix

Table 8: Wall-clock time under different threads.

Metric	Avg. Time
EX	1.22s
FLEX ${}_{\textit{o3-2504}}$ (thread-1)	18.19s
ROSE ${}_{\textit{o3-2504}}$ (thread-1)	22.48s
ROSE ${}_{\textit{o3-2504}}$ (thread-8)	3.35s

Appendix

Table 9: Average time on ROSE-VEC-BIRD.

Metric	Total Cost (USD)	Avg. Cost (USD)
FLEX ${}_{\textit{o3-2504}}$	3.819	0.0118
ROSE ${}_{\textit{o3-2504}}$	2.249	0.0070

Appendix

Table 10: Monetary cost on ROSE-VEC-BIRD.

#Calls	#Questions	Percentage (%)
1	176	54.66
2	146	45.34
Avg.	1.45	–

Appendix

Table 11: Number of LLM calls by ROSE

{}_{\textit{o3-2504}}

on ROSE-VEC-BIRD.

Label	Model	Count	Precision (%)
GoldX	OpenAI o3	35	82.86
	Gemini-2.5 Pro	58	63.79
	DeepSeek-R1	59	57.63
AmbQ	OpenAI o3	16	75.00
	Gemini-2.5 Pro	17	35.29
	DeepSeek-R1	61	27.87

Appendix

Table 12: Precision of ROSE diagnostic labelling for GoldX and AmbQ on ROSE-VEC-BIRD.

Backbone	Metric	$\boldsymbol{\kappa}$ (%)	Acc (%)	MCC (%)	F1 (%)
Deterministic	EM	0.00	32.73	0.00	0.00
	ETM	8.60	41.30	21.19	22.22
	EX	43.56	69.57	51.36	71.18
OpenAI o3	LLM-SQL-Solver	10.51	43.65	23.56	25.75
	FLEX	55.20	76.87	60.53	79.18
	ROSE w/o Refuter	66.76	85.67	67.25	89.57

Appendix

Table 13: Performance of NL2SQL metrics on ROSE-VEC-BIRD. Bold indicates the best metric within each LLM block. Underline indicates the second best within each block. Open-source models are represented with , while closed-source models are represented with .

Backbone	Metric	$\boldsymbol{\kappa}$ (%)	Acc (%)	MCC (%)	F1 (%)
Deterministic	EM	0.77	21.59	6.22	3.72
	ETM	4.47	28.41	15.11	18.88
	EX	11.09	39.02	24.23	38.31
OpenAI o3	LLM-SQL-Solver	13.06	41.83	26.43	42.70
	FLEX	57.85	82.13	63.23	87.47
	ROSE w/o Refuter	45.87	84.79	47.29	90.87

Appendix

Table 14: Performance of NL2SQL metrics on ROSE-VEC-Spider. Bold indicates the best metric within each LLM block. Underline indicates the second best within each block. Open-source models are represented with , while closed-source models are represented with .

Label	Model	Count	Precision (%)
GoldX	OpenAI o3	16	87.50
	Gemini-2.5 Pro	18	83.33
	DeepSeek-R1	17	76.47
AmbQ	OpenAI o3	41	97.56
	Gemini-2.5 Pro	46	86.96
	DeepSeek-R1	47	82.98

Appendix

Table 15: Precision of ROSE diagnostic labelling for GoldX and AmbQ on ROSE-VEC-Spider.

Prompting Methods
Method	Model	Simple	Moderate	Challenge	Overall
GPT-5	–	21.24	36.21	43.44	33.19
Alpha-SQL-32B	Qwen2.5-Coder	10.22	10.18	17.52	11.74
OpenSearch-SQL	DeepSeek-Chat	13.24	10.66	11.46	11.59
RSL-SQL	DeepSeek-Chat	12.14	11.96	10.20	11.65
DeepSeek-Chat	–	11.65	23.99	22.78	20.08

Appendix

Table 16: Score gap on BIRD Mini-Dev across difficulty levels. Underlined methods denote base models.

Figure 6: Streamlit interface for inspecting SQL predictions.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 173 papers

LLM NL2SQL Robustness: Surface Noise vs. Linguistic Variation in Traditional and Agentic Settings
Mar 17, 2026Lifu Tu, Rongguang Wang, Tao Sheng +2Text-to-SQL Language Model Robustness
Oracle AI
SOMA-SQL: Resolving Multi-Source Ambiguity in NL-to-SQL via Synthetic Log and Execution Probing
Jun 9, 2026Sai Ashish Somayajula, Marianne Menglin Liu, Chuan Lei +9Execution-Guided Text-to-SQL Text-to-SQL
Oracle AI
AgentNLQ: A General-Purpose Agent for Natural Language to SQL
May 18, 2026Olena Bogdanov, Yeunji Jung, Chandra Dhir +5Multi-Agent LLM Systems LLM Agent Orchestration
JPMorganChase

Mar 17, 2026J/K move · Enter open · S save

LLM NL2SQL Robustness: Surface Noise vs. Linguistic Variation in Traditional and Agentic Settings

Lifu Tu, Rongguang Wang, Tao Sheng, Sujjith Ravi+1

Oracle AI

Text-to-SQL Language Model Robustness LLM Evaluation

Open paper AskPDF arXiv

Robustness evaluation for Natural Language to SQL (NL2SQL) systems is essential because real-world database environments are dynamic, noisy, and continuously evolving, whereas conventional benchmark evaluations typically assume static schemas and well-formed user inputs. In this work, we introduce a robustness evaluation benchmark containing approximately ten types of perturbations and conduct evaluations under both traditional and agentic settings. We assess multiple state-of-the-art large language models (LLMs), including Grok-4.1, Gemini-3-Pro, Claude-Opus-4.6, and GPT-5.2. Our results show that these models generally maintain strong performance under several perturbations; however, notable performance degradation is observed for surface-level noise (e.g., character-level corruption) and linguistic variation that preserves semantics while altering lexical or syntactic forms. Furthermore, we observe that surface-level noise causes larger performance drops in traditional pipelines, whereas linguistic variation presents greater challenges in agentic settings. These findings highlight the remaining challenges in achieving robust NL2SQL systems, particularly in handling linguistic variability.

Abstract

Text-to-SQL

Explore similar work

LLM NL2SQL Robustness: Surface Noise vs. Linguistic Variation in Traditional and Agentic Settings

SOMA-SQL: Resolving Multi-Source Ambiguity in NL-to-SQL via Synthetic Log and Execution Probing

AgentNLQ: A General-Purpose Agent for Natural Language to SQL