LLM NL2SQL Robustness: Surface Noise vs. Linguistic Variation in Traditional and Agentic Settings

Lifu Tu; Rongguang Wang; Tao Sheng; Sujjith Ravi; Dan Roth

cs.CLMar 17, 2026

LLM NL2SQL Robustness: Surface Noise vs. Linguistic Variation in Traditional and Agentic Settings

Authors: Lifu Tu, Rongguang Wang, Tao Sheng, Sujjith Ravi, Dan Roth

Organizations: Oracle AI

Abstract

Robustness evaluation for Natural Language to SQL (NL2SQL) systems is essential because real-world database environments are dynamic, noisy, and continuously evolving, whereas conventional benchmark evaluations typically assume static schemas and well-formed user inputs. In this work, we introduce a robustness evaluation benchmark containing approximately ten types of perturbations and conduct evaluations under both traditional and agentic settings. We assess multiple state-of-the-art large language models (LLMs), including Grok-4.1, Gemini-3-Pro, Claude-Opus-4.6, and GPT-5.2. Our results show that these models generally maintain strong performance under several perturbations; however, notable performance degradation is observed for surface-level noise (e.g., character-level corruption) and linguistic variation that preserves semantics while altering lexical or syntactic forms. Furthermore, we observe that surface-level noise causes larger performance drops in traditional pipelines, whereas linguistic variation presents greater challenges in agentic settings. These findings highlight the remaining challenges in achieving robust NL2SQL systems, particularly in handling linguistic variability.

Figures & tables

Perturbations	Query
origin	In which year were the two most common causes of traffic accidents different from those in other years?
BackTranslation (bt)	In which year were the two most common causes of traffic accidents different from other years?
Chinese (zh)	在哪一年，交通事故最常见的两个原因与其他年份不同?
Past	In which year was the two most common causes of traffic accidents different from those in other years?
Future	In which year will be the two most common causes of traffic accidents different from those in other years?
InflectionalVariation	In which year been the two most commoner causes of traffic accident different from those in other year?

Table 1: Perturbations of a user query: the last two blocks present linguistic variation (preserves semantics while altering lexical or syntactic forms) and surface-level noise (e.g., character-level corruption).

Figure 1: Results on R-NL2SQL-Tradition (adapt from Spider 1 ( Yu et al., 2018 ) ). LLMs generally maintain strong performance under several perturbations; however, notable performance degradation is still observed.

Figure 2: Results for R-NL2SQL-Agentic (adapted from Spider 2 ( Lei et al., 2024 ) ). Using the Spider-Agent framework ( Lei et al., 2024 ) (following ReAct ( Yao et al., 2023 ) )for evaluation, we substitute different LLMs. Earlier LLMs (e.g., GPT-4.1) achieved very low results. Traditional pipelines degrade more under surface-level noises, whereas agentic setups are more challenged by linguistic variations.

Figure 3: Execution accuracy of “one-perturbation” and “all-perturbation” in the agentic setting, confirming that linguistic variations larger performance differences.

Figure 4: Execution accuracy when extra databases are added in the agentic setting.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

db	5 database names that are most similar
bank_sales_trading	electronic_sales, E_commerce, Brazilian_E_Commerce, northwind, AdventureWorks
Pagila	sqlite-sakila, chinook, Db-IMDB, imdb_movies, northwind
electronic_sales	E_commerce, Brazilian_E_Commerce, bank_sales_trading, northwind, AdventureWorks
E_commerce	Brazilian_E_Commerce, electronic_sales, bank_sales_trading, northwind, AdventureWorks
Brazilian_E_Commerce	E_commerce, electronic_sales, bank_sales_trading, northwind, AdventureWorks
complex_oracle	oracle_sql, modern_data, log, AdventureWorks, northwind

Appendix

Table 2: Illustrations of the five most similar databases for each database. The similarity are based on judged by LLMs.

Figure 5: Execution accuracy of GPT-4.1 in the agentic setting. The low accuracy indicates that earlier LLMs have much weaker agentic capabilities.

Text-to-SQL

Text-to-SQL translates natural-language questions into executable database queries, linking user intent to schemas, values, and SQL logic.

See all 173 papers

NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions
Apr 13, 2026Shizheng Hou, Wenqi Pei, Nuo Chen +3Text-to-SQL Benchmark Design
National University of Singapore · Zhejiang University
SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks
Apr 20, 2026Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid +2LLM Evaluation Benchmark Contamination
Oracle AI
PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems
Oct 15, 2026Shuo Lin, Jinzhe Li, Chenlu Guo +3Text-to-SQL Text-to-SQL Evaluation
School of Artificial Intelligence, Jilin University · International Center of Future Science, Jilin University · Bilibili Inc. +2

Apr 13, 2026J/K move · Enter open · S save

NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions

Shizheng Hou, Wenqi Pei, Nuo Chen, Quang-Trung Ta+2

National University of Singapore · Zhejiang University

Text-to-SQL Benchmark Design LLM Evaluation LLM Inference Efficiency

Open paper AskPDF arXiv

Natural Language to SQL (NL2SQL) technology empowers non-expert users to query relational databases without requiring SQL expertise. While large language models (LLMs) have greatly improved NL2SQL algorithms, their rapid development outpaces systematic evaluation, leaving a critical gap in understanding their effectiveness, efficiency, and limitations. To this end, we present NL2SQLBench, the first modular evaluation and benchmarking framework for LLM-enabled NL2SQL approaches. Specifically, we dissect NL2SQL systems into three core modules: Schema Selection, Candidate Generation, and Query Revision. For each module, we comprehensively review existing strategies and propose novel fine-grained metrics that systematically quantify module-level effectiveness and efficiency. We further implement these metrics in a flexible multi-agent framework, allowing configurable benchmarking across diverse NL2SQL approaches. Leveraging NL2SQLBench, we rigorously evaluate ten representative open-source methods on two datasets, the BIRD development set and the ScienceBenchmark development set, using two LLMs, DeepSeek-V3 and GPT-4o mini. We systematically assess each approach across the three core modules and evaluate multiple critical performance dimensions. Our evaluation reveals significant gaps in existing NL2SQL methods, highlighting not only substantial room for accuracy improvements but also the significant computational inefficiency, which severely hampers real-world adoption. Furthermore, our analysis identifies critical shortcomings in current benchmark datasets and evaluation rules, emphasizing issues such as inaccurate gold SQL annotations and limitations in existing evaluation rules. By synthesizing these insights into a unified benchmarking, our study establishes a clear reference point for fair comparison and serves as essential guidance for future targeted innovation in NL2SQL technology.

Abstract

Text-to-SQL

Explore similar work

NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions

SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks

PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems