Organizations: Harbin Institute of Technology, Shenzhen, Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, Shenzhen, China · Huawei Hong Kong Research Center, Hongkong, China
Database Management Systems (DBMSs) are fundamental infrastructure for modern data-driven applications, where thorough testing with high-quality SQL test cases is essential for ensuring system reliability. Traditional approaches such as fuzzing can be effective for specific DBMSs, but adapting them to different proprietary dialects requires substantial manual effort. Large Language Models (LLMs) present promising opportunities for automated SQL test generation, but face critical challenges in industrial environments. First, lightweight models are widely used in organizations due to security and privacy constraints, but they struggle to generate syntactically valid queries for proprietary SQL dialects. Second, LLM-generated queries are often semantically similar and exercise only shallow execution paths, thereby quickly reaching a coverage plateau. To address these challenges, we propose MIST, an LLM-based test case generatIon framework for DBMS through Monte Carlo Tree search. MIST consists of two stages: Feature-Guided Error-Driven Test Case Synthetization, which constructs a hierarchical feature tree and uses error feedback to guide LLM generation, aiming to produce syntactically valid and semantically diverse queries for different DBMS dialects, and Monte Carlo Tree Search-Based Test Case Mutation, which jointly optimizes seed query selection and mutation rule application guided by coverage feedback, aiming at boosting code coverage by exploring deeper execution paths. Experiments on three widely-used DBMSs with four lightweight LLMs show that MIST achieves average improvements of 43.3% in line coverage, 32.3% in function coverage, and 46.4% in branch coverage compared to the baseline approach with the highest line coverage of 69.3% in the Optimizer module.
Figures & tables
Figure 1. An illustrative example of using Qwen2.5-7B to generate a SQL test case for DBMS.
Figure 2. Illustrating challenges in LLM-based DBMS test case generation with Qwen2.5-7B.
Figure 3. The overview of MIST.
Figure 4. The prompt template for test case synthetization. [DBMS_TYPE] is the target DBMS, [FEATURE_LIST] contains selected features from the hierarchical tree, and [ERROR_MEMORY] provides feedback from previous execution failures.
Figure 5. Illustrating the MCTS for test case mutation.
DuckDB
PostgreSQL
SQLite
Model
Approach
Line
Function
Branch
Line
Function
Branch
Line
Function
Branch
Fuzz4All
29.9%
29.7%
17.0%
23.1%
25.2%
16.2%
29.9%
38.9%
22.5%
Qwen2.5-7B
MIST
42.0%
38.2%
24.5%
30.3%
38.6%
21.3%
39.7%
46.1%
31.3%
Fuzz4All
32.3%
31.7%
18.3%
24.2%
31.8%
16.9%
28.3%
36.6%
21.1%
Llama3.1-8B
MIST
43.2%
39.3%
25.4%
29.3%
37.7%
20.5%
39.9%
45.2%
32.1%
Fuzz4All
26.4%
26.8%
14.6%
23.6%
31.3%
16.4%
21.6%
30.3%
15.6%
Table 1. Overall code coverage comparison between baseline and MIST.
Model
Approach
Parser
Optimizer
Executor
Storage
DuckDB
Qwen2.5 7B
Fuzz4All
26.6%
51.6%
31.0%
20.0%
MIST
43.2%
68.4%
42.2%
31.3%
Llama3.1 8B
Fuzz4All
30.1%
52.8%
35.1%
22.7%
MIST
48.9%
69.3%
47.3%
32.9%
Qwen2.5 14B
Fuzz4All
24.2%
43.4%
22.9%
22.1%
Table 2. Module-level line coverage comparison between baseline and MIST.
Synthetization
Mutation
DuckDB
PostgreSQL
SQLite
Line
Function
Branch
Line
Function
Branch
Line
Function
Branch
Simple Instruction
✗
29.9%
29.7%
17.0%
23.1%
25.2%
16.2%
29.9%
38.9%
22.5%
Random Feature
✗
32.3%
31.7%
18.7%
25.2%
30.1%
16.9%
32.8%
41.7%
24.6%
Hierarchical Feature (MIST)
✗
36.2%
34.7%
20.6%
27.4%
32.1%
17.8%
35.5%
43.2%
26.3%
Hierarchical Feature (MIST)
Simple Instruction
37.8%
35.7%
21.6%
28.3%
33.9%
18.1%
36.9%
44.2%
27.5%
Hierarchical Feature (MIST)
Random Rule
38.7%
36.7%
22.4%
28.8%
35.4%
19.5%
37.5%
44.8%
29.2%
Table 3. Ablation study results showing the impact of different components in MIST. Each row represents a different combination of synthetization strategy and mutation strategy.
Figure 6. Case study on test case generation using Qwen2.5-7B on SQLite.
Translating natural language to SQL for data retrieval has become more accessible thanks to code generation LLMs. But how hard is it to generate SQL code? While databases can become unbounded in complexity, the complexity of queries is bounded by real life utility and human needs. With a sample of 376 databases, we show that SQL queries, as translations of natural language questions are finite in practical complexity. There is no clear monotonic relationship between increases in database table count and increases in complexity of SQL queries. In their template forms, SQL queries follow a Power Law-like distribution of frequency where 70% of our tested queries can be covered with just 13% of all template types, indicating that the high majority of SQL queries are predictable. This suggests that while LLMs for code generation can be useful, in the domain of database access, they may be operating in a narrow, highly formulaic space where templates could be safer, cheaper, and auditable.
We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based tests. Existing LLM evaluations largely target general-purpose code generation or declarative text-to-SQL, leaving procedural database programming underexplored. PLSQLBench contains 2,865 instances: 2,594 single-turn tasks and 271 multi-turn conversations spanning 978 turns. The benchmark combines complex schema-grounded tasks over enterprise-style Spider 2 databases, simpler schema-grounded tasks derived from Spider, and MBPP-derived procedural problems, covering varying levels of database grounding and procedural complexity. Experiments with eight LLMs reveal recurring difficulties in schema grounding, PL/SQL dialect fidelity, procedural control flow, exception handling, and cross-turn consistency. Tool-augmented LLM agents improve performance on several schema-grounded evaluations, although substantial gaps remain. These results highlight procedural database programming capabilities not directly assessed by conventional code generation or text-to-SQL benchmarks. Our code is available at https://github.com/oracle-samples/plsqlbench.
Marianne Menglin Liu, Leonid Boytsov, Daniel W. Peterson +13
Large Language Models (LLMs) have been increasingly adopted in Text-to-SQL systems, yet SQL errors remain a major obstacle in real-world Text-to-SQL inference pipelines. Existing SQL correction approaches either rely on large-scale, high-quality training data with substantial overhead, or adopt single-path agentic workflows that are brittle to early mistakes and prone to error propagation. To develop a practical SQL correctness system for industrial scenarios, we present a training-free framework that formulates SQL correction as a plan-guided, tree-structured debugging process. By maintaining multiple correction strategies and enabling backtracking, the framework mitigates error accumulation during iterative refinement. We further integrate execution-based verification and clause-level diagnostic tools to support strategy pruning and precise error localization. We evaluate the system on the BIRD-Critic benchmark and observe consistent accuracy gains over strong LLM backbones and representative agent-based baselines, achieving a 9.42% improvement over the previous state-of-the-art method. The framework is also deployed in the Torch Log Service (TLS) of Volcano Engine to support an online Text-to-TLS API. In production, it improves execution accuracy from 36.77% to 53.61% on real user queries with a representative strong LLM backbone (GPT-5). These results demonstrate the effectiveness and stability of our approach in real-world deployments.