PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming

Marianne Menglin Liu; Leonid Boytsov; Daniel W. Peterson; Pramuditha Perera; Rongguang Wang; Sai Ashish Somayajula; Syed Hamza Rafique; Rohit Saini

cs.CLAug 16, 2026

PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming

Authors: Marianne Menglin Liu, Leonid Boytsov, Daniel W. Peterson, Pramuditha Perera, Rongguang Wang, Sai Ashish Somayajula, Syed Hamza Rafique, Rohit Saini,

Organizations: Oracle AI · Turing Enterprise Inc

Abstract

We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based tests. Existing LLM evaluations largely target general-purpose code generation or declarative text-to-SQL, leaving procedural database programming underexplored. PLSQLBench contains 2,865 instances: 2,594 single-turn tasks and 271 multi-turn conversations spanning 978 turns. The benchmark combines complex schema-grounded tasks over enterprise-style Spider 2 databases, simpler schema-grounded tasks derived from Spider, and MBPP-derived procedural problems, covering varying levels of database grounding and procedural complexity. Experiments with eight LLMs reveal recurring difficulties in schema grounding, PL/SQL dialect fidelity, procedural control flow, exception handling, and cross-turn consistency. Tool-augmented LLM agents improve performance on several schema-grounded evaluations, although substantial gaps remain. These results highlight procedural database programming capabilities not directly assessed by conventional code generation or text-to-SQL benchmarks. Our code is available at https://github.com/oracle-samples/plsqlbench.

Figures & tables

Figure 1: Overview of PLSQLBench . PLSQLBench combines PL/SQL artifacts, database schemas, development tasks, and executable tests to evaluate whether LLM systems can generate, modify, debug, and repair procedural programs while preserving conversational context. It contains both single-turn tasks and multi-turn interactive workflows requiring generation, modification, debugging, validation, and repair of executable PL/SQL programs.

Benchmark	Inst.	#DB	#Test (Avg.)
Spider2-ST-dev	407	26	4.9
Spider2-ST-test	103	8	4.7
Spider2-MT-dev	208	26	13.7
Spider2-MT-test	63	7	13.0
Spider-PLSQL	970	20	1.0
MBPP-PLSQL †	806	–	1.0

Table 1: Dataset statistics for PLSQLBench . Inst. denotes the number of single-turn tasks or multi-turn conversations, #DB the number of distinct databases, and #Test (Avg.) the average number of executable unit tests per instance.

	Development				Test			Overall
Model	MBPP	Spider- PLSQL	Spider2- ST	Spider2- MT	MBPP+	Spider2- ST	Spider2- MT	Test Mean
Open-weight models
Llama-4-Maverick	57.82	75.46	60.10	48.38	26.30	61.24	52.85	46.80
Gemma-4-31B	80.40	81.44	73.90	67.90	39.94	76.26	70.57	62.26
Proprietary models
Gemini-2.5-Flash-Lite	63.77	73.92	60.99	50.16	33.44	60.94	56.78	50.39

Table 2: Main PLSQLBench results, with overall means computed only across the three test sets. All values are Mean Test Pass@1 (%). Best results in each column are shown in bold.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

		Development		Test
Backbone	Setting	S2-ST	S2-MT	MBPP+	S2-ST	S2-MT
GPT-5.4 Mini	Direct	63.14	61.29	38.96	70.90	65.94
	Codex Agent	75.36	70.21	45.13	77.80	75.72
GPT-5.6-Sol	Direct	61.77	64.69	46.75	71.57	66.81
	Codex Agent	77.53	73.94	46.10	83.56	81.29

Appendix

Table 3: Direct generation vs. Codex DB Agent with medium reasoning. Results are Mean Test Pass@1 (%); bold indicates the better setting for each backbone. MBPP+ is evaluated on its test split. S2-ST and S2-MT denote Spider2 single- and multi-turn.

Quality Dimension	Check
Prompt–answer alignment	Required PL/SQL features, such as procedures, cursors, exceptions, object types, or dynamic SQL.
Metadata accuracy	Required tables/columns, difficulty, reasoning types, and PL/SQL features are correct.
Syntax and compilation	Reference PL/SQL compiles in the target Oracle environment.
Execution correctness	Program output matches the expected result.
Test-case validity	Tests run without modification and validate the reference answer.
Idiomatic PL/SQL	Code is readable, maintainable, Oracle-dialect compliant, and avoids anti-patterns.

Appendix

Table 4: Quality dimensions used in PLSQLBench review.

Panel of Figure 3 — Figure 3: A sample MBPP-style test suite.

Dataset	Prompt condition	Pass rate
MBPP+	No hints	45.1%
MBPP+	Canonical solution	56.8%
MBPP+	Canonical solution + 20% tests	68.8%
MBPP+	Canonical solution + 100% tests	87.3%

Appendix

Table 5: PL/SQL pass rates for MBPP+ under different prompt conditions.

Conversation Length	Dev Conv.	Dev Turns	Test Conv.	Test Turns	All Conv.	All Turns
3-turn	117	351	38	114	155	465
4-turn	56	224	11	44	67	268
5-turn	35	175	14	70	49	245
Total	208	750	63	228	271	978

Appendix

Table 6: Turn-length distribution of Spider2-MT conversations in PLSQLBench .

Difficulty	ST Dev	ST Test	MT Dev	MT Test
Simple	229 (56.3)	65 (63.1)	459 (61.2)	134 (58.8)
Intermediate	169 (41.5)	37 (35.9)	254 (33.9)	66 (28.9)
Advanced	9 (2.2)	1 (1.0)	37 (4.9)	28 (12.3)

Appendix

Table 7: Difficulty distribution for Spider2-ST and Spider2-MT dev/test splits. Parenthesized values are percentages within each split; Spider2-MT counts are over turns.

Reasoning Type	Spider2-ST Dev	Spider2-ST Test	Spider2-MT Dev	Spider2-MT Test
Exception Handling	407 (100.0)	103 (100.0)	750 (100.0)	226 (99.1)
Data Retrieval	407 (100.0)	103 (100.0)	746 (99.5)	227 (99.6)
Control Flow	398 (97.8)	102 (99.0)	716 (95.5)	221 (96.9)
Debugging	336 (82.6)	81 (78.6)	665 (88.7)	210 (92.1)
Iterative Processing	283 (69.5)	66 (64.1)	436 (58.1)	143 (62.7)
Aggregation	162 (39.8)	38 (36.9)	330 (44.0)	116 (50.9)

Appendix

Table 8: Top-10 reasoning-type distribution for Spider2-ST and Spider2-MT dev/test splits, ranked by Spider2-MT dev frequency. Counts are multi-label; parenthesized values are percentages within each split, and Spider2-MT counts are over turns.

	Development			Test		Overall
Model	MBPP	Spider-PLSQL	Spider2-ST	MBPP+	Spider2-ST	Test Mean
Open-weight models
Llama-4-Maverick	57.82	75.46	42.75	26.30	50.49	38.39
Gemma-4-31B	80.40	81.44	55.77	39.94	61.17	50.55
Proprietary models
Gemini-2.5-Flash-Lite	63.77	73.92	45.45	33.44	47.57	40.51

Appendix

Table 9: Single-turn Suite Pass@1 (%), with overall means computed only across the two test sets. Codex Agent rows use medium reasoning; unavailable evaluations are shown as –. Best results in each column are shown in bold.

Open-weight models
	Episode Pass@1		Turn Suite Pass@1
Model	Dev	Test	Dev	Test
Llama-4-Maverick	17.31	15.87	36.93	39.47
Gemma-4-31B	26.44	30.16	50.53	54.39
Proprietary models
Gemini-2.5-Flash-Lite	15.38	19.05	35.07	42.98

Appendix

Table 10: Spider2-MT strict multi-turn metrics (%). Episode Pass@1 requires every turn in a conversation to pass all unit tests, while Turn Suite Pass@1 measures suite-level success at the individual turn level. Codex Agent rows use medium reasoning. Best results are shown in bold.

Panel of Figure 8 — Figure 8: Prompt for PL/SQL generation.

Module	Hyperparameter	Value / Notes
Models	Evaluated models	Gemma-4-31B, Llama-4-Maverick, Claude-Opus-4.8, Gemini-2.5-Flash-Lite, Grok-4.3, GPT-5.4-Mini, GPT-5.4, and GPT-5.6-Sol
Prompts	System prompt	Oracle PL/SQL program-unit generation prompt, see Appendix F
	Prompt inputs	Natural-language request, schema DDL context, and task requirements
	Multi-turn context	Previous turn question and generated answer appended to subsequent turns
	Output format	Executable Oracle PL/SQL only; trailing / ; no markdown explanations
Inference	Temperature	0.0

Appendix

Table 11: Hyperparameter and infrastructure settings for PLSQLBench .

Code Generation Benchmarks

Code generation benchmarks measure how well AI models produce code across programming tasks, languages, and real-world settings.

See all 154 papers

ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language
Sep 6, 2026Hang Zhang, Chaokun Wang, Yuzhi Pan +12Code Generation LLM Code Generation
Tsinghua University, Beijing, China · Lenovo Group Limited, Beijing, China
Are LLMs Overkill for Databases?: A Study on the Finiteness of SQL
Mar 26, 2026Yue Li, David Mimno, Unso Eun Seo JoText-to-SQL Query Complexity
Cornell University
Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks
Aug 4, 2026Shunfan Zheng, Dongsheng Shi, Yue Li +3LLM Evaluation Text-to-SQL
1East China Normal University · 2Hasso Plattner Institute/University of Potsdam

Sep 6, 2026J/K move · Enter open · S save

ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language

Hang Zhang, Chaokun Wang, Yuzhi Pan, Ziyao Zhong+11

Tsinghua University, Beijing, China · Lenovo Group Limited, Beijing, China

Code Generation LLM Code Generation Code Generation Benchmarks

Open paper AskPDF arXiv

Large language models (LLMs) have shown strong potential for translating natural-language (NL) requirements into PL/SQL programs, attracting increasing attention from the database community. However, existing NL-to-PL/SQL efforts primarily focus on directly generating PL/SQL from complete NL requirements. In practice, PL/SQL development involves diverse scenarios, such as from-scratch development, code modification, debugging, and optimization, and may require either direct generation or multi-turn interaction. Yet, no comprehensive benchmark evaluates multi-scenario, direct and interactive, and multi-dialect NL-to-PL/SQL development. In this paper, we present ProcArena, an execution-based benchmark covering both Direct and Interactive modes. ProcArena comprises 3,998 executable tasks over 157 databases, spanning nine development subscenarios in PostgreSQL and Oracle. We construct challenging Direct tasks through Iterative Logic Enhancement and scenario-specific adapters, and derive paired Interactive tasks through Knowledge Integration and Requirement Perturbation while preserving executable targets. We further design a controlled Solver-User Simulator protocol that allows models to clarify user intent and inspect the database environment without exposing hidden execution feedback. Evaluating seven language models, we find that the best average scores are only 62.2% and 57.8% in Direct and Interactive, respectively, demonstrating that realistic NL-to-PL/SQL development remains challenging, particularly in interactive settings.

Abstract

Code Generation Benchmarks

Explore similar work

ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language

Are LLMs Overkill for Databases?: A Study on the Finiteness of SQL

Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks