Large language models (LLMs) have shown strong potential for translating natural-language (NL) requirements into PL/SQL programs, attracting increasing attention from the database community. However, existing NL-to-PL/SQL efforts primarily focus on directly generating PL/SQL from complete NL requirements. In practice, PL/SQL development involves diverse scenarios, such as from-scratch development, code modification, debugging, and optimization, and may require either direct generation or multi-turn interaction. Yet, no comprehensive benchmark evaluates multi-scenario, direct and interactive, and multi-dialect NL-to-PL/SQL development. In this paper, we present ProcArena, an execution-based benchmark covering both Direct and Interactive modes. ProcArena comprises 3,998 executable tasks over 157 databases, spanning nine development subscenarios in PostgreSQL and Oracle. We construct challenging Direct tasks through Iterative Logic Enhancement and scenario-specific adapters, and derive paired Interactive tasks through Knowledge Integration and Requirement Perturbation while preserving executable targets. We further design a controlled Solver-User Simulator protocol that allows models to clarify user intent and inspect the database environment without exposing hidden execution feedback. Evaluating seven language models, we find that the best average scores are only 62.2% and 57.8% in Direct and Interactive, respectively, demonstrating that realistic NL-to-PL/SQL development remains challenging, particularly in interactive settings.
Figures & tables
Figure 1. ProcArena evaluates the same PL/SQL development target under two modes. Left: the nine subscenarios, five under Synthesis and four under Editing, and a Direct episode that maps a complete requirement to PL/SQL code in a single turn. Right: an Interactive episode that clarifies an underspecified request via ask_selector and ask_user turns with the User , execute turns against the Database Environment before the code is submitted. On the left, nine PL/SQL development subscenarios are grouped into five Synthesis subscenarios and four Editing subscenarios, below which a Direct episode passes a complete requirement to the Solver, which returns Oracle PL/SQL code in one turn. On the right, an Interactive episode starts from an underspecified checkout-flow request, proceeds through four numbered turns that propose an interpretation, ask for a foreign-key relationship, propose an execution sequence, and execute a scratch statement against the database, and ends with the Solver submitting a PL/pgSQL procedure.
Figure 2. The ProcArena construction pipeline.
Subscenario
Applicable seed task
Constructed user input
Additional acceptance criteria
A1: From-Scratch
Any validated seed task.
Complete behavioral requirement; no attachment .
❶ The task has no attachment. ❷ The requirement fully specifies the target behavior.
A2: Document-Grounded
The target behavior depends on facts that can form a separate document.
Task request without the document facts; document stating them .
❶ Some implementation that ignores a document fact differs from the reference under the call suite. ❷ The document exposes no code.
A3: Example-Guided
The call suite contains inputs that illustrate the target behavior.
Task request; calls selected from the call suite with their observed behavior .
❶ The reference code reproduces every example. ❷ A leakage check rejects PL/SQL implementation text.
A4: Template Instantiation
The target admits a reusable skeleton and task-specific bindings.
Task-specific bindings; reusable skeleton with unfilled placeholders .
❶ A placeholder-coverage check matches every binding to the template. ❷ The check rejects a target-specific body.
A5: Code Composition
Existing callable components can implement the target when coordinated.
Composition request; callable PL/SQL components without the coordination code .
❶ The components compile and are callable. ❷ A structural check confirms the target coordination code is absent.
B1: Functional Modification
Reference code has a business rule that functional reversion can invert.
Original rule as the requested change; reference code with that rule reverted .
❶ At least one call distinguishes the unchanged code from the reference behavior.
Table 1. Scenario-specific adapters. A1–A5 cover Synthesis and B1–B4 cover Editing. All adapters preserve the database description, reference code, call suite, and initial database state. Each constructed user input is written as requirement; attachment . The last column lists the checks applied after the three common checks.
Figure 3. Controlled requirement edits on one example task. Each row pairs the original sentence with the rewritten one produced by a single operator, with the changed span highlighted: Iterative Logic Enhancement, Knowledge Injection, Information Omission, and Contradiction Injection.
Action
Env.
Arguments
Returned observation
Cost
ask_user
User
question
Natural-language answer
2
ask_selector
User
options
Index of the intended option
2
get_schema
DB
–
Schema of the whole database
1
describe_table
DB
table
Metadata of one table
0.5
sample_rows
DB
table , limit
Sample rows of one table
0.5
compile_plsql
DB
code
Compiler diagnostics
0.5
Table 2. Nine actions available to the Solver in an Interactive episode. Every issued action consumes budget units, even if malformed or failed.
Scale & coverage
Per-task difficulty
Dataset
Tasks
DBs
Dialects
Scen.
Interactive
Stmt
Cyclo
PLForge Simple
300
130
1
1
0
2.8
3.0
PLForge Hard
300
113
1
1
0
10.7
6.8
ProcBench
300
123
1
1
0
3.3
3.5
WeBridge
209
40
1
1
0
29.5
9.4
ProcArena
3,998
157
2
9
1,879
32.1
18.6
Table 3. ProcArena and prior NL-to-PL/SQL datasets. Tasks reports all evaluation instances, and Interactive reports the Interactive subset. Stmt and Cyclo report the mean statement count and cyclomatic complexity per reference procedure.
A. Synthesis
B. Editing
Metric
A1
A2
A3
A4
A5
B1
B2
B3
B4
All
Scale & coverage
Direct, PG
114
119
118
119
111
119
120
120
120
117.8
Direct, Oracle
115
116
120
116
118
118
120
120
116
117.7
Interactive, PG
114
119
118
119
111
119
–
120
120
117.5
Interactive, Oracle
115
116
120
116
118
118
–
120
116
117.4
Table 4. ProcArena corpus by subscenario. All gives the per-subscenario mean for task counts, the corpus-wide distinct count for databases, and the per-applicable-task mean for other metrics. A dash denotes a missing Interactive task or a subscenario without a code attachment.
A. Synthesis
B. Editing
Fact kind
A1
A2
A3
A4
A5
B1
B3
B4
Avg
Knowledge
Jargon (terminology)
1.89
3.01
2.05
1.00
1.56
1.81
0.00
1.07
1.54
Reasoning
0.84
0.78
0.86
0.43
0.76
0.84
0.00
0.85
0.67
Commonsense
0.84
0.79
0.82
0.37
0.74
0.79
0.00
0.83
0.65
Contradiction
Table 5. Interactive requirement transformations per task and subscenario. B2 has no Interactive task. Avg reports the mean across Interactive tasks.
PostgreSQL
Oracle
A. Synthesis
B. Editing
Overall
A. Synthesis
B. Editing
Overall
Model
A1
A2
A3
A4
A5
B1
B3
B4
EX
A1
A2
A3
A4
A5
B1
B3
B4
EX
Direct
DeepSeek V4 Pro
39.5
36.1
27.1
44.5
33.3
40.3
19.2
41.7
35.2
46.1
34.5
41.7
44.8
30.5
44.1
15.0
50.9
38.3
GLM-5.2
44.7
45.4
33.9
51.3
37.8
49.6
31.7
53.3
43.5
47.0
40.5
48.3
50.9
28.0
45.8
32.5
51.7
43.0
GLM-5.3
62.3
52.1
50.0
59.7
57.7
61.3
29.2
65.8
54.7
52.2
47.4
52.5
49.1
28.0
52.5
46.7
54.3
47.8
Table 6. Interaction lowers accuracy on Synthesis and raises it on Editing. Execution accuracy (%) per subscenario and dialect, with Direct above the shaded band and Interactive below it. Subscenarios run along each category’s ordering axis of Section 3.2 . Within each mode block, the best value per column is in bold and the second best is underlined .
Figure 4. Most candidates match or beat the gold procedure, but two models buy their speed with far more database work. Each model contributes one bar or marker pair per dialect, PostgreSQL then Oracle. Panel (a) stacks the faster speedup bins above the axis and the slower ones below. Panel (b) uses a log axis on which 1.0 matches the gold procedure, and its connector spans the acceleration and work means. In panel (c), a zero marks a model with no rejected task.
Figure 5. Interaction cost does not predict retention. Panel (a): Direct and Interactive EX as bars, their ratio as a line. Panel (b): mean turns as bars, mean tokens per episode as a line.
Figure 6. Models split between iterating against the database and inspecting it. Share of all recorded actions per model. Tool names are verbatim from the traces; names outside the protocol are grouped as malformed/other .
Figure 7. Models follow visibly different interaction trajectories. Action counts by recorded step, two models per row with shared axes. The bands use the tool names of Figure 6 .
Figure 8. Control flow and persistent state dominate, and neither exists in declarative SQL. Failures by error type in the Interactive mode, as a percentage of the tasks each model ran.
Condition
GPT-5.5
Gemini-3.1 Pro
GPT-5.2
GLM-5.2
Direct task
86.67
82.92
72.92
73.33
+ Knowledge injection
-16.27
-9.09
-13.40
-25.36
+ Contradiction injection
-10.92
-13.87
-10.92
-15.13
+ Requirement omission
-10.82
-3.90
-7.79
-25.54
Direct task + all three methods
-23.35
-17.18
-22.91
-39.21
Table 7. Every construction operation lowers EX on its own, and the full pipeline lowers it most. Base is EX (%); indented rows are paired percentage-point changes against the same tasks.
Figure 9. Human audit (a) Pass rate and failure reasons over all audited ask_user replies. (b) The same audit restricted to the sampled LOC pool.
We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based tests. Existing LLM evaluations largely target general-purpose code generation or declarative text-to-SQL, leaving procedural database programming underexplored. PLSQLBench contains 2,865 instances: 2,594 single-turn tasks and 271 multi-turn conversations spanning 978 turns. The benchmark combines complex schema-grounded tasks over enterprise-style Spider 2 databases, simpler schema-grounded tasks derived from Spider, and MBPP-derived procedural problems, covering varying levels of database grounding and procedural complexity. Experiments with eight LLMs reveal recurring difficulties in schema grounding, PL/SQL dialect fidelity, procedural control flow, exception handling, and cross-turn consistency. Tool-augmented LLM agents improve performance on several schema-grounded evaluations, although substantial gaps remain. These results highlight procedural database programming capabilities not directly assessed by conventional code generation or text-to-SQL benchmarks. Our code is available at https://github.com/oracle-samples/plsqlbench.
Marianne Menglin Liu, Leonid Boytsov, Daniel W. Peterson +13
Large Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs). However, current evaluation benchmarks remain disproportionately fixated on Text-to-SQL tasks, neglecting the holistic Database Lifecycle-from initial schema design to post-deployment maintenance. This narrow focus fails to capture the diverse capabilities required for real-world database management. To bridge this gap, we introduce DBLifeBench, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance. Furthermore, addressing the cognitive mismatch between ambiguous natural language and complex SQL logic, we propose Progressive-Text2SQL, a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving. Our extensive evaluation reveals a critical insight: while general-purpose models demonstrate balanced performance, specialized Text-to-SQL models suffer from ``catastrophic forgetting'' in non-coding phases like design and maintenance. DBLifeBench serves as a foundational step toward evaluating and building true full-stack database intelligence.
Shunfan Zheng, Dongsheng Shi, Yue Li +3
1East China Normal University · 2Hasso Plattner Institute/University of Potsdam
Natural language to SQL (NL2SQL) conversion is an important problem for researchers and enterprises due to the ubiquitous importance of relational databases in broad-ranging practical problems. Despite the rapid advancements in the capabilities of LLMs, NL2SQL has not reached parity in accuracy with human expert SQL writers, hence needing additional improvements in NL2SQL algorithms. This study presents a new multi-agent method for NL2SQL that achieves 78.1% semantic accuracy on the BIg Bench for LaRge-scale Database (BIRD) benchmark. Our method leverages a semantically enriched representation of user-provided schema, adds user-provided business rules, and produces accurate SQL queries. The main contributions of this study are (a) We designed an optimized new orchestrator in a multi-agent solution that uses LLMs to plan, orchestrate, reflect, and self-correct to generate accurate SQL queries, (b) We developed an advanced schema enrichment method that creates context-aware metadata to improve accuracy, and (c) We demonstrated the accuracy and generalizability of the method across different domains and datasets by evaluating it on the BIRD-SQL benchmark.