cs.CLSep 6, 2026
SaveProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language
Organizations: Tsinghua University, Beijing, China · Lenovo Group Limited, Beijing, China
Abstract
Large language models (LLMs) have shown strong potential for translating natural-language (NL) requirements into PL/SQL programs, attracting increasing attention from the database community. However, existing NL-to-PL/SQL efforts primarily focus on directly generating PL/SQL from complete NL requirements. In practice, PL/SQL development involves diverse scenarios, such as from-scratch development, code modification, debugging, and optimization, and may require either direct generation or multi-turn interaction. Yet, no comprehensive benchmark evaluates multi-scenario, direct and interactive, and multi-dialect NL-to-PL/SQL development. In this paper, we present ProcArena, an execution-based benchmark covering both Direct and Interactive modes. ProcArena comprises 3,998 executable tasks over 157 databases, spanning nine development subscenarios in PostgreSQL and Oracle. We construct challenging Direct tasks through Iterative Logic Enhancement and scenario-specific adapters, and derive paired Interactive tasks through Knowledge Integration and Requirement Perturbation while preserving executable targets. We further design a controlled Solver-User Simulator protocol that allows models to clarify user intent and inspect the database environment without exposing hidden execution feedback. Evaluating seven language models, we find that the best average scores are only 62.2% and 57.8% in Direct and Interactive, respectively, demonstrating that realistic NL-to-PL/SQL development remains challenging, particularly in interactive settings.
Figures & tables
Figure 1. ProcArena evaluates the same PL/SQL development target under two modes. Left: the nine subscenarios, five under Synthesis and four under Editing, and a Direct episode that maps a complete requirement to PL/SQL code in a single turn. Right: an Interactive episode that clarifies an underspecified request via ask_selector and ask_user turns with the User , execute turns against the Database Environment before the code is submitted. On the left, nine PL/SQL development subscenarios are grouped into five Synthesis subscenarios and four Editing subscenarios, below which a Direct episode passes a complete requirement to the Solver, which returns Oracle PL/SQL code in one turn. On the right, an Interactive episode starts from an underspecified checkout-flow request, proceeds through four numbered turns that propose an interpretation, ask for a foreign-key relationship, propose an execution sequence, and execute a scratch statement against the database, and ends with the Solver submitting a PL/pgSQL procedure.
Figure 2. The ProcArena construction pipeline.
| Subscenario | Applicable seed task | Constructed user input | Additional acceptance criteria |
| A1: From-Scratch | Any validated seed task. | Complete behavioral requirement; no attachment . | ❶ The task has no attachment. ❷ The requirement fully specifies the target behavior. |
| A2: Document-Grounded | The target behavior depends on facts that can form a separate document. | Task request without the document facts; document stating them . | ❶ Some implementation that ignores a document fact differs from the reference under the call suite. ❷ The document exposes no code. |
| A3: Example-Guided | The call suite contains inputs that illustrate the target behavior. | Task request; calls selected from the call suite with their observed behavior . | ❶ The reference code reproduces every example. ❷ A leakage check rejects PL/SQL implementation text. |
| A4: Template Instantiation | The target admits a reusable skeleton and task-specific bindings. | Task-specific bindings; reusable skeleton with unfilled placeholders . | ❶ A placeholder-coverage check matches every binding to the template. ❷ The check rejects a target-specific body. |
| A5: Code Composition | Existing callable components can implement the target when coordinated. | Composition request; callable PL/SQL components without the coordination code . | ❶ The components compile and are callable. ❷ A structural check confirms the target coordination code is absent. |
| B1: Functional Modification | Reference code has a business rule that functional reversion can invert. | Original rule as the requested change; reference code with that rule reverted . | ❶ At least one call distinguishes the unchanged code from the reference behavior. |
Table 1. Scenario-specific adapters. A1–A5 cover Synthesis and B1–B4 cover Editing. All adapters preserve the database description, reference code, call suite, and initial database state. Each constructed user input is written as requirement; attachment . The last column lists the checks applied after the three common checks.
Figure 3. Controlled requirement edits on one example task. Each row pairs the original sentence with the rewritten one produced by a single operator, with the changed span highlighted: Iterative Logic Enhancement, Knowledge Injection, Information Omission, and Contradiction Injection.
| Action | Env. | Arguments | Returned observation | Cost |
| ask_user | User | question | Natural-language answer | 2 |
| ask_selector | User | options | Index of the intended option | 2 |
| get_schema | DB | – | Schema of the whole database | 1 |
| describe_table | DB | table | Metadata of one table | 0.5 |
| sample_rows | DB | table , limit | Sample rows of one table | 0.5 |
| compile_plsql | DB | code | Compiler diagnostics | 0.5 |
Table 2. Nine actions available to the Solver in an Interactive episode. Every issued action consumes budget units, even if malformed or failed.
| Scale & coverage | Per-task difficulty | ||||||
|---|---|---|---|---|---|---|---|
| Dataset | Tasks | DBs | Dialects | Scen. | Interactive | Stmt | Cyclo |
| PLForge Simple | 300 | 130 | 1 | 1 | 0 | 2.8 | 3.0 |
| PLForge Hard | 300 | 113 | 1 | 1 | 0 | 10.7 | 6.8 |
| ProcBench | 300 | 123 | 1 | 1 | 0 | 3.3 | 3.5 |
| WeBridge | 209 | 40 | 1 | 1 | 0 | 29.5 | 9.4 |
| ProcArena | 3,998 | 157 | 2 | 9 | 1,879 | 32.1 | 18.6 |
Table 3. ProcArena and prior NL-to-PL/SQL datasets. Tasks reports all evaluation instances, and Interactive reports the Interactive subset. Stmt and Cyclo report the mean statement count and cyclomatic complexity per reference procedure.
| A. Synthesis | B. Editing | |||||||||
| Metric | A1 | A2 | A3 | A4 | A5 | B1 | B2 | B3 | B4 | All |
| Scale & coverage | ||||||||||
| Direct, PG | 114 | 119 | 118 | 119 | 111 | 119 | 120 | 120 | 120 | 117.8 |
| Direct, Oracle | 115 | 116 | 120 | 116 | 118 | 118 | 120 | 120 | 116 | 117.7 |
| Interactive, PG | 114 | 119 | 118 | 119 | 111 | 119 | – | 120 | 120 | 117.5 |
| Interactive, Oracle | 115 | 116 | 120 | 116 | 118 | 118 | – | 120 | 116 | 117.4 |
Table 4. ProcArena corpus by subscenario. All gives the per-subscenario mean for task counts, the corpus-wide distinct count for databases, and the per-applicable-task mean for other metrics. A dash denotes a missing Interactive task or a subscenario without a code attachment.
| A. Synthesis | B. Editing | ||||||||
| Fact kind | A1 | A2 | A3 | A4 | A5 | B1 | B3 | B4 | Avg |
| Knowledge | |||||||||
| Jargon (terminology) | 1.89 | 3.01 | 2.05 | 1.00 | 1.56 | 1.81 | 0.00 | 1.07 | 1.54 |
| Reasoning | 0.84 | 0.78 | 0.86 | 0.43 | 0.76 | 0.84 | 0.00 | 0.85 | 0.67 |
| Commonsense | 0.84 | 0.79 | 0.82 | 0.37 | 0.74 | 0.79 | 0.00 | 0.83 | 0.65 |
| Contradiction | |||||||||
Table 5. Interactive requirement transformations per task and subscenario. B2 has no Interactive task. Avg reports the mean across Interactive tasks.
| PostgreSQL | Oracle | |||||||||||||||||
| A. Synthesis | B. Editing | Overall | A. Synthesis | B. Editing | Overall | |||||||||||||
| Model | A1 | A2 | A3 | A4 | A5 | B1 | B3 | B4 | EX | A1 | A2 | A3 | A4 | A5 | B1 | B3 | B4 | EX |
| Direct | ||||||||||||||||||
| DeepSeek V4 Pro | 39.5 | 36.1 | 27.1 | 44.5 | 33.3 | 40.3 | 19.2 | 41.7 | 35.2 | 46.1 | 34.5 | 41.7 | 44.8 | 30.5 | 44.1 | 15.0 | 50.9 | 38.3 |
| GLM-5.2 | 44.7 | 45.4 | 33.9 | 51.3 | 37.8 | 49.6 | 31.7 | 53.3 | 43.5 | 47.0 | 40.5 | 48.3 | 50.9 | 28.0 | 45.8 | 32.5 | 51.7 | 43.0 |
| GLM-5.3 | 62.3 | 52.1 | 50.0 | 59.7 | 57.7 | 61.3 | 29.2 | 65.8 | 54.7 | 52.2 | 47.4 | 52.5 | 49.1 | 28.0 | 52.5 | 46.7 | 54.3 | 47.8 |
Table 6. Interaction lowers accuracy on Synthesis and raises it on Editing. Execution accuracy (%) per subscenario and dialect, with Direct above the shaded band and Interactive below it. Subscenarios run along each category’s ordering axis of Section 3.2 . Within each mode block, the best value per column is in bold and the second best is underlined .
Figure 4. Most candidates match or beat the gold procedure, but two models buy their speed with far more database work. Each model contributes one bar or marker pair per dialect, PostgreSQL then Oracle. Panel (a) stacks the faster speedup bins above the axis and the slower ones below. Panel (b) uses a log axis on which matches the gold procedure, and its connector spans the acceleration and work means. In panel (c), a zero marks a model with no rejected task.
Figure 5. Interaction cost does not predict retention. Panel (a): Direct and Interactive EX as bars, their ratio as a line. Panel (b): mean turns as bars, mean tokens per episode as a line.
Figure 6. Models split between iterating against the database and inspecting it. Share of all recorded actions per model. Tool names are verbatim from the traces; names outside the protocol are grouped as malformed/other .
Figure 7. Models follow visibly different interaction trajectories. Action counts by recorded step, two models per row with shared axes. The bands use the tool names of Figure 6 .
Figure 8. Control flow and persistent state dominate, and neither exists in declarative SQL. Failures by error type in the Interactive mode, as a percentage of the tasks each model ran.
| Condition | GPT-5.5 | Gemini-3.1 Pro | GPT-5.2 | GLM-5.2 |
|---|---|---|---|---|
| Direct task | 86.67 | 82.92 | 72.92 | 73.33 |
| + Knowledge injection | -16.27 | -9.09 | -13.40 | -25.36 |
| + Contradiction injection | -10.92 | -13.87 | -10.92 | -15.13 |
| + Requirement omission | -10.82 | -3.90 | -7.79 | -25.54 |
| Direct task + all three methods | -23.35 | -17.18 | -22.91 | -39.21 |
Table 7. Every construction operation lowers EX on its own, and the full pipeline lowers it most. Base is EX (%); indented rows are paired percentage-point changes against the same tasks.
Figure 9. Human audit (a) Pass rate and failure reasons over all audited ask_user replies. (b) The same audit restricted to the sampled LOC pool.
More in this topic
Code Generation
Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.
See all 316 papers