SemPlan: Benchmarking Structured Semantic Planning for LLM-Based Queries over Enterprise Data

Bruno Santos Teixeira

cs.AIAug 12, 2026

SemPlan: Benchmarking Structured Semantic Planning for LLM-Based Queries over Enterprise Data

Authors: Bruno Santos Teixeira

Organizations: Universidade Federal de Ouro Preto (UFOP), Brazil

Abstract

Natural-language interfaces to enterprise data must translate underspecified requests into governed, executable behavior while controlling invalid queries, policy failures, cost, and nondeterminism. SemPlan Benchmark evaluates this architectural design space with a deterministic synthetic bilingual benchmark containing 1,800 cases in English and Brazilian Portuguese; 1,200 cases form the frozen scientific evaluation subset. Four architectures are compared under the same model configuration: direct SQL generation (A1), a bounded tool-agent baseline (A2), structured semantic-request generation followed by deterministic planning and execution (A3), and a clarification/stateful semantic-plan variant (A4). Across 4,800 primary records, answer correctness was low in absolute terms: 22.25% for A1, 22.58% for A2, 25.67% for A3, and 24.25% for A4. A3 had the highest observed correctness and significantly exceeded A1, A2, and A4 in the pre-specified paired correctness analysis, while A1 retained the highest policy-correct rate and the lowest unsafe-or-invalid rate. A4 had the lowest mean API cost and lowest false-refusal rate. On a preselected 150-case stability subset, answer-correct repeatability ranged from 92.00% to 98.67%. The results support a trade-off interpretation rather than a universal ranking: additional structural constraints changed failure modes and efficiency, but did not monotonically improve correctness or solve ambiguity and multi-turn state consistency.

Figures & tables

Dimension	Value	Cases
Split	Adversarial	200
	Development	300
	Multi-turn	200
	Hidden test	300
	Public test	500
	Validation	300

Table 1: Composition of the frozen SemPlan Benchmark version 1.0.0-rc.2.

Panel of Figure 1 — Figure 1: Where structure enters the SemPlan comparison. Only A1 permits model-authored SQL. A2 constrains model actions to typed tools. A3 and A4 emit typed semantic requests that are normalized and executed by deterministic software; A4 additionally exposes structured state and clarification behavior.

Approach	Answer correct	Policy correct	False refusal	Unsafe/invalid	Mean cost
A1	22.25%	43.67%	21.42%	31.00%	$0.000918
A2	22.58%	35.42%	1.83%	62.25%	$0.000927
A3	25.67%	37.33%	1.58%	60.58%	$0.000512
A4	24.25%	35.25%	0.17%	64.08%	$0.000469

Table 2: Primary metrics over 1,200 scientific cases per approach. Absolute correctness is reported before pairwise significance.

Figure 2: Primary answer correctness from the frozen primary-results table. The figure shows absolute rates; paired uncertainty is reported in Table 3 .

Metric	Contrast	n	Risk diff.	95% CI	Holm p
answer_correct	A3 vs A1	1200	+0.034167	[+0.018534, +0.049799]	0.000110
answer_correct	A4 vs A1	1200	+0.020000	[+0.004720, +0.035280]	0.041250
answer_correct	A3 vs A2	1200	+0.030833	[+0.016796, +0.044870]	0.000110
answer_correct	A4 vs A2	1200	+0.016667	[+0.003231, +0.030102]	0.041250
answer_correct	A4 vs A3	1200	-0.014167	[-0.025094, -0.003239]	0.041250
policy_correct	A3 vs A1	1200	-0.063333	[-0.090032, -0.036635]	0.000020

Table 3: Selected paired binary contrasts. Risk difference is left minus right.

Approach	en-US	pt-BR
A1	24.67%	19.83%
A2	28.50%	16.67%
A3	31.00%	20.33%
A4	29.33%	19.17%

Table 4: Exploratory primary answer correctness by language. Each language subgroup has

n=600

cases per approach.

Error code	All records	Primary records
EXECUTION_FAILED	1762	1402
OUTPUT_SCHEMA_INVALID	1051	845
POLICY_VIOLATION	368	287
CATALOG_UNKNOWN_ID	78	65
CFG_INVALID	24	16

Table 5: Typed model-quality failure categories. These are scientific outcomes rather than missing provider evidence.

Approach	Answered	Clarify	Out-of-scope	Error	False refusal
A1	292	0	536	372	257
A2	210	29	214	747	22
A3	194	33	246	727	19
A4	194	9	228	769	2

Table 6: Primary terminal-outcome counts by approach.

Approach	Repeatability	Cases	Additional records
A1	98.00%	150	300
A2	92.00%	150	300
A3	98.67%	150	300
A4	98.00%	150	300

Table 7: Answer-correct repeatability on the 150-case stability subset.

Approach	Clarification correct	Ambiguity n	Sequence-state correct	Multi-turn n
A1	0.00%	84	20.00%	200
A2	26.19%	84	17.00%	200
A3	27.38%	84	14.00%	200
A4	8.33%	84	14.50%	200

Table 8: Ambiguity and multi-turn endpoints. Weak absolute rates are treated as primary scientific limitations.

Figure 3: Observed cost-correctness trade-off from frozen primary tables. A3 has the highest correctness, while A4 has the lowest mean API cost.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Artifact	Identifier
Benchmark version	1.0.0-rc.2
Benchmark SHA-256	60fe39c51a71637afd2a88f9fd44c3eee620a6e5467591d3c36a0a39a16792ed
Scientific results SHA-256	5f06bc8bad81a8c0c151d4890a34432bb0166e73331371fb5c7fb8536aefb816
Model	gpt-5.6-luna , reasoning low , max output 1,200
Primary design	1,200 cases $\times$ 4 approaches $=$ 4,800 records
Stability design	150 cases $\times$ 4 approaches $\times$ 2 additional runs $=$ 1,200 records

Appendix

Table 9: Key immutable identifiers for the reported experiment.

LLM Planning

Large language model planning studies how models decompose goals, choose actions, and revise plans under constraints.

See all 142 papers

Bounded Semantic Planning and Deterministic Compilation for Reliable Enterprise Text-to-SQL
Aug 17, 2026Yi AiText-to-SQL Semantic Parsing
Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
Jun 3, 2026Haoyu Sun, Wenxuan Wang, Mingyang Song +5Tool-Augmented Language Model Agents LLM Agent Evaluation
Tongji University · Shanghai AI Laboratory · Harbin Institute of Technology +5
PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models
May 20, 2026Ziliang Zhao, Zenan Xu, Shuting Wang +7LLM Evaluation LLM Planning
Gaoling School of Artificial Intelligence, Renmin University of China · LLM Department, Hunyuan Team, Tencent · Beijing Academy of Artificial Intelligence +1

Aug 17, 2026J/K move · Enter open · S save

Bounded Semantic Planning and Deterministic Compilation for Reliable Enterprise Text-to-SQL

Yi Ai

Text-to-SQL Semantic Parsing LLM Planning Compiler-Guided Program Synthesis

Open paper AskPDF arXiv

Direct text-to-SQL asks a language model to do two jobs: interpret the business question and construct the complete relational query. In enterprise schemas, SQL can execute successfully while using the wrong relationship role or aggregation grain. We study an alternative placement of the stochastic boundary. A multi-turn planner grounds phrases and selects from question-specific governed options; graph traversal, role predicates, grain lowering, SQL construction, and deterministic checks are implemented in code. We evaluate this semantic path compilation (SPC) system against direct DDL-to-SQL generation on the ACME insurance benchmark. On a 38-question adjudicated comparison set with three runs per question, SPC was adjudicated correct on every run for 37 questions (97.4%), compared with 21 (55.3%) for the baseline. The paired discordance was 16 questions in favor of SPC and none in favor of the baseline (two-sided exact McNemar p=3.05x10^-5). SPC answered all 38 questions correctly at least once and produced one refusal and no adjudicated wrong-but-executed run across 114 run outcomes; the baseline produced 29 adjudicated wrong runs and seven additional judge-flagged data-only coincidences on the same set. A strict-equivalence sensitivity analysis increased the paired difference. Additional SPC runs with GPT-5.4 and Gemini-3.6-Flash showed similar question-level robustness, although their per-run verdict artifacts were not preserved. Six additional benchmark items are retained in an all-item analysis and documented separately by failure class. The study supports an end-to-end systems result, not a causal claim that compilation alone produced the gain, because SPC receives governed semantic artifacts that the DDL baseline does not.

Abstract

LLM Planning

Explore similar work

Bounded Semantic Planning and Deterministic Compilation for Reliable Enterprise Text-to-SQL

Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents

PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models