How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective

Ma'ayan Armony; Albert Meroño-Peñuela; Gerard Canal

cs.AIAug 2, 2025

How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective

Authors: Ma'ayan Armony, Albert Meroño-Peñuela, Gerard Canal

Organizations: Department of Informatics, King’s College London

Abstract

AI planning provides agents the ability to act autonomously towards achieving a goal in a given environment. Recent advances in Large Language Models (LLMs) have introduced the prospect of using unstructured input for planning, making LLMs in planning an active area of research, especially for supporting autonomous agents. However, LLMs remain unreliable as long-horizon planners, since some of the plans they produce contain mistaken or hallucinated actions; executing such plans could lead to unexpected and harmful outcomes. Existing benchmarks for LLMs in planning primarily rely on binary evaluation metrics, providing limited insight into the quality of plans that do not achieve the goal, and how far those are from becoming valid. In this paper, we frame the evaluation of LLMs in planning as a natural language processing (NLP) task, as LLMs are themselves NLP models. We propose a pipeline comprising three NLP-based steps that assess plan quality from complementary perspectives, followed by a symbolic planning step to achieve a valid plan, to provide a finer-grained understanding of the quality of invalid plans. In our experiments, we find no clear evidence of logical derivation or state-tracking during LLM plan generation -- on average, for all non-empty plans, only 3.55 actions per plan are executable, despite symbolically-generated optimal plans having an average length of 9.47 actions, which suggests that LLMs are still far from the reliability of symbolic planners. Our suite of metrics is further used to investigate patterns in LLM plan generation over time, and can provide a useful metric for assessing the quality of individual invalid plans.

Figures & tables

Panel of Figure 1 — Figure 1. Overview of the pipeline - intermediate plans

Panel of Figure 2 — Figure 2. BW 3 problem for running example - instance 100

Panel of Figure 3 — Figure 3. Overview of the pipeline - evaluation metrics

Panel of Figure 4 — Figure 4. Quality Analysis through plan refinement

Domain	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Blocksworld	0.3	0.34	0.43	0.49	9.52	0.37	2.33	0.32	5.41	0.46	1.0
Blocksworld 3	0.56	0.7	0.73	0.83	2.8	0.77	3.07	0.58	1.91	0.81	1.0
Elevator	0.47	0.71	0.6	0.53	5.61	0.75	1.02	0.47	5.02	0.8	1.0
Logistics	0.12	0.15	0.33	0.32	14.54	0.17	0.43	0.13	12.71	0.25	1.0
Satellite	0.22	0.56	0.39	0.38	12.9	0.61	0.58	0.22	11.58	0.55	1.0

Table 2. Comparison of core metrics by domain

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	0.39	0.74	0.53	0.33	13.29	0.79	0.52	0.39	12.4	0.63	1.0
GPT 6 Sol	0.39	0.97	0.6	0.44	8.35	0.97	1.38	0.39	8.16	0.71	1.0
Gemini 3.1 Pro	0.39	0.54	0.53	0.43	9.03	0.55	0.58	0.39	9.16	0.5	1.0
Gemini 3.8 Flash	0.0	0.59	0.06	0.46	14.16	0.59	0.12	0.0	14.15	0.67	1.0
Gemma 4 12B	0.02	0.25	0.26	0.31	16.56	0.37	0.24	0.02	13.65	0.39	1.0
Gemma 4 26B	0.0	0.17	0.12	0.23	20.92	0.31	0.13	0.0	13.66	0.51	1.0

Table 3. Comparison of core metrics by model - Satellite

Prompt Type	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.1	0.44	0.33	0.39	14.3	0.53	0.6	0.1	12.71	0.49	1.0
PDDL	0.57	0.65	0.63	0.33	9.78	0.68	0.3	0.57	8.29	0.77	1.0
CoT	0.17	0.96	0.42	0.41	13.52	0.96	1.13	0.17	12.99	0.57	1.0
Zero-Shot	0.03	0.51	0.19	0.39	13.98	0.54	0.65	0.03	13.1	0.4	1.0

Table 4. Comparison of core metrics by prompt type - Satellite

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Blocksworld	0.3	0.34	0.73	0.79	6.2	0.37	1.71	0.31	3.74	0.4	1.0
Blocksworld 3	0.56	0.7	0.72	0.82	2.8	0.77	3.04	0.58	1.85	0.8	1.0
Elevator	0.47	0.71	0.6	0.55	5.3	0.75	1.39	0.47	4.35	0.81	1.0
Logistics	0.12	0.16	0.33	0.31	14.18	0.17	0.38	0.13	12.47	0.24	1.0
Satellite	0.22	0.56	0.39	0.38	12.87	0.61	0.81	0.22	11.53	0.54	1.0

Appendix

Table A.1 . Comparison of core metrics by domain - optimal plan as GT

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	0.54	0.65	0.72	0.64	6.57	0.68	2.28	0.55	5.74	0.67	1.0
GPT 6 Sol	0.52	0.73	0.75	0.68	4.96	0.73	2.16	0.52	4.99	0.69	1.0
Gemini 3.1 Pro	0.46	0.59	0.71	0.68	5.23	0.6	1.75	0.46	5.23	0.6	1.0
Gemini 3.8 Flash	0.41	0.64	0.54	0.8	6.36	0.64	2.22	0.41	6.36	0.67	1.0
Gemma 4 12B	0.1	0.23	0.38	0.4	11.22	0.32	0.62	0.12	9.22	0.37	1.0
Gemma 4 26B	0.08	0.18	0.27	0.37	17.04	0.29	0.71	0.11	8.33	0.43	1.0

Appendix

Table A.2 . Comparison of core metrics by model (all domains) - optimal plan as GT

Task	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.23	0.34	0.54	0.56	9.2	0.38	1.05	0.24	7.68	0.4	1.0
PDDL	0.58	0.69	0.68	0.64	7.07	0.73	2.32	0.6	4.73	0.82	1.0
State Tracking	0.39	0.57	0.58	0.61	6.38	0.62	2.1	0.4	6.03	0.64	1.0
Zero-Shot	0.16	0.39	0.43	0.54	9.43	0.41	0.87	0.16	7.87	0.37	1.0

Appendix

Table A.3 . Comparison of core metrics by prompt types (all domains) - optimal plan as GT

Action	(pick-up c)	(stack c a)	(stack b c)	(stack a b)	$\pi_{\text{LLM}}$
Score	0.28	1.00	0.50	0.09	0.47

Appendix

Table A.4 .

\pi_{\text{LLM}}

similarity scores per action

$\pi_{\text{GT}}$		$\pi_{\text{LLM}}$		AQM		NP-AQM		$\pi_{\text{NLP}}$		$\pi_{\text{LCS}}$		$\pi_{\text{NLP-LCS}}$
1	(unstack c a)	1	(pick-up c)	1	same_act	1	same_act	1	(stack a b)	1	(stack c a)	1	(pick-up b)
2	(stack c a)	2	(stack c a)	2	correct	2	correct	2	(stack c a)	2	(stack b c)	2	(stack b c)
3	(pick-up b)	3	(stack b c)	3	misplaced	3	correct	3	(pick-up b)
4	(stack b c)	4	(stack a b)	4	diff_act	4	diff_act	4	(stack b c)

Appendix

Table A.5 . Consolidated running example

Metric	$\pi_{\text{LLM}}$ Score $\uparrow$	$\pi_{\text{NLP}}$ Score $\uparrow$	AQM $\uparrow$	LLM StV $\downarrow$	$\pi_{\text{NLP}}$ StV $\downarrow$	$\pi_{\text{LLM}}$ LEA $\uparrow$	$\pi_{\text{NLP}}$ LEA $\uparrow$
Value	0.4004	0.773	0.4688	3	1	0.0	0.0
Metric	$\pi_{\text{LLM}}$ Validity	$\pi_{\text{NLP}}$ Validity	$\pi_{\text{LCS}}$ Validity	$\pi_{\text{NLP-LCS}}$ Validity	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA $\uparrow$	$\pi_{\text{VAL}}$ Validity
Value	False	False	False	True	0	1.0	True

Appendix

Table A.6 . Core metric values for running example

Metric	Description	Formal Definition
SR / Validity	Fraction of plans that reach the goal from all plans in an experiment.	$\text{SR}=\frac{1}{N}\sum_{i=1}^{N}[\pi_{i}\in\Pi_{\text{valid}}]$
ER / Executability	Fraction of plans that follow domain constraints from all plans in an experiment.	$\text{ER}=\frac{1}{N}\sum_{i=1}^{N}[\pi_{i}\in\Pi_{\text{exec}}]$
Score	Plan quality score based on similarity to $\pi^{*}$ and plan potential. The score of an experiment is the mean score for all $\pi_{\text{LLM}}$ s in that experiment.	$\text{Score}(\pi_{\text{LLM}})=\frac{\operatorname{S}(\pi_{\text{LLM}},\pi_{\text{GT}})+\operatorname{S}(\pi_{\text{NLP}},\pi_{\text{GT}})}{2}$ where $S(\pi_{i},\pi_{\text{GT}})$ is the similarity score between $\pi_{i}$ and $\pi_{\text{GT}}$ . $\text{Score}=\frac{1}{N}\sum_{i=1}^{N}\text{Score}(\pi_{\text{LLM}})$
AQM Score	$\text{AQM}(\pi_{\text{LLM}})$ is the average quality of actions in the generated plan based on their assigned label. The AQM of an experiment is the mean AQM score for all $\pi_{\text{LLM}}$ s in that experiment.	$\text{AQM}(\pi)=\frac{1}{\|\pi\|}\sum_{j=1}^{\|\pi\|}q(a_{j})$ , where $q(a_{j})$ is the label’s rank. $\text{AQM}=\frac{1}{N}\sum_{i=1}^{N}\text{AQM}(\pi_{\text{LLM}})$
LEA	Propotion of the index of the last action in the plan that can be executed without violating domain constraints, out of the full plan length. The LEA of an experiment is the mean LEA for all $\pi_{\text{LLM}}$ s in that experiment.	$\text{LEA}(\pi)=\frac{\max\{k\mid s_{k}\models\text{pre}(a_{k}),\,s_{k+1}=\gamma(s_{k},a_{k})\}}{\|\pi\|}$ , where $\text{pre}(a)$ are the preconditions for action $a$ . $\text{LEA}=\frac{1}{N}\sum_{i=1}^{N}\text{LEA}(\pi_{\text{LLM}})$
StV	Number of stepwise modifications needed to transform the plan into $\pi_{\text{GT}}$ . The StV of an experiment is the mean StV for all $\pi_{\text{LLM}}$ s in that experiment.	$\text{StV}(\pi)=\|d(\pi,\pi^{})\|$ , where $d(\pi,\pi^{})$ are the operations to achieve $\pi_{\text{GT}}$ . $\text{StV}=\frac{1}{N}\sum_{i=1}^{N}\text{StV}(\pi_{\text{LLM}})$

Appendix

Table A.7 . Metrics and their formal definitions.

Table 15

Parameter	Meaning	Values tried	Chosen
$k$	candidates assessed in full	3, 5, 10, 20, 40	3
$m_{\min}$	minimum shifts kept after preliminary scoring	fixed	7
$m_{\max}$	maximum shifts kept after preliminary scoring	8, 15, 25, 50	25
$\tau$	keep shifts within this fraction of the best score	0.05, 0.1, 0.2, 0.4	0.1
$r$	alternation iterations	0, 1, 2, 4	2

Appendix

Table A.9. Hyperparameters of the NLP manipulation search.

Domain	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Blocksworld 3	0.43	0.53	0.7	0.75	2.98	0.64	2.74	0.51	2.2	0.74	1.0
Logistics	0.32	0.45	0.5	0.5	12.54	0.54	1.25	0.34	11.31	0.63	1.0

Appendix

Table A.10 . Comparison of core metrics by domain - 2025 LLMs

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude 3.5 Haiku	0.23	0.3	0.56	0.58	7.59	0.47	1.72	0.33	6.69	0.64	1.0
Claude 3.7 Sonnet	0.46	0.69	0.65	0.76	6.49	0.78	2.6	0.49	5.92	0.8	1.0
Gemini 2.5 Flash	0.26	0.5	0.53	0.66	8.33	0.66	1.96	0.3	7.33	0.7	1.0
Gemini 2.5 Pro	0.27	0.51	0.54	0.66	8.3	0.66	1.91	0.31	7.41	0.68	1.0
Gemma 2 (2B)	0.06	0.12	0.4	0.41	9.03	0.22	0.69	0.12	7.41	0.47	1.0
O3 Mini	0.74	0.74	0.83	0.75	6.08	0.8	3.17	0.8	5.27	0.85	1.0

Appendix

Table A.11 . Comparison of core metrics by model (all domains) - 2025 LLMs

Task	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.36	0.41	0.61	0.6	7.34	0.53	1.98	0.44	6.23	0.66	1.0
PDDL	0.42	0.44	0.64	0.61	7.6	0.55	2.09	0.47	6.47	0.67	1.0
State Tracking	0.26	0.66	0.5	0.7	7.22	0.74	1.83	0.29	6.97	0.73	1.0
Zero-Shot	0.47	0.48	0.65	0.62	8.1	0.57	2.23	0.51	6.6	0.69	1.0

Appendix

Table A.12 . Comparison of core metrics by prompt types (all domains) - 2025 LLMs

Figure A.1 . Validity improvement comparison across LLMs

Figure A.2 . Validity improvement comparison across LLMs

Table A.13 . Correlation results, grouped by domain and prompt type

Prompt Type	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.23	0.33	0.41	0.43	10.71	0.38	0.92	0.24	8.46	0.42	1.0
PDDL	0.58	0.68	0.68	0.65	7.29	0.73	2.35	0.59	5.05	0.82	1.0
State Tracking	0.39	0.57	0.58	0.63	6.33	0.62	2.52	0.4	5.98	0.65	1.0
Zero-Shot	0.16	0.39	0.32	0.42	10.83	0.41	0.92	0.16	8.92	0.4	1.0

Appendix

Table A.14 . Comparison of core metrics by prompt type (all domains)

Prompt Type	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.51	0.64	0.71	0.8	3.01	0.72	2.84	0.54	2.23	0.74	1.0
PDDL	0.63	0.76	0.79	0.88	2.52	0.82	3.53	0.67	1.44	0.87	1.0
State Tracking	0.49	0.68	0.69	0.81	2.31	0.76	2.81	0.5	2.12	0.8	1.0
Zero-Shot	0.64	0.78	0.73	0.86	3.45	0.8	3.28	0.65	1.67	0.84	1.0

Appendix

Table A.15 . Comparison of core metrics by prompt type - BW3

Prompt Type	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.0	0.0	0.14	0.18	14.43	0.01	0.01	0.02	8.24	0.12	1.0
PDDL	0.62	0.73	0.73	0.83	7.26	0.78	4.87	0.64	2.47	0.88	1.0
State Tracking	0.69	0.75	0.79	0.85	2.28	0.81	5.18	0.71	1.99	0.84	1.0
Zero-Shot	0.0	0.0	0.14	0.21	12.09	0.01	0.0	0.01	7.98	0.11	1.0

Appendix

Table A.16 . Comparison of core metrics by prompt type - BW

Prompt Type	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.0	0.02	0.23	0.23	16.46	0.02	0.02	0.0	14.63	0.03	1.0
PDDL	0.47	0.58	0.59	0.57	11.66	0.63	1.66	0.5	8.81	0.78	1.0
State Tracking	0.0	0.0	0.25	0.25	14.01	0.0	0.03	0.0	13.45	0.16	1.0
Zero-Shot	0.0	0.01	0.23	0.24	15.56	0.01	0.01	0.0	13.71	0.03	1.0

Appendix

Table A.17 . Comparison of core metrics by prompt type - Logistics

Prompt Type	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.57	0.61	0.68	0.53	4.76	0.67	1.09	0.57	4.34	0.77	1.0
PDDL	0.6	0.71	0.68	0.62	4.92	0.73	1.27	0.61	4.02	0.81	1.0
State Tracking	0.39	0.91	0.63	0.53	5.96	0.93	1.42	0.39	5.66	0.93	1.0
Zero-Shot	0.02	0.82	0.23	0.38	8.6	0.89	0.2	0.02	8.04	0.79	1.0

Appendix

Table A.18 . Comparison of core metrics by prompt type - Elevator

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	0.54	0.65	0.66	0.59	7.47	0.68	2.32	0.55	6.36	0.68	1.0
GPT 6 Sol	0.52	0.73	0.66	0.6	6.23	0.73	2.3	0.52	5.74	0.72	1.0
Gemini 3.1 Pro	0.46	0.59	0.6	0.59	6.8	0.6	1.83	0.46	6.24	0.61	1.0
Gemini 3.8 Flash	0.41	0.64	0.48	0.68	6.65	0.64	2.34	0.41	6.6	0.68	1.0
Gemma 4 12B	0.1	0.23	0.35	0.37	11.7	0.32	0.59	0.12	9.63	0.38	1.0
Gemma 4 26B	0.07	0.18	0.26	0.36	17.47	0.28	0.71	0.1	8.82	0.44	1.0

Appendix

Table A.19 . Comparison of core metrics by model (all domains)

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	1.0	1.0	0.99	1.0	0.03	1.0	5.0	1.0	0.02	1.0	1.0
GPT 6 Sol	1.0	1.0	1.0	1.0	0.0	1.0	4.95	1.0	0.0	1.0	1.0
Gemini 3.1 Pro	0.34	0.95	0.57	0.99	2.6	0.95	2.22	0.34	2.6	1.0	1.0
Gemini 3.8 Flash	1.0	1.0	1.0	1.0	0.0	1.0	5.0	1.0	0.0	1.0	1.0
Gemma 4 12B	0.16	0.32	0.56	0.61	4.11	0.45	1.37	0.22	3.8	0.53	1.0
Gemma 4 26B	0.14	0.36	0.43	0.62	8.89	0.51	1.46	0.2	3.78	0.61	1.0

Appendix

Table A.20 . Comparison of core metrics by model - BW3

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	0.55	0.55	0.63	0.64	5.24	0.57	3.99	0.58	3.57	0.65	1.0
GPT 6 Sol	0.44	0.44	0.53	0.54	7.12	0.45	3.13	0.46	4.57	0.52	1.0
Gemini 3.1 Pro	0.5	0.5	0.58	0.59	6.36	0.51	3.66	0.51	4.13	0.57	1.0
Gemini 3.8 Flash	0.32	0.46	0.44	0.62	5.12	0.47	2.41	0.32	4.96	0.55	1.0
Gemma 4 12B	0.02	0.03	0.25	0.29	12.22	0.12	0.46	0.05	7.72	0.2	1.0
Gemma 4 26B	0.03	0.11	0.2	0.3	26.94	0.21	0.83	0.06	7.18	0.38	1.0

Appendix

Table A.21 . Comparison of core metrics by model - BW

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	0.14	0.14	0.4	0.34	14.89	0.18	0.56	0.16	12.22	0.23	1.0
GPT 6 Sol	0.25	0.25	0.51	0.41	11.76	0.26	0.78	0.25	11.99	0.44	1.0
Gemini 3.1 Pro	0.25	0.25	0.48	0.37	11.85	0.25	0.7	0.25	11.73	0.25	1.0
Gemini 3.8 Flash	0.03	0.33	0.13	0.45	13.78	0.33	0.31	0.03	13.78	0.38	1.0
Gemma 4 12B	0.06	0.08	0.25	0.26	15.91	0.1	0.27	0.08	13.47	0.19	1.0
Gemma 4 26B	0.02	0.02	0.17	0.22	19.64	0.06	0.22	0.03	13.61	0.19	1.0

Appendix

Table A.22 . Comparison of core metrics by model - Logistics

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	0.57	0.84	0.69	0.56	4.52	0.87	1.0	0.58	4.26	0.89	1.0
GPT 6 Sol	0.53	0.99	0.66	0.63	3.52	0.99	1.34	0.53	3.45	0.94	1.0
Gemini 3.1 Pro	0.97	0.99	0.97	0.68	1.05	0.99	1.83	0.97	1.05	0.99	1.0
Gemini 3.8 Flash	0.09	0.64	0.21	0.64	7.95	0.64	0.66	0.09	7.91	0.63	1.0
Gemma 4 12B	0.24	0.49	0.41	0.37	8.61	0.57	0.57	0.24	7.7	0.64	1.0
Gemma 4 26B	0.18	0.25	0.34	0.32	11.47	0.38	0.47	0.18	7.87	0.58	1.0

Appendix

Table A.23 . Comparison of core metrics by model - Elevator

Domain	Model	prompt type	$\pi_{\text{LLM}}$ SR $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	LEA $\uparrow$	$\pi_{\text{NLP}}$ SR $\uparrow$	$\pi_{\text{LCS}}$ LEA $\uparrow$	$\pi_{\text{LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{VAL}}$ SR $\uparrow$
Blocksworld	Claude Opus 4.8	One-Shot	0.00	0.15	0.18	12.85	0.00	0.01	0.00	0.00	0.01	0.03	8.25	1.00
Blocksworld	Claude Opus 4.8	PDDL	0.90	0.92	0.92	0.98	6.60	0.96	0.90	0.95	0.92	0.92	0.70	1.00
Blocksworld	Claude Opus 4.8	State Tracking	0.90	0.94	0.94	0.74	6.54	0.92	0.90	0.92	0.98	0.95	0.49	1.00
Blocksworld	Claude Opus 4.8	Zero-Shot	0.00	0.20	0.23	9.24	0.00	0.00	0.00	0.00	0.00	0.03	8.00	1.00
Blocksworld 3	Claude Opus 4.8	One-Shot	1.00	0.99	1.00	0.02	5.08	1.00	1.00	1.00	1.00	1.00	0.02	1.00
Blocksworld 3	Claude Opus 4.8	PDDL	0.99	0.98	0.99	0.05	5.04	0.99	0.99	0.99	1.00	1.00	0.00	1.00

Appendix

Table A.24 . Mean values of evaluation metrics per experiment (domain x model × prompt type) for high SR models. Bold highlights the best value per model.

Domain	Model	prompt type	$\pi_{\text{LLM}}$ SR $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	LEA $\uparrow$	$\pi_{\text{NLP}}$ SR $\uparrow$	$\pi_{\text{LCS}}$ LEA $\uparrow$	$\pi_{\text{LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{VAL}}$ SR $\uparrow$
Blocksworld	Gemma 4 12B	One-Shot	0.00	0.15	0.17	14.40	0.01	0.01	0.00	0.01	0.01	0.02	8.35	1.00
Blocksworld	Gemma 4 12B	PDDL	0.05	0.43	0.48	9.68	1.36	0.31	0.05	0.24	0.09	0.09	6.91	1.00
Blocksworld	Gemma 4 12B	State Tracking	0.08	0.52	0.59	5.50	1.58	0.39	0.08	0.25	0.17	0.17	6.08	1.00
Blocksworld	Gemma 4 12B	Zero-Shot	0.00	0.17	0.17	12.00	0.00	0.03	0.00	0.03	0.00	0.00	6.00	1.00
Blocksworld 3	Gemma 4 12B	One-Shot	0.20	0.60	0.62	4.23	1.54	0.47	0.24	0.42	0.24	0.28	3.60	1.00
Blocksworld 3	Gemma 4 12B	PDDL	0.12	0.56	0.57	4.00	1.20	0.34	0.17	0.33	0.17	0.24	4.14	1.00

Appendix

Table A.25 . Mean values of evaluation metrics per experiment (domain x model × prompt type) for low SR models. Bold highlights the best value per model.

LLM Planning

Large language model planning studies how models decompose goals, choose actions, and revise plans under constraints.

See all 142 papers

Planning in the LLM Era: Building for Reliability and Efficiency
May 21, 2026Michael Katz, Harsha Kokel, Kavitha Srinivas +1LLM Inference Efficiency LLM Planning
IBM Research
SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model
Jun 12, 2026Xiaoxin Lu, Ranran Haoran Zhang, Rui ZhangLarge Language Model-Based Robot Planning LLM Planning
The Pennsylvania State University, State College, PA, USA
Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution
Sep 29, 2026Subba Reddy Oota, Francisco Herrera, Jordi Cabot Sagrera +2LLM Agents Long-Horizon LLM Agents
ADIA Lab, Abu Dhabi, United Arab Emirates · University of Granada, Granada, Spain · Luxembourg Institute of Science and Technology, Luxembourg +2

May 21, 2026J/K move · Enter open · S save

Planning in the LLM Era: Building for Reliability and Efficiency

Michael Katz, Harsha Kokel, Kavitha Srinivas, Shirin Sohrabi

IBM Research

LLM Inference Efficiency LLM Planning Symbolic Planning

Open paper AskPDF arXiv

Growing attention to intelligent agents has put a spotlight on one of their central capabilities: planning. Early attempts to leverage large language models (LLMs) for planning relied on single-shot plan generation, followed by hybrid approaches that coupled LLMs with limited external search. These methods, unsound and incomplete by their very nature, often require substantial resources without yielding better solutions on unseen problems. As the limitations of LLMs become clearer, recent work has shifted toward using them at solution construction time -- generating symbolic solvers for a family of problems that can be verified and then used efficiently at inference time. This trend reflects the growing need for agents that are both reliable and resource-efficient. It also offers a path towards generating maintainable planners with minimal dependence on language models at inference time. In this paper, we argue that this shift reflects a broader realignment of the planning field in the LLM era. We examine three major categories of planner-generation methods, discuss their current limitations, and outline research steps towards a more reliable and efficient LLM-based generation of planners.

Abstract

LLM Planning

Explore similar work

Planning in the LLM Era: Building for Reliability and Efficiency

SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution