How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective

Ma'ayan Armony; Albert Meroño-Peñuela; Gerard Canal

cs.AIAug 2, 2025

How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective

Authors: Ma'ayan Armony, Albert Meroño-Peñuela, Gerard Canal

Organizations: Department of Informatics, King’s College London

Abstract

AI planning provides agents the ability to act autonomously towards achieving a goal in a given environment. Recent advances in Large Language Models (LLMs) have introduced the prospect of using unstructured input for planning, making LLMs in planning an active area of research, especially for supporting autonomous agents. However, LLMs remain unreliable as long-horizon planners, since some of the plans they produce contain mistaken or hallucinated actions; executing such plans could lead to unexpected and harmful outcomes. Existing benchmarks for LLMs in planning primarily rely on binary evaluation metrics, providing limited insight into the quality of plans that do not achieve the goal, and how far those are from becoming valid. In this paper, we frame the evaluation of LLMs in planning as a natural language processing (NLP) task, as LLMs are themselves NLP models. We propose a pipeline comprising three NLP-based steps that assess plan quality from complementary perspectives, followed by a symbolic planning step to achieve a valid plan, to provide a finer-grained understanding of the quality of invalid plans. In our experiments, we find no clear evidence of logical derivation or state-tracking during LLM plan generation -- on average, for all non-empty plans, only 3.55 actions per plan are executable, despite symbolically-generated optimal plans having an average length of 9.47 actions, which suggests that LLMs are still far from the reliability of symbolic planners. Our suite of metrics is further used to investigate patterns in LLM plan generation over time, and can provide a useful metric for assessing the quality of individual invalid plans.

Figures & tables

Panel of Figure 1 — Figure 1. Overview of the pipeline - intermediate plans

Panel of Figure 2 — Figure 2. BW 3 problem for running example - instance 100

Panel of Figure 3 — Figure 3. Overview of the pipeline - evaluation metrics

Panel of Figure 4 — Figure 4. Quality Analysis through plan refinement

Domain	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Blocksworld	0.3	0.34	0.43	0.49	9.52	0.37	2.33	0.32	5.41	0.46	1.0
Blocksworld 3	0.56	0.7	0.73	0.83	2.8	0.77	3.07	0.58	1.91	0.81	1.0
Elevator	0.47	0.71	0.6	0.53	5.61	0.75	1.02	0.47	5.02	0.8	1.0
Logistics	0.12	0.15	0.33	0.32	14.54	0.17	0.43	0.13	12.71	0.25	1.0
Satellite	0.22	0.56	0.39	0.38	12.9	0.61	0.58	0.22	11.58	0.55	1.0

Table 2. Comparison of core metrics by domain

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	0.39	0.74	0.53	0.33	13.29	0.79	0.52	0.39	12.4	0.63	1.0
GPT 6 Sol	0.39	0.97	0.6	0.44	8.35	0.97	1.38	0.39	8.16	0.71	1.0
Gemini 3.1 Pro	0.39	0.54	0.53	0.43	9.03	0.55	0.58	0.39	9.16	0.5	1.0
Gemini 3.8 Flash	0.0	0.59	0.06	0.46	14.16	0.59	0.12	0.0	14.15	0.67	1.0
Gemma 4 12B	0.02	0.25	0.26	0.31	16.56	0.37	0.24	0.02	13.65	0.39	1.0
Gemma 4 26B	0.0	0.17	0.12	0.23	20.92	0.31	0.13	0.0	13.66	0.51	1.0

Table 3. Comparison of core metrics by model - Satellite

Prompt Type	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.1	0.44	0.33	0.39	14.3	0.53	0.6	0.1	12.71	0.49	1.0
PDDL	0.57	0.65	0.63	0.33	9.78	0.68	0.3	0.57	8.29	0.77	1.0
CoT	0.17	0.96	0.42	0.41	13.52	0.96	1.13	0.17	12.99	0.57	1.0
Zero-Shot	0.03	0.51	0.19	0.39	13.98	0.54	0.65	0.03	13.1	0.4	1.0

Table 4. Comparison of core metrics by prompt type - Satellite

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Blocksworld	0.3	0.34	0.73	0.79	6.2	0.37	1.71	0.31	3.74	0.4	1.0
Blocksworld 3	0.56	0.7	0.72	0.82	2.8	0.77	3.04	0.58	1.85	0.8	1.0
Elevator	0.47	0.71	0.6	0.55	5.3	0.75	1.39	0.47	4.35	0.81	1.0
Logistics	0.12	0.16	0.33	0.31	14.18	0.17	0.38	0.13	12.47	0.24	1.0
Satellite	0.22	0.56	0.39	0.38	12.87	0.61	0.81	0.22	11.53	0.54	1.0

Appendix

Table A.1 . Comparison of core metrics by domain - optimal plan as GT

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	0.54	0.65	0.72	0.64	6.57	0.68	2.28	0.55	5.74	0.67	1.0
GPT 6 Sol	0.52	0.73	0.75	0.68	4.96	0.73	2.16	0.52	4.99	0.69	1.0
Gemini 3.1 Pro	0.46	0.59	0.71	0.68	5.23	0.6	1.75	0.46	5.23	0.6	1.0
Gemini 3.8 Flash	0.41	0.64	0.54	0.8	6.36	0.64	2.22	0.41	6.36	0.67	1.0
Gemma 4 12B	0.1	0.23	0.38	0.4	11.22	0.32	0.62	0.12	9.22	0.37	1.0
Gemma 4 26B	0.08	0.18	0.27	0.37	17.04	0.29	0.71	0.11	8.33	0.43	1.0

Appendix

Table A.2 . Comparison of core metrics by model (all domains) - optimal plan as GT

Task	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.23	0.34	0.54	0.56	9.2	0.38	1.05	0.24	7.68	0.4	1.0
PDDL	0.58	0.69	0.68	0.64	7.07	0.73	2.32	0.6	4.73	0.82	1.0
State Tracking	0.39	0.57	0.58	0.61	6.38	0.62	2.1	0.4	6.03	0.64	1.0
Zero-Shot	0.16	0.39	0.43	0.54	9.43	0.41	0.87	0.16	7.87	0.37	1.0

Appendix

Table A.3 . Comparison of core metrics by prompt types (all domains) - optimal plan as GT

Action	(pick-up c)	(stack c a)	(stack b c)	(stack a b)	$\pi_{\text{LLM}}$
Score	0.28	1.00	0.50	0.09	0.47

Appendix

Table A.4 .

\pi_{\text{LLM}}

similarity scores per action

$\pi_{\text{GT}}$		$\pi_{\text{LLM}}$		AQM		NP-AQM		$\pi_{\text{NLP}}$		$\pi_{\text{LCS}}$		$\pi_{\text{NLP-LCS}}$
1	(unstack c a)	1	(pick-up c)	1	same_act	1	same_act	1	(stack a b)	1	(stack c a)	1	(pick-up b)
2	(stack c a)	2	(stack c a)	2	correct	2	correct	2	(stack c a)	2	(stack b c)	2	(stack b c)
3	(pick-up b)	3	(stack b c)	3	misplaced	3	correct	3	(pick-up b)
4	(stack b c)	4	(stack a b)	4	diff_act	4	diff_act	4	(stack b c)

Appendix

Table A.5 . Consolidated running example

Metric	$\pi_{\text{LLM}}$ Score $\uparrow$	$\pi_{\text{NLP}}$ Score $\uparrow$	AQM $\uparrow$	LLM StV $\downarrow$	$\pi_{\text{NLP}}$ StV $\downarrow$	$\pi_{\text{LLM}}$ LEA $\uparrow$	$\pi_{\text{NLP}}$ LEA $\uparrow$
Value	0.4004	0.773	0.4688	3	1	0.0	0.0
Metric	$\pi_{\text{LLM}}$ Validity	$\pi_{\text{NLP}}$ Validity	$\pi_{\text{LCS}}$ Validity	$\pi_{\text{NLP-LCS}}$ Validity	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA $\uparrow$	$\pi_{\text{VAL}}$ Validity
Value	False	False	False	True	0	1.0	True

Appendix

Table A.6 . Core metric values for running example

Metric	Description	Formal Definition
SR / Validity	Fraction of plans that reach the goal from all plans in an experiment.	$\text{SR}=\frac{1}{N}\sum_{i=1}^{N}[\pi_{i}\in\Pi_{\text{valid}}]$
ER / Executability	Fraction of plans that follow domain constraints from all plans in an experiment.	$\text{ER}=\frac{1}{N}\sum_{i=1}^{N}[\pi_{i}\in\Pi_{\text{exec}}]$
Score	Plan quality score based on similarity to $\pi^{*}$ and plan potential. The score of an experiment is the mean score for all $\pi_{\text{LLM}}$ s in that experiment.	$\text{Score}(\pi_{\text{LLM}})=\frac{\operatorname{S}(\pi_{\text{LLM}},\pi_{\text{GT}})+\operatorname{S}(\pi_{\text{NLP}},\pi_{\text{GT}})}{2}$ where $S(\pi_{i},\pi_{\text{GT}})$ is the similarity score between $\pi_{i}$ and $\pi_{\text{GT}}$ . $\text{Score}=\frac{1}{N}\sum_{i=1}^{N}\text{Score}(\pi_{\text{LLM}})$
AQM Score	$\text{AQM}(\pi_{\text{LLM}})$ is the average quality of actions in the generated plan based on their assigned label. The AQM of an experiment is the mean AQM score for all $\pi_{\text{LLM}}$ s in that experiment.	$\text{AQM}(\pi)=\frac{1}{\|\pi\|}\sum_{j=1}^{\|\pi\|}q(a_{j})$ , where $q(a_{j})$ is the label’s rank. $\text{AQM}=\frac{1}{N}\sum_{i=1}^{N}\text{AQM}(\pi_{\text{LLM}})$
LEA	Propotion of the index of the last action in the plan that can be executed without violating domain constraints, out of the full plan length. The LEA of an experiment is the mean LEA for all $\pi_{\text{LLM}}$ s in that experiment.	$\text{LEA}(\pi)=\frac{\max\{k\mid s_{k}\models\text{pre}(a_{k}),\,s_{k+1}=\gamma(s_{k},a_{k})\}}{\|\pi\|}$ , where $\text{pre}(a)$ are the preconditions for action $a$ . $\text{LEA}=\frac{1}{N}\sum_{i=1}^{N}\text{LEA}(\pi_{\text{LLM}})$
StV	Number of stepwise modifications needed to transform the plan into $\pi_{\text{GT}}$ . The StV of an experiment is the mean StV for all $\pi_{\text{LLM}}$ s in that experiment.	$\text{StV}(\pi)=\|d(\pi,\pi^{})\|$ , where $d(\pi,\pi^{})$ are the operations to achieve $\pi_{\text{GT}}$ . $\text{StV}=\frac{1}{N}\sum_{i=1}^{N}\text{StV}(\pi_{\text{LLM}})$

Appendix

Table A.7 . Metrics and their formal definitions.

Table 15

Parameter	Meaning	Values tried	Chosen
$k$	candidates assessed in full	3, 5, 10, 20, 40	3
$m_{\min}$	minimum shifts kept after preliminary scoring	fixed	7
$m_{\max}$	maximum shifts kept after preliminary scoring	8, 15, 25, 50	25
$\tau$	keep shifts within this fraction of the best score	0.05, 0.1, 0.2, 0.4	0.1
$r$	alternation iterations	0, 1, 2, 4	2

Appendix

Table A.9. Hyperparameters of the NLP manipulation search.

Domain	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Blocksworld 3	0.43	0.53	0.7	0.75	2.98	0.64	2.74	0.51	2.2	0.74	1.0
Logistics	0.32	0.45	0.5	0.5	12.54	0.54	1.25	0.34	11.31	0.63	1.0

Appendix

Table A.10 . Comparison of core metrics by domain - 2025 LLMs

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude 3.5 Haiku	0.23	0.3	0.56	0.58	7.59	0.47	1.72	0.33	6.69	0.64	1.0
Claude 3.7 Sonnet	0.46	0.69	0.65	0.76	6.49	0.78	2.6	0.49	5.92	0.8	1.0
Gemini 2.5 Flash	0.26	0.5	0.53	0.66	8.33	0.66	1.96	0.3	7.33	0.7	1.0
Gemini 2.5 Pro	0.27	0.51	0.54	0.66	8.3	0.66	1.91	0.31	7.41	0.68	1.0
Gemma 2 (2B)	0.06	0.12	0.4	0.41	9.03	0.22	0.69	0.12	7.41	0.47	1.0
O3 Mini	0.74	0.74	0.83	0.75	6.08	0.8	3.17	0.8	5.27	0.85	1.0

Appendix

Table A.11 . Comparison of core metrics by model (all domains) - 2025 LLMs

Task	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.36	0.41	0.61	0.6	7.34	0.53	1.98	0.44	6.23	0.66	1.0
PDDL	0.42	0.44	0.64	0.61	7.6	0.55	2.09	0.47	6.47	0.67	1.0
State Tracking	0.26	0.66	0.5	0.7	7.22	0.74	1.83	0.29	6.97	0.73	1.0
Zero-Shot	0.47	0.48	0.65	0.62	8.1	0.57	2.23	0.51	6.6	0.69	1.0

Appendix

Table A.12 . Comparison of core metrics by prompt types (all domains) - 2025 LLMs

Figure A.1 . Validity improvement comparison across LLMs

Figure A.2 . Validity improvement comparison across LLMs

Table A.13 . Correlation results, grouped by domain and prompt type

Prompt Type	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.23	0.33	0.41	0.43	10.71	0.38	0.92	0.24	8.46	0.42	1.0
PDDL	0.58	0.68	0.68	0.65	7.29	0.73	2.35	0.59	5.05	0.82	1.0
State Tracking	0.39	0.57	0.58	0.63	6.33	0.62	2.52	0.4	5.98	0.65	1.0
Zero-Shot	0.16	0.39	0.32	0.42	10.83	0.41	0.92	0.16	8.92	0.4	1.0

Appendix

Table A.14 . Comparison of core metrics by prompt type (all domains)

Prompt Type	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.51	0.64	0.71	0.8	3.01	0.72	2.84	0.54	2.23	0.74	1.0
PDDL	0.63	0.76	0.79	0.88	2.52	0.82	3.53	0.67	1.44	0.87	1.0
State Tracking	0.49	0.68	0.69	0.81	2.31	0.76	2.81	0.5	2.12	0.8	1.0
Zero-Shot	0.64	0.78	0.73	0.86	3.45	0.8	3.28	0.65	1.67	0.84	1.0

Appendix

Table A.15 . Comparison of core metrics by prompt type - BW3

Prompt Type	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.0	0.0	0.14	0.18	14.43	0.01	0.01	0.02	8.24	0.12	1.0
PDDL	0.62	0.73	0.73	0.83	7.26	0.78	4.87	0.64	2.47	0.88	1.0
State Tracking	0.69	0.75	0.79	0.85	2.28	0.81	5.18	0.71	1.99	0.84	1.0
Zero-Shot	0.0	0.0	0.14	0.21	12.09	0.01	0.0	0.01	7.98	0.11	1.0

Appendix

Table A.16 . Comparison of core metrics by prompt type - BW

Prompt Type	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.0	0.02	0.23	0.23	16.46	0.02	0.02	0.0	14.63	0.03	1.0
PDDL	0.47	0.58	0.59	0.57	11.66	0.63	1.66	0.5	8.81	0.78	1.0
State Tracking	0.0	0.0	0.25	0.25	14.01	0.0	0.03	0.0	13.45	0.16	1.0
Zero-Shot	0.0	0.01	0.23	0.24	15.56	0.01	0.01	0.0	13.71	0.03	1.0

Appendix

Table A.17 . Comparison of core metrics by prompt type - Logistics

Prompt Type	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
One-Shot	0.57	0.61	0.68	0.53	4.76	0.67	1.09	0.57	4.34	0.77	1.0
PDDL	0.6	0.71	0.68	0.62	4.92	0.73	1.27	0.61	4.02	0.81	1.0
State Tracking	0.39	0.91	0.63	0.53	5.96	0.93	1.42	0.39	5.66	0.93	1.0
Zero-Shot	0.02	0.82	0.23	0.38	8.6	0.89	0.2	0.02	8.04	0.79	1.0

Appendix

Table A.18 . Comparison of core metrics by prompt type - Elevator

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	0.54	0.65	0.66	0.59	7.47	0.68	2.32	0.55	6.36	0.68	1.0
GPT 6 Sol	0.52	0.73	0.66	0.6	6.23	0.73	2.3	0.52	5.74	0.72	1.0
Gemini 3.1 Pro	0.46	0.59	0.6	0.59	6.8	0.6	1.83	0.46	6.24	0.61	1.0
Gemini 3.8 Flash	0.41	0.64	0.48	0.68	6.65	0.64	2.34	0.41	6.6	0.68	1.0
Gemma 4 12B	0.1	0.23	0.35	0.37	11.7	0.32	0.59	0.12	9.63	0.38	1.0
Gemma 4 26B	0.07	0.18	0.26	0.36	17.47	0.28	0.71	0.1	8.82	0.44	1.0

Appendix

Table A.19 . Comparison of core metrics by model (all domains)

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	1.0	1.0	0.99	1.0	0.03	1.0	5.0	1.0	0.02	1.0	1.0
GPT 6 Sol	1.0	1.0	1.0	1.0	0.0	1.0	4.95	1.0	0.0	1.0	1.0
Gemini 3.1 Pro	0.34	0.95	0.57	0.99	2.6	0.95	2.22	0.34	2.6	1.0	1.0
Gemini 3.8 Flash	1.0	1.0	1.0	1.0	0.0	1.0	5.0	1.0	0.0	1.0	1.0
Gemma 4 12B	0.16	0.32	0.56	0.61	4.11	0.45	1.37	0.22	3.8	0.53	1.0
Gemma 4 26B	0.14	0.36	0.43	0.62	8.89	0.51	1.46	0.2	3.78	0.61	1.0

Appendix

Table A.20 . Comparison of core metrics by model - BW3

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	0.55	0.55	0.63	0.64	5.24	0.57	3.99	0.58	3.57	0.65	1.0
GPT 6 Sol	0.44	0.44	0.53	0.54	7.12	0.45	3.13	0.46	4.57	0.52	1.0
Gemini 3.1 Pro	0.5	0.5	0.58	0.59	6.36	0.51	3.66	0.51	4.13	0.57	1.0
Gemini 3.8 Flash	0.32	0.46	0.44	0.62	5.12	0.47	2.41	0.32	4.96	0.55	1.0
Gemma 4 12B	0.02	0.03	0.25	0.29	12.22	0.12	0.46	0.05	7.72	0.2	1.0
Gemma 4 26B	0.03	0.11	0.2	0.3	26.94	0.21	0.83	0.06	7.18	0.38	1.0

Appendix

Table A.21 . Comparison of core metrics by model - BW

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	0.14	0.14	0.4	0.34	14.89	0.18	0.56	0.16	12.22	0.23	1.0
GPT 6 Sol	0.25	0.25	0.51	0.41	11.76	0.26	0.78	0.25	11.99	0.44	1.0
Gemini 3.1 Pro	0.25	0.25	0.48	0.37	11.85	0.25	0.7	0.25	11.73	0.25	1.0
Gemini 3.8 Flash	0.03	0.33	0.13	0.45	13.78	0.33	0.31	0.03	13.78	0.38	1.0
Gemma 4 12B	0.06	0.08	0.25	0.26	15.91	0.1	0.27	0.08	13.47	0.19	1.0
Gemma 4 26B	0.02	0.02	0.17	0.22	19.64	0.06	0.22	0.03	13.61	0.19	1.0

Appendix

Table A.22 . Comparison of core metrics by model - Logistics

Model	$\pi_{\text{LLM}}$ SR $\uparrow$	$\pi_{\text{LLM}}$ ER $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	LEA $\uparrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{NLP-LCS}}$ LEA	$\pi_{\text{VAL}}$ SR $\uparrow$
Claude Opus 4.8	0.57	0.84	0.69	0.56	4.52	0.87	1.0	0.58	4.26	0.89	1.0
GPT 6 Sol	0.53	0.99	0.66	0.63	3.52	0.99	1.34	0.53	3.45	0.94	1.0
Gemini 3.1 Pro	0.97	0.99	0.97	0.68	1.05	0.99	1.83	0.97	1.05	0.99	1.0
Gemini 3.8 Flash	0.09	0.64	0.21	0.64	7.95	0.64	0.66	0.09	7.91	0.63	1.0
Gemma 4 12B	0.24	0.49	0.41	0.37	8.61	0.57	0.57	0.24	7.7	0.64	1.0
Gemma 4 26B	0.18	0.25	0.34	0.32	11.47	0.38	0.47	0.18	7.87	0.58	1.0

Appendix

Table A.23 . Comparison of core metrics by model - Elevator

Domain	Model	prompt type	$\pi_{\text{LLM}}$ SR $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	LEA $\uparrow$	$\pi_{\text{NLP}}$ SR $\uparrow$	$\pi_{\text{LCS}}$ LEA $\uparrow$	$\pi_{\text{LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{VAL}}$ SR $\uparrow$
Blocksworld	Claude Opus 4.8	One-Shot	0.00	0.15	0.18	12.85	0.00	0.01	0.00	0.00	0.01	0.03	8.25	1.00
Blocksworld	Claude Opus 4.8	PDDL	0.90	0.92	0.92	0.98	6.60	0.96	0.90	0.95	0.92	0.92	0.70	1.00
Blocksworld	Claude Opus 4.8	State Tracking	0.90	0.94	0.94	0.74	6.54	0.92	0.90	0.92	0.98	0.95	0.49	1.00
Blocksworld	Claude Opus 4.8	Zero-Shot	0.00	0.20	0.23	9.24	0.00	0.00	0.00	0.00	0.00	0.03	8.00	1.00
Blocksworld 3	Claude Opus 4.8	One-Shot	1.00	0.99	1.00	0.02	5.08	1.00	1.00	1.00	1.00	1.00	0.02	1.00
Blocksworld 3	Claude Opus 4.8	PDDL	0.99	0.98	0.99	0.05	5.04	0.99	0.99	0.99	1.00	1.00	0.00	1.00

Appendix

Table A.24 . Mean values of evaluation metrics per experiment (domain x model × prompt type) for high SR models. Bold highlights the best value per model.

Domain	Model	prompt type	$\pi_{\text{LLM}}$ SR $\uparrow$	Score $\uparrow$	AQM $\uparrow$	StV $\downarrow$	$\|\pi_{\text{corr}}\|$ $\uparrow$	LEA $\uparrow$	$\pi_{\text{NLP}}$ SR $\uparrow$	$\pi_{\text{LCS}}$ LEA $\uparrow$	$\pi_{\text{LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ SR $\uparrow$	$\pi_{\text{NLP-LCS}}$ StV $\downarrow$	$\pi_{\text{VAL}}$ SR $\uparrow$
Blocksworld	Gemma 4 12B	One-Shot	0.00	0.15	0.17	14.40	0.01	0.01	0.00	0.01	0.01	0.02	8.35	1.00
Blocksworld	Gemma 4 12B	PDDL	0.05	0.43	0.48	9.68	1.36	0.31	0.05	0.24	0.09	0.09	6.91	1.00
Blocksworld	Gemma 4 12B	State Tracking	0.08	0.52	0.59	5.50	1.58	0.39	0.08	0.25	0.17	0.17	6.08	1.00
Blocksworld	Gemma 4 12B	Zero-Shot	0.00	0.17	0.17	12.00	0.00	0.03	0.00	0.03	0.00	0.00	6.00	1.00
Blocksworld 3	Gemma 4 12B	One-Shot	0.20	0.60	0.62	4.23	1.54	0.47	0.24	0.42	0.24	0.28	3.60	1.00
Blocksworld 3	Gemma 4 12B	PDDL	0.12	0.56	0.57	4.00	1.20	0.34	0.17	0.33	0.17	0.24	4.14	1.00

Appendix

Table A.25 . Mean values of evaluation metrics per experiment (domain x model × prompt type) for low SR models. Bold highlights the best value per model.

LLM Planning

Large language model planning studies how models decompose goals, choose actions, and revise plans under constraints.

See all 142 papers

May 21, 2026

Planning in the LLM Era: Building for Reliability and Efficiency

Growing attention to intelligent agents has put a spotlight on one of their central capabilities: planning. Early attempts to leverage large language models (LLMs) for planning relied on single-shot plan generation, followed by hybrid approaches that coupled LLMs with limited external search. These methods, unsound and incomplete by their very nature, often require substantial resources without yielding better solutions on unseen problems. As the limitations of LLMs become clearer, recent work has shifted toward using them at solution construction time -- generating symbolic solvers for a family of problems that can be verified and then used efficiently at inference time. This trend reflects the growing need for agents that are both reliable and resource-efficient. It also offers a path towards generating maintainable planners with minimal dependence on language models at inference time. In this paper, we argue that this shift reflects a broader realignment of the planning field in the LLM era. We examine three major categories of planner-generation methods, discuss their current limitations, and outline research steps towards a more reliable and efficient LLM-based generation of planners.

Michael Katz, Harsha Kokel, Kavitha Srinivas +1

IBM Research

LLM Inference Efficiency LLM Planning Symbolic Planning

Ask PDF

Jun 12, 2026

SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model

Large language models (LLMs) are increasingly deployed as planners for autonomous agents in household environments. While existing benchmarks evaluate whether LLM-generated plans execute successfully, they overlook a critical type of failure: latent failures. Unlike immediate failures that trigger instant feedback at execution time and enable timely correction, latent failures do not immediately halt plan execution but silently compromise goal achievement. In severe cases, they cause irreversible harm. To address this gap, we introduce SIMMER, a benchmark for evaluating latent failures in LLM planning through a human-curated symbolic world model grounded in the kitchen domain. SIMMER defines a world model comprising 77 actions, 262 unique objects, and approximately 46,800 possible interactions that are semantically realistic, derived from real-world cooking scripts. It then leverages a state machine executor that validates plans against the world model and detects immediate precondition violations, latent hazards, and irreversible failures. Experiments across six LLMs show that even frontier models achieve at most 17% error-free plans. Moreover, up to 56% of plans contain latent failures, the majority of which lead to irreversible consequences. We further demonstrate that explicit state reasoning via counterfactual foresight simulation can reduce latent failures by up to 72% and irreversible cases by up to 75%, suggesting a promising direction for more robust LLM planners.

Xiaoxin Lu, Ranran Haoran Zhang, Rui Zhang

The Pennsylvania State University, State College, PA, USA

Large Language Model-Based Robot Planning LLM Planning World Model-Based Planning AI Agent Safety Benchmarks

Ask PDF

Sep 29, 2026

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

Large language models (LLMs) enable agents to solve long-horizon tasks by generating a plan and then executing it in an environment. However, successful planning requires two distinct capabilities: selecting an appropriate plan for the task and executing it faithfully. Existing planner--executor systems can fail at either stage, while final task success alone cannot distinguish selection from execution failures. We therefore study the Plan Declaration--Execution Gap and introduce Planning-as-Routing, where an LLM declares one of four planning modes: Predefined, Sequential, Hierarchical, or Search, and a deterministic router dispatches the task to the corresponding pattern-specific executor. Across four benchmarks and three LLMs, we find three consistent patterns. First, generic Plan+ReAct often fails to preserve declared planning structure, especially for longer plans: across three benchmarks, only (22)--(45%) of trajectories preserve it, whereas pattern-specific executors enforce the intended structure. Second, planning-mode effectiveness varies across environments and models: Search performs best on ALFWorld, Hierarchical on SWE-bench, and the strongest pattern can vary across models within the same benchmark. Third, the largest gains come from execution: pattern-specific executors improve task success from (0.48) to (0.92) on ALFWorld and from (0.36) to (0.44) on SWE-bench Verified over Plan+ReAct. Current LLMs, however, do not reliably select the strongest mode for each task, although few-shot examples improve selection in some benchmark--model combinations. Overall, reliable agent planning requires both effective mode selection and faithful execution: routing substantially closes the execution gap, while task-specific mode selection remains open.

Subba Reddy Oota, Francisco Herrera, Jordi Cabot Sagrera +2

ADIA Lab, Abu Dhabi, United Arab Emirates · University of Granada, Granada, Spain · Luxembourg Institute of Science and Technology, Luxembourg +2

LLM Agents Long-Horizon LLM Agents LLM Agent Planning LLM Agent Routing

Ask PDF

Abstract

LLM Planning

Explore similar work

Planning in the LLM Era: Building for Reliability and Efficiency

SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution