From Solver Feedback to Faithful Plans: Multi-Role Reinforcement Learning for Symbolic Planning

Chenghao Zhang; Yikai Mao; Haoyu Gao; Saisai Hu; Yuxi Cheng; Shanqi Liu; Dan Roth

cs.AIAug 22, 2026

From Solver Feedback to Faithful Plans: Multi-Role Reinforcement Learning for Symbolic Planning

Authors: Chenghao Zhang, Yikai Mao, Haoyu Gao, Saisai Hu, Yuxi Cheng, Shanqi Liu, Dan Roth

Organizations: University of Pennsylvania · Georgia Institute of Technology · Pace University · Unaffiliated

Abstract

Reliable planning requires converting natural-language instructions into executable symbolic specifications, yet large language models remain brittle without costly PDDL annotations and may exploit solver success in semantically unfaithful ways. We study how to learn faithful natural-language-to-PDDL formalization using only solver feedback, without human-written demonstrations. We propose a solvergrounded multi-role reinforcement learning framework where a single language model acts as an Actor, Judge, and Editor for generation, verification, and repair. The Actor proposes PDDL specifications, the Judge provides a solver-calibrated quality signal, and the Editor performs bounded diagnostic-conditioned refinement. On PlanBench, our method improves average success from 35.5% for LLM+P to 70.8%, achieves 66.3% faithful success, and reduces semantic drift to 6.4%. These results show that organizing solver feedback into generation, verification, and repair roles enables more scalable and faithful annotation-free symbolic planning

Figures & tables

Figure 1: One Brain Three Roles framework. A single LLM plays Actor, Judge, and Editor roles with shared parameters, grounded by deterministic PDDL solver feedback.

Figure 2: Solver-Grounded Multi-Role RL for NL-to-PDDL Planning. A single shared LLM backbone is conditioned into Actor, Judge, and Editor roles. The Actor generates an initial PDDL specification. A deterministic PDDL Solver verifies executability, producing a success label and diagnostics on failure. The Judge predicts a calibrated solver score, and the Editor uses diagnostics to perform bounded local edits, feeding back to the solver.

Method	BW	MBW	Logistics	Gripper	Avg.	ProntoQA	Trip	Calendar
LLM ${}^{\text{CoT}}$	$24.0{\scriptstyle\pm 4.3}$	$0.0{\scriptstyle\pm 0.0}$	$3.0{\scriptstyle\pm 1.7}$	$23.0{\scriptstyle\pm 4.2}$	$12.5{\scriptstyle\pm 1.6}$	$66.0{\scriptstyle\pm 4.7}$	$8.0{\scriptstyle\pm 2.7}$	$34.0{\scriptstyle\pm 4.7}$
LLM ${}^{\text{ToT}}$	$3.0{\scriptstyle\pm 1.7}$	$3.0{\scriptstyle\pm 1.7}$	$7.0{\scriptstyle\pm 2.6}$	$13.0{\scriptstyle\pm 3.4}$	$6.5{\scriptstyle\pm 1.2}$	–	–	–
LLM+P	$87.0{\scriptstyle\pm 3.4}$	$37.0{\scriptstyle\pm 4.8}$	$18.0{\scriptstyle\pm 3.8}$	$0.0{\scriptstyle\pm 0.0}$	$35.5{\scriptstyle\pm 1.8}$	$4.0{\scriptstyle\pm 2.0}$	$16.0{\scriptstyle\pm 3.7}$	$12.0{\scriptstyle\pm 3.2}$
Ours	$\mathbf{98.0}{\scriptstyle\pm 1.4}$	$\mathbf{71.0}{\scriptstyle\pm 4.5}$	$\mathbf{58.0}{\scriptstyle\pm 4.9}$	$\mathbf{56.0}{\scriptstyle\pm 5.0}$	$\mathbf{70.8}{\scriptstyle\pm 2.1}$	$\mathbf{90.0}{\scriptstyle\pm 3.0}$	$\mathbf{48.0}{\scriptstyle\pm 5.0}$	$\mathbf{40.0}{\scriptstyle\pm 4.9}$

Table 1: Planning success rate (%) on PlanBench and zero-shot transfer benchmarks. BW denotes BlocksWorld and MBW denotes Mystery BlocksWorld. Avg. is the mean over the four PlanBench domains.

Method	Backbone	BW	MBW	Logistics	Gripper	Avg.	Avg. Calls	Faithful	Drift $\downarrow$
Qwen-CoT	Qwen2.5-7B	31	1	5	25	15.5	5.8	11.6	25.2
Qwen-ToT	Qwen2.5-7B	16	5	9	18	12.0	5.9	9.1	24.4
Qwen-LLM+P	Qwen2.5-7B	80	32	20	7	34.8	5.2	27.1	22.1
Qwen-Self-Refine+Solver	Qwen2.5-7B	85	43	36	35	49.8	4.7	37.5	24.7
GPT-4o-Self-Refine+Solver	GPT-4o	91	47	42	37	54.3	4.5	43.0	20.8
Ours	Qwen2.5-7B	98	71	58	56	70.8	3.2	66.3	6.4

Table 2: Controlled comparison under matched solver-call budget. All methods are allowed at most

K=6

solver calls per test instance. Faithful denotes solver-successful and semantically faithful outputs. Drift is the conditional fraction of solver-successful outputs that fail semantic checking.

Figure 3: Repair and diagnostic analysis. Left: cumulative success rate over Editor repair steps. Step $0$ denotes the initial Actor output, and steps $1$ – $5$ denote successive repairs. Right: diagnostic heatmap over remaining unsolved cases after each step.

Figure 4: Solvability–faithfulness gap analysis. Left: paired bars compare solver success and faithful success under matched-budget settings. Right: domain-level slope chart comparing solver success and faithful success.

Figure 5: Judge score distribution over post-hoc semantic categories. Left: aggregate violin plot over all PlanBench domains. Right: domain-level grouped violin plot.

Domain	#Cand.	Unsolved	Solved-Unfaithful	Solved-Faithful	AUC
BlocksWorld	700	$0.16\pm 0.09$	$0.42\pm 0.13$	$0.80\pm 0.10$	0.94
Mystery BW	830	$0.18\pm 0.10$	$0.47\pm 0.14$	$0.77\pm 0.11$	0.90
Logistics	800	$0.20\pm 0.11$	$0.50\pm 0.15$	$0.74\pm 0.12$	0.87
Gripper	780	$0.21\pm 0.11$	$0.51\pm 0.15$	$0.73\pm 0.13$	0.85
All	3110	$0.19\pm 0.11$	$0.48\pm 0.15$	$0.76\pm 0.12$	0.89

Table 5: Post-hoc Judge separation analysis. Scores are reported as mean

\pm

standard deviation. AUC measures how well the Judge separates faithful outputs from solved-but-unfaithful outputs among solver-successful candidates.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Base model	Qwen2.5-7B
Environment	PlanBench
Solver	Fast Downward, 60s timeout
Learning rate	$3\times 10^{-5}$
Batch size	32
PPO clip	$\epsilon=0.2$

Appendix

Table 6: Hyperparameters and settings.

Component	Setting
Base model	Qwen2.5-7B
Solver	Fast Downward
Solver timeout	60 seconds
Actor decoding during training	nucleus sampling, $\tau_{\mathrm{temp}}=0.7$
Actor decoding during evaluation	greedy decoding
Editor decoding during training	nucleus sampling, $\tau_{\mathrm{temp}}=0.7$

Appendix

Table 7: Hyperparameters used in the solver-grounded multi-role training procedure.

Method	Backbone	Max Calls	BW	MBW	Logistics	Gripper	Avg.
Qwen-CoT	Qwen2.5-7B	6	31	1	5	25	15.5
Qwen-ToT	Qwen2.5-7B	6	16	5	9	18	12.0
Qwen-LLM+P	Qwen2.5-7B	6	80	32	20	7	34.8
Qwen-Self-Refine+Solver	Qwen2.5-7B	6	85	43	36	35	49.8
GPT-4o-Self-Refine+Solver	GPT-4o	6	91	47	42	37	54.3
Ours	Qwen2.5-7B	6	98	71	58	56	70.8

Appendix

Table 8: Planning success rate (%) under matched backbone and matched solver-call budget. All Qwen baselines use Qwen2.5-7B and at most

K=6

solver calls per test instance. Avg. denotes the mean across the four PlanBench domains.

Method	Avg. Calls	Solv. (%)	Faithful (%)	Drift (%) $\downarrow$
Qwen-CoT	$5.8\pm 0.4$	$15.5\pm 2.5$	$11.6\pm 2.0$	$25.2\pm 3.0$
Qwen-ToT	$5.9\pm 0.4$	$12.0\pm 2.0$	$9.1\pm 1.5$	$24.4\pm 3.0$
Qwen-LLM+P	$5.2\pm 0.3$	$34.8\pm 3.5$	$27.1\pm 3.0$	$22.1\pm 2.5$
Qwen-Self-Refine+Solver	$4.7\pm 0.3$	$49.8\pm 4.0$	$37.5\pm 3.5$	$24.7\pm 3.0$
GPT-4o-Self-Refine+Solver	$4.5\pm 0.3$	$54.3\pm 4.0$	$43.0\pm 3.5$	$20.8\pm 2.5$
Ours	$\mathbf{3.2\pm 0.2}$	$\mathbf{70.8\pm 4.5}$	$\mathbf{66.3\pm 4.0}$	$\mathbf{6.4\pm 1.0}$

Appendix

Table 9: Budget efficiency and semantic faithfulness under the matched

K=6

solver-call setting. Avg. Calls is the average number of solver invocations actually used before success or budget exhaustion. Faithful denotes the fraction of all tasks that are both solver-successful and semantically faithful according to the reference checker. Drift is the conditional fraction of solver-successful outputs that fail semantic checking.

Explore similar work

Grounded Evaluation and Repair for NL-to-PDDL Problem Generation
Sep 9, 2026Joana Rosa, Pedro Santos, Valdemar Oliveira +3Large Language Model Planning Large Language Model Generation
INESC INOV, Lisbon, Portugal · Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal · Motamineral Minerais Industriais S.A. +1
Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback
Jun 29, 2026Jiamei Jiang, Jiajing Zhang, Feifei Mo +2Large Language Model Planning Formalization
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · School of Industry-education Integration, University of Chinese Academy of Sciences, Beijing, China
Plan Right, Then Plan Tight: Symbolic RL for Efficient Embodied Reasoning
Jun 30, 2026Xiangli Shi, Xiaomeng Zhu, Ye Tian +5
1Tsinghua University · 2The Hong Kong University of Science and Technology · 3Tencent +1

Sep 9, 2026, cs.AIJ/K move · Enter open · S save

Grounded Evaluation and Repair for NL-to-PDDL Problem Generation

Joana Rosa, Pedro Santos, Valdemar Oliveira, Romão Silva+2

INESC INOV, Lisbon, Portugal · Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal · Motamineral Minerais Industriais S.A.+1

Large Language Model Planning Large Language Model Generation Repair Data Generation Motion Reconstruction Branch Multi-Dimensional Evaluation+1

Open paper AskPDF arXiv

Large Language Models (LLMs) have shown promise for translating Natural Language (NL) planning descriptions into PDDL problem instances. However, standard evaluation criteria such as syntactic validity or planner success can substantially overestimate faithfulness to the described task: a generated problem may be parseable and solvable while misrepresenting the intended initial state, goal, object structure, or optimization target. This paper studies an end-to-end NL-to-PDDL pipeline that combines LLM generation, checks in terms of PDDL parsing, planning and validation, a domain-conformance checker, an LLM critic, and iterative repair. Fine-grained repair feedback is constructed from the domain description, the generated problem, the natural language problem description, and operational diagnostics. Reference-based comparisons against curated benchmark PDDL problem descriptions are used for post-hoc benchmark analysis, and these offline checks include renaming-invariant structural matching and semantic equivalence, where domain support is available. Across Planetarium, AutoPlanBench, and curated PDDL~~2.1 problems, results show that operational success and benchmark-reference reconstruction can diverge substantially. Results also show that structured repair can be useful, and that PDDL~~2.1 remains challenging for reference reconstruction, even when operational success improves.

Abstract

Explore similar work

Grounded Evaluation and Repair for NL-to-PDDL Problem Generation

Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback

Plan Right, Then Plan Tight: Symbolic RL for Efficient Embodied Reasoning