From Solver Feedback to Faithful Plans: Multi-Role Reinforcement Learning for Symbolic Planning

Chenghao Zhang; Yikai Mao; Haoyu Gao; Saisai Hu; Yuxi Cheng; Shanqi Liu; Dan Roth

cs.AIAug 22, 2026

From Solver Feedback to Faithful Plans: Multi-Role Reinforcement Learning for Symbolic Planning

Authors: Chenghao Zhang, Yikai Mao, Haoyu Gao, Saisai Hu, Yuxi Cheng, Shanqi Liu, Dan Roth

Organizations: University of Pennsylvania · Georgia Institute of Technology · Pace University · Unaffiliated

Abstract

Reliable planning requires converting natural-language instructions into executable symbolic specifications, yet large language models remain brittle without costly PDDL annotations and may exploit solver success in semantically unfaithful ways. We study how to learn faithful natural-language-to-PDDL formalization using only solver feedback, without human-written demonstrations. We propose a solvergrounded multi-role reinforcement learning framework where a single language model acts as an Actor, Judge, and Editor for generation, verification, and repair. The Actor proposes PDDL specifications, the Judge provides a solver-calibrated quality signal, and the Editor performs bounded diagnostic-conditioned refinement. On PlanBench, our method improves average success from 35.5% for LLM+P to 70.8%, achieves 66.3% faithful success, and reduces semantic drift to 6.4%. These results show that organizing solver feedback into generation, verification, and repair roles enables more scalable and faithful annotation-free symbolic planning

Figures & tables

Figure 1: One Brain Three Roles framework. A single LLM plays Actor, Judge, and Editor roles with shared parameters, grounded by deterministic PDDL solver feedback.

Figure 2: Solver-Grounded Multi-Role RL for NL-to-PDDL Planning. A single shared LLM backbone is conditioned into Actor, Judge, and Editor roles. The Actor generates an initial PDDL specification. A deterministic PDDL Solver verifies executability, producing a success label and diagnostics on failure. The Judge predicts a calibrated solver score, and the Editor uses diagnostics to perform bounded local edits, feeding back to the solver.

Method	BW	MBW	Logistics	Gripper	Avg.	ProntoQA	Trip	Calendar
LLM ${}^{\text{CoT}}$	$24.0{\scriptstyle\pm 4.3}$	$0.0{\scriptstyle\pm 0.0}$	$3.0{\scriptstyle\pm 1.7}$	$23.0{\scriptstyle\pm 4.2}$	$12.5{\scriptstyle\pm 1.6}$	$66.0{\scriptstyle\pm 4.7}$	$8.0{\scriptstyle\pm 2.7}$	$34.0{\scriptstyle\pm 4.7}$
LLM ${}^{\text{ToT}}$	$3.0{\scriptstyle\pm 1.7}$	$3.0{\scriptstyle\pm 1.7}$	$7.0{\scriptstyle\pm 2.6}$	$13.0{\scriptstyle\pm 3.4}$	$6.5{\scriptstyle\pm 1.2}$	–	–	–
LLM+P	$87.0{\scriptstyle\pm 3.4}$	$37.0{\scriptstyle\pm 4.8}$	$18.0{\scriptstyle\pm 3.8}$	$0.0{\scriptstyle\pm 0.0}$	$35.5{\scriptstyle\pm 1.8}$	$4.0{\scriptstyle\pm 2.0}$	$16.0{\scriptstyle\pm 3.7}$	$12.0{\scriptstyle\pm 3.2}$
Ours	$\mathbf{98.0}{\scriptstyle\pm 1.4}$	$\mathbf{71.0}{\scriptstyle\pm 4.5}$	$\mathbf{58.0}{\scriptstyle\pm 4.9}$	$\mathbf{56.0}{\scriptstyle\pm 5.0}$	$\mathbf{70.8}{\scriptstyle\pm 2.1}$	$\mathbf{90.0}{\scriptstyle\pm 3.0}$	$\mathbf{48.0}{\scriptstyle\pm 5.0}$	$\mathbf{40.0}{\scriptstyle\pm 4.9}$

Table 1: Planning success rate (%) on PlanBench and zero-shot transfer benchmarks. BW denotes BlocksWorld and MBW denotes Mystery BlocksWorld. Avg. is the mean over the four PlanBench domains.

Method	Backbone	BW	MBW	Logistics	Gripper	Avg.	Avg. Calls	Faithful	Drift $\downarrow$
Qwen-CoT	Qwen2.5-7B	31	1	5	25	15.5	5.8	11.6	25.2
Qwen-ToT	Qwen2.5-7B	16	5	9	18	12.0	5.9	9.1	24.4
Qwen-LLM+P	Qwen2.5-7B	80	32	20	7	34.8	5.2	27.1	22.1
Qwen-Self-Refine+Solver	Qwen2.5-7B	85	43	36	35	49.8	4.7	37.5	24.7
GPT-4o-Self-Refine+Solver	GPT-4o	91	47	42	37	54.3	4.5	43.0	20.8
Ours	Qwen2.5-7B	98	71	58	56	70.8	3.2	66.3	6.4

Table 2: Controlled comparison under matched solver-call budget. All methods are allowed at most

K=6

solver calls per test instance. Faithful denotes solver-successful and semantically faithful outputs. Drift is the conditional fraction of solver-successful outputs that fail semantic checking.

Figure 3: Repair and diagnostic analysis. Left: cumulative success rate over Editor repair steps. Step $0$ denotes the initial Actor output, and steps $1$ – $5$ denote successive repairs. Right: diagnostic heatmap over remaining unsolved cases after each step.

Figure 4: Solvability–faithfulness gap analysis. Left: paired bars compare solver success and faithful success under matched-budget settings. Right: domain-level slope chart comparing solver success and faithful success.

Figure 5: Judge score distribution over post-hoc semantic categories. Left: aggregate violin plot over all PlanBench domains. Right: domain-level grouped violin plot.

Domain	#Cand.	Unsolved	Solved-Unfaithful	Solved-Faithful	AUC
BlocksWorld	700	$0.16\pm 0.09$	$0.42\pm 0.13$	$0.80\pm 0.10$	0.94
Mystery BW	830	$0.18\pm 0.10$	$0.47\pm 0.14$	$0.77\pm 0.11$	0.90
Logistics	800	$0.20\pm 0.11$	$0.50\pm 0.15$	$0.74\pm 0.12$	0.87
Gripper	780	$0.21\pm 0.11$	$0.51\pm 0.15$	$0.73\pm 0.13$	0.85
All	3110	$0.19\pm 0.11$	$0.48\pm 0.15$	$0.76\pm 0.12$	0.89

Table 5: Post-hoc Judge separation analysis. Scores are reported as mean

\pm

standard deviation. AUC measures how well the Judge separates faithful outputs from solved-but-unfaithful outputs among solver-successful candidates.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Base model	Qwen2.5-7B
Environment	PlanBench
Solver	Fast Downward, 60s timeout
Learning rate	$3\times 10^{-5}$
Batch size	32
PPO clip	$\epsilon=0.2$

Appendix

Table 6: Hyperparameters and settings.

Component	Setting
Base model	Qwen2.5-7B
Solver	Fast Downward
Solver timeout	60 seconds
Actor decoding during training	nucleus sampling, $\tau_{\mathrm{temp}}=0.7$
Actor decoding during evaluation	greedy decoding
Editor decoding during training	nucleus sampling, $\tau_{\mathrm{temp}}=0.7$

Appendix

Table 7: Hyperparameters used in the solver-grounded multi-role training procedure.

Method	Backbone	Max Calls	BW	MBW	Logistics	Gripper	Avg.
Qwen-CoT	Qwen2.5-7B	6	31	1	5	25	15.5
Qwen-ToT	Qwen2.5-7B	6	16	5	9	18	12.0
Qwen-LLM+P	Qwen2.5-7B	6	80	32	20	7	34.8
Qwen-Self-Refine+Solver	Qwen2.5-7B	6	85	43	36	35	49.8
GPT-4o-Self-Refine+Solver	GPT-4o	6	91	47	42	37	54.3
Ours	Qwen2.5-7B	6	98	71	58	56	70.8

Appendix

Table 8: Planning success rate (%) under matched backbone and matched solver-call budget. All Qwen baselines use Qwen2.5-7B and at most

K=6

solver calls per test instance. Avg. denotes the mean across the four PlanBench domains.

Method	Avg. Calls	Solv. (%)	Faithful (%)	Drift (%) $\downarrow$
Qwen-CoT	$5.8\pm 0.4$	$15.5\pm 2.5$	$11.6\pm 2.0$	$25.2\pm 3.0$
Qwen-ToT	$5.9\pm 0.4$	$12.0\pm 2.0$	$9.1\pm 1.5$	$24.4\pm 3.0$
Qwen-LLM+P	$5.2\pm 0.3$	$34.8\pm 3.5$	$27.1\pm 3.0$	$22.1\pm 2.5$
Qwen-Self-Refine+Solver	$4.7\pm 0.3$	$49.8\pm 4.0$	$37.5\pm 3.5$	$24.7\pm 3.0$
GPT-4o-Self-Refine+Solver	$4.5\pm 0.3$	$54.3\pm 4.0$	$43.0\pm 3.5$	$20.8\pm 2.5$
Ours	$\mathbf{3.2\pm 0.2}$	$\mathbf{70.8\pm 4.5}$	$\mathbf{66.3\pm 4.0}$	$\mathbf{6.4\pm 1.0}$

Appendix

Table 9: Budget efficiency and semantic faithfulness under the matched

K=6

solver-call setting. Avg. Calls is the average number of solver invocations actually used before success or budget exhaustion. Faithful denotes the fraction of all tasks that are both solver-successful and semantically faithful according to the reference checker. Drift is the conditional fraction of solver-successful outputs that fail semantic checking.

Explore similar work

Sep 9, 2026cs.AI

Grounded Evaluation and Repair for NL-to-PDDL Problem Generation

Large Language Models (LLMs) have shown promise for translating Natural Language (NL) planning descriptions into PDDL problem instances. However, standard evaluation criteria such as syntactic validity or planner success can substantially overestimate faithfulness to the described task: a generated problem may be parseable and solvable while misrepresenting the intended initial state, goal, object structure, or optimization target. This paper studies an end-to-end NL-to-PDDL pipeline that combines LLM generation, checks in terms of PDDL parsing, planning and validation, a domain-conformance checker, an LLM critic, and iterative repair. Fine-grained repair feedback is constructed from the domain description, the generated problem, the natural language problem description, and operational diagnostics. Reference-based comparisons against curated benchmark PDDL problem descriptions are used for post-hoc benchmark analysis, and these offline checks include renaming-invariant structural matching and semantic equivalence, where domain support is available. Across Planetarium, AutoPlanBench, and curated PDDL~~2.1 problems, results show that operational success and benchmark-reference reconstruction can diverge substantially. Results also show that structured repair can be useful, and that PDDL~~2.1 remains challenging for reference reconstruction, even when operational success improves.

Joana Rosa, Pedro Santos, Valdemar Oliveira +3

INESC INOV, Lisbon, Portugal · Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal · Motamineral Minerais Industriais S.A. +1

Large Language Model Planning Large Language Model Generation Repair Data Generation Motion Reconstruction Branch Multi-Dimensional Evaluation+1

Ask PDF

Jun 29, 2026cs.AI

Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback

Planning often requires symbolic specifications that are both executable and verifiable. For large language models deployed in autonomous or decision-support systems, failures in such formalization may lead to unverifiable decisions, execution failures, or unsafe downstream behavior. We present NL-PDDL-Bench, a multi-domain benchmark for natural-language-to-PDDL specification construction with planner-verified executability and controlled difficulty scaling by object count. We further propose a planner-in-the-loop framework that uses validator and planner diagnostics to revise non-executable specifications through localized edits. Building on this infrastructure, we develop a planner-grounded optimization recipe that combines parameter-efficient Low-Rank Adaptation supervised fine-tuning, offline planner-derived preference pairs for Direct Preference Optimization, and inference-time planner-in-the-loop repair, without requiring online planner calls during training. We also provide a unified evaluation suite for parseability, solvability, specification similarity, and outcome-aware plan-level consistency against planner references. Experiments on representative model families show substantial gains in planner success and plan-level agreement, with improved robustness under difficulty scaling and cross-domain variation. These results highlight the value of externally verifiable formalization for reliable deployment of LLMs in safety- or security-sensitive planning systems. Code and data are available at: https://github.com/ibasicplan/NL-PDDL-Bench

Jiamei Jiang, Jiajing Zhang, Feifei Mo +2

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · School of Industry-education Integration, University of Chinese Academy of Sciences, Beijing, China

Large Language Model Planning Formalization Large-Scale Benchmark

Ask PDF

Jun 30, 2026cs.RO

Plan Right, Then Plan Tight: Symbolic RL for Efficient Embodied Reasoning

Embodied task planning asks an agent to turn a natural-language instruction into an executable sequence of actions in a physical scene, and is a building block for household, assistive, and service robots. Recent prompting-based and reinforcement-learning planners generate fluent action text but lack a cheap deterministic check that the produced plan is valid in the target world, while high-fidelity simulation is too slow to serve as an inner-loop training signal. The general problem is therefore how to obtain verifiable supervision and rewards for embodied planners without relying on string-level matching or full simulation. Here we show that a single BDDL specification, automatically constructed from open-world video evidence or curated tasks, can serve as a shared interface for data construction, plan verification, and reward design. A video-to-BDDL parser, an LLM verifier, and a lightweight symbolic engine together supply dense feedback at millisecond latency. We further introduce GroupAdapt, a difficulty-aware length schedule that uses the in-batch group pass rate as a zero-cost signal so that hard prompts get wider length tolerance and automatically tighten as their pass rate improves. Under the guidance of the proposed verifier and GroupAdapt schedule, the 8B planner attains a Strict-Pass score of 97.3 on BEHAVIOR-1000, yielding a 25.9 percent relative improvement over the Qwen3-8B baseline. This result exceeds the strongest large-model baseline by 3.5 percent, while simultaneously compressing the response length by 79 percent to 207 tokens, demonstrating both effectiveness and efficiency.

Xiangli Shi, Xiaomeng Zhu, Ye Tian +5

1Tsinghua University · 2The Hong Kong University of Science and Technology · 3Tencent +1

Topic pending

Ask PDF