TGPO: Trace-Guided Policy Optimization for Robot Task Planning via Verifiable Subgoal Generation

Zhihong Liu; Yang Li; Renming Huang; Chendong Zeng; Cewu Lu; Panpan Cai

cs.ROFeb 12, 2026

TGPO: Trace-Guided Policy Optimization for Robot Task Planning via Verifiable Subgoal Generation

Authors: Zhihong Liu, Yang Li, Renming Huang, Chendong Zeng, Cewu Lu, Panpan Cai

Organizations: Shanghai Innovation Institute · Xi’an Jiao Tong University · Nankai University · Shanghai Jiao Tong University

Abstract

Robot task planning in real-world environments requires mapping abstract natural language instructions to executable action sequences under long horizons and complex constraints. While large language models (LLMs) provide strong commonsense reasoning, they often fail to generate reliable and feasible plans. In contrast, symbolic planners ensure feasibility and optimality but require well-specified goals and cannot directly interpret high-level human intent.We formulate robot task planning as learning to generate verifiable subgoals in the Planning Domain Definition Language (PDDL), bridging language understanding and symbolic planning. To address the challenges of sparse and noisy supervision, we propose Trace-Guided Policy Optimization (TGPO), a reinforcement learning framework that improves structured subgoal generation through (i) verifier-grounded rewards, (ii) external correction of intermediate reasoning traces, and (iii) constrained policy updates that incorporate corrected traces into training.We evaluate TGPO on large-scale household planning tasks with long horizons, abstract instructions, and complex constraints. TGPO significantly outperforms prompting-based and reinforcement learning baselines, with the largest gains on abstract tasks. Furthermore, TGPO integrates naturally with symbolic planners and language-conditioned executors, enabling robust long-horizon planning and combinatorial generalization in realistic environments.

Figures & tables

Figure 1: Overview of TGPO. (a) Data Generation : Construction of a long-horizon planning dataset with task decomposition traces and grounded subgoals. (b) Policy Initialization : Supervised fine-tuning (SFT) to initialize a structured generation policy. (c) Trace-Guided Policy Optimization : A reinforcement learning framework that improves structured generation by incorporating external trace correction and constrained sampling, enabling effective learning under sparse and noisy rewards.

		In-Distribution			Out-of-Distribution
Method	T (s)	Easy	Complex	Abstract	Human	PARTNR	BEHAVIOR-1K
GPT-5	8.0	38.3 $\pm$ 3.4	24.6 $\pm$ 1.0	9.1 $\pm$ 3.2	21.7 $\pm$ 2.7	69.2 $\pm$ 1.2	35.1 $\pm$ 1.0
Gemini-3.0-pro	23.5	30.0 $\pm$ 2.3	16.8 $\pm$ 1.4	19.9 $\pm$ 2.9	13.0 $\pm$ 3.0	71.8 $\pm$ 1.4	26.0 $\pm$ 1.1
SayPlan	63.7	47.5 $\pm$ 2.8	33.5 $\pm$ 2.2	22.4 $\pm$ 2.1	26.5 $\pm$ 3.4	72.2 $\pm$ 1.8	43.8 $\pm$ 0.7
DELTA	37.8	52.7 $\pm$ 2.6	22.0 $\pm$ 2.8	3.9 $\pm$ 1.3	26.5 $\pm$ 2.1	69.3 $\pm$ 1.1	39.3 $\pm$ 1.5
LLM+P	12.3	12.6 $\pm$ 1.0	2.1 $\pm$ 0.4	4.4 $\pm$ 0.8	3.9 $\pm$ 1.8	67.2 $\pm$ 1.7	22.5 $\pm$ 1.2

Table 1: Comparison with prompting-based baselines. Numbers are success rates (%).

T

denotes the average planning time per task.

	In-Distribution			Out-of-Distribution
Method	Easy	Complex	Abstract	Human	PARTNR	BEHAVIOR-1K
Qwen2.5-7B (Base)	0.0	0.0	0.0	0.0	27.8 $\pm$ 1.0	12.5 $\pm$ 0.7
SFT	83.2 $\pm$ 0.5	71.1 $\pm$ 0.6	38.1 $\pm$ 1.7	55.8 $\pm$ 2.2	71.7 $\pm$ 1.3	63.7 $\pm$ 1.1
GRPO	84.1 $\pm$ 0.8	76.0 $\pm$ 0.2	51.1 $\pm$ 2.4	65.9 $\pm$ 2.6	76.4 $\pm$ 0.9	67.1 $\pm$ 0.8
Reinforce++	83.7 $\pm$ 0.6	77.1 $\pm$ 0.3	54.0 $\pm$ 1.5	65.5 $\pm$ 3.3	75.2 $\pm$ 1.2	68.3 $\pm$ 1.3
TGPO (E2E)	72.4 $\pm$ 0.5	55.4 $\pm$ 0.2	44.1 $\pm$ 2.0	54.1 $\pm$ 2.2	62.3 $\pm$ 0.6	54.4 $\pm$ 1.0

Table 2: Comparison with learning-based baselines. Numbers are success rates (%).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : In large-scale environments, the subgoal policy receives abstract instructions and a scene graph, and generates a decomposition trace with corresponding subgoals. These subgoals are then solved using a PDDL planner, resulting in an executable long-horizon plan that satisfies the user’s requirements.

Entity type	State attributes
Furniture / Objects	is_open , can_be_opened is_powered_on , is_powerable is_clean , is_cleaning_tool , require_floor_cleaner , requires_water_to_clean has_faucet , is_filled is_heating_device , is_heater , is_heated
Person	agent_at , p_holding
Agent	agent_at , holding , handempty

Appendix

Table A.1: State attributes stored in the scene graph for planning.

Hyperparameter	Value
Base model	Qwen2.5-7B
Batch size	32
SFT epochs	3
Cutoff length ( cutoff_len )	15000
Learning rate	$5\times 10^{-6}$
Warm-up ratio	0.1

Appendix

Table B.1 : SFT hyperparameters.

Hyperparameter	Value
Learning rate	$5\times 10^{-6}$
Batch size	128
Mini-batch size	32
Rollout number	8
TGPO_fix_num_per_rollout	2

Appendix

Table B.2 : TGPO and GRPO hyperparameters.

Dimension	Weight	Input	Evaluation Focus & Prompt Link
Goal Achievement	0.4	Final state + Instruction	Judges purely by outcome: whether the execution satisfies the instruction goals. ( View Prompt )
Intent Understanding	0.2	Scene + Plan + Instruction	Enables constraint-aware evaluation: selecting the best available option demonstrates correct intent. ( View Prompt )
Completeness	0.2	Plan + Instruction	Evaluates subtask coverage at the goal level (e.g., “set the table” counts as one covered subtask). ( View Prompt )
Execution Quality	0.2	Plan + Instruction	Checks for unreverted state changes, self-contradictions, navigation efficiency, and action count proportionality. ( View Prompt )

Appendix

Table C.1 : Multi-perspective evaluation dimensions ensuring orthogonal assessment.

Figure D.1 : Scalability Test Results showing the variation of success rate with: (a) Scene Graph Size, (b) Plan Length, (c) Task Abstractness, and (d) Constraint Complexity.

Figure E.1 : Visualization of the “Preparing popcorn for a radio evening” task.

Combined Task	Instruction	Source Tasks
Preparing popcorn for a radio evening	Heat a popcorn bag in the microwave, then turn on the radio in the living room.	make microwave popcorn & turning on radio
Clean up kitchen after party	Collect soda cans into the trash, move pizzas to the fridge, and put bowls into the sink.	cleaning up plates and food & picking up trash
Unload groceries and store food	Bring the grocery sack from the car to the kitchen, then store items in the fridge and cabinets.	carrying in groceries & storing food
Tidy up children’s room toys	Put board games, puzzles, and a ball into the toy box, then place dice, teddy bears, and games into the bookcase.	collecting children’s toys & picking up toys
Prep vegetables and cook	Dice vegetables on the chopping board, then cook cabbage and chili in the frying pan.	cook cabbage & slicing vegetables
Evening relaxation setup	Bring bottles from the fridge to the coffee table, then light a fire in the fireplace.	bringing water & setting the fire

Appendix

Table E.1 : Combined task descriptions and their corresponding source tasks.

Ours	Behavior-1K
navigate	NAVIGATE_TO
pick_from_surface	GRASP
pick_from_receptacle	GRASP
place_on_surface	PLACE_ONTOP
place_in_receptacle	PLACE_INSIDE
open	OPEN

Appendix

Table F.1: Action space mapping between our fixed PDDL domain and the Behavior-1K benchmark.

Figure F.1 : Screenshot of the web-based dialogue interface used to collect real human household task instructions. The interface visualizes the scene graph to provide environmental context, and participants are informed that their conversations may be used for research purposes.

ID	Instruction
E1	I’m in the bathroom — can you bring me a towel?
E2	Move a musical instrument from the living room to the bedroom.
E3	I want to do some crafts — can you bring scissors, colored paper and glue to the couch?
E4	Could you fill my reusable water bottle and give it to the person on the stairs, please?

Appendix

Table F.2: Easy task examples.

ID	Instruction	Category
H1	I’m thirsty. Give me a glass of water.	Explicit short-horizon
H2	I’m kind of in the mood for coffee.	Abstract long-horizon
H3	Give me some tea.	Explicit short-horizon
H4	Mop the floor and clean it all up.	Explicit short-horizon
H5	Prepare breakfast.	Abstract long-horizon

Appendix

Table F.5 : Human-authored task examples.

Baseline	Avg. Time (s)	Budget / Setting
GPT-5	8.0	No extra reasoning effort (latency-matched).
Gemini-3.0-pro	23.5	Low reasoning level (latency-matched).
SayPlan	63.7	Iterative prompting / replanning pipeline.
DELTA	37.8	Decompose–formulate–solve per-subtask pipeline.

Appendix

Table G.1 : Baseline prompting pipelines and latency-matched inference budgets used in evaluation.

Panel of Figure 17 — Figure H.1 : Entity frequency statistics over the 67.1k enriched scene instances. We visualize the top-20 most frequent rooms, furniture categories, and object categories, respectively.

Panel of Figure 18 — Figure H.2 : Statistical overview of the synthesized personas, including age distribution, profession distribution, lifestyle tags, and preference attributes.

Category	Instruction
Easy	I’m in the bathroom, can you bring me a towel?
Complex	Collect scissors, tape, and colored pencils from the closet, office, and bedrooms. Set up all materials on the kitchen table for a family craft night, then power on the multiport hub in the living room so everyone can charge their devices during the activity. Afterwards, clean up by putting all used materials back in their original rooms.
Abstract	Create a classy dinner atmosphere and tidy afterward
Human Task	Give me some tea.

Appendix

Table I.1 : Task examples used in the qualitative analysis. We report the instruction and the associated qualitative category. Detailed decomposition, grounding, and plans are shown in subsequent listings.

Robot Task Planning

Robot task planning turns goals and observations into executable actions while coordinating skills, motion, uncertainty, and safety.

See all 239 papers

When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Execution
May 14, 2026Zilin Zhu, Longteng Guo, Yanghong Mei +5Long-Horizon Planning Long-Horizon Agent Evaluation
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Zhongguancun Academy, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
HEART: Coordination of Heterogeneous Expert Agents for Physically Grounded Robotic Task Planning
Jun 24, 2026Junho Lee, Seabin Lee, Wonjong Lee +3Large Language Model-Based Robot Planning Robot Task Planning
Dept. of Electronic Engineering, Sogang University, Seoul, Korea
Scale-Plan: Scalable Language-Enabled Task Planning for Heterogeneous Multi-Robot Teams
Mar 9, 2026Piyush Gupta, Sangjae Bae, Jiachen Li +1Large Language Model-Based Robot Planning Symbolic Planning
Honda Research Institute, San Jose, CA 95134, USA. · Georgia Institute of Technology, Atlanta, GA 30332, USA.

May 14, 2026J/K move · Enter open · S save

When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Execution

Zilin Zhu, Longteng Guo, Yanghong Mei, Bowen Pang+4

Institute of Automation, Chinese Academy of Sciences, Beijing, China · Zhongguancun Academy, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Long-Horizon Planning Long-Horizon Agent Evaluation Robot Task Planning Embodied AI VLMs for Robotics Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Long-horizon household tasks demand robust high-level planning and sustained reasoning capabilities, which are largely overlooked by existing embodied AI benchmarks that emphasize short-horizon navigation or manipulation and rely on fixed task categories. We introduce LongAct, a benchmark designed to evaluate planning-level autonomy in long-horizon household tasks specified through free-form instructions. By abstracting away embodiment-specific low-level control, LongAct isolates high-level cognitive capabilities such as instruction understanding, dependency management, memory maintenance, and adaptive planning. We further propose HoloMind, a VLM-driven agent with a DAG-based long-horizon hierarchical planner, a Multimodal Spatial Memory for persistent world modeling, an Episodic Memory for experience reuse, and a global Critic for reflective supervision. Experiments with GPT-5 and Qwen3-VL models show that HoloMind substantially improves long-horizon performance while reducing reliance on model scale. Even top models achieve only 59% goal completion and 16% full-task success, underscoring the difficulty of LongAct and the need for stronger long-horizon planning in embodied agents.

Abstract

Robot Task Planning

Explore similar work

When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Execution

HEART: Coordination of Heterogeneous Expert Agents for Physically Grounded Robotic Task Planning

Scale-Plan: Scalable Language-Enabled Task Planning for Heterogeneous Multi-Robot Teams