TGPO: Trace-Guided Policy Optimization for Robot Task Planning via Verifiable Subgoal Generation
Organizations: Shanghai Innovation Institute · Xi’an Jiao Tong University · Nankai University · Shanghai Jiao Tong University
Abstract
Robot task planning in real-world environments requires mapping abstract natural language instructions to executable action sequences under long horizons and complex constraints. While large language models (LLMs) provide strong commonsense reasoning, they often fail to generate reliable and feasible plans. In contrast, symbolic planners ensure feasibility and optimality but require well-specified goals and cannot directly interpret high-level human intent.We formulate robot task planning as learning to generate verifiable subgoals in the Planning Domain Definition Language (PDDL), bridging language understanding and symbolic planning. To address the challenges of sparse and noisy supervision, we propose Trace-Guided Policy Optimization (TGPO), a reinforcement learning framework that improves structured subgoal generation through (i) verifier-grounded rewards, (ii) external correction of intermediate reasoning traces, and (iii) constrained policy updates that incorporate corrected traces into training.We evaluate TGPO on large-scale household planning tasks with long horizons, abstract instructions, and complex constraints. TGPO significantly outperforms prompting-based and reinforcement learning baselines, with the largest gains on abstract tasks. Furthermore, TGPO integrates naturally with symbolic planners and language-conditioned executors, enabling robust long-horizon planning and combinatorial generalization in realistic environments.
Figures & tables
| In-Distribution | Out-of-Distribution | ||||||
| Method | T (s) | Easy | Complex | Abstract | Human | PARTNR | BEHAVIOR-1K |
| GPT-5 | 8.0 | 38.3 3.4 | 24.6 1.0 | 9.1 3.2 | 21.7 2.7 | 69.2 1.2 | 35.1 1.0 |
| Gemini-3.0-pro | 23.5 | 30.0 2.3 | 16.8 1.4 | 19.9 2.9 | 13.0 3.0 | 71.8 1.4 | 26.0 1.1 |
| SayPlan | 63.7 | 47.5 2.8 | 33.5 2.2 | 22.4 2.1 | 26.5 3.4 | 72.2 1.8 | 43.8 0.7 |
| DELTA | 37.8 | 52.7 2.6 | 22.0 2.8 | 3.9 1.3 | 26.5 2.1 | 69.3 1.1 | 39.3 1.5 |
| LLM+P | 12.3 | 12.6 1.0 | 2.1 0.4 | 4.4 0.8 | 3.9 1.8 | 67.2 1.7 | 22.5 1.2 |
| In-Distribution | Out-of-Distribution | |||||
| Method | Easy | Complex | Abstract | Human | PARTNR | BEHAVIOR-1K |
| Qwen2.5-7B (Base) | 0.0 | 0.0 | 0.0 | 0.0 | 27.8 1.0 | 12.5 0.7 |
| SFT | 83.2 0.5 | 71.1 0.6 | 38.1 1.7 | 55.8 2.2 | 71.7 1.3 | 63.7 1.1 |
| GRPO | 84.1 0.8 | 76.0 0.2 | 51.1 2.4 | 65.9 2.6 | 76.4 0.9 | 67.1 0.8 |
| Reinforce++ | 83.7 0.6 | 77.1 0.3 | 54.0 1.5 | 65.5 3.3 | 75.2 1.2 | 68.3 1.3 |
| TGPO (E2E) | 72.4 0.5 | 55.4 0.2 | 44.1 2.0 | 54.1 2.2 | 62.3 0.6 | 54.4 1.0 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Entity type | State attributes |
| Furniture / Objects | is_open , can_be_opened is_powered_on , is_powerable is_clean , is_cleaning_tool , require_floor_cleaner , requires_water_to_clean has_faucet , is_filled is_heating_device , is_heater , is_heated |
| Person | agent_at , p_holding |
| Agent | agent_at , holding , handempty |
| Hyperparameter | Value |
| Base model | Qwen2.5-7B |
| Batch size | 32 |
| SFT epochs | 3 |
| Cutoff length ( cutoff_len ) | 15000 |
| Learning rate | |
| Warm-up ratio | 0.1 |
| Hyperparameter | Value |
| Learning rate | |
| Batch size | 128 |
| Mini-batch size | 32 |
| Rollout number | 8 |
| TGPO_fix_num_per_rollout | 2 |
| Dimension | Weight | Input | Evaluation Focus & Prompt Link |
| Goal Achievement | 0.4 | Final state + Instruction | Judges purely by outcome: whether the execution satisfies the instruction goals. ( View Prompt ) |
| Intent Understanding | 0.2 | Scene + Plan + Instruction | Enables constraint-aware evaluation: selecting the best available option demonstrates correct intent. ( View Prompt ) |
| Completeness | 0.2 | Plan + Instruction | Evaluates subtask coverage at the goal level (e.g., “set the table” counts as one covered subtask). ( View Prompt ) |
| Execution Quality | 0.2 | Plan + Instruction | Checks for unreverted state changes, self-contradictions, navigation efficiency, and action count proportionality. ( View Prompt ) |
| Combined Task | Instruction | Source Tasks |
| Preparing popcorn for a radio evening | Heat a popcorn bag in the microwave, then turn on the radio in the living room. | make microwave popcorn & turning on radio |
| Clean up kitchen after party | Collect soda cans into the trash, move pizzas to the fridge, and put bowls into the sink. | cleaning up plates and food & picking up trash |
| Unload groceries and store food | Bring the grocery sack from the car to the kitchen, then store items in the fridge and cabinets. | carrying in groceries & storing food |
| Tidy up children’s room toys | Put board games, puzzles, and a ball into the toy box, then place dice, teddy bears, and games into the bookcase. | collecting children’s toys & picking up toys |
| Prep vegetables and cook | Dice vegetables on the chopping board, then cook cabbage and chili in the frying pan. | cook cabbage & slicing vegetables |
| Evening relaxation setup | Bring bottles from the fridge to the coffee table, then light a fire in the fireplace. | bringing water & setting the fire |
| Ours | Behavior-1K |
| navigate | NAVIGATE_TO |
| pick_from_surface | GRASP |
| pick_from_receptacle | GRASP |
| place_on_surface | PLACE_ONTOP |
| place_in_receptacle | PLACE_INSIDE |
| open | OPEN |
| ID | Instruction |
| E1 | I’m in the bathroom — can you bring me a towel? |
| E2 | Move a musical instrument from the living room to the bedroom. |
| E3 | I want to do some crafts — can you bring scissors, colored paper and glue to the couch? |
| E4 | Could you fill my reusable water bottle and give it to the person on the stairs, please? |
| ID | Instruction | Category |
| H1 | I’m thirsty. Give me a glass of water. | Explicit short-horizon |
| H2 | I’m kind of in the mood for coffee. | Abstract long-horizon |
| H3 | Give me some tea. | Explicit short-horizon |
| H4 | Mop the floor and clean it all up. | Explicit short-horizon |
| H5 | Prepare breakfast. | Abstract long-horizon |
| Baseline | Avg. Time (s) | Budget / Setting |
| GPT-5 | 8.0 | No extra reasoning effort (latency-matched). |
| Gemini-3.0-pro | 23.5 | Low reasoning level (latency-matched). |
| SayPlan | 63.7 | Iterative prompting / replanning pipeline. |
| DELTA | 37.8 | Decompose–formulate–solve per-subtask pipeline. |
| Category | Instruction |
| Easy | I’m in the bathroom, can you bring me a towel? |
| Complex | Collect scissors, tape, and colored pencils from the closet, office, and bedrooms. Set up all materials on the kitchen table for a family craft night, then power on the multiport hub in the living room so everyone can charge their devices during the activity. Afterwards, clean up by putting all used materials back in their original rooms. |
| Abstract | Create a classy dinner atmosphere and tidy afterward |
| Human Task | Give me some tea. |
Robot Task Planning
Robot task planning turns goals and observations into executable actions while coordinating skills, motion, uncertainty, and safety.
See all 239 papers