SGR-OPSD: Learning from Subgoal Plans with Graph-Regularized Credit Assignment for Agentic Reinforcement Learning
Organizations: School of Computer Science and Technology, East China Normal University · MeiTuan
Abstract
OPSD provides dense supervision for agentic reinforcement learning through privileged information feedback. However, existing privileged information often lacks explicit task-progression planning, while teacher-student log-probability gaps reflect shifts in model preference rather than true task progress. To address these limitations, we propose Subgoal-Guided Graph-Regularized OPSD (SGR-OPSD), a framework that combines subgoal supervision with interaction-derived evidence for turn-level credit assignment. Specifically, SGR-OPSD uses LLM-generated subgoal plans to provide dense action-level guidance, converting them into advantage weights via a constrained, KL-regularized objective. To ground credit in observed outcomes, we construct a Subgoal Progress Graph that estimates empirical returns from sampled transition frequencies. The graph compares empirical action returns against the state average to derive preferences that calibrate the teacher's credit weights. Evaluated on ALFWorld, WebShop, and Search-QA across two model scales (Qwen2.5-3B and 7B), SGR-OPSD consistently outperforms GRPO and strong self-distillation baselines, improving success rates over GRPO by up to 7.6 percentage points and reaching 85.8% on ALFWorld with Qwen2.5-7B. Ablation studies confirm that subgoal guidance and graph-based weight calibration are key to these performance gains.
Figures & tables
| ALFWorld | SearchQA | WebShop | |||||||||||||||
| Method | Pick | Look | Clean | Heat | Cool | Pick2 | Avg | NQ | Triv | Pop | Hotp | 2Wk | MuS | Bam | Avg | Score | Succ. |
| Closed-Source Models | |||||||||||||||||
| Qwen3.5-Plus | 82.9 | 30.8 | 66.7 | 6.3 | 44.0 | 66.7 | 49.5 | 28.5 | 55.1 | 35.0 | 28.6 | 37.2 | 23.5 | 42.8 | 35.8 | 9.8 | 9.5 |
| DeepSeek-V3.1 | 91.4 | 69.2 | 48.1 | 50.0 | 32.0 | 54.2 | 57.5 | 20.6 | 64.0 | 35.6 | 35.4 | 31.0 | 18.6 | 54.4 | 37.1 | 10.2 | 8.4 |
| Qwen2.5-3B-Instruct | |||||||||||||||||
| Vanilla | 44.4 | 11.1 | 6.2 | 15.4 | 28.6 | 12.5 | 19.7 | 24.6 | 48.1 | 31.0 | 26.3 | 25.3 | 7.2 | 21.5 | 26.3 | 6.7 | 0.8 |
| Method | ALFWorld | SearchQA | WebShop |
| SGR-OPSD | 76.9 | 40.6 | 68.8 |
| w/o Graph | 75.4 | 39.5 | 64.8 |
| w/o Teacher Signal | 74.1 | 40.8 | 62.2 |
| Skill instead of Subgoal | 74.6 | 37.7 | 60.4 |
| Method | ALFWorld | SearchQA | WebShop |
| SGR-OPSD | 85.8 | 46.4 | 78.2 |
| w/o Graph | 76.7 | 43.8 | 74.2 |
| w/o Teacher Signal | 79.9 | 44.4 | 72.6 |
| Skill instead of Subgoal | 75.9 | 44.2 | 68.4 |
| Skill: reusable workflow |
| Always execute clean tasks in the fixed sequence: |
| (1) locate and acquire target object, |
| (2) bring it to an available water source or sink to clean, |
| (3) navigate to final location, |
| (4) place object. |
| Subgoal: task-specific requirement |
| Trajectory | Action | Base | |||
| go to garbagecan 1 | 1.0000 | 0.9596 | 0.9738 | ||
| go to cabinet 1 | 1.0000 | 1.0397 | 1.0540 | ||
| go to cabinet 1 | 1.0000 | 1.0192 | 1.0366 |
| Qwen2.5-3B | Qwen2.5-7B | |||
| Method | Seen | Unseen | Seen | Unseen |
| GRPO | 69.3 | 67.5 | 78.2 | 79.6 |
| AgentOPSD | 68.6 | 68.2 | 80.3 | 78.7 |
| AlignOPSD | 65.9 | 72.1 | 75.0 | 65.8 |
| SGR-OPSD | 76.9 | 73.1 | 85.8 | 82.3 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Task-instance specific | Task-specific subgoal plan | No successful rollout required | No task-specific interaction required |
| OPSD (Skill) ( Zhao et al., 2026 ) | ||||
| RLSD ( Yang et al., 2026a ) | ||||
| Skill-SD ( Wang et al., 2026a ) | ||||
| SDAR ( Lu et al., 2026 ) | ||||
| AgentOPSD ( Wang et al., 2026b ) | ||||
| UniOPSD ( Fu et al., 2026 ) |
| Task: clean some soapbar and put it in countertop. | |
| Skill guidance: verbatim excerpts | Subgoal guidance: complete recorded plan |
| 10. Progressive Goal Decomposition: Break tasks into ordered sub-goals: (1) locate object, (2) obtain or transform it, (3) locate destination or reference object, (4) satisfy spatial relation or placement. Complete each sub-goal before moving to the next. Apply this at the start of every episode. 1. Phase-Ordered Plan: Always execute clean tasks in the fixed sequence: (1) locate and acquire target object, (2) bring it to an available water source or sink to clean, (3) navigate to final location, (4) place object. Apply this as soon as the goal specifies the object must be clean before placement. | 1. Find a soapbar in the environment Preconditions: None 2. Pick up the soapbar Preconditions: Soapbar is found and reachable 3. Clean the soapbar (e.g., rinse or wash it) so it is clean Preconditions: Soapbar is held 4. Find a countertop receptacle Preconditions: Soapbar is clean and held 5. Put the clean soapbar on the countertop Preconditions: Countertop is found and reachable, soapbar is clean and held |
| Setting | ALFWorld | WebShop | SearchQA |
| Tasks per update | 16 | 16 | 128 |
| Maximum interaction turns | 50 | 15 | 4 |
| Maximum prompt tokens | 2048 | 4096 | 4096 |
| Interaction history (turns) | 2 | 2 | 4 |
| Reference-policy KL coefficient | 0.01 | 0.01 | 0.001 |
| Learning-rate warmup ratio | 0 | 0 | 0.1 |
| Qwen2.5-3B-Instruct | Qwen2.5-7B-Instruct | |||||
| Method | ALFWorld | SearchQA | WebShop | ALFWorld | SearchQA | WebShop |
| GRPO | ||||||
| RLSD | ||||||
| SDAR | ||||||
| AgentOPSD | ||||||
| TASPO | ||||||
| Configuration | Seen | Unseen |
| SGR-OPSD | 76.9 | 73.1 |
| Length-matched Skill | 71.0 | 68.4 |
| Randomized graph preferences | 72.2 | 63.9 |
| Measure | Estimate | 95% interval |
| Teacher return agreement | 50.0% | [37.7, 62.3] |
| Graph return agreement | 81.7% | [70.1, 89.4] |
| Mean return difference | [0.00054, 0.08394] | |
| Success difference (pp) | [0.05, 5.89] |
| Metric | Count / total | Fraction |
| Direct task coverage | 947 / 2,396 | 39.52% |
| Direct trajectory coverage | 7,364 / 19,200 | 38.35% |
| Direct turn coverage | 13,878 / 698,343 | 1.99% |
| Minimum matched support = 1 | 5,545 / 6,539 | 84.80% |
| Minimum value-pool support = 1 | 5,545 / 6,539 | 84.80% |
| Covered unit | Count / total | Fraction |
| Tasks | 94 / 140 | 67.1% |
| Trajectories | 658 / 1,120 | 58.8% |
| Turns | 1,055 / 26,717 | 3.95% |
| Signal | Correct | Incorrect | Tied |
| All 60 pairs | |||
| Teacher evidence | 30 | 30 | 0 |
| Graph preference | 49 | 11 | 0 |
| Teacher-reference credit | 11 | 14 | 35 |
| Calibrated credit | 19 | 6 | 35 |
| Same 21 directly eligible pairs | |||
| Action preference | Credit allocation | |||
| Teacher | Graph | Reference | Calibrated | |
| 0.1 | 30/60 | 49/60 | 8/21 | 16/21 |
| 0.3 | 28/60 | 49/60 | 8/21 | 16/21 |
| 0.5 | 28/60 | 49/60 | 8/21 | 16/21 |
| 0.7 | 28/60 | 49/60 | 8/21 | 16/21 |
| 0.9 | 29/60 | 46/60 | 10/21 | 14/21 |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,852 papers