AgentOPSD: Turn-Level Credit Assignment via Recursive Belief Revision
Organizations: Beijing Institute of Technology
Abstract
Reinforcement learning with verifiable rewards ordinarily spreads a single advantage over a whole trajectory, a granularity too coarse to isolate the few decisions on which success in a long, multi-turn interaction actually turns. Privileged self-distillation supplies denser supervision, but it leaves unanswered how a set of local teacher-student discrepancies should be converted into credit along a sequence. We introduce AgentOPSD, a critic-free scheme that assigns credit recursively, one interaction turn at a time. Inside a turn, the token-level log-probability gaps between a skill-conditioned self-teacher and the student are summed, and that sum acts as evidence in a recursive Bayesian update carried out in log-odds space; the resulting revisions reallocate the outcome-derived advantage across turns through a multiplier that is bounded and preserves its sign. With Qwen2.5-3B and Qwen2.5-7B as policy backbones on ALFWorld, the method improves consistently on GRPO and on self-distillation baselines, and attains 89.1% success at the 7B scale. Ablations attribute the gain to aggregating evidence at turn boundaries and to making the belief revision depend on history, rather than to the local gap by itself.
Figures & tables
| ALFWorld | |||||||
| Method | Pick | Look | Clean | Heat | Cool | Pick2 | Avg |
| Qwen2.5-3B-Instruct | |||||||
| Vanilla | 44.4 | 11.1 | 6.2 | 15.4 | 28.6 | 12.5 | 21.9 |
| Skill-Prompt* | 51.7 | 66.7 | 48.4 | 0.0 | 4.3 | 10.0 | 28.9 |
| OPSD | 48.8 | 41.7 | 16.7 | 0.0 | 15.8 | 16.7 | 28.1 |
| GRPO | 91.2 | 62.5 | 96.2 | 61.9 | 65.0 | 47.4 | 75.0 |
| Ablated component | Alteration | ALFWorld |
|---|---|---|
| AgentOPSD (full) | bounded reshaping of turn-level credit, | 89.1 |
| Turn-level granularity of credit | score tokens individually rather than turns | 85.9 |
| Recursive revision of the belief state ( 8 ) | use the raw local gap in place of | 82.8 |
| Outcome-signed credit direction ( 10 ) | keep only the magnitude and drop the outcome sign | 80.5 |
| Anchor at the state prior | initialize without the empirical rate | 78.9 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | SRS | |||||||
|---|---|---|---|---|---|---|---|---|
| AgentOPSD | 8 | 0.2/0.24 | 0.01 | 0.5 | 0.2 | 0.95 | KM |
| ALFWorld | |
|---|---|
| Training steps | 150 |
| Train batch size | 16 |
| Rollout group size | 8 |
| Max prompt length | 2048 |
| Max response length | 512 |
| Max interaction turns | 50 |
Credit Assignment in RL
Credit assignment in reinforcement learning studies how to trace rewards and failures back to the actions, states, or agents that caused them.
See all 225 papers