cs.CLNov 9, 2026

AgentOPSD: Turn-Level Credit Assignment via Recursive Belief Revision

Authors: Yizhuo Li

Organizations: Beijing Institute of Technology

Abstract

Reinforcement learning with verifiable rewards ordinarily spreads a single advantage over a whole trajectory, a granularity too coarse to isolate the few decisions on which success in a long, multi-turn interaction actually turns. Privileged self-distillation supplies denser supervision, but it leaves unanswered how a set of local teacher-student discrepancies should be converted into credit along a sequence. We introduce AgentOPSD, a critic-free scheme that assigns credit recursively, one interaction turn at a time. Inside a turn, the token-level log-probability gaps between a skill-conditioned self-teacher and the student are summed, and that sum acts as evidence in a recursive Bayesian update carried out in log-odds space; the resulting revisions reallocate the outcome-derived advantage across turns through a multiplier that is bounded and preserves its sign. With Qwen2.5-3B and Qwen2.5-7B as policy backbones on ALFWorld, the method improves consistently on GRPO and on self-distillation baselines, and attains 89.1% success at the 7B scale. Ablations attribute the gain to aggregating evidence at turn boundaries and to making the belief revision depend on history, rather than to the local gap by itself.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

    Aug 6, 2026Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao +10Agentic RLReinforcement Learning

  2. StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning

    May 26, 2026Yanfei Zhang, Xu Lin, Chenglin WuCredit Assignment in RLStep-Level Credit Assignment in RL

  3. UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learning

    Sep 28, 2026Zenghuang Fu, Zhaoyang Li, Qiuyuan Ai +8Agentic RLCredit Assignment in RL