Step-Level Credit Assignment in RL

RL: Reinforcement Learning

Step-level credit assignment studies how to distribute a task’s final reward across the actions, decisions, or reasoning steps that contributed to it. Common approaches use value functions and Bellman updates, eligibility traces, comparisons between alternative rollouts, hindsight analysis, or process rewards tied to verifiable progress. The problem is especially active in long-horizon language-model agents, where methods are tested on interactive environments such as ALFWorld and WebShop as well as math, code, and robotics tasks. A central tension is that denser feedback can improve learning, but inaccurate local credit can mislead the policy, while reliable labels, extra rollouts, and learned critics can be costly or unavailable.

Latest papers

167 papers55 papers in the last four weeks, up 206% on the four weeks before. 0.3% of all new papers.

SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting
Apr 12, 2026Binbin Zheng, Xing Ma, Yiheng Liang +6On-Policy Distillation RL for Language Model Reasoning
University of Science and Technology of China · Meituan LongCat Interaction Team · Nanjing University +2
Reason in Chains, Learn in Trees: Self-Rectification and Grafting for Multi-turn Agent Policy Optimization
Apr 8, 2026Yu Li, Sizhe Tang, Tian LanRL for Language Model Reasoning RL for LLM Agents
Department of Electrical and Computer Engineering, George Washington University
Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation
Apr 4, 2026Fei Ding, Yongkang Zhang, youwei wang +1RL for Language Model Reasoning Step-Level Credit Assignment in RL
Alibaba Group · Tsinghua University
Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration
Apr 3, 2026Wachiravit Modecrua, Krittanon Kaewtawee, Krittin Pachtrachai +1Multi-Turn RL RL for Tool Use
Amity Research and Application Center (ARAC)
Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
Apr 3, 2026Tianze Xu, Yanzhao Zheng, Pengrui Lu +11RL for Language Models Rubric-Based RL
Shanghai Jiao Tong University · GAIR · Alibaba Group +2
Stepwise Credit Assignment for GRPO on Flow-Matching Models
Mar 30, 2026Yash Savani, Branislav Kveton, Yuchen Liu +5RL for Diffusion Models Step-Level Credit Assignment in RL
Carnegie Mellon University · Adobe Research
SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation
Mar 17, 2026Long Li, Zhijian Zhou, Jiangxuan Long +5Text-to-SQL Step-Level Credit Assignment in RL
Griffith University, Brisbane, Australia · Fudan University, Shanghai, China · Shanghai Innovation Institute, China +4
APEX-Searcher: Refining Credit Assignment with Subgoaling for Agentic Retrieval-Augmented Generation
Mar 14, 2026Kun Chen, Qingchao Kong, Zhao Feifei +1Agentic RAG RL for RAG
University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences · Wenge Technology Co., Ltd
On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agents
Mar 12, 2026Deyu Zou, Yongqiang Chen, Fan Feng +4RL for LLM Agents Active Information Gathering
The Chinese University of Hong Kong · University of California San Diego · Georgia Institute of Technology +1
Hindsight Credit Assignment for Long-Horizon LLM Agents
Mar 7, 2026Hui-Ze Tan, Xiao-Wen Yang, Hao Chen +7Credit Assignment in RL Long-Horizon LLM Agents
State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · FiT, Tencent, Shenzhen, China +2
MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue
Mar 6, 2026Naifan Zhang, Ruihan Sun, Jinwei Su +4RL for Language Models Credit Assignment in RL
NatureSelect · Tsinghua University · South China Normal University +1
Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents
Mar 2, 2026Haojin Yang, Ai Jian, Xinyue Huang +5RL for LLM Agents Reinforcement Learning
Peking University · Beijing University of Posts and Telecommunications · Sun Yat-sen University +2
Truncated Step-Level Sampling with Process Rewards for Retrieval-Augmented Reasoning
Feb 26, 2026Chris Samarinas, Haw-Shiuan Chang, Hamed ZamaniReinforcement Learning Process Reward Models
Center for Intelligent Information Retrieval University of Massachusetts Amherst Amherst, MA, United States
Proximity-Based Multi-Turn Optimization: Practical Credit Assignment for LLM Agent Training
Feb 22, 2026Yangyi Fang, Jiaye Lin, Xiaoliang Fu +4LLM Agent Training RL for LLM Agents
Tsinghua University · Fudan University · Peking University +3
Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models
Feb 19, 2026Wen-Tse Chen, Jiayu Chen, Fahim Tajwar +4Step-Level Credit Assignment in RL Sparse-Reward RL
Carnegie Mellon University · The University of Hong Kong · Stanford University
HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
Feb 18, 2026Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou +4RL for LLM Agents Hierarchical RL
University of Minnesota · Northwestern University · Amazon AGI +2
TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels
Apr 17, 2025Siow Meng Low, Ze Gong, Akshat KumarRL Control Credit Assignment in RL
School of Computing and Information Systems, Singapore Management University, Singapore · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

Apr 12, 2026J/K move · Enter open · S save

SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting

Binbin Zheng, Xing Ma, Yiheng Liang, Jingqing Ruan+5

University of Science and Technology of China · Meituan LongCat Interaction Team · Nanjing University+2

On-Policy Distillation RL for Language Model Reasoning Step-Level Credit Assignment in RL Uncertainty-Aware Knowledge Distillation

Open paper AskPDF arXiv

On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult. On-Policy Distillation (OPD) alleviates this by introducing dense, token-level KL supervision from a teacher model, but typically applies this supervision uniformly across all rollouts, ignoring fundamental differences in signal quality. We propose Signal-Calibrated On-Policy Distillation Enhancement (SCOPE), a dual-path adaptive training framework that routes on-policy rollouts by correctness into two complementary supervision paths. For incorrect trajectories, SCOPE performs teacher-perplexity-weighted KL distillation to prioritize instances where the teacher demonstrates genuine corrective capability, while down-weighting unreliable guidance. For correct trajectories, it applies student-perplexity-weighted MLE to concentrate reinforcement on low-confidence samples at the capability boundary rather than over-reinforcing already mastered ones. Both paths employ a group-level normalization to adaptively calibrate weight distributions, accounting for the intrinsic difficulty variance across prompts. Extensive experiments on six reasoning benchmarks show that SCOPE achieves an average relative improvement of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines, demonstrating its consistent effectiveness.