Step-Level Credit Assignment in RL

RL: Reinforcement Learning

Step-level credit assignment studies how to distribute a task’s final reward across the actions, decisions, or reasoning steps that contributed to it. Common approaches use value functions and Bellman updates, eligibility traces, comparisons between alternative rollouts, hindsight analysis, or process rewards tied to verifiable progress. The problem is especially active in long-horizon language-model agents, where methods are tested on interactive environments such as ALFWorld and WebShop as well as math, code, and robotics tasks. A central tension is that denser feedback can improve learning, but inaccurate local credit can mislead the policy, while reliable labels, extra rollouts, and learned critics can be costly or unavailable.

Latest papers

167 papers55 papers in the last four weeks, up 206% on the four weeks before. 0.3% of all new papers.

PACE: Phase-Progress-Aware Credit for Long-Horizon Embodied Manipulation
Aug 15, 2026Chengye Song, Jiawei Zhang, Rui Song +5Step-Level Credit Assignment in RL Long-Horizon Robotic Manipulation
Intelligence Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Ganjingzi District, Dalian 116024, China · Jilin University, No. 2699 Qianjin Street, Changchun 130012, China
Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents
Aug 13, 2026Zechuan Wang, Siyuan Lu, Hongxuan Zhang +3Token-Level Credit Assignment Step-Level Credit Assignment in RL
Zhejiang University · AWorld Team, Inclusion AI · Shanghai Innovation Institute +2
Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning
Aug 13, 2026Yao Zhou, Hang Gao, Fengge Wu +2Group Relative Policy Optimization Step-Level Credit Assignment in RL
Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents
Aug 13, 2026Haoze Wu, Chuqiao Kuang, Tianyi Zhuang +1Group Relative Policy Optimization Credit Assignment in RL
The Hong Kong University of Science and Technology · Huawei Technologies Ltd.
Redistribution-based Cost Inference Improves Sparse Safe Offline RL
Aug 12, 2026Ebenezer Gelo, Geraud Nangue Tasse, Steven James +1Credit Assignment in RL Step-Level Credit Assignment in RL
University of the Witwatersrand
Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation
Aug 10, 2026Xuan-Phi Nguyen, Zeyu Leo Liu, Yang Li +4On-Policy Self-Distillation Token-Level Credit Assignment
Salesforce AI Research
How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning
Aug 7, 2026Lichao Ma, Yang Sun, Shuaitao Zhao +9Token-Level Credit Assignment Credit Assignment in RL
1Peking University · 2Meituan LongCat Interaction Team · 3Fudan University +4
Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents
Aug 7, 2026Hongxi Yan, Ziyue Huang, Shichao Fan +1Credit Assignment in RL Step-Level Credit Assignment in RL
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Zhongguancun Laboratory, Beijing, China · Qualcomm, Beijing, China
AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning
Aug 6, 2026Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao +10Agentic RL Reinforcement Learning
Tsinghua University · Meituan · Zhejiang University
ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment
Aug 5, 2026Yijun Lu, Rui Ye, Jiajun Wang +4Credit Assignment in RL Step-Level Credit Assignment in RL
Shanghai Jiao Tong University
Agentic Reinforcement Learning with Self-Distilled Reward Shaping
Aug 4, 2026Ranxu Zhang, Guinan Chen, Chenshaodong +5Reward Shaping Token-Level Credit Assignment
Alibaba Group · University of Science and Technology of China Hefei, China · Alibaba Group Hangzhou, China
TCPO: Turn-Level Credit Policy Optimization
Aug 3, 2026Sicong Liao, Zhi Chen, Yaohua TangRL for Language Model Reasoning Credit Assignment in RL
Moore Threads AI
BiCAA: Bidirectional Credit Assignment for Search-Augmented Agent
Aug 2, 2026Yibin Huang, Bin Xu, Hailong Cao +1Process Reward Models Credit Assignment in RL
Faculty of Computing, Harbin Institute of Technology
Gated Q-learning: Add Off-Policy Bias to Taste
Jul 31, 2026Brett DaleyCredit Assignment in RL Step-Level Credit Assignment in RL
Meta
Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning
Jul 30, 2026Qiangqiang He, Zhongheng Wu, ZiJian WangToken-Level Credit Assignment RL for Language Model Reasoning
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Institute of Wireless Communications Technology, Shanghai Jiao Tong University, Shanghai, China
CAST: Game Solvers as Turn-Level Teachers for LLM Agents
Jul 28, 2026Yu Wang, Yi-Kai Zhang, Wentao Shi +8Credit Assignment in RL Reinforcement Learning
University of Science and Technology of China · 4Meituan, China · 2Nanjing University +1
TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
Jul 15, 2026Leitian Tao, Baolin Peng, Wenlin Yao +5Credit Assignment in RL Step-Level Credit Assignment in RL
University of Wisconsin–Madison · Microsoft Research
STAMP: Provenance-Guided Credit Assignment for Deep Search Agents
Jul 13, 2026Ke Xu, Han Xu, Xinran Chen +6Inference-Time Search Step-Level Credit Assignment in RL
Baidu Inc. · Peking University
ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
Jul 3, 2026Zijun Xie, Yuyang You, Yongzhi Li +6Token-Level Credit Assignment RL for Language Model Reasoning
Peking University · Baidu Inc. · Kuaishou Inc.
TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning
Jun 30, 2026Yuanda Xu, Zhengze Zhou, Hejian Sang +6Agentic RL Step-Level Credit Assignment in RL
1LinkedIn Corporation · 2Harvard University · 3Johns Hopkins University +1
CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning
Jun 28, 2026Zibin Meng, Kani ChenAgentic RL KL-Regularized RL
The Hong Kong University of Science and Technology
On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse
Jun 28, 2026Amritansh Mishra, Supriyo Chakraborty, Berkcan KapusuzogluPolicy Gradient Token-Level Credit Assignment
Capital One
Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents
Jun 24, 2026Peng Xu, Sijia Chen, Junzhuo Li +1Reward Shaping Credit Assignment in RL
†The Hong Kong University of Science and Technology (Guangzhou) · ‡The Hong Kong University of Science and Technology
Learning with a Single Rollout via Pass@k Policy Optimization
Jun 24, 2026Fengdi Che, Yang Liu, Lei Yu +4RL for Language Model Reasoning Credit Assignment in RL
University of Alberta · BIGAI · University of Toronto +4
ARCO: Adaptive Rubric with Co-Evolution for Multi-Step LLM-Based Agents
Jun 19, 2026Zihang Tian, Jingsen Zhang, Rui Li +3Process Reward Models Credit Assignment in RL
Renmin University of China
Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards
Jun 17, 2026Yingyu Shan, Yuhang Guo, Zihao Cheng +7Token-Level Credit Assignment RL for Language Model Reasoning
Beijing Institute of Technology · Beihang University · Independent Researcher
STRIDE: Strategic Trajectory Reasoning via Discriminative Estimation for Verifiable Reinforcement Learning
Jun 14, 2026Qinjian Zhao, Zhihao Dou, Dinggen Zhang +10RL for Language Model Reasoning Credit Assignment in RL
Kean University · Case Western Reserve University · University of Texas at Austin +4
Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning
Jun 14, 2026Yu Li, Shu Hong, Tian LanRL for Language Model Reasoning Credit Assignment in RL
Department of Electrical and Computer Engineering, George Washington University
3SPO: State-Score-Supervised Policy Optimization for LLM Agents
Jun 8, 2026Yu Han, Kailing Li, Yang Jiao +4Credit Assignment in RL Step-Level Credit Assignment in RL
School of Computer Science and Technology, East China Normal University · Fudan University · KAUST +1
PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment
Jun 8, 2026Yang Tian, Rui Wang, Xumeng Wen +5Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards
School of AI, Shanghai Jiao Tong University · 2XYZ AI Lab

Aug 15, 2026J/K move · Enter open · S save

PACE: Phase-Progress-Aware Credit for Long-Horizon Embodied Manipulation

Chengye Song, Jiawei Zhang, Rui Song, Shengqi Wang+4

Intelligence Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Ganjingzi District, Dalian 116024, China · Jilin University, No. 2699 Qianjin Street, Changchun 130012, China

Step-Level Credit Assignment in RL Long-Horizon Robotic Manipulation RL Post-Training VLMs for Robotics Policy Distillation

Open paper AskPDF arXiv

Post-training of vision-language-action (VLA) models typically relies on expert demonstrations and policy interaction trajectories. However, in long-horizon manipulation, a single episode often spans hundreds of control steps and multiple phases, while success or failure is only revealed at episode termination. Policy improvement therefore requires step-level credit signals to distinguish behaviors that advance the task from those that stall or regress. We present PACE, a credit-assignment framework for post-training on long-horizon manipulation, centered on a phase-progress-aware critic. PACE consists of two key modules: (1) the Global-Local Cooperative Value-Correction Critic (GLC-Critic) aggregates visual and motion-difference features within local temporal windows to infer the phase and intra-phase progress of each step, and applies residual correction to a discretized remaining-cost distribution accordingly, enabling step-level credit assignment; (2) Progressive Policy Distillation (PPD) converts credit into positive and negative conditions via task-wise thresholds and trains a credit-conditioned action generation policy: it first protects the pretrained policy with high-credit positive samples, then incorporates all positive and negative credits to learn the quality boundary, and at inference amplifies high-credit behaviors through the difference between conditional outputs. Extensive simulation experiments and diverse real-world robotic-arm experiments demonstrate that PACE consistently achieves significant improvements over the strongest baseline.