Step-Level Credit Assignment in RL

RL: Reinforcement Learning

Step-level credit assignment studies how to distribute a task’s final reward across the actions, decisions, or reasoning steps that contributed to it. Common approaches use value functions and Bellman updates, eligibility traces, comparisons between alternative rollouts, hindsight analysis, or process rewards tied to verifiable progress. The problem is especially active in long-horizon language-model agents, where methods are tested on interactive environments such as ALFWorld and WebShop as well as math, code, and robotics tasks. A central tension is that denser feedback can improve learning, but inaccurate local credit can mislead the policy, while reliable labels, extra rollouts, and learned critics can be costly or unavailable.

Latest papers

167 papers55 papers in the last four weeks, up 206% on the four weeks before. 0.3% of all new papers.

The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning
Jun 8, 2026Aakriti Agrawal, Souradip Chakraborty, Armin Saghafian +6Process Reward Models RL for Language Model Reasoning
University of Maryland, College Park · Capital One · University of Central Florida
RREDCoT: Segment-Level Reward Redistribution for Reasoning Models
Jun 4, 2026Mykyta Ielanskyi, Kajetan Schweighofer, Lukas Aichberger +1Reinforcement Learning RL for Language Model Reasoning
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria · Cognizant AI Lab, San Francisco, USA · Work done while at Johannes Kepler University Linz. +1
When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training
Jun 4, 2026Yuanfan Li, Qi Zhou, Wenjing Duan +1Credit Assignment in RL Step-Level Credit Assignment in RL
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Faculty of Electronic and Information Engineering, Xi’an Jiaotong University
SPADER: Step-wise Peer Advantage with Diversity-Aware Exploration Rewards for Multi-Answer Question Answering
May 30, 2026Qiming Shi, Zhaolu Kang, Yunfan Zhou +2Credit Assignment in RL Step-Level Credit Assignment in RL
State Key Lab of CAD&CG, Zhejiang University · School of Software and Microelectronics, Peking University · School of Software Technology, Zhejiang University
Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modeling
May 28, 2026Yuchen Liu, Yingjie Feng, Lixiong Qin +5Reward Modeling Agentic RL
Beijing University of Posts and Telecommunications · Li Auto Inc.
StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning
May 26, 2026Yanfei Zhang, Xu Lin, Chenglin WuCredit Assignment in RL Step-Level Credit Assignment in RL
Independent Researcher · Tencent · DeepWisdom
Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning
May 26, 2026Xin Cheng, Shuo He, Lang Feng +4Group Relative Policy Optimization Credit Assignment in RL
Nanyang Technological University, Singapore · Tongyi Lab, Alibaba Group · Southeast University, China
RICE-PO: Turning Retrieval Interactions into Credit Signals for Reasoning Agents
May 25, 2026Mingchen Li, Hansi Zeng, Zhuo Qian +3RL for Language Model Reasoning Credit Assignment in RL
University of Massachusetts, Amherst · Texas Tech University · University of Connecticut
Credit Assignment with Resets in Language Model Reasoning
May 25, 2026Ankur Samanta, Akshayaa Magesh, Ayush Jain +7RL for Language Model Reasoning Credit Assignment in RL
1Meta AI · 2Columbia University · *Work done at Meta +2
From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning
May 21, 2026Xitai Jiang, Zihan Tang, Wenze Lin +3RL for Language Model Reasoning Credit Assignment in RL
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · B Gao Huang
OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning
May 21, 2026Yu Li, Rui Miao, Tian Lan +1Value Function Estimation Token-Level Credit Assignment
1George Washington University · 2The University of Texas at Dallas
What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents
May 19, 2026Xiaozhe Li, Tianyi Lyu, Yang Li +6Reinforcement Learning Credit Assignment in RL
1Tongji University · 2Shanghai AI Laboratory · 5Independent +2
AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment
May 18, 2026Zhenlin Wei, Pu Jian, Yingzhuo Deng +6Token-Level Credit Assignment RL for Language Model Reasoning
Meituan Beijing, China · University of Chinese Academy of Sciences
PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization
May 18, 2026Wonjoong Kim, Yeonjun In, Sangwu Park +2Reward Modeling Credit Assignment in RL
KAIST · Yonsei University
Step-wise Rubric Rewards for LLM Reasoning
May 17, 2026Weichu Xie, Haozhe Zhao, Wenpu Liu +15Reward Modeling Credit Assignment in RL
Peking University · JD Explore Academy · Shanghai Jiao Tong University +2
GAGPO: Generalized Advantage Grouped Policy Optimization
May 13, 2026Siyuan Zhu, Chao Yu, Rongxin Yang +4Reinforcement Learning Group Relative Policy Optimization
School of Computer Science and Engineering, Sun Yat-sen University · Meituan
Verifiable Process Rewards for Agentic Reasoning
May 11, 2026Huining Yuan, Zelai Xu, Huaijie Wang +6Process Reward Models Credit Assignment in RL
Tsinghua University
PiCA: Pivot-Based Credit Assignment for Search Agentic Reinforcement Learning
May 10, 2026Dongyi Liu, Yifan Niu, Qinwen Wang +2Reinforcement Learning Reward Shaping
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking
May 9, 2026Zhida He, Xiaoyu Wen, Han Qi +7RL for Language Models Credit Assignment in RL
Shanghai AI Laboratory · Fudan University · Shanghai Jiao Tong University
DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
May 8, 2026Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade +4Group Relative Policy Optimization Credit Assignment in RL
The Ohio State University · Apple
Selective Successor Eligibility Traces for RLVR
May 7, 2026Chaoli Mou, Xinning Chen, Zhan Zhuang +3RL for Language Model Reasoning Credit Assignment in RL
Southern University of Science and Technology · City University of Hong Kong
Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifiers
May 6, 2026Senkang Hu, Yong Dai, Xudong Han +4Agentic RL Reinforcement Learning
Hong Kong JC STEM Lab of Smart City · City University of Hong Kong · University of Oxford +4
DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignment
May 5, 2026Hongbo Jin, Rongpeng Zhu, Zhongjing Du +4Token-Level Credit Assignment RL for Language Model Reasoning
Peking University · SJTU · Tsinghua University
SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attribution
Apr 21, 2026Xiachong Feng, Yi Jiang, Xiaocheng Feng +9Credit Assignment in RL Step-Level Credit Assignment in RL
The University of Hong Kong · Harbin Institute of Technology · Harbin Institute of Technology, Shenzhen
Reducing Credit Assignment Variance via Counterfactual Reasoning Paths
Apr 20, 2026Fei Ding, Yongkang Zhang, Youwei Wang +1RL for Language Model Reasoning Credit Assignment in RL
Alibaba Group · Tsinghua University
Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning
Apr 19, 2026Fei Ding, Yongkang Zhang, Runhao Liu +4RL for Language Model Reasoning Credit Assignment in RL
Alibaba Group · Tsinghua University
Soft $Q(λ)$ : A multi-step off-policy method for entropy regularised reinforcement learning using eligibility traces
Apr 15, 2026Pranav Mahajan, Ben SeymourReinforcement Learning Q-Learning
University of Oxford
Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization
Apr 14, 2026Shiping Gao, Hongzhan Chen, Xiaojun Quan +2Process Reward Models RL for Language Model Reasoning
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, Guangdong, China · Shenzhen Loop Area Institute, Shenzhen, Guangdong, China · Meta AI, Menlo Park, California, USA +1
MindGames Arena Generalization Track: In2AI Solution with Delayed Per-Step Reward Attribution
Apr 13, 2026Aliaksei Korshuk, Alexander Buyantuev, Ilya MakarovRL for Game AI RL for Language Models
iMak AI Lab · Coframe
RTMC: Step-Level Credit Assignment via Rollout Trees
Apr 13, 2026Tao Wang, Suhang Zheng, Xiaoxiao XuStep-Level Credit Assignment in RL Agentic RL
Alibaba Group

Jun 8, 2026J/K move · Enter open · S save

The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning

Aakriti Agrawal, Souradip Chakraborty, Armin Saghafian, Nihal Sharma+5

University of Maryland, College Park · Capital One · University of Central Florida

Process Reward Models RL for Language Model Reasoning Step-Level Credit Assignment in RL Step-Level Credit Assignment

Open paper AskPDF arXiv

Process Reward Models (PRMs) improve credit assignment for reasoning by providing step-level feedback. However, we identify a hidden bias in PRMs caused by severe imbalance in step-level training data. Standard cross-entropy training amplifies this bias, causing PRMs to overcredit plausible but incorrect steps and produce high false-positive rates. We show that these false positives have an asymmetric downstream effect: false negatives mainly slow exploration, whereas false positives actively steer Best-of-N selection, guided decoding, and policy optimization toward flawed reasoning. This suggests that PRM training should shift from pointwise label fitting to reliable relative comparisons. To address this, we propose PRISM (Precision Ranking for Improved Step Modeling), a policy-aware PRM training framework that learns from contrastive step-level comparisons and hard negatives generated by a temporal lookahead strategy, requiring no new human labels. We further use a difficulty-aware curriculum to optimize the contrastive step margin. Across PRMBench and ProcessBench, PRISM substantially reduces false positives (22% on PRMBench) and improves macro F1 over strong discriminative PRMs. When applied to policy optimization and search tasks, including guided decoding and Best-of-N selection, it consistently improves accuracy (up to 22% for guided decoding and 33% for Best-of-N) and robustness. More broadly, trustworthy process supervision is not just about assigning high rewards, but about rewarding the right reasoning for the right reasons.