Step-Level Credit Assignment in RL

RL: Reinforcement Learning

Step-level credit assignment studies how to distribute a task’s final reward across the actions, decisions, or reasoning steps that contributed to it. Common approaches use value functions and Bellman updates, eligibility traces, comparisons between alternative rollouts, hindsight analysis, or process rewards tied to verifiable progress. The problem is especially active in long-horizon language-model agents, where methods are tested on interactive environments such as ALFWorld and WebShop as well as math, code, and robotics tasks. A central tension is that denser feedback can improve learning, but inaccurate local credit can mislead the policy, while reliable labels, extra rollouts, and learned critics can be costly or unavailable.

Latest papers

167 papers55 papers in the last four weeks, up 206% on the four weeks before. 0.3% of all new papers.

Cheap Retrieval and Observation Credit Assignment from One Quantity: The Value of Information
Oct 29, 2026Daniel Strauss, Nikolaos Sotirakis, Zhenshan Bing +1Reinforcement Learning Memory-Augmented RL
Technical University of Munich · Nanjing University · Harbin Institute of Technology
Where to Learn and How Far to Roll Out: Efficient Adaptive Training for GUI Agents
Oct 29, 2026Yuxiang Chai, Quyu Kong, Panrong Tong +2RL for GUI Agents Reinforcement Learning
Alibaba Token Hub, Alibaba Group · MMLab, The Chinese University of Hong Kong
Action-Value Calibration via Continuation Sharing for Long-Horizon Agentic Reinforcement Learning
Oct 27, 2026Yingjian Chen, Lang Zhou, Zixin Zhao +3Agentic RL Step-Level Credit Assignment in RL
Sun Yat-sen University · Shenzhen Loop Area Institute
DTCA: Denoising Token Credit Assignment for Reinforcement Learning in Diffusion Language Models
Oct 26, 2026Hongze Chen, Miao Pan, Lizhou Liang +7Diffusion Language Model Training RL for Diffusion Models
Zhejiang University · National University of Singapore · SAP
From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning
Oct 23, 2026Yuhang Han, Dingning Liu, Yujie Chen +7RL for VLMs Step-Level Credit Assignment in RL
The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Jiao Tong University · Shadoweave +2
Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning
Oct 22, 2026Zeropy Surio, Tobias Engel, Mei-Lin Chou +2Step-Level Credit Assignment in RL Counterfactual Credit Assignment
University of Sheffield · Carnegie Mellon University · University of Michigan
RIPPLE: Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit for Long-Horizon Agentic Coding
Oct 22, 2026Jingxuan Xu, Zihang Xu, Yuechen Luo +9Code Generation RL for LLM Agents
National University of Singapore, Singapore · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Tsinghua University, Beijing, China +1
Self-Credit: Credit Assignment for Long-Horizon Agentic RL via Intra-group Rescaling
Oct 20, 2026Fangcong Yin, Ming Yin, Tanvi Bhandarkar +7Agentic RL Step-Level Credit Assignment in RL
Meta · New York University · Duke University
TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment
Oct 20, 2026Yi Wen, Hao Chen, Wanyu Wang +9RL for Language Model Reasoning RL Post-Training
City University of Hong Kong · Tencent · Nanjing University
From Outcomes to Steps: Reliable Success-Aware Critic Learning for Long-Horizon Language Agents
Oct 20, 2026Jiaxi Liu, Shixiong Kai, Zhentao Tang +4Reinforcement Learning RL for LLM Agents
Shanghai Jiao Tong University · Huawei Noah’s Ark Lab
Policy-Adaptive Credit Assignment for Spatial Reasoning with Verifiable Rewards
Oct 19, 2026Hailiang Yang, Shengchao ChenProcess Reward Models Step-Level Credit Assignment in RL
School of Computer Science and Engineering, University of New South Wales · Australian AI Institute, University of Technology Sydney
ClueFlow: Learning Long-Horizon Search through Attributed Research-State Transitions
Oct 19, 2026Yicheng Wang, Wentao Tan, Yidan Huang +2Long-Horizon LLM Agents RL for LLM Agents
Joy Future Academy
Learning to Reason Efficiently with Interventional Step Credit
Oct 19, 2026Guanda Li, Xuefeng Bai, Yulong Chen +3Efficient Language Model Reasoning Step-Level Credit Assignment in RL
Harbin Institute of Technology, Shenzhen · University of Aberdeen · University of Cambridge
ToolOPD: Credit-Aware On-Policy Distillation for Tool-Use Agents
Oct 19, 2026Chu Zhao, Lei Tang, Yanming Wang +5On-Policy Distillation Step-Level Credit Assignment in RL
Northeastern University · Honor Device Co., Ltd
Outcome-Anchored Credit Reallocation for Agentic Knowledge Graph Question Answering
Oct 18, 2026Mingyuan Li, Yucheng Li, Xiaxia Wang +1Knowledge Graph Question Answering RL for LLM Agents
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Department of Computer Science, University of Oxford
EG-GRPO: Evidence-Guided Step-level Credit Assignment for Deep Search Agents
Oct 18, 2026Sijia Luo, Mengjie Gao, Yuxiao He +4RL for LLM Agents Step-Level Credit Assignment in RL
Renmin University of China, Beijing 100872, China · Key Laboratory of Data Engineering and Knowledge Engineering, MOE, China · Ant Group, Beijing, China +2
On Reparameterizing the Score Function
Oct 16, 2026Mohammad Hossein Amani, Leonardo Bocchieri, Maxime Peyrard +2Policy Gradient Policy Gradient Methods
EPFL · CNRS, Univ. Grenoble Alpes · XENT Labs
Visual Causal-chain Bookmarking: Retrieval-Weighted Credit Assignment for Reinforcement Learning from Delayed Reward
Oct 16, 2026Prathap SelvakumarCredit Assignment in RL Step-Level Credit Assignment in RL
University of Manchester
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
Oct 16, 2026Heming Zou, Qi Wang, Yixiu Mao +7RL for LLM Agents Long-Horizon RL
Tsinghua University · Foundation Model Department, Tencent
Distill Until Divergence: Error-Truncated Hindsight Distillation for Agentic RL
Oct 16, 2026Yibin Huang, Conghui ZhuAgentic RL Step-Level Credit Assignment in RL
Faculty of Computing, Harbin Institute of Technology
OpCredit: Operation-Aware Policy Learning for Agentic Knowledge Graph Question Answering
Oct 15, 2026Yucheng Li, Mingyuan Li, Xiaxia Wang +1Knowledge Graph Question Answering Policy Learning
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Department of Computer Science, University of Oxford
Bellman-Consistent Trajectory RL for LLM Arithmetic Reasoning
Oct 15, 2026Lianfa LiRL for Language Model Reasoning Reinforcement Learning
Department of Computer Science, Stanford University · LREIS, Institute of Geographical Sciences and Natural Resources Research, Chinese Academy of Sciences
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
Oct 14, 2026Wei Chen, Yubing Wu, Zhou Xu +9RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
South China University of Technology · Tsinghua University · The Hong Kong University of Science and Technology +3
C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learning
Oct 14, 2026Ai Jian, Haoran Pan, Eryu Guo +5Multi-Turn RL Reinforcement Learning
Beijing University of Posts and Telecommunications · Peking University · Institute of Automation, Chinese Academy of Sciences +2
Contextual Credit Policy Optimization
Oct 13, 2026Huiwen Han, Ziyao Zeng, Zihao Zhu +6RL for LLM Agents Step-Level Credit Assignment in RL
Stanford University · Texas A&M University · Yale University +3
ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning
Oct 13, 2026Bowen Ding, Mingda Wan, Yuhan Chen +7Agentic RL Reinforcement Learning
Zhejiang University · School of Engineering, Westlake University · Huawei Noah’s Ark Lab +1
CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning
Oct 13, 2026Guangcheng Zhu, Yingfan Ma, Xing Zheng +7RL for Language Models Rubric-Based RL
Zhejiang University · Ant Group
Adaptive-Group Process Rewards for GUI Agents
Oct 13, 2026Zirui Shang, Yang Liu, Xin Shu +3RL for GUI Agents Process Reward Models
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · State Key Laboratory of General Artificial Intelligence, BIGAI · Wuhan University +1
VAttnPO: Visual Attention-Guided Policy Optimization
Oct 12, 2026Lihao Zheng, Zhenwei Shao, Zhou Yu +5RL for VLMs Reinforcement Learning with Verifiable Rewards
School of Computer Science and Technology, Hangzhou Dianzi University · Base Model, Li Auto
LLM Value Functions for Credit Assignment in Agentic RL
Oct 11, 2026Kaijie Zhu, Weixiang Yan, Omar Attia +3RL for LLM Agents Agentic RL
Apple

Oct 29, 2026J/K move · Enter open · S save

Cheap Retrieval and Observation Credit Assignment from One Quantity: The Value of Information

Daniel Strauss, Nikolaos Sotirakis, Zhenshan Bing, Alois Knoll

Technical University of Munich · Nanjing University · Harbin Institute of Technology

Reinforcement Learning Memory-Augmented RL Step-Level Credit Assignment in RL RL for IR

Open paper AskPDF arXiv

Memory in reinforcement learning raises two problems. Attention finds the relevant item by scoring the whole store, which is costly for robots under latency and energy limits. When a retrieved memory pays off, credit for acquiring it must cross the whole sequence. We argue that both are governed by one quantity, the value of information, and propose to estimate it from archived rollouts with a learned world model. We obtain that, when read at the observation, it yields a provable lower bound on the optimal value of the acquiring action, which floors its TD target and attributes this floor to the observation that earned it. Read at the decision, it defines the optimal retrieval set. A pilot retrieval system, to be trained and improved with this signal, solves a memory task with reduced context.