Credit Assignment in RL

RL: Reinforcement Learning

Credit assignment asks which decisions deserve credit or blame when rewards arrive late or apply to an entire trajectory, a central problem in long-horizon and cooperative tasks. Traditional approaches include temporal-difference learning, eligibility traces, critics and advantage estimation, while reward shaping, counterfactual comparisons, and Shapley values offer ways to make contributions more explicit. Recent work on language-model agents often assigns feedback at the token, step, or turn level using process rewards, hindsight, trajectory graphs, or comparisons between alternative rollouts; related methods address multi-agent control and robotics. The key tension is that denser signals can improve learning but may be noisy, biased, or expensive, and it remains difficult to establish that a local action actually caused progress rather than merely accompanied it.

Latest papers

219 papers47 papers in the last four weeks, up 88% on the four weeks before. 0.3% of all new papers.

GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation
May 12, 2026Sijia Li, Yuchen Huang, Zifan Liu +7RL for Language Model Reasoning Credit Assignment in RL
Hong Kong University of Science and Technology · Microsoft Research Asia
Verifiable Process Rewards for Agentic Reasoning
May 11, 2026Huining Yuan, Zelai Xu, Huaijie Wang +6Process Reward Models Credit Assignment in RL
Tsinghua University
PiCA: Pivot-Based Credit Assignment for Search Agentic Reinforcement Learning
May 10, 2026Dongyi Liu, Yifan Niu, Qinwen Wang +2Reinforcement Learning Reward Shaping
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Models
May 9, 2026Yuanhao Li, Hongbo Wang, Xiaotang Shang +3Automated Program Repair Process Reward Models
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China · University of Luxembourg
Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking
May 9, 2026Zhida He, Xiaoyu Wen, Han Qi +7RL for Language Models Credit Assignment in RL
Shanghai AI Laboratory · Fudan University · Shanghai Jiao Tong University
Learning CLI Agents with Structured Action Credit under Selective Observation
May 8, 2026Haoyang Su, Ying WenTerminal-Agent Benchmarks Computer-Use Agents
Fudan University · Shanghai Innovation Institute · Shanghai Jiao Tong University
The Reciprocity Gradient
May 8, 2026Yue Lin, Pascal Poupart, Shuhui Zhu +5Credit Assignment in RL Policy Gradient Methods
The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · University of Waterloo +2
DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
May 8, 2026Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade +4Group Relative Policy Optimization Credit Assignment in RL
The Ohio State University · Apple
SHARP: A Self-Evolving Human-Auditable Rubric Policy for Financial Trading Agents
May 7, 2026Xiwen Chen, Wenhui Zhu, Songzhu Zheng +3Reinforcement Learning Credit Assignment in RL
Morgan Stanley · Arizona State University · Columbia University +1
A $^2$ TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping
May 7, 2026Dingwei Chen, Zefang Zong, Zhipeng Ma +5Credit Assignment in RL Policy Gradient Methods
Tencent Inc · The Chinese University of Hong Kong · Shenzhen MSU-BIT University
Milestone-Guided Policy Learning for Long-Horizon Language Agents
May 7, 2026Zixuan Wang, Yuchen Yan, Hongxing Li +7Credit Assignment in RL Long-Horizon Agent Tasks
Zhejiang University · Baidu Inc.
Selective Successor Eligibility Traces for RLVR
May 7, 2026Chaoli Mou, Xinning Chen, Zhan Zhuang +3RL for Language Model Reasoning Credit Assignment in RL
Southern University of Science and Technology · City University of Hong Kong
EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance
May 6, 2026Song Yu, Li Li, Wenwen Zhao +1RL for Language Model Reasoning Group Relative Policy Optimization
School of Computer and Information Science, Southwest University, Chongqing 400715, China
Every Step Counts: Step-Level Credit Assignment for Tool-Integrated Text-to-SQL
May 6, 2026Yaxun Dai, Baolin Sun, Junying Wang +6Credit Assignment in RL Text-to-SQL
Institute of Computer Science and Technology, Soochow University, China · Ant Digital Technologies, Ant Group · School of Management, University of Science and Technology of China
Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards
May 5, 2026Tianyang Han, Hengyu Shi, Junjie Hu +3RL for Code Generation Process Reward Models
1D4 Lab · 2Independent Researcher
DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignment
May 5, 2026Hongbo Jin, Rongpeng Zhu, Zhongjing Du +4Token-Level Credit Assignment RL for Language Model Reasoning
Peking University · SJTU · Tsinghua University
AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning
May 1, 2026Haotian Zhao, Songlin Zhou, Yuxin Zhang +9Agentic RL Credit Assignment in RL
Baidu · Tsinghua University · Fudan University
SOLAR-RL: Semi-Online Long-horizon Assignment Reinforcement Learning
Apr 24, 2026Jichao Wang, Liuyang Bian, Yufeng Zhou +9RL for GUI Agents Credit Assignment in RL
1vivo AI Lab · 2Zhejiang Lab · 5Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences +2
SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attribution
Apr 21, 2026Xiachong Feng, Yi Jiang, Xiaocheng Feng +9Credit Assignment in RL Step-Level Credit Assignment in RL
The University of Hong Kong · Harbin Institute of Technology · Harbin Institute of Technology, Shenzhen
StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning
Apr 20, 2026Daoyu Wang, Qingchuan Li, Mingyue Cheng +4Agentic RL Credit Assignment in RL
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China
Reducing Credit Assignment Variance via Counterfactual Reasoning Paths
Apr 20, 2026Fei Ding, Yongkang Zhang, Youwei Wang +1RL for Language Model Reasoning Credit Assignment in RL
Alibaba Group · Tsinghua University
Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought
Apr 20, 2026Muhammed Emrullah Ildiz, Halil Alperen Gozeten, Ege Onur Taga +1CoT Reasoning RL for Language Model Reasoning
University of Michigan, Ann Arbor · Google Research
COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams
Apr 20, 2026Shripad Deshmukh, Jayakumar Subramanian, Raghavendra Addanki +1Sequential MARL Credit Assignment in RL
College of Information and Computer Sciences University of Massachusetts Amherst · Adobe Research San Jose, CA
Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning
Apr 19, 2026Fei Ding, Yongkang Zhang, Runhao Liu +4RL for Language Model Reasoning Credit Assignment in RL
Alibaba Group · Tsinghua University
IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning
Apr 16, 2026Zihan Liang, Yufei Ma, Ben Chen +6Retrieval-Augmented Generation RL for Language Model Reasoning
Kuaishou Technology
Representation over Routing: Diagnosing Temporal Routing Pathologies in Multi-Timescale PPO
Apr 15, 2026Jing SunReinforcement Learning Credit Assignment in RL
Information Engineering School, Chengyi College, Jimei University, Xiamen 361000
Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning
Apr 12, 2026Zikang Shan, Han Zhong, Liwei Wang +1RL for Language Models Reinforcement Learning
Peking University · Microsoft Research Asia · Shanghai Jiao Tong University
From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
Apr 10, 2026Chenchen ZhangCredit Assignment in RL RL for Language Models
Independent Researcher
Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control
Apr 3, 2026Siwei Ju, Jan Tauberschmidt, Oleg Arenz +2Constrained RL RL for Autonomous Driving
Intelligent Autonomous Systems Lab, Department of Computer Science, TU Darmstadt, Germany. · Porsche AG, Stuttgart, Germany. · German Research Center for Artificial Intelligence (DFKI), Germany.
ShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Training
Mar 31, 2026Rui Ai, Yu Pan, David Simchi-Levi +1RL Post-Training Reward Design for RL
MIT · University of Sydney · University of Texas at Dallas

May 12, 2026J/K move · Enter open · S save

GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation

Sijia Li, Yuchen Huang, Zifan Liu, Yanping Li+6

Hong Kong University of Science and Technology · Microsoft Research Asia

RL for Language Model Reasoning Credit Assignment in RL Policy Learning Reward Design for RL RL for LLM Agents RL for Tool Use

Open paper AskPDF arXiv

Reinforcement learning has become a widely used post-training approach for LLM agents, where training commonly relies on outcome-level rewards that provide only coarse supervision. While finer-grained credit assignment is promising for effective policy updates, obtaining reliable local credit and assigning it to the right parts of the long-horizon trajectory remains an open challenge. In this paper, we propose Granularity-adaptivE Advantage Reweighting (GEAR), an adaptive-granularity credit assignment framework that reshapes the trajectory-level GRPO advantage using token- and segment-level signals derived from self-distillation. GEAR compares an on-policy student with a ground-truth-conditioned teacher to obtain a reference-guided divergence signal for identifying adaptive segment boundaries and modulating local advantage weights. This divergence often spikes at the onset of a semantic deviation, while later tokens in the same autoregressive continuation may return to low divergence. GEAR therefore treats such spikes as anchors for adaptive credit regions: where the student remains aligned with the teacher, token-level resolution is preserved; where it departs, GEAR groups the corresponding continuation into an adaptive segment and uses the divergence at the departure point to modulate the segment' s advantage. Experiments across eight mathematical reasoning and agentic tool-use benchmarks with Qwen3 4B and 8B models show that GEAR consistently outperforms standard GRPO, self-distillation-only baselines, and token- or turn-level credit-assignment methods. The gains are especially strong on benchmarks with lower GRPO baseline accuracy, reaching up to around 20% over GRPO, suggesting that the proposed adaptive reweighting scheme is especially useful in more challenging long-horizon settings.