Step-Level Credit Assignment in RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

25 papers in the last four weeks, up 178% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 90

All topics

DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training
Sep 3, 2026Shubham Gandhi, Saurabh Goyal, Kiran Kate +1Agentic RL Step-Level Credit Assignment in RL
1Carnegie Mellon University · 2IBM Research
Dense Process Supervision for Search Agents via Fact Utility Estimation
Sep 1, 2026Rongzhi Zhu, Xiangyu Liu, Yi Liu +7Reinforcement Learning Credit Assignment in RL
State Key Laboratory for Novel Software Technology, Nanjing University, China · Ant Group, China · National Institute of Healthcare Data Science, Nanjing University, China
A^2Agent: Action-Aware Reinforcement Learning for Repository-Level Code Localization Agents
Aug 30, 2026Doyeon Kim, Suyoung Bae, Yumin Lee +1Software Engineering Agents Credit Assignment in RL
College of Computing and Informatics Sungkyunkwan University, South Korea
Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents
Aug 13, 2026Zechuan Wang, Siyuan Lu, Hongxuan Zhang +3Token-Level Credit Assignment Step-Level Credit Assignment in RL
Zhejiang University · AWorld Team, Inclusion AI · Shanghai Innovation Institute +2
Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning
Aug 13, 2026Yao Zhou, Hang Gao, Fengge Wu +2Group Relative Policy Optimization Step-Level Credit Assignment in RL
Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents
Aug 13, 2026Haoze Wu, Chuqiao Kuang, Tianyi Zhuang +1Group Relative Policy Optimization Credit Assignment in RL
The Hong Kong University of Science and Technology · Huawei Technologies Ltd.
Redistribution-based Cost Inference Improves Sparse Safe Offline RL
Aug 12, 2026Ebenezer Gelo, Geraud Nangue Tasse, Steven James +1Credit Assignment in RL Step-Level Credit Assignment in RL
University of the Witwatersrand · University of the Witwatersrand Johannesburg, South Africa
Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation
Aug 10, 2026Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao +3On-Policy Self-Distillation Token-Level Credit Assignment
Salesforce AI Research
How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning
Aug 7, 2026Lichao Ma, Yang Sun, Shuaitao Zhao +9Token-Level Credit Assignment Credit Assignment in RL
1Peking University · 2Meituan LongCat Interaction Team · 3Fudan University +4
Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents
Aug 7, 2026Hongxi Yan, Ziyue Huang, Shichao Fan +1Credit Assignment in RL Step-Level Credit Assignment in RL
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Zhongguancun Laboratory, Beijing, China · Qualcomm, Beijing, China
AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning
Aug 6, 2026Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao +10Agentic RL Reinforcement Learning
Tsinghua University · Meituan · Zhejiang University
ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment
Aug 5, 2026Yijun Lu, Rui Ye, Jiajun Wang +4Credit Assignment in RL Step-Level Credit Assignment in RL
Shanghai Jiao Tong University
Agentic Reinforcement Learning with Self-Distilled Reward Shaping
Aug 4, 2026Ranxu Zhang, Guinan Chen, Chenshaodong +5Reward Shaping Token-Level Credit Assignment
Alibaba Group · University of Science and Technology of China Hefei, China · Alibaba Group Hangzhou, China
TCPO: Turn-Level Credit Policy Optimization
Aug 3, 2026Sicong Liao, Zhi Chen, Yaohua TangRL for Language Model Reasoning Credit Assignment in RL
Moore Threads AI
BiCAA: Bidirectional Credit Assignment for Search-Augmented Agent
Aug 2, 2026Yibin Huang, Bin Xu, Hailong Cao +1Process Reward Models Credit Assignment in RL
Faculty of Computing, Harbin Institute of Technology
Gated Q-learning: Add Off-Policy Bias to Taste
Jul 31, 2026Brett DaleyCredit Assignment in RL Step-Level Credit Assignment in RL
Meta
Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning
Jul 30, 2026Qiangqiang He, Zhongheng Wu, ZiJian WangToken-Level Credit Assignment RL for Language Model Reasoning
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Institute of Wireless Communications Technology, Shanghai Jiao Tong University, Shanghai, China
CAST: Game Solvers as Turn-Level Teachers for LLM Agents
Jul 28, 2026Yu Wang, Yi-Kai Zhang, Wentao Shi +8Reinforcement Learning Game-Playing Agents
University of Science and Technology of China · 4Meituan, China · 2Nanjing University +1
TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
Jul 15, 2026Leitian Tao, Baolin Peng, Wenlin Yao +5Credit Assignment in RL Step-Level Credit Assignment in RL
University of Wisconsin–Madison · Microsoft Research
STAMP: Provenance-Guided Credit Assignment for Deep Search Agents
Jul 13, 2026Ke Xu, Han Xu, Xinran Chen +6Inference-Time Search Step-Level Credit Assignment in RL
Baidu Inc. · Peking University
ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
Jul 3, 2026Zijun Xie, Yuyang You, Yongzhi Li +6Token-Level Credit Assignment RL for Language Model Reasoning
Peking University · Baidu Inc. · Kuaishou Inc.
TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning
Jun 30, 2026Yuanda Xu, Zhengze Zhou, Hejian Sang +6Agentic RL Step-Level Credit Assignment in RL
1LinkedIn Corporation · 2Harvard University · 3Johns Hopkins University +1
CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning
Jun 28, 2026Zibin Meng, Kani ChenAgentic RL KL-Regularized RL
The Hong Kong University of Science and Technology
On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse
Jun 28, 2026Amritansh Mishra, Supriyo Chakraborty, Berkcan KapusuzogluPolicy Gradient Token-Level Credit Assignment
Capital One
Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents
Jun 24, 2026Peng Xu, Sijia Chen, Junzhuo Li +1Reward Shaping Credit Assignment in RL
†The Hong Kong University of Science and Technology (Guangzhou) · ‡The Hong Kong University of Science and Technology
Learning with a Single Rollout via Monte Carlo Pass@k Critic
Jun 24, 2026Fengdi Che, Yang Liu, Lei Yu +4RL for Language Model Reasoning Credit Assignment in RL
University of Alberta · BIGAI · University of Toronto +4
ARCO: Adaptive Rubric with Co-Evolution for Multi-Step LLM-Based Agents
Jun 19, 2026Zihang Tian, Jingsen Zhang, Rui Li +3Process Reward Models Credit Assignment in RL
Renmin University of China
Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards
Jun 17, 2026Yingyu Shan, Yuhang Guo, Zihao Cheng +7Token-Level Credit Assignment RL for Language Model Reasoning
Beijing Institute of Technology · Beihang University · Independent Researcher
STRIDE: Strategic Trajectory Reasoning via Discriminative Estimation for Verifiable Reinforcement Learning
Jun 14, 2026Qinjian Zhao, Zhihao Dou, Dinggen Zhang +10RL for Language Model Reasoning Credit Assignment in RL
Kean University · Case Western Reserve University · University of Texas at Austin +4
Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning
Jun 14, 2026Yu Li, Shu Hong, Tian LanRL for Language Model Reasoning Credit Assignment in RL
Department of Electrical and Computer Engineering, George Washington University

Sep 3, 2026, cs.AIJ/K move · Enter open · S save

DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training

Shubham Gandhi, Saurabh Goyal, Kiran Kate, Yara Rizk

1Carnegie Mellon University · 2IBM Research

Agentic RL Step-Level Credit Assignment in RL Rubric-Based RL RL Post-Training Policy Learning Fine-Grained Credit Assignment

Open paper AskPDF arXiv

Reinforcement Learning from Verifiable Rewards works well when a task has a programmatic checker, but most long-horizon agent domains have none. We work in the outcome-blind setting, where ground-truth success signals are not available. Multi-criteria rubrics are a popular way to supply such a reward; they are scored once per trajectory, but a single scalar is a poor signal across tens of steps. We propose DRACO: Distributing Rubric-based Advantage for Credit Optimization. It generates rubrics dynamically during training to track the policy's evolving capability, scores those rubrics once per completed trajectory, and redistributes that judgment over the steps responsible for annotated rubrics to produce differentiated per-step advantages in GRPO. The redistribution is closed-form and does not introduce any trained attribution module. On AppWorld, DRACO gains 15.9 points over the base model and 5.3 points over GRPO trained with a sparse ground-truth reward, despite not using any verifiers itself. On out-of-domain Tau-Bench, it gains 5.3 points over the base model even without a frontier judge, beating both ground-truth-reward training and other rubric-based training settings. The code for DRACO is available at https://github.com/IBM/draco.

Momentum

Related topics

Latest papers 90