Credit Assignment in RL

RL: Reinforcement Learning

Credit assignment asks which decisions deserve credit or blame when rewards arrive late or apply to an entire trajectory, a central problem in long-horizon and cooperative tasks. Traditional approaches include temporal-difference learning, eligibility traces, critics and advantage estimation, while reward shaping, counterfactual comparisons, and Shapley values offer ways to make contributions more explicit. Recent work on language-model agents often assigns feedback at the token, step, or turn level using process rewards, hindsight, trajectory graphs, or comparisons between alternative rollouts; related methods address multi-agent control and robotics. The key tension is that denser signals can improve learning but may be noisy, biased, or expensive, and it remains difficult to establish that a local action actually caused progress rather than merely accompanied it.

Latest papers

219 papers47 papers in the last four weeks, up 88% on the four weeks before. 0.3% of all new papers.

FOCUS: Active Critic Learning for Long-Horizon Language Model Agents
Oct 26, 2026Zhengping Jiang, William Jurayj, Liaoyaqi Wang +3RL for LLM Agents Long-Horizon RL
Johns Hopkins University
When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning
Oct 23, 2026Zongyao Zhao, Cheng-Jie Zhao, Ruihang Jiang +3Policy Optimization Group Relative Policy Optimization
The University of Hong Kong · Tsinghua University
ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents
Oct 22, 2026Rumei Li, Xunliang CaiLong-Horizon LLM Agents RL for LLM Agents
Meituan, China
RECAP: Looking Once Is Not Enough for Vision-Language Reasoning
Oct 22, 2026Zhaolu Kang, Tailong Luo, Chenxin Li +13Reasoning for VLMs Visual Evidence Sensitivity in VLMs
Peking University · Chengdu Minto Tech
AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation
Oct 21, 2026Likui Zhang, Zhe Zhu, Zhihao Zhan +7RL for Robotics Long-Horizon Robotic Manipulation
Sun Yat-sen University · JD Explore Academy · The Hong Kong Polytechnic University
One Act, Many Masks: Effect-Marginal Policy Optimization for Long-Horizon Agents
Oct 21, 2026Shaojin Ma, Ruitao Wang, Pengyi Li +6Policy Optimization RL for LLM Agents
Tianjin University · Fudan University · Renmin University of China
TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment
Oct 20, 2026Yi Wen, Hao Chen, Wanyu Wang +9RL for Language Model Reasoning RL Post-Training
City University of Hong Kong · Tencent · Nanjing University
Cooperative Multi-Agent Reinforcement Learning for Market Making on Prediction-Market Order Books
Oct 19, 2026Skander LejmiAlgorithmic Trading Cooperative MARL
Department of Statistics, London School of Economics and Political Science
Rubrics Beyond Evaluation: Exploiting Rubric Structure in Reinforcement Learning
Oct 19, 2026Yuxun Huang, Yunyao Zhang, ZeLiang Chen +3Rubric-Based RL Credit Assignment in RL
Huazhong University of Science and Technology, Wuhan, China · La Trobe University, Melbourne, Australia
Visual Causal-chain Bookmarking: Retrieval-Weighted Credit Assignment for Reinforcement Learning from Delayed Reward
Oct 16, 2026Prathap SelvakumarCredit Assignment in RL Step-Level Credit Assignment in RL
University of Manchester
Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents
Oct 16, 2026Qirui Liu, Hao Chen, Yi Wen +7Credit Assignment in RL Reinforcement Learning with Verifiable Rewards
South China University of Technology · FiT, Tencent · City University of Hong Kong +5
What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds
Oct 14, 2026Caixin Kang, Ke Wang, Yuchuan Wu +5LLM Agent Evaluation LLM Agent Training
Alibaba Token Hub, Alibaba Group · The University of Tokyo
ForkGRPO: Efficient Exploration and Fine-Grained Credit Assignment for Diffusion Language Models
Oct 14, 2026Zhaoyan Jia, Chengwei Zhou, Gourav Datta +1RL for Language Model Reasoning RL for Diffusion Models
University of Southern California, Los Angeles, USA · Case Western Reserve University, Cleveland, USA
SkillScaffold: Scaffolding Agentic Reinforcement Learning with Policy-Adaptive Skill Credit Assignment and Selective Injection
Oct 13, 2026Wenlin Zhong, Hao Wang, Kun Yuan +6Reinforcement Learning Agentic RL
Zhejiang University · Kuaishou Technology · University of the Chinese Academy of Sciences +1
FlowCred: Hierarchical Credit Assignment and Functional Coordination for Joint Text-to-Video Post-Training
Oct 13, 2026Mingxuan Zhu, Yanwei Huang, Shiyuan Liu +4T2V Generation Prompt Optimization for T2V Generation
Shanghai Jiao Tong University · KlingAI · The Hong Kong University of Science and Technology +2
WorkRover: Scaling Long-Horizon Cowork Agent Training Via Recursive Environment Evolution
Oct 13, 2026Hang He, Linting Wang, Guanming Liu +14LLM Agent Training Long-Horizon LLM Agents
CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning
Oct 13, 2026Guangcheng Zhu, Yingfan Ma, Xing Zheng +7RL for Language Models Rubric-Based RL
Zhejiang University · Ant Group
DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter
Oct 13, 2026Xuyan Ye, Xiyan Fu, Shengda Fan +3RL for LLM Agents Reinforcement Learning
College of Computing and Data Science, Nanyang Technological University · Gaoling School of Artificial Intelligence, Renmin University of China
Double-Rollout Reinforcement Learning with Sampling-based Reward Estimation
Oct 12, 2026Haifeng Qian, David Mosallanezhad, Christian Munley +2Reinforcement Learning RL for Language Model Reasoning
NVIDIA, 2788 San Tomas Expressway, Santa Clara, CA 95051, USA
Equal-information evaluation of credit assignment in modular financial reinforcement learning
Oct 10, 2026Gennadiy Chistiakov, Yury YanovichCredit Assignment in RL Reinforcement Learning
Independent Researcher, Bangkok, 10100, Thailand · Skolkovo Institute of Science and Technology, Moscow, 121205, Russia
Matched-Context Group Policy Optimization for LLM Agent Training
Oct 9, 2026Chenxu Zhang, Yaorui Shi, Wenlong Zhang +3RL for LLM Agents Preference-Based RL
Imperial College London · Meituan · University of Science and Technology of China +1
SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents
Oct 9, 2026Shengze Huang, Zihao Tang, Jinluan Yang +11Claim Attribution in Language Models RL for Language Model Reasoning
Zhejiang University · Microsoft
TreeHCA: Hindsight-Guided Exploration and Credit Assignment for Multi-Turn LLM Agents
Oct 9, 2026Debajoy Mukherjee, Jeremy Carleton, Sushil Vemuri +2LLM Agent Training RL for LLM Agents
Department of Electrical and Computer Engineering, Texas A&M University, College Station, Texas, USA
Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards
Oct 8, 2026Xiaobing Chen, Zhiqi PangRL for Language Model Reasoning RL Post-Training
Harbin Engineering University · Tencent · Harbin Institute of Technology
RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptation
Oct 7, 2026Hengbo Xiao, Boyao Zhang, Purui Liu +4RL for LLM Agents Rubric-Based RL
AGAR: a reinforcement learning substrate for LLM program evolution
Oct 6, 2026Haoran Li, Zengle Ge, Xiaomin Yuan +11LLM-Based Program Synthesis RL for Code Generation
University of the Chinese Academy of Sciences · Baidu · Tsinghua University +2
VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents
Oct 6, 2026Jiaju Chen, Min Yang, Jinghua Piao +4Token-Level Credit Assignment Credit Assignment in RL
University of Science and Technology of China · Zhongguancun Academy · Shandong University +2
RELACE: retrospective likelihood-based action credit estimation for long-horizon language agents
Oct 5, 2026Sayak Chakrabarti, Sathish Reddy IndurthiGroup Relative Policy Optimization Credit Assignment in RL
Columbia University · Zoom Communications, Inc.
Minimal Witness Reinforcement Learning
Oct 5, 2026T. Y. Tsui, Zihao Ye, Pengxiang Cai +3RL for Language Model Reasoning Credit Assignment in RL
University of Pennsylvania · Shanghai AI Laboratory
Residual Visual Credit Optimization: Conserved Evidence Routing for Multimodal Reinforcement Learning
Oct 4, 2026Lin Qiu, Yao Liu, Diyi Hu +6Credit Assignment in RL Step-Level Credit Assignment in RL
Meta Superintelligence Labs · University of Southern California · Meta Recommendation System +1

Oct 26, 2026J/K move · Enter open · S save

FOCUS: Active Critic Learning for Long-Horizon Language Model Agents

Zhengping Jiang, William Jurayj, Liaoyaqi Wang, Chunsheng Zuo+2

Johns Hopkins University

RL for LLM Agents Long-Horizon RL Credit Assignment in RL Value Function Estimation

Open paper AskPDF arXiv

Sparse outcome rewards make it difficult for long-horizon language-model agents to assign credit to earlier actions. Learned critics provide value estimates for intermediate states, but these estimates can be inaccurate, while estimating intermediate values with repetitive continuation samples from points within a trajectory incurs substantial costs. To mitigate these issues, we introduce FOCUS, an RL method that uses selective branching to help the critic network learn its value estimator. The rollout controller facilitates active learning for the critic by sampling continuations from states that are expected to most reduce the critic's estimation error. These refined value estimates then score the segment between consecutive states with a TD(

λ

) advantage, while the sampled evidence supplies training targets for the critic. FOCUS supports long-horizon agentic reinforcement learning through a modular context-management interface that decouples context management from rollout allocation. Our experiments show that using compaction-induced segment boundaries for value estimation and credit assignment, FOCUS outperforms GRPO and VinePPO with matched per-step generation budgets on legal reasoning, deep search, and code localization benchmarks, improving held-out accuracy at Qwen3.5-9B scale by 7.2 points over the strongest baseline on BrowseComp-Plus (55.4 vs. 48.2) and by 4.1 points on DeonticBench (56.5 vs. 52.4).