Critic-Free RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 17

All topics

RELACE: retrospective likelihood-based action credit estimation for long-horizon language agents
Oct 5, 2026Sayak Chakrabarti, Sathish Reddy IndurthiGroup Relative Policy Optimization Credit Assignment in RL
Columbia University · Zoom Communications, Inc.
Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learning
Sep 28, 2026Yangyang Ren, Haodong Zhu, Linlin Yang +3Agentic RL Step-Level Credit Assignment in RL
Beihang University · Zhongguancun Academy · Communication University of China +2
GR2PO: Group Relative Return Policy Optimization for Continuous Robot Control
Sep 17, 2026Pengqin Wang, Qiming Zhang, Shaojie Shen +1Reinforcement Learning Group Relative Policy Optimization
Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Division of Emerging Interdisciplinary Areas, the Hong Kong University of Science and Technology, Hong Kong SAR, China · Intelligent Transportation Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China +1
Bellman Policy Optimization
Sep 14, 2026Zhuoqing Song, Haotian Xu, Xikun Zhang +1Reinforcement Learning RL for Language Model Reasoning
1Apodex US, Inc. · 2Princeton University
Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning
Sep 11, 2026Taoran Liang, Yang Liu, Shang Luo +10Credit Assignment in RL Step-Level Credit Assignment in RL
Nankai University · Peking University · Supply Chain Tech Team Y, JD.com +2
Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning
Aug 11, 2026Daoyi Li, Yixian Zhang, Wenbo Ding +2Reinforcement Learning Critic-Free RL
Tsinghua University
When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning
Jul 8, 2026Xiuyi Lou, Zicheng Xu, Yu-Neng Chuang +4RL for Language Models Token-Level Credit Assignment
Johns Hopkins University · Rice University · Workato +1
PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF
Jun 29, 2026Doo Hwan Hwang, Kee-Eung KimReinforcement Learning RL for Language Model Reasoning
Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea.
BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards
Jun 27, 2026Yupeng Chang, Yuan Wu, Yi ChangReinforcement Learning RL for Language Model Reasoning
School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · 3International Center of Future Science, Jilin University
VIMPO: Value-Implicit Policy Optimization for LLMs
Jun 18, 2026Zhewei Kang, Aosong Feng, Sergey Levine +2RL for Language Model Reasoning Policy Optimization
UC Berkeley · Yale University
Rethinking Groups in Critic-Free RLVR
Jun 15, 2026Yihong Wu, Liheng Ma, Lingfeng Xiao +4Reinforcement Learning RL for Language Model Reasoning
1Université de Montréal · 2McGill University · 3Mila - Quebec AI Institute +3
BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
May 26, 2026Shijin Gong, Erhan Xu, Kai Ye +3RL for Language Model Reasoning Critic-Free RL
School of Management, University of Science and Technology of China · Department of Statistics, London School of Economics and Political Science · Department of Engineering Science, University of Oxford
Value-Gradient Hypothesis of RL for LLMs
May 20, 2026Arip Asadulaev, Daniil Ognev, Karim Salta +1RL for Language Models Policy Gradient
MBZUAI · Independent
GAGPO: Generalized Advantage Grouped Policy Optimization
May 13, 2026Siyuan Zhu, Chao Yu, Rongxin Yang +4Reinforcement Learning Group Relative Policy Optimization
School of Computer Science and Engineering, Sun Yat-sen University · Meituan
The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits
May 9, 2026Tianhao Cheng, Zeyu Huang, Zihan Qiu +5Reinforcement Learning Token-Level Credit Assignment
1Fudan University · 3The University of Edinburgh · 6Qwen Team, Alibaba Group +4
Exploring Pass-Rate Reward in Reinforcement Learning for Code Generation
May 1, 2026Xin-Ye Li, Ren-Biao Liu, Yun-Ji Zhang +3RL for Code Generation Reinforcement Learning
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China
Constrained Group Relative Policy Optimization
Feb 5, 2026Roger Girgis, Rodrigue de Schaetzen, Luke Rowe +3Group Relative Policy Optimization Constrained RL
Mila – Quebec AI Institute Université de Montréal

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

RELACE: retrospective likelihood-based action credit estimation for long-horizon language agents

Sayak Chakrabarti, Sathish Reddy Indurthi

Columbia University · Zoom Communications, Inc.

Group Relative Policy Optimization Credit Assignment in RL Step-Level Credit Assignment in RL Critic-Free RL RL for LLM Agents

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) avoids a separate critic by estimating advantages from rollout groups. For multi-turn agents, however, trajectory-level supervision provides coarse, noisy credit: terminal rewards do not locate errors and can penalize useful actions alongside mistakes. Group-in-Group Policy Optimization (GiGPO) and subsequent methods refine supervision through state-conditioned comparisons, but their credit estimates remain sensitive to downstream decisions and outcomes. We introduce RELACE, Retrospective Likelihood-based Action, a critic-free framework that integrates retrospective action assessment with state-conditioned advantage estimation. RELACE evaluates executed actions through teacher-forced likelihood scoring under both their original contexts and outcome-augmented contexts. Comparing these likelihoods yields a trajectory-normalized retrospective factor that captures outcome-dependent changes in action plausibility, rather than hindsight plausibility alone. We use this factor to reweight discounted task returns and construct local advantages by comparing weighted returns among actions from equivalent states within a task. This couples retrospective relevance with observed reward, producing fine-grained credit that complements trajectory-level GRPO supervision. Temporal smoothing and success-protecting masking further stabilize the local signal. RELACE requires neither auxiliary value nor reward models nor additional autoregressive rollouts for credit estimation. Experiments on ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct demonstrate substantial improvements over GRPO, GiGPO, and HCAPO. With the 1.5B model, RELACE achieves

96.35\%

success on ALFWorld and

79.43\%

on WebShop, surpassing GiGPO by

5.47

and

5.60

percentage points, respectively.

Momentum

Related topics

Latest papers 17