Critic-Free RL

RL: Reinforcement Learning

Critic-free reinforcement learning studies how to improve a policy without training a separate value model to estimate expected returns. Common approaches compare groups of sampled trajectories, share information across rollouts or prompts, use discounted returns, or derive value signals from the policy itself. These methods are applied to language-model reasoning and multi-turn agents, as well as routing and robot control, with benchmarks such as MATH, ALFWorld, WebShop, and continuous-control tasks. A central challenge is getting reliable step-level credit from sparse or noisy outcomes while balancing rollout cost, learning stability, and the risk that coarse trajectory rewards reinforce unhelpful actions.

Latest papers

23 papers6 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

Occupancy Modeling via Drifting: One-Step Generation Enables Direct Policy Extraction
Oct 14, 2026Robin Uhrich, Marco Bagatella, Núria Armengol Urpí +2Reinforcement Learning Policy Learning
University of Tübingen · ETH Zürich · Max Planck Institute for Intelligent Systems
ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning
Oct 13, 2026Bowen Ding, Mingda Wan, Yuhan Chen +7Agentic RL Reinforcement Learning
Zhejiang University · School of Engineering, Westlake University · Huawei Noah’s Ark Lab +1
On KL-Regularized Policy Optimization
Oct 6, 2026Yifan ZhangRL for LLM Agents KL-Regularized RL
Princeton University
RELACE: retrospective likelihood-based action credit estimation for long-horizon language agents
Oct 5, 2026Sayak Chakrabarti, Sathish Reddy IndurthiGroup Relative Policy Optimization Credit Assignment in RL
Columbia University · Zoom Communications, Inc.
Follow the Winners: Conservative Policy Improvement with the Cross-Entropy Method for Critic-Free RFT
Oct 2, 2026Joery Ariën de Vries, Neil David Lawrence, Zhenwen DaiCritic-Free RL RL Fine-Tuning
Trent AI Limited, London, United Kingdom · Computer Laboratory, University of Cambridge, Cambridge, United Kingdom
Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learning
Sep 28, 2026Yangyang Ren, Haodong Zhu, Linlin Yang +3Critic-Free RL Step-Level Credit Assignment in RL
Beihang University · Zhongguancun Academy · Communication University of China +2
GR2PO: Group Relative Return Policy Optimization for Continuous Robot Control
Sep 17, 2026Pengqin Wang, Qiming Zhang, Shaojie Shen +1Reinforcement Learning Group Relative Policy Optimization
Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Division of Emerging Interdisciplinary Areas, the Hong Kong University of Science and Technology, Hong Kong SAR, China · Intelligent Transportation Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China +1
Bellman Policy Optimization
Sep 14, 2026Zhuoqing Song, Haotian Xu, Xikun Zhang +1Reinforcement Learning RL for Language Model Reasoning
1Apodex US, Inc. · 2Princeton University
Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning
Sep 11, 2026Taoran Liang, Yang Liu, Shang Luo +10Credit Assignment in RL Step-Level Credit Assignment in RL
Nankai University · Peking University · Supply Chain Tech Team Y, JD.com +2
Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning
Aug 11, 2026Daoyi Li, Yixian Zhang, Wenbo Ding +2Reinforcement Learning Critic-Free RL
Tsinghua University
Right Answer, Wrong Tokens: Tail-Aware Credit Calibration for RLVR
Jul 8, 2026Xiuyi Lou, Zicheng Xu, Yu-Neng Chuang +4RL for Language Models Token-Level Credit Assignment
Johns Hopkins University · Rice University · Workato +1
PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF
Jun 29, 2026Doo Hwan Hwang, Kee-Eung KimReinforcement Learning RL for Language Model Reasoning
Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea.
BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards
Jun 27, 2026Yupeng Chang, Yuan Wu, Yi ChangReinforcement Learning RL for Language Model Reasoning
School of Artificial Intelligence, Jilin University · 2Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · 3International Center of Future Science, Jilin University
VIMPO: Value-Implicit Policy Optimization for LLMs
Jun 18, 2026Zhewei Kang, Aosong Feng, Sergey Levine +2RL for Language Model Reasoning Policy Optimization
UC Berkeley · Yale University
Rethinking Groups in Critic-Free RLVR
Jun 15, 2026Yihong Wu, Liheng Ma, Lingfeng Xiao +4Reinforcement Learning RL for Language Model Reasoning
1Université de Montréal · 2McGill University · 3Mila - Quebec AI Institute +3
BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
May 26, 2026Shijin Gong, Erhan Xu, Kai Ye +3RL for Language Model Reasoning Critic-Free RL
School of Management, University of Science and Technology of China · Department of Statistics, London School of Economics and Political Science · Department of Engineering Science, University of Oxford
Value-Gradient Hypothesis of RL for LLMs
May 20, 2026Arip Asadulaev, Daniil Ognev, Karim Salta +1RL for Language Models Policy Gradient
MBZUAI · Independent
GAGPO: Generalized Advantage Grouped Policy Optimization
May 13, 2026Siyuan Zhu, Chao Yu, Rongxin Yang +4Reinforcement Learning Group Relative Policy Optimization
School of Computer Science and Engineering, Sun Yat-sen University · Meituan
The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits
May 9, 2026Tianhao Cheng, Zeyu Huang, Zihan Qiu +5Reinforcement Learning Token-Level Credit Assignment
1Fudan University · 3The University of Edinburgh · 6Qwen Team, Alibaba Group +4
Exploring Pass-Rate Reward in Reinforcement Learning for Code Generation
May 1, 2026Xin-Ye Li, Ren-Biao Liu, Yun-Ji Zhang +3RL for Code Generation Reinforcement Learning
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China
RTMC: Step-Level Credit Assignment via Rollout Trees
Apr 13, 2026Tao Wang, Suhang Zheng, Xiaoxiao XuStep-Level Credit Assignment in RL Agentic RL
Alibaba Group
Critic-Free Deep Reinforcement Learning for Maritime Coverage Path Planning on Irregular Hexagonal Grids
Mar 30, 2026Carlos S. Sepúlveda, Gonzalo A. RuzPointer Networks Coverage Path Planning
Facultad de Ingeniería y Ciencias, Universidad Adolfo Ibáñez, Av. Diagonal las Torres 2640, Santiago, Chile · Dirección de Programas, Investigación y Desarrollo, Armada de Chile, General del Canto 398, Valparaíso, Chile · Millennium Nucleus for Social Data Science (SODAS), Santiago, Chile +1
Constrained Group Relative Policy Optimization
Feb 5, 2026Roger Girgis, Rodrigue de Schaetzen, Luke Rowe +3Group Relative Policy Optimization Constrained RL
Mila – Quebec AI Institute Université de Montréal

Oct 14, 2026J/K move · Enter open · S save

Occupancy Modeling via Drifting: One-Step Generation Enables Direct Policy Extraction

Robin Uhrich, Marco Bagatella, Núria Armengol Urpí, Pavel Kolev+1

University of Tübingen · ETH Zürich · Max Planck Institute for Intelligent Systems

Reinforcement Learning Policy Learning Reward-Free RL Critic-Free RL

Open paper AskPDF arXiv

The growing interest in behavior foundation models, combined with recent advancements in generative modeling, has led to an appealing recipe for reinforcement learning from unlabeled data: pre-train a generative model to model occupancies, i.e., discounted state visitations, on reward-free trajectories, such that a policy can be easily extracted once a small reward-labeled dataset becomes available. We decompose this recipe along three axes: (i) the degree to which the multi-modal structure of occupancies must be captured, (ii) the class of generative model, and (iii) the method used for policy extraction. Along each axis, we consider existing choices, propose simpler alternatives, and validate each decision experimentally. This analysis culminates in a novel, streamlined method that leverages drifting, a modern one-step generative sampler, to model the occupancy. We show that this enables policy extraction via direct differentiation through the occupancy model, and bypasses the need for critic distillation. Overall, we observe that this nimble setup leads to improvements in performance on standard benchmarks.