Critic-Free RL
RL: Reinforcement Learning
Overview
Critic-free reinforcement learning studies how to improve a policy without training a separate value model to estimate expected returns. Common approaches compare groups of sampled trajectories, share information across rollouts or prompts, use discounted returns, or derive value signals from the policy itself. These methods are applied to language-model reasoning and multi-turn agents, as well as routing and robot control, with benchmarks such as MATH, ALFWorld, WebShop, and continuous-control tasks. A central challenge is getting reliable step-level credit from sparse or noisy outcomes while balancing rollout cost, learning stability, and the risk that coarse trajectory rewards reinforce unhelpful actions.
Latest papers
24 papers1 paper in the last four weeks, down 83% on the four weeks before. 0.0% of all new papers.