Reinforcement Learning with Verifiable Rewards

Also known as RLVR

This field studies how to improve language and multimodal models by rewarding outputs whose correctness or task progress can be checked, for example with unit tests, mathematical answers, formal proofs, or simulated environments. Common approaches use policy-gradient methods such as PPO and GRPO, with outcome rewards supplemented by step-level feedback, rollout exploration, or replay to improve credit assignment and data efficiency. Work spans math and coding benchmarks as well as search, GUI, and other interactive agents. A central tension is that rewards must be scalable yet faithful: imperfect or narrow verifiers can be exploited, and a correct final answer does not by itself establish sound reasoning, visual grounding, or robust generalization.

Latest papers

692 papers137 papers in the last four weeks, up 111% on the four weeks before. 0.8% of all new papers.

Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
Aug 13, 2025Maxime Heuillet, Yufei Cui, Boxing Chen +2RL for Language Model Reasoning Efficient Language Model Training
Universit´e Laval (IID), Canada · Mila - Qu´ebec AI Institute, Canada · Huawei Noah’s Ark Lab (Montreal Research Center), Canada +1
The Invisible Leash: Why RLVR May or May Not Escape Its Origin
Jul 20, 2025Fang Wu, Weihao Xuan, Danny Xing +5RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Stanford University, CA, USA · The University of Tokyo, Japan · RIKEN AIP, Tokyo, Japan +2

Aug 13, 2025J/K move · Enter open · S save

Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts

Maxime Heuillet, Yufei Cui, Boxing Chen, Audrey Durand+1

Universit´e Laval (IID), Canada · Mila - Qu´ebec AI Institute, Canada · Huawei Noah’s Ark Lab (Montreal Research Center), Canada+1

RL for Language Model Reasoning Efficient Language Model Training Reinforcement Learning with Verifiable Rewards RL Fine-Tuning Off-Policy RL

Open paper AskPDF arXiv

Advanced reasoning in LLMs on challenging domains like mathematical reasoning can be tackled using verifiable rewards based reinforced fine-tuning (ReFT). In standard ReFT frameworks, a behavior model generates multiple completions with answers per problem, for the answer to be then scored by a reward function. While such RL post-training methods demonstrate significant performance improvements across challenging reasoning domains, the computational cost of generating completions during training with multiple inference steps makes the training cost non-trivial. To address this, we draw inspiration from off-policy RL, and speculative decoding to introduce a novel ReFT framework, dubbed Nested-ReFT, where a subset of layers of the target model acts as the behavior model to generate off-policy completions during training. The behavior model configured with dynamic layer skipping per batch during training decreases the inference cost compared to the standard ReFT frameworks. Our theoretical analysis shows that Nested-ReFT yields unbiased gradient estimates with controlled variance. Our empirical analysis demonstrates improved computational efficiency measured as tokens/sec across multiple math reasoning benchmarks and model sizes. Additionally, we explore three variants of bias mitigation to minimize the off-policyness in the gradient updates that allows for maintaining performance that matches the baseline ReFT performance.

Latest papers

Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts

The Invisible Leash: Why RLVR May or May Not Escape Its Origin