Off-Policy Value-Based Reinforcement Learning for Large Language Models

Peng-Yuan Wang; Ziniu Li; Tian Xu; Bohan Yang; Tian-Shuo Liu; ChenYang Wang; Xiong-Hui Chen; Yi-Chen Li

cs.LGMar 24, 2026

Off-Policy Value-Based Reinforcement Learning for Large Language Models

Authors: Peng-Yuan Wang, Ziniu Li, Tian Xu, Bohan Yang, Tian-Shuo Liu, ChenYang Wang, Xiong-Hui Chen, Yi-Chen Li,

Organizations: National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data · Shenzhen Loop Area Institute, Shenzhen

Abstract

Improving data utilization efficiency is critical for scaling reinforcement learning (RL) for long-horizon tasks where generating trajectories is expensive. However, the dominant RL methods for LLMs are largely on-policy: they update each batch of data only once, discard it, and then collect fresh samples, resulting in poor sample efficiency. In this work, we explore an alternative value-based RL framework for LLMs that naturally enables off-policy learning. We propose ReVal, a Bellman-update-based method that combines stepwise signals capturing internal consistency with trajectory-level signals derived from outcome verification. ReVal naturally supports replay-buffer-based training, allowing efficient reuse of past trajectories. Experiments on standard mathematical reasoning benchmarks show that ReVal not only converges faster but also outperforms GRPO in final performance. On DeepSeek-R1-Distill-1.5B, ReVal improves training efficiency and achieves improvement of 2.7% in AIME24 and 4.5% in out-of-domain benchmark GPQA over GRPO. These results suggest that value-based RL is a practical alternative to policy-based methods for LLM training.

Figures & tables

Figure 1: Framework of ReVal. By interpreting LLM logits as $Q$ -values, ReVal unifies policy and value within a single model and enables replay-based off-policy updates.

Figure 2: Performance of GRPO across different difficulty levels.

Panel of Figure 3 — Figure 3: Performance under different data reuse frequencies on tasks with varying difficulty levels.

	AIME24	AIME25	AMC	MATH	MINERVA	Olympiad	GPQA	Avg
DPSK-R1-Distill-1.5B	19.8	20.0	50.7	76.3	22.9	37.5	15.8	34.7
+ GRPO	29.4	24.4	65.0	82.6	27.6	46.3	28.8	43.4
+ TBRM	26.9	22.1	65.1	81.3	28.8	44.6	27.3	42.3
+ ReVal	32.1	23.8	68.6	84.6	30.3	46.6	33.3	45.6
Qwen2.5-Math-7B	19.0	6.9	43.6	60.1	10.9	26.3	12.8	25.7
+ GRPO	34.8	12.3	59.4	74.0	30.8	37.0	20.4	38.4

Table 1: Evaluation performance ( avg@16 ) comparison across different models and benchmarks.

Panel of Figure 4 — Figure 4: Training curves of DPSK-R1-Distill-1.5B and Qwen2.5-Math-7B. Curves show the accuracy across seven benchmarks (AIME, AIME25, AMC, MATH, Minerva, Olympiad, and GPQA) as well as the average accuracy.

Figure 5: Training Curves of DPSK-R1-Distill-1.5B with N=1

Figure 6: The Average Generation Rounds and Average Total Wall-clock Time (h).

Figure 7: Comparison of different update frequency of reference model.

Panel of Figure 8 — Figure 8: Effect of the hyperparameter $\beta$ . A smaller $\beta$ leads to a larger KL divergence between the policy and the reference model.

Figure 9: Comparison of different reward designs.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Training behavior of TBRM when the reward is fixed to

0

. The KL divergence between the current policy and the reference policy does not converge to zero.

Panel of Figure 11 — Figure 11: Comparison of different objectives. (a) Average benchmark performance across training steps. (b) Corresponding gradient norm during training.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

LLMs Can Learn to Reason Via Off-Policy RL
Feb 22, 2026Daniel Ritter, Owen Oertell, Bradley Guo +3RL for Language Model Reasoning RL Post-Training
Cornell University · Databricks · Harvard University
Le Critique: Privileged Value Functions for LLM Reinforcement Learning
Aug 17, 2026Siddarth Venkatraman, Matthieu Dinot, Laurence AitchisonRL for Language Models RL for Language Model Reasoning
Mistral AI · Mila – Quebec AI Institute · Université de Montréal
Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
Feb 24, 2026Xu Wan, Yansheng Wang, Wenqi Huang +1RL for Language Model Reasoning RL Post-Training
Zhejiang University · Bytedance Seed Robotics · Peking University +1

Feb 22, 2026J/K move · Enter open · S save

LLMs Can Learn to Reason Via Off-Policy RL

Daniel Ritter, Owen Oertell, Bradley Guo, Jonathan Chang+2

Cornell University · Databricks · Harvard University

RL for Language Model Reasoning RL Post-Training Off-Policy RL LLM Reasoning RL for Code Generation

Open paper AskPDF arXiv

Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training architectures and differences between the training and inference policies break this assumption, making the data off-policy by design. To rectify this, prior work has focused on making this off-policy data appear more on-policy, either via importance sampling (IS), or by more closely aligning the training and inference policies by explicitly modifying the inference engine. In this work, we embrace off-policyness and propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL). We show that OAPL outperforms GRPO with importance sampling on competition math benchmarks, and can match the performance of a publicly available coding model, DeepCoder, on LiveCodeBench, while using 3x fewer generations during training. We further empirically demonstrate that models trained via OAPL have improved test time scaling under the Pass@k metric. OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.

Abstract

Reinforcement Learning

Explore similar work

LLMs Can Learn to Reason Via Off-Policy RL

Le Critique: Privileged Value Functions for LLM Reinforcement Learning

Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning