Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning

Xu Wan; Yansheng Wang; Wenqi Huang; Mingyang Sun

cs.AIFeb 24, 2026

Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning

Authors: Xu Wan, Yansheng Wang, Wenqi Huang, Mingyang Sun

Organizations: Zhejiang University · Bytedance Seed Robotics · Peking University · China Southern Power Grid

Abstract

Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7% of problems that base models consistently fail to solve.

Figures & tables

Figure 1: Tracking the sample counts across accuracy groups of the mathematical dataset before and after GRPO post-training.

Figure 2: The overview of the (a) on-policy and (b) off-policy RL Post-training framework

Figure 3: The workflow of (a) off-policy rollout and (b) off-policy training in our RLVR framework

Figure 4: Training Curves of Reward Changes for mathematics, planning, and geometry tasks using DeepSeek Distilled Qwen 1.5B, Qwen2.5 Math 1.5B, and Qwen2.5 VL 3B, respectively.

(a) Mathematics Benchmarks
Method	AIME24	AMC	MATH500	Minerva.	Olympiad.	Avg. $\uparrow$	Rollouts $\downarrow$	Type
DeepSeek R1 Distill Qwen 1.5B	28.80	62.90	82.80	26.50	44.42	48.90	-	-
+GRPO ( Guo et al., 2025 )	30.73	67.47	85.40	28.95	45.33	51.58	677k	on
+DAPO ( Yu et al., 2025 )	35.73	70.08	86.05	30.70	48.48	54.20	1921k	on
+MoPPS ∗ ( Qu et al., 2025 )	33.33	65.29	84.94	28.88	45.93	51.67	737k	on
+GRPO ( $v=5$ ) ( Mroueh et al., 2025 )	30.49	65.09	86.72	28.16	46.18	51.57	677k	off

Table 1: Comprehensive Evaluation Results. ’ + ’ indicates fine-tuning via the corresponding method. Accuracy is averaged over 32 runs. The bold value denotes the top result, and the underlined value denotes the second-top result.

Figure 5: Test Curves of Group Accuracy Changes on AIME for different RLVR methods based on Qwen3 8B . Left: Standard BAPO vs. GRPO. Medium: BAPO (mini test) vs. GRPO . Right: Standard BAPO vs. DAPO.

Figure 6: Ablation Studies for BAPO. The first column presents ablations on frequency-related hyperparameters ( $m,v$ ). The second column shows ablations on buffer subsets ( $\mathcal{X}_{2},\mathcal{X}_{3}$ ). The third column compares fixed vs. adaptive difficulty thresholds.

Figure 7: Tracking changes in the Number of Different Accuracy Bins on the DeepScalerR training subset. Special attention is paid to the reduction of bad samples (red bars).

Figure 8: Dynamic Sample Distribution. The composition of BAPO’s $\mathcal{X}_{1},\mathcal{X}_{2},\mathcal{X}_{3}$ and the total samples compared to the fixed GRPO batch size (Red line).

Figure 9: Cumulative Rollout Batches Comparison between BAPO and DAPO. The maximum rollout time for DAPO is set to 4.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Term	Definition
$c_{1},c_{2},c_{3}$	Thresholds for classifying historical samples by difficulty (group mean reward).
$\mathcal{X}_{1},\mathcal{X}_{2},\mathcal{X}_{3}$	Subsets of training batch: fresh, re-evaluated difficult, and historical high-quality samples.
$m$	Re-evaluation frequency for historically difficult samples.
$v$	Delay steps for updating the rollout policy.
$G$	Group size, number of responses generated per prompt during rollout.
$\mathcal{B}$	Replay buffer storing historical samples.

Appendix

Table 11

Figure 10: Ablation on Online Filtering Strategies. Comparison of Range Filter, Gaussian Filter, and Uniform Filter on training stability (Grad Norm) and performance (Mean@8). The star symbol indicates the best checkpoint for BAPO.

Figure 11: Training Dynamics during BAPO, GRPO, and DAPO post-training, including training rewards, training entropy, and response lengths.

Figure 12: Test Curves of Group Accuracy Changes for mathematics, planning, and geometry tasks among AMC, CD-4 test set, and Geo-3K test set, respectively.

Tasks	Methods	Batch Size	Num Epoch	Time
Mathematics	GRPO	(256, 64)	3	1d 16h 58m
Mathematics	DAPO	(256, 64)	3	2d 15h 30m
	BAPO	(256, 64)	3	1d 22h 37m
Planning	GRPO	(256, 64)	3	3h 47m
	DAPO	(256, 64)	3	6h 35m
	BAPO	(256, 64)	3	3h 23m

Appendix

Table 2: Computational Overhead Analysis. “ Batch size ”

(a,b)

represents the sample batch size

a

and train mini batch size

b

. “ Time ” is measured in total training time (d=days, h=hours, m=minutes) on 8 A100 GPUs.

Figure 13: Tracking changes in the Number of Different Accuracy Bins on the Countdown (upper) and Geometry3K training sets (lower) for the baseline model, GRPO, and our BAPO method. Special attention is paid to the change in the number of bad samples (red bars) that the base model fails to handle.

Figure 14: Batch Distribution Visualization of $\mathcal{X}_{1}$ , $\mathcal{X}_{2}$ , $\mathcal{X}_{3}$ for Mathematics, Planning, and Visual Geometry Tasks (left to right) during BAPO’s training.

Figure 15: Accuracy Migration Matrix Analysis. We track a fixed subset of 1,000 randomly selected prompts from the training set and visualize their movement between accuracy bins (0/8 to 8/8) at Steps 0, 150, 300, and 471 (the last step). The y-axis represents the initial accuracy bin at Step 0, while the x-axis represents the current accuracy bin. The scarcity of samples in the lower triangle demonstrates that performance degradation is rare.

Rollout Configuration
Parameter	Value	Parameter	Value	Parameter	Value
Top-p	1	Top-k	-1	Temperature	1
Group size ( $G$ )	8	Max prompt length	2048	Max response length	8192
Rollout workers	8	Sample batch size	256	Seed	42
Training Configuration
Learning rate	1e-6	Train mini batch size	64	GAE lambda	1.0

Appendix

Table 3: Hyperparameter Configuration for BAPO Framework on Mathematics Task. For planning and visual geometry tasks, some parameters differ slightly; specific configuration scripts are provided in our code repository.

Dataset	#Train	#Test	Task Type	Domain	License	Source
Training Datasets
DeepScaleR-1.5B-Preview	40,000	–	Math reasoning	Mathematics	Apache 2.0	Link
Countdown-Tasks-3to4	49,000	–	Logic reasoning	Planning	Apache 2.0	Link
Geometry3k	2,100	–	Visual reasoning	Visual Geometry	Apache 2.0	Link
Test Benchmarks
AIME24	–	30	Math competition	Mathematics	MIT	Link

Appendix

Table 4: Datasets and Benchmarks used in this study.

Figure 16: Generalization to Actor-Critic Algorithms (BA-PPO). Performance comparison between standard PPO (orange triangles) and BA-PPO (purple circles) on the AIME 2024 benchmark using Qwen3-4B. The star ( $\star$ ) marks the peak performance of BA-PPO ( $0.325$ ).

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,057 papers

LLMs Can Learn to Reason Via Off-Policy RL
Feb 22, 2026Daniel Ritter, Owen Oertell, Bradley Guo +3RL for Language Model Reasoning RL Post-Training
Cornell University · Databricks · Harvard University
Bellman Policy Optimization
Sep 14, 2026Zhuoqing Song, Haotian Xu, Xikun Zhang +1Reinforcement Learning RL for Language Model Reasoning
1Apodex US, Inc. · 2Princeton University
The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning
Jun 6, 2026Zhanke Zhou, Xiangyu Lu, Chentao Cao +4RL for Language Model Reasoning Policy Optimization
1TMLR Group, Department of Computer Science, Hong Kong Baptist University · 2Stanford University · 3Sydney AI Centre, The University of Sydney

Feb 22, 2026J/K move · Enter open · S save

LLMs Can Learn to Reason Via Off-Policy RL

Daniel Ritter, Owen Oertell, Bradley Guo, Jonathan Chang+2

Cornell University · Databricks · Harvard University

RL for Language Model Reasoning RL Post-Training Off-Policy RL LLM Reasoning RL for Code Generation

Open paper AskPDF arXiv

Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training architectures and differences between the training and inference policies break this assumption, making the data off-policy by design. To rectify this, prior work has focused on making this off-policy data appear more on-policy, either via importance sampling (IS), or by more closely aligning the training and inference policies by explicitly modifying the inference engine. In this work, we embrace off-policyness and propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL). We show that OAPL outperforms GRPO with importance sampling on competition math benchmarks, and can match the performance of a publicly available coding model, DeepCoder, on LiveCodeBench, while using 3x fewer generations during training. We further empirically demonstrate that models trained via OAPL have improved test time scaling under the Pass@k metric. OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.

Abstract

RL for Language Model Reasoning

Explore similar work

LLMs Can Learn to Reason Via Off-Policy RL

Bellman Policy Optimization

The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning