Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7% of problems that base models consistently fail to solve.
Figures & tables
Figure 1: Tracking the sample counts across accuracy groups of the mathematical dataset before and after GRPO post-training.
Figure 2: The overview of the (a) on-policy and (b) off-policy RL Post-training framework
Figure 3: The workflow of (a) off-policy rollout and (b) off-policy training in our RLVR framework
Figure 4: Training Curves of Reward Changes for mathematics, planning, and geometry tasks using DeepSeek Distilled Qwen 1.5B, Qwen2.5 Math 1.5B, and Qwen2.5 VL 3B, respectively.
(a) Mathematics Benchmarks
Method
AIME24
AMC
MATH500
Minerva.
Olympiad.
Avg. ↑
Rollouts ↓
Type
DeepSeek R1 Distill Qwen 1.5B
28.80
62.90
82.80
26.50
44.42
48.90
-
-
+GRPO ( Guo et al., 2025 )
30.73
67.47
85.40
28.95
45.33
51.58
677k
on
+DAPO ( Yu et al., 2025 )
35.73
70.08
86.05
30.70
48.48
54.20
1921k
on
+MoPPS ∗ ( Qu et al., 2025 )
33.33
65.29
84.94
28.88
45.93
51.67
737k
on
+GRPO ( v=5 ) ( Mroueh et al., 2025 )
30.49
65.09
86.72
28.16
46.18
51.57
677k
off
Table 1: Comprehensive Evaluation Results. ’ + ’ indicates fine-tuning via the corresponding method. Accuracy is averaged over 32 runs. The bold value denotes the top result, and the underlined value denotes the second-top result.
Figure 5: Test Curves of Group Accuracy Changes on AIME for different RLVR methods based on Qwen3 8B . Left: Standard BAPO vs. GRPO. Medium: BAPO (mini test) vs. GRPO . Right: Standard BAPO vs. DAPO.
Figure 6: Ablation Studies for BAPO. The first column presents ablations on frequency-related hyperparameters ( m,v ). The second column shows ablations on buffer subsets ( X2,X3 ). The third column compares fixed vs. adaptive difficulty thresholds.
Figure 7: Tracking changes in the Number of Different Accuracy Bins on the DeepScalerR training subset. Special attention is paid to the reduction of bad samples (red bars).
Figure 8: Dynamic Sample Distribution. The composition of BAPO’s X1,X2,X3 and the total samples compared to the fixed GRPO batch size (Red line).
Figure 9: Cumulative Rollout Batches Comparison between BAPO and DAPO. The maximum rollout time for DAPO is set to 4.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Term
Definition
c1,c2,c3
Thresholds for classifying historical samples by difficulty (group mean reward).
X1,X2,X3
Subsets of training batch: fresh, re-evaluated difficult, and historical high-quality samples.
m
Re-evaluation frequency for historically difficult samples.
v
Delay steps for updating the rollout policy.
G
Group size, number of responses generated per prompt during rollout.
B
Replay buffer storing historical samples.
Appendix
Table 11
Figure 10: Ablation on Online Filtering Strategies. Comparison of Range Filter, Gaussian Filter, and Uniform Filter on training stability (Grad Norm) and performance (Mean@8). The star symbol indicates the best checkpoint for BAPO.
Figure 11: Training Dynamics during BAPO, GRPO, and DAPO post-training, including training rewards, training entropy, and response lengths.
Figure 12: Test Curves of Group Accuracy Changes for mathematics, planning, and geometry tasks among AMC, CD-4 test set, and Geo-3K test set, respectively.
Tasks
Methods
Batch Size
Num Epoch
Time
Mathematics
GRPO
(256, 64)
3
1d 16h 58m
DAPO
(256, 64)
3
2d 15h 30m
BAPO
(256, 64)
3
1d 22h 37m
Planning
GRPO
(256, 64)
3
3h 47m
DAPO
(256, 64)
3
6h 35m
BAPO
(256, 64)
3
3h 23m
Appendix
Table 2: Computational Overhead Analysis. “ Batch size ” (a,b) represents the sample batch size a and train mini batch size b . “ Time ” is measured in total training time (d=days, h=hours, m=minutes) on 8 A100 GPUs.
Figure 13: Tracking changes in the Number of Different Accuracy Bins on the Countdown (upper) and Geometry3K training sets (lower) for the baseline model, GRPO, and our BAPO method. Special attention is paid to the change in the number of bad samples (red bars) that the base model fails to handle.
Figure 14: Batch Distribution Visualization of X1 , X2 , X3 for Mathematics, Planning, and Visual Geometry Tasks (left to right) during BAPO’s training.
Figure 15: Accuracy Migration Matrix Analysis. We track a fixed subset of 1,000 randomly selected prompts from the training set and visualize their movement between accuracy bins (0/8 to 8/8) at Steps 0, 150, 300, and 471 (the last step). The y-axis represents the initial accuracy bin at Step 0, while the x-axis represents the current accuracy bin. The scarcity of samples in the lower triangle demonstrates that performance degradation is rare.
Parameter
Value
Parameter
Value
Parameter
Value
Rollout Configuration
Top-p
1
Top-k
-1
Temperature
1
Group size ( G )
8
Max prompt length
2048
Max response length
8192
Rollout workers
8
Sample batch size
256
Seed
42
Training Configuration
Learning rate
1e-6
Train mini batch size
64
GAE lambda
1.0
Appendix
Table 3: Hyperparameter Configuration for BAPO Framework on Mathematics Task. For planning and visual geometry tasks, some parameters differ slightly; specific configuration scripts are provided in our code repository.
Dataset
#Train
#Test
Task Type
Domain
License
Source
Training Datasets
DeepScaleR-1.5B-Preview
40,000
–
Math reasoning
Mathematics
Apache 2.0
Link
Countdown-Tasks-3to4
49,000
–
Logic reasoning
Planning
Apache 2.0
Link
Geometry3k
2,100
–
Visual reasoning
Visual Geometry
Apache 2.0
Link
Test Benchmarks
AIME24
–
30
Math competition
Mathematics
MIT
Link
Appendix
Table 4: Datasets and Benchmarks used in this study.
Figure 16: Generalization to Actor-Critic Algorithms (BA-PPO). Performance comparison between standard PPO (orange triangles) and BA-PPO (purple circles) on the AIME 2024 benchmark using Qwen3-4B. The star ( ⋆ ) marks the peak performance of BA-PPO ( 0.325 ).
Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training architectures and differences between the training and inference policies break this assumption, making the data off-policy by design. To rectify this, prior work has focused on making this off-policy data appear more on-policy, either via importance sampling (IS), or by more closely aligning the training and inference policies by explicitly modifying the inference engine. In this work, we embrace off-policyness and propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL). We show that OAPL outperforms GRPO with importance sampling on competition math benchmarks, and can match the performance of a publicly available coding model, DeepCoder, on LiveCodeBench, while using 3x fewer generations during training. We further empirically demonstrate that models trained via OAPL have improved test time scaling under the Pass@k metric. OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.
Daniel Ritter, Owen Oertell, Bradley Guo +3
Cornell University · Databricks · Harvard University
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective. The reformulation avoids estimating state values at intermediate states. We prove that it has the same unique optimal solution as the original PMD objective. We derive the practical BPO loss by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO.
RL with verifiable rewards can substantially improve LLM reasoning, yet standard GRPO-style training often treats easy, hard, and learnable questions alike through uniform sampling and weighting, leading to inefficient compute allocation. We study GRPO by tracking token log-probabilities, group-normalized advantages, and the induced token-level update weights. This reveals three recurring dynamics as training proceeds: (1) confidence inflation, (2) advantage contraction, and (3) hierarchical convergence. These findings suggest that the utility of each update depends strongly on both question difficulty and the model's current competence. Motivated by this, we propose Confidence and Difficulty-adaptive Policy Optimization (CoDaPO), which assigns each question a bounded value from rollout confidence and empirical difficulty. CoDaPO then uses this value to reweight policy updates and resample high-value learnable questions within mini-batches, thereby increasing discovery within the learnable band under a fixed compute budget. Across twelve benchmarks, CoDaPO consistently improves accuracy over existing RL methods. Our code is publicly available at https://github.com/tmlr-group/CoDaPO.
Zhanke Zhou, Xiangyu Lu, Chentao Cao +4
1TMLR Group, Department of Computer Science, Hong Kong Baptist University · 2Stanford University · 3Sydney AI Centre, The University of Sydney