Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training architectures and differences between the training and inference policies break this assumption, making the data off-policy by design. To rectify this, prior work has focused on making this off-policy data appear more on-policy, either via importance sampling (IS), or by more closely aligning the training and inference policies by explicitly modifying the inference engine. In this work, we embrace off-policyness and propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL). We show that OAPL outperforms GRPO with importance sampling on competition math benchmarks, and can match the performance of a publicly available coding model, DeepCoder, on LiveCodeBench, while using 3x fewer generations during training. We further empirically demonstrate that models trained via OAPL have improved test time scaling under the Pass@k metric. OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.
Figures & tables
Figure 1: OAPL and GRPO on math reasoning benchmarks. Bars show the average of the maximum accuracy across three runs, with error bars indicating standard error. We report Pass@1 (computed via averaging over 10 rollouts per prompt), Pass@5, and Pass@10 on ( Left ) HMMT-25 (Feb & Nov), ( Middle ) AIME-25, and ( Right ) BRUMO-25.
Figure 2: Training curves on competition math. Curves show mean accuracy across three benchmarks (AIME25, HMMT25, BRUMO25) and shaded regions denote standard error. ( Left ) Pass@1, ( Middle ) Pass@5, and ( Right ) Pass@10. OAPL converges to higher accuracy and remains more stable than GRPO over training.
Figure 3: Training dynamics and robustness to policy lag in competitive math. ( Left ) The training entropy for both OAPL and GRPO (mean across three runs; shaded region is standard error). ( Right ) Accuracy over training for OAPL with a larger synchronization interval ( K=100 ), averaged over AIME, HMMT, and BRUMO; dashed/dotted lines show Pass@1/5/10 computed from 10 rollouts per prompt. OAPL remains stable even under substantially lagged rollouts.
Figure 4: Scaling behaviors of OAPL and GRPO for Pass@k. We observe RL training increases Pass@k for all k ranging from 1 to 256 . OAPL improves scaling relative to GRPO and the base model. ( Left ) Average across all benchmarks; remaining panels show per-benchmark results (AIME25, HMMT25 Nov, HMMT25 Feb, BRUMO25).
Figure 5: Code generation results on LiveCodeBench. ( Left ) Pass@k scaling for OAPL, DeepCoder, and the shared base model. ( Right ) Sample efficiency: Pass@1 Accuracy versus the number of training generations, highlighting that OAPL matches DeepCoder while using substantially fewer samples. All metrics are computed from 20 rollouts per prompt using the same evaluation protocol as DeepCoder.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Optimizer
AdamW
Learning rate
1×10−6
β1
0.9
β2
0.95
Weight decay
1×10−2
Gradient clipping type
Norm
Appendix
Table 1: Optimizer hyperparameters used for both OAPL and GRPO for the math task. Note that β1 and β2 here are for AdamW, not OAPL.
Parameter
OAPL
GRPO
β1
1
–
β2
1×10−3
–
L
50
–
Clip ratio
–
0.2
Length normalization
–
True
Max async iterations
–
1
Appendix
Table 2: Method-specific hyperparameters for OAPL and GRPO on math task.
Category
Parameter
Value
Training
Generations per prompt (G)
8
Batches per update
2
Global train batch size
128
Evaluation
Temperature
0.6
Top- p
0.95
Max tokens
16384
Appendix
Table 3: Shared hyperparameters for OAPL and GRPO.
Parameter
Value
Optimizer
AdamW
β1
0.9
β2
0.999
Weight decay
1×10−2
Gradient clipping type
Norm
Gradient clipping threshold
1.0
Appendix
Table 4: Optimizer hyperparameters for code generation experiments
Parameter
Value
β1
1
β2
1×10−3
L
418
Appendix
Table 5: OAPL Hyperparameters for code generation experiments
Category
Parameter
Value
Training
Generations per prompt (G)
8
Batches per update
1
Global train batch size
256
Evaluation
Temperature
0.6
Top- p
0.95
Max tokens
65536
Appendix
Table 6: Training and Eval hyperparameters for code generation experiments
Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7% of problems that base models consistently fail to solve.
Xu Wan, Yansheng Wang, Wenqi Huang +1
Zhejiang University · Bytedance Seed Robotics · Peking University +1
Improving data utilization efficiency is critical for scaling reinforcement learning (RL) for long-horizon tasks where generating trajectories is expensive. However, the dominant RL methods for LLMs are largely on-policy: they update each batch of data only once, discard it, and then collect fresh samples, resulting in poor sample efficiency. In this work, we explore an alternative value-based RL framework for LLMs that naturally enables off-policy learning. We propose ReVal, a Bellman-update-based method that combines stepwise signals capturing internal consistency with trajectory-level signals derived from outcome verification. ReVal naturally supports replay-buffer-based training, allowing efficient reuse of past trajectories. Experiments on standard mathematical reasoning benchmarks show that ReVal not only converges faster but also outperforms GRPO in final performance. On DeepSeek-R1-Distill-1.5B, ReVal improves training efficiency and achieves improvement of 2.7% in AIME24 and 4.5% in out-of-domain benchmark GPQA over GRPO. These results suggest that value-based RL is a practical alternative to policy-based methods for LLM training.
Peng-Yuan Wang, Ziniu Li, Tian Xu +8
National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data +1
Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses. We argue that this is a misleading inductive bias: the optimization-optimal trajectory distribution favors informative gradients, whereas the inference-optimal response distribution emphasizes accuracy and consistency. Forcing both into a single policy entangles their gradients and suppresses exploration. We propose R2PO (Residual Rollout Policy Optimization), which attaches a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, diversifying rollouts during training while keeping inference generation intact. Experiments show that R2PO consistently outperforms baselines, with average accuracy gains of 3.4% on MATH-500 and 1.3% on APPS, alongside more diverse rollouts and reduced length bias. Our code is available at https://github.com/RRPO-ARR/Code.
Jingchu Wang, Bingbing Xu, Yige Yuan +4
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · National University of Singapore