Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training architectures and differences between the training and inference policies break this assumption, making the data off-policy by design. To rectify this, prior work has focused on making this off-policy data appear more on-policy, either via importance sampling (IS), or by more closely aligning the training and inference policies by explicitly modifying the inference engine. In this work, we embrace off-policyness and propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL). We show that OAPL outperforms GRPO with importance sampling on competition math benchmarks, and can match the performance of a publicly available coding model, DeepCoder, on LiveCodeBench, while using 3x fewer generations during training. We further empirically demonstrate that models trained via OAPL have improved test time scaling under the Pass@k metric. OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.
Figures & tables
Figure 1: OAPL and GRPO on math reasoning benchmarks. Bars show the average of the maximum accuracy across three runs, with error bars indicating standard error. We report Pass@1 (computed via averaging over 10 rollouts per prompt), Pass@5, and Pass@10 on ( Left ) HMMT-25 (Feb & Nov), ( Middle ) AIME-25, and ( Right ) BRUMO-25.
Figure 2: Training curves on competition math. Curves show mean accuracy across three benchmarks (AIME25, HMMT25, BRUMO25) and shaded regions denote standard error. ( Left ) Pass@1, ( Middle ) Pass@5, and ( Right ) Pass@10. OAPL converges to higher accuracy and remains more stable than GRPO over training.
Figure 3: Training dynamics and robustness to policy lag in competitive math. ( Left ) The training entropy for both OAPL and GRPO (mean across three runs; shaded region is standard error). ( Right ) Accuracy over training for OAPL with a larger synchronization interval ( K=100 ), averaged over AIME, HMMT, and BRUMO; dashed/dotted lines show Pass@1/5/10 computed from 10 rollouts per prompt. OAPL remains stable even under substantially lagged rollouts.
Figure 4: Scaling behaviors of OAPL and GRPO for Pass@k. We observe RL training increases Pass@k for all k ranging from 1 to 256 . OAPL improves scaling relative to GRPO and the base model. ( Left ) Average across all benchmarks; remaining panels show per-benchmark results (AIME25, HMMT25 Nov, HMMT25 Feb, BRUMO25).
Figure 5: Code generation results on LiveCodeBench. ( Left ) Pass@k scaling for OAPL, DeepCoder, and the shared base model. ( Right ) Sample efficiency: Pass@1 Accuracy versus the number of training generations, highlighting that OAPL matches DeepCoder while using substantially fewer samples. All metrics are computed from 20 rollouts per prompt using the same evaluation protocol as DeepCoder.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Optimizer
AdamW
Learning rate
1×10−6
β1
0.9
β2
0.95
Weight decay
1×10−2
Gradient clipping type
Norm
Appendix
Table 1: Optimizer hyperparameters used for both OAPL and GRPO for the math task. Note that β1 and β2 here are for AdamW, not OAPL.
Parameter
OAPL
GRPO
β1
1
–
β2
1×10−3
–
L
50
–
Clip ratio
–
0.2
Length normalization
–
True
Max async iterations
–
1
Appendix
Table 2: Method-specific hyperparameters for OAPL and GRPO on math task.
Category
Parameter
Value
Training
Generations per prompt (G)
8
Batches per update
2
Global train batch size
128
Evaluation
Temperature
0.6
Top- p
0.95
Max tokens
16384
Appendix
Table 3: Shared hyperparameters for OAPL and GRPO.
Parameter
Value
Optimizer
AdamW
β1
0.9
β2
0.999
Weight decay
1×10−2
Gradient clipping type
Norm
Gradient clipping threshold
1.0
Appendix
Table 4: Optimizer hyperparameters for code generation experiments
Parameter
Value
β1
1
β2
1×10−3
L
418
Appendix
Table 5: OAPL Hyperparameters for code generation experiments
Category
Parameter
Value
Training
Generations per prompt (G)
8
Batches per update
1
Global train batch size
256
Evaluation
Temperature
0.6
Top- p
0.95
Max tokens
65536
Appendix
Table 6: Training and Eval hyperparameters for code generation experiments
National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data +1
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · National University of Singapore