Off-policy problems such as policy staleness and training--inference mismatch have become a major bottleneck for training stability and further exploration in LLM RL. The distribution gap between the inference and updated policies grows because of the techniques to enhance inference efficiency, leading to heavy-tailed importance ratios. Heavy-tailed ratios arise when the policy is locally sharp, which further inflates gradients and can push updates outside the trust region. To address this, we propose Adaptive Layerwise Perturbation (ALP), which injects small learnable perturbations into the input hidden states of each layer during updates and uses the resulting perturbed policy as the numerator of the importance ratio against the unchanged inference policy in the objective. Intuitively, by adding controlled noise to intermediate representations, ALP prevents the updated policy from deviating too sharply from the inference policy and enlarges the policy family to cover inference-time mismatch noise. Hence, the flattened distribution can naturally tighten the gap between the updated and inference policies and reduce the tail of importance ratios, thus maintaining training stability. This is further validated empirically. Experiments on single-turn math and multi-turn tool-integrated reasoning tasks show that ALP not only improves final performance, but also avoids blow-up in the importance-ratio tail and KL spikes during iterative training, along with boosted exploration. Ablations show that representation-level perturbations across all layers are most effective, substantially outperforming partial-layer and logits-only variants.
Figures & tables
Figure 1: Left: ALP injects small layerwise perturbations during training to cover training-inference bias. Right: compared with Bypass over 1840 steps, ALP yields a smoother policy and a tighter importance-ratio envelope, where lines P 2 , P 98 represent the 2 th and 98 th quantile of the ratio.
Figure 2: Effects of perturbation. Left: perturbation smooths a sharp objective into a flatter surrogate. Middle/right: in a controlled multi-turn comparison from the same checkpoint and rollout batch, perturbation shrinks the mismatch envelope, especially in the low-probability tail.
Method
Math500
Minerva Math
Olympiad Bench
AIME24
AIME25
Average
GRPO
75.91
36.43
38.82
16.77
10.94
35.77
Seq-Bypass
76.21
35.23
38.52
16.35
7.81
34.82
Seq-MIS
77.08
36.75
39.06
15.21
9.58
35.54
Token-MIS
77.84
35.94
40.06
17.40
10.83
36.41
Token-ALP
78.10
37.27
40.77
21.46
11.77
37.87
Seq-ALP
77.84
37.06
40.28
16.98
11.98
36.83
Table 1: Single-turn evaluation (average@32, temperature 1.0 ) across five math benchmarks. Token-ALP attains the highest average score.
Figure 3: Single-turn training dynamics under training-inference mismatch: reward, gradient norm, entropy, and rollout-training KL over policy-update steps. Reward is smoothed with a 10-step moving average.
Method
Math500
Minerva Math
Olympiad Bench
AIME24
AIME25
Average
GRPO
94.49
56.96
68.28
60.29
51.17
66.24
Token-MIS
94.28
53.66
66.36
62.35
50.00
65.33
Seq-Bypass
93.70
52.00
64.21
63.80
46.88
64.12
Seq-ALP
95.18
57.44
69.49
64.32
51.17
67.52
Table 2: Single-turn validation on Qwen3-4B under the same evaluation protocol. Seq-ALP achieves the best average score across benchmarks.
Algorithm
Math500
Minerva Math
Olympiad Bench
AIME24
AIME25
Average
GRPO
80.90
42.13
49.59
35.42
24.79
46.57
Seq-Bypass
81.74
39.21
51.08
37.19
24.06
46.66
Token-MIS
83.29
41.45
50.65
39.48
28.85
48.74
Seq-MIS
80.61
42.05
48.38
39.48
24.17
46.94
Token-ALP
83.48
43.18
51.55
38.65
31.25
49.62
Seq-ALP
84.29
43.10
52.75
43.85
28.65
50.53
Table 3: TIR evaluation: the test accuracy average@32 under temperature 1.0 across the five benchmarks. ALP has the best performance.
Figure 4: Multi-turn training dynamics. Seq-ALP maintains higher but controlled entropy and stable convergence in both KL metrics, while Token-MIS is unstable in train-inference KL and Seq-MIS shows spikes in policy-update KL.
Figure 5: Pass@ k on AIME 2024 and AIME 2025 for TIR. ALP achieves the highest accuracy over rollout budgets k=16∼256 , indicating improved exploration efficiency and solution diversity.
Figure 6: Perturbation-layer ablation in the single-turn setting. The results reveal a clear performance hierarchy: All layers > Partial layers > Logits only; Early layers > Late layers.
Algorithm
Math500
Minerva Math
Olympiad Bench
AIME24
AIME25
Average
Layers 0-5
83.58
43.62
49.88
38.85
25.31
48.25
Layers 12-17
82.19
41.70
51.69
39.69
27.29
48.51
Layers 23-27
82.77
43.97
49.60
40.10
26.87
48.66
Layers 0-27
84.29
43.10
52.75
43.85
28.65
50.53
Table 4: Perturbation-layer ablation in the multi-turn setting. All-layer perturbation outperforms partial-layer perturbations.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Baselines
πθtrain/πθoldtrain
πθoldtrain/πθoldinfer
πθtrain/πθoldinfer
GRPO
token
None
None
Token-MIS
token
seq
None
Seq-MIS
seq
seq
None
Seq-Bypass
None
None
seq
Appendix
Table 5: Settings of parameters of importance ratios
Figure 7: Single-turn evaluation on the test dataset. We compare final-task performance of GRPO, MIS, Bypass, and ALP, highlighting that stable optimization dynamics translate into improved reward/generalization.
Figure 8: Layer-wise perturbation diagnostics across target choices: (left) mean adaptive noise scale σ , (middle) distribution of perturbation-induced probability differences ∣Δp∣ , and (right) tail behavior measured by the 75th percentile of ∣Δp∣ over training steps.
Figure 9: Hyperparameter sensitivity in the single-turn setting. We vary the initial Gaussian noise standard deviation (std) and its learning rate (lr), and report average@32 across Math500, Minerva Math, OlympiadBench, AIME24, and AIME25.
Figure 10: Token-level training-inference mismatch at step 140 in the multi-turn setting. Each point is one response token, with x-axis denoting the training probability πθoldtrain and y-axis denoting the rollout-time token probability πθoldinfer . The red dashed diagonal is y=x ; tighter concentration around this line indicates smaller training-inference mismatch. The Pearson correlation coefficient r is reported in each panel title.
Reinforcement learning (RL) has become an effective post-training paradigm for long-horizon large language model (LLM) agents. However, we find that the resulting policies can be sensitive to various policy perturbations, such as hidden-state noise, pruning, and quantization. In this work, we study how to improve perturbation robustness during policy optimization. We first introduce the notion of a perturbation robust policy and analyze conditions under which perturbed policy updates preserve stable monotonic improvement. Based on this analysis, we introduce Stable Perturbation-Robust Policy Optimization (SPrPO), which applies adaptive and sensitivity-aware perturbations during RL training. We evaluate SPrPO on ALFWorld and WebShop and conduct systematic experiments across multiple perturbation types and scales, showing improved perturbation robustness while maintaining stable policy optimization.
Pengxin Wang, Yuanzhe LI, Yuxin Ren +2
Department of Electrical and Computer Engineering University of Arizona
GRPO-style reinforcement learning (RL)-based LLM fine-tuning algorithms have recently gained popularity. Relying on heuristic trust-region approximations, however, they can lead to brittle optimization behavior, as global importance-ratio clipping and group-wise normalization fail to regulate samples whose importance ratios fall outside the clipping range. We propose Query-Adaptive Trust-Region policy Optimization (QUATRO), which directly enforces trust-region constraints through a principled optimization. This yields a clear and interpretable objective that enables explicit control over policy updates and stable, entropy-controlled optimization, with a stabilizer terms arising intrinsically from the exact trust-region formulation. Empirically verified on diverse mathematical reasoning benchmarks, QUATRO shows stable training under increased policy staleness and aggressive learning rates, maintaining well-controlled entropy throughout training.
Doyeon Lee, Eunyi Lyou, Hyunsoo Cho +3
Seoul National University · Department of Artificial Intelligence / Institute for Multiscale Matter and Systems, Ewha Woman University · Geogia Institute of Technology
Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse. One vital cause is training-inference mismatch: LLM adopts separate inference and training engines for generation efficiency and training precision, which in practice exhibits inconsistent probabilities for the same trajectories on training and inference sides, even with synchronized model parameters. This naturally induces a special type of off-policyness ever existing and poisoning the training. Prior works have made various efforts in addressing the off-policyness to stabilize the training policies under the mismatch. In this paper, we point out the objective misalignment neglected by existing works that an effective update to the policy in the training engine not necessarily ensures the improvement of the inference policy, i.e., the one used in deployment. To this end, we propose a new policy optimization objective for LLM RL, named Monotonic Inference Policy Improvement (MIPI). Following this principle, we introduce Monotonic Inference Policy Update (MIPU), a two-step LLM RL framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates using an inference-side gap proxy. Experiments conducted on two model scales under high mismatch show that MIPU improves average reasoning performance and training stability.