Learning Perturbation Robust Policies for LLM Agents with Stable Optimization

Pengxin Wang; Yuanzhe LI; Yuxin Ren; Huanrui Yang; Jingdi Chen

cs.LGSep 28, 2026

Learning Perturbation Robust Policies for LLM Agents with Stable Optimization

Authors: Pengxin Wang, Yuanzhe LI, Yuxin Ren, Huanrui Yang, Jingdi Chen

Organizations: Department of Electrical and Computer Engineering University of Arizona

Abstract

Reinforcement learning (RL) has become an effective post-training paradigm for long-horizon large language model (LLM) agents. However, we find that the resulting policies can be sensitive to various policy perturbations, such as hidden-state noise, pruning, and quantization. In this work, we study how to improve perturbation robustness during policy optimization. We first introduce the notion of a perturbation robust policy and analyze conditions under which perturbed policy updates preserve stable monotonic improvement. Based on this analysis, we introduce Stable Perturbation-Robust Policy Optimization (SPrPO), which applies adaptive and sensitivity-aware perturbations during RL training. We evaluate SPrPO on ALFWorld and WebShop and conduct systematic experiments across multiple perturbation types and scales, showing improved perturbation robustness while maintaining stable policy optimization.

Figures & tables

Figure 1: LLM agents trained by RL are sensitive to policy perturbations. Even with mild perturbation magnitude, success rates degrade substantially under these perturbations on both WebShop and ALFWorld. $\sigma$ denotes the standard deviation for Gaussian distribution, $p$ denotes the dropout probability and $r$ denotes pruning ratio.

Figure 2: Illustrative example: different perturbation strategies can lead to distinct training dynamics during policy optimization. Isotropic noise (left) applies the same perturbation scale across directions, whereas SAM-style perturbations (middle) adversarially push the policy along the sensitive direction. Both can disrupt the optimization trajectory and undermine training stability. Controlled noise (right) instead limits perturbation magnitude along sensitive directions to reduce the induced variance in policy improvement and promote a more stable training process.

Panel of Figure 3 — Figure 3: Evaluation of perturbation robustness on ALFWorld and WebShop across increasing perturbation magnitudes. The x-axis denotes perturbation strength, measured by Gaussian standard deviation $\sigma$ , dropout probability $p$ , pruning ratio $r$ , and quantization bit-width. SPrPO generally exhibits less severe performance degradation and maintains higher success rates.

WebShop	Clean	Gaussian ( $\sigma=.20$ )	Dropout ( $p=.05$ )	Structured Pruning ( $r=.20$ )	Unstructured Pruning ( $r=.40$ )	Quantization (INT6)
Vanilla	$57.00_{\scriptscriptstyle\pm 4.39}$	$18.73_{\scriptscriptstyle\pm 3.03}$ (32.9)	$12.20_{\scriptscriptstyle\pm 3.30}$ (21.4)	$2.27_{\scriptscriptstyle\pm 1.86}$ (4.0)	$42.00_{\scriptscriptstyle\pm 8.31}$ (73.7)	$45.20_{\scriptscriptstyle\pm 9.18}$ (79.3)
Gaussian	$50.60_{\scriptscriptstyle\pm 3.99}$	$17.87_{\scriptscriptstyle\pm 5.12}$ (35.3)	$11.33_{\scriptscriptstyle\pm 2.60}$ (22.4)	$0.27_{\scriptscriptstyle\pm 0.12}$ (0.5)	$34.33_{\scriptscriptstyle\pm 9.00}$ (67.9)	$39.80_{\scriptscriptstyle\pm 9.90}$ (78.7)
SAM	$52.27_{\scriptscriptstyle\pm 4.02}$	$15.20_{\scriptscriptstyle\pm 2.65}$ (29.1)	$8.87_{\scriptscriptstyle\pm 2.14}$ (17.0)	$0.60_{\scriptscriptstyle\pm 0.87}$ (1.1)	$29.53_{\scriptscriptstyle\pm 5.74}$ (56.5)	$34.93_{\scriptscriptstyle\pm 4.54}$ (66.8)
SPrPO (Ours)	$\mathbf{60.00}_{\scriptscriptstyle\pm 0.72}$	$\mathbf{30.13}_{\scriptscriptstyle\pm 1.81}$ (50.2)	$\mathbf{20.87}_{\scriptscriptstyle\pm 2.00}$ (34.8)	$\mathbf{12.40}_{\scriptscriptstyle\pm 3.65}$ (20.7)	$\mathbf{50.20}_{\scriptscriptstyle\pm 2.99}$ (83.7)	$\mathbf{55.67}_{\scriptscriptstyle\pm 1.60}$ (92.8)
ALFWorld	Clean	Gaussian ( $\sigma=.20$ )	Dropout ( $p=.05$ )	Structured Pruning ( $r=.20$ )	Unstructured Pruning ( $r=.70$ )	Quantization (INT6)
Vanilla	$87.59_{\scriptscriptstyle\pm 0.97}$	$64.96_{\scriptscriptstyle\pm 2.55}$ (74.2)	$54.62_{\scriptscriptstyle\pm 1.65}$ (62.4)	$41.73_{\scriptscriptstyle\pm 5.32}$ (47.6)	$41.12_{\scriptscriptstyle\pm 4.35}$ (46.9)	$86.98_{\scriptscriptstyle\pm 0.84}$ (99.3)

Table 1: Evaluation of perturbation robustness on WebShop and ALFWorld. Results are reported as success rate std (retention) over three seeds, where retention denotes the percentage of performance preserved relative to the corresponding clean policy. SPrPO achieves the strongest performance, maintaining both higher absolute success rates and higher relative retention.

Panel of Figure 4 — Figure 4: Reward landscape of policies trained with Vanilla and SPrPO on ALFWorld and WebShop. Vanilla exhibits sharp performance cliffs, while SPrPO shows flatter reward surfaces.

Panel of Figure 5 — Figure 5: Ablation and hyperparameter study. (a) Comparison of perturbation strategies during training. (b) Effect of gradient-norm scaling. (c,d) Effects of perturbation scale $\rho$ and sensitivity exponent $\alpha$ . Starred and $\rho=0$ results reuse the main-experiment estimates in Table 1 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Base model	Qwen2.5-1.5B-Instruct
LoRA rank / $\alpha_{\mathrm{LoRA}}$	$64$ / $64$
Optimizer	Adam
Learning rate	$3\times 10^{-6}$
Training steps	$200$
Tasks per step	$16$

Appendix

Table 2: Shared training hyperparameters used across all methods.

Method	Hyperparameter	Value
Gaussian	Noise scale $\sigma$	$0.10$
SAM	Perturbation scale $\rho$	$0.005$
SPrPO	Base perturbation scale $\rho$	$0.20$
SPrPO	Sensitivity exponent $\alpha$	$1.0$

Appendix

Table 3: Perturbation-specific training hyperparameters used in the main experiments.

Environment	Method	Rollout	Update	Total
ALFWorld	Vanilla	143.8	198.2	343.6
ALFWorld	SPrPO	171.5	400.1	572.9
WebShop	Vanilla	50.1	60.0	110.6
WebShop	SPrPO	58.2	138.7	197.4

Appendix

Table 4: Observed training cost of Vanilla and SPrPO on Qwen2.5-1.5B-Instruct. Times are averaged seconds per training step and exclude periodic validation and checkpoint saving.

Perturbation	Strength	Vanilla	Gaussian	SAM	SPrPO
Clean	BF16	$87.59\,{\scriptstyle\pm\,0.97}$	$89.05\,{\scriptstyle\pm\,0.97}$	$86.86\,{\scriptstyle\pm\,0.73}$	$\mathbf{91.85}\,{\scriptstyle\pm\,0.42}$
Gaussian	$\sigma=0.02$	$88.44\,{\scriptstyle\pm\,1.38}$	$90.02\,{\scriptstyle\pm\,0.21}$	$86.01\,{\scriptstyle\pm\,0.76}$	$\mathbf{92.70}\,{\scriptstyle\pm\,0.36}$
	$\sigma=0.05$	$87.23\,{\scriptstyle\pm\,1.67}$	$89.54\,{\scriptstyle\pm\,1.28}$	$86.74\,{\scriptstyle\pm\,0.92}$	$\mathbf{91.12}\,{\scriptstyle\pm\,0.56}$
	$\sigma=0.10$	$84.79\,{\scriptstyle\pm\,2.23}$	$85.89\,{\scriptstyle\pm\,0.56}$	$86.13\,{\scriptstyle\pm\,0.73}$	$\mathbf{89.78}\,{\scriptstyle\pm\,0.63}$
	$\sigma=0.15$	$79.81\,{\scriptstyle\pm\,0.92}$	$81.51\,{\scriptstyle\pm\,1.11}$	$81.63\,{\scriptstyle\pm\,2.48}$	$\mathbf{86.13}\,{\scriptstyle\pm\,1.90}$
	$\sigma=0.20$	$64.96\,{\scriptstyle\pm\,2.55}$	$72.26\,{\scriptstyle\pm\,0.97}$	$66.79\,{\scriptstyle\pm\,2.55}$	$\mathbf{75.91}\,{\scriptstyle\pm\,0.97}$

Appendix

Table 5: ALFWorld: overall success rate (%).

		ID				OOD
Perturbation	Strength	Vanilla	Gaussian	SAM	SPrPO	Vanilla	Gaussian	SAM	SPrPO
Clean	BF16	$89.05\,{\scriptstyle\pm\,1.09}$	$89.29\,{\scriptstyle\pm\,1.24}$	$90.48\,{\scriptstyle\pm\,2.06}$	$\mathbf{94.05}\,{\scriptstyle\pm\,0.82}$	$86.07\,{\scriptstyle\pm\,1.55}$	$88.81\,{\scriptstyle\pm\,1.97}$	$83.08\,{\scriptstyle\pm\,2.62}$	$\mathbf{89.55}\,{\scriptstyle\pm\,0.00}$
Gaussian	$\sigma=0.02$	$89.52\,{\scriptstyle\pm\,2.70}$	$90.24\,{\scriptstyle\pm\,0.82}$	$88.33\,{\scriptstyle\pm\,0.41}$	$\mathbf{93.33}\,{\scriptstyle\pm\,1.65}$	$87.31\,{\scriptstyle\pm\,0.75}$	$89.80\,{\scriptstyle\pm\,0.43}$	$83.58\,{\scriptstyle\pm\,1.49}$	$\mathbf{92.04}\,{\scriptstyle\pm\,1.14}$
	$\sigma=0.05$	$86.90\,{\scriptstyle\pm\,1.80}$	$90.48\,{\scriptstyle\pm\,1.49}$	$89.29\,{\scriptstyle\pm\,1.43}$	$\mathbf{92.14}\,{\scriptstyle\pm\,1.43}$	$87.56\,{\scriptstyle\pm\,1.55}$	$88.56\,{\scriptstyle\pm\,1.14}$	$84.08\,{\scriptstyle\pm\,2.83}$	$\mathbf{90.05}\,{\scriptstyle\pm\,0.43}$
	$\sigma=0.10$	$86.90\,{\scriptstyle\pm\,2.89}$	$85.95\,{\scriptstyle\pm\,1.49}$	$88.33\,{\scriptstyle\pm\,1.80}$	$\mathbf{90.48}\,{\scriptstyle\pm\,2.30}$	$82.59\,{\scriptstyle\pm\,3.02}$	$85.82\,{\scriptstyle\pm\,1.49}$	$83.83\,{\scriptstyle\pm\,0.43}$	$\mathbf{89.05}\,{\scriptstyle\pm\,2.28}$
	$\sigma=0.15$	$77.62\,{\scriptstyle\pm\,1.65}$	$82.86\,{\scriptstyle\pm\,2.14}$	$\mathbf{85.71}\,{\scriptstyle\pm\,3.11}$	$\mathbf{85.71}\,{\scriptstyle\pm\,2.14}$	$82.09\,{\scriptstyle\pm\,0.75}$	$80.10\,{\scriptstyle\pm\,1.88}$	$77.36\,{\scriptstyle\pm\,2.28}$	$\mathbf{86.57}\,{\scriptstyle\pm\,4.48}$

Appendix

Table 6: ALFWorld: ID and OOD success rates (%).

Perturbation	Strength	Vanilla	Gaussian	SAM	SPrPO
Clean	BF16	$57.00\,{\scriptstyle\pm\,4.39}$	$50.60\,{\scriptstyle\pm\,3.99}$	$52.27\,{\scriptstyle\pm\,4.02}$	$\mathbf{60.00}\,{\scriptstyle\pm\,0.72}$
Gaussian	$\sigma=0.02$	$57.00\,{\scriptstyle\pm\,3.82}$	$51.07\,{\scriptstyle\pm\,3.35}$	$52.13\,{\scriptstyle\pm\,2.95}$	$\mathbf{59.73}\,{\scriptstyle\pm\,1.55}$
	$\sigma=0.05$	$53.47\,{\scriptstyle\pm\,4.88}$	$48.47\,{\scriptstyle\pm\,4.80}$	$49.00\,{\scriptstyle\pm\,2.31}$	$\mathbf{58.67}\,{\scriptstyle\pm\,1.29}$
	$\sigma=0.10$	$45.87\,{\scriptstyle\pm\,6.65}$	$41.53\,{\scriptstyle\pm\,5.35}$	$40.00\,{\scriptstyle\pm\,4.13}$	$\mathbf{53.07}\,{\scriptstyle\pm\,1.29}$
	$\sigma=0.15$	$32.73\,{\scriptstyle\pm\,2.95}$	$30.33\,{\scriptstyle\pm\,7.83}$	$27.27\,{\scriptstyle\pm\,3.69}$	$\mathbf{44.33}\,{\scriptstyle\pm\,0.50}$
	$\sigma=0.20$	$18.73\,{\scriptstyle\pm\,3.03}$	$17.87\,{\scriptstyle\pm\,5.12}$	$15.20\,{\scriptstyle\pm\,2.65}$	$\mathbf{30.13}\,{\scriptstyle\pm\,1.81}$

Appendix

Table 7: WebShop: success rate (%).

Value	Clean	Gaussian $\sigma=.20$	Dropout $p=.05$	Struct. $r=.20$	Unstruct. $r=.40$	INT6
$\rho$ sweep ( $\alpha=1$ )
0	$57.00\,{\scriptstyle\pm\,4.39}$	$18.73\,{\scriptstyle\pm\,3.03}$	$12.20\,{\scriptstyle\pm\,3.30}$	$2.27\,{\scriptstyle\pm\,1.86}$	$42.00\,{\scriptstyle\pm\,8.31}$	$45.20\,{\scriptstyle\pm\,9.18}$
0.1	$55.33\,{\scriptstyle\pm\,5.16}$	$17.67\,{\scriptstyle\pm\,5.52}$	$10.73\,{\scriptstyle\pm\,2.93}$	$4.27\,{\scriptstyle\pm\,4.45}$	$40.80\,{\scriptstyle\pm\,9.71}$	$39.53\,{\scriptstyle\pm\,9.98}$
0.2	$\mathbf{60.00}\,{\scriptstyle\pm\,0.72}$	$\mathbf{30.13}\,{\scriptstyle\pm\,1.81}$	$\mathbf{20.87}\,{\scriptstyle\pm\,2.00}$	$\mathbf{12.40}\,{\scriptstyle\pm\,3.65}$	$\mathbf{50.20}\,{\scriptstyle\pm\,2.99}$	$\mathbf{55.67}\,{\scriptstyle\pm\,1.60}$
0.3	$48.27\,{\scriptstyle\pm\,2.50}$	$25.93\,{\scriptstyle\pm\,2.02}$	$17.00\,{\scriptstyle\pm\,1.97}$	$7.60\,{\scriptstyle\pm\,2.62}$	$40.47\,{\scriptstyle\pm\,3.56}$	$39.27\,{\scriptstyle\pm\,4.75}$
$\alpha$ sweep ( $\rho=0.2$ )

Appendix

Table 8: Hyperparameter study. Success rate (%).

Training variant	Clean	Gaussian $\sigma=.20$	Dropout $p=.05$	Struct. $r=.20$	Unstruct. $r=.40$	INT6
w/o perturbation ( $\rho=0$ )	$57.00\,{\scriptstyle\pm\,4.39}$	$18.73\,{\scriptstyle\pm\,3.03}$	$12.20\,{\scriptstyle\pm\,3.30}$	$2.27\,{\scriptstyle\pm\,1.86}$	$42.00\,{\scriptstyle\pm\,8.31}$	$45.20\,{\scriptstyle\pm\,9.18}$
w/o sensitivity ( $\alpha=0$ )	$48.80\,{\scriptstyle\pm\,0.80}$	$27.07\,{\scriptstyle\pm\,3.72}$	$18.93\,{\scriptstyle\pm\,3.42}$	$11.07\,{\scriptstyle\pm\,6.92}$	$47.07\,{\scriptstyle\pm\,3.11}$	$48.27\,{\scriptstyle\pm\,3.06}$
w/o scheduling	$52.13\,{\scriptstyle\pm\,0.42}$	$26.40\,{\scriptstyle\pm\,4.70}$	$19.33\,{\scriptstyle\pm\,5.32}$	$3.60\,{\scriptstyle\pm\,2.88}$	$45.00\,{\scriptstyle\pm\,4.20}$	$49.73\,{\scriptstyle\pm\,2.42}$
Full SPrPO	$\mathbf{60.00}\,{\scriptstyle\pm\,0.72}$	$\mathbf{30.13}\,{\scriptstyle\pm\,1.81}$	$\mathbf{20.87}\,{\scriptstyle\pm\,2.00}$	$\mathbf{12.40}\,{\scriptstyle\pm\,3.65}$	$\mathbf{50.20}\,{\scriptstyle\pm\,2.99}$	$\mathbf{55.67}\,{\scriptstyle\pm\,1.60}$

Appendix

Table 9: Component ablations on WebShop. Success rate (%).

Explore similar work

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning
Jun 28, 2026Jing Liang, Hongyao Tang, Yi Ma +9Large Language Model Reinforcement Learning Training-Inference Mismatch
1Tianjin · 2Alibaba
When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?
May 29, 2026Stephane Hatgis-Kessell, Emma BrunskillLarge Language Model Policy Optimization Frictive Policy Optimization
Department of Computer Science, Stanford University
Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
Nov 25, 2025Chenliang Li, Adel Elmahdy, Alex Boyd +7Large Language Model Reinforcement Learning
Texas A&M University · GE HealthCare · Independent Researcher +1

Jun 28, 2026, cs.LGJ/K move · Enter open · S save

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

Jing Liang, Hongyao Tang, Yi Ma, Yancheng He+8

1Tianjin · 2Alibaba

Large Language Model Reinforcement Learning Training-Inference Mismatch Boltzmann Policies

Open paper AskPDF arXiv

Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse. One vital cause is training-inference mismatch: LLM adopts separate inference and training engines for generation efficiency and training precision, which in practice exhibits inconsistent probabilities for the same trajectories on training and inference sides, even with synchronized model parameters. This naturally induces a special type of off-policyness ever existing and poisoning the training. Prior works have made various efforts in addressing the off-policyness to stabilize the training policies under the mismatch. In this paper, we point out the objective misalignment neglected by existing works that an effective update to the policy in the training engine not necessarily ensures the improvement of the inference policy, i.e., the one used in deployment. To this end, we propose a new policy optimization objective for LLM RL, named Monotonic Inference Policy Improvement (MIPI). Following this principle, we introduce Monotonic Inference Policy Update (MIPU), a two-step LLM RL framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates using an inference-side gap proxy. Experiments conducted on two model scales under high mismatch show that MIPU improves average reasoning performance and training stability.

Abstract

Explore similar work

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?

Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization