Ordered Policy Optimization

Dongkyu Derek Cho; Pan Xu

cs.LGNov 6, 2026

Ordered Policy Optimization

Authors: Dongkyu Derek Cho, Pan Xu

Organizations: Department of Statistical Science Duke University · Department of Biostatistics & Bioinformatics Department of Computer Science Department of Electrical & Computer Engineering Duke University

Abstract

Proximal Policy Optimization (PPO) stabilizes policy updates by clipping the importance ratio between the new policy and the data-collecting policy. The standard rule uses one fixed clipping range for all samples. While this choice offers simplicity and effectiveness, it fails to distinguish statistically harmful extreme ratio values from large, yet tolerable changes. We study how to adapt the clipping radius to the reliability of the induced importance ratio distribution. Our case study demonstrates that reliable updates require controlling these extreme ratios, rather than merely verifying that individual ratios fall within a fixed interval. Motivated by this observation, we propose Ordered Policy Optimization (OPO). In each minibatch, OPO sorts the importance ratios, selects the extreme ratios that potentially have the largest effect on the update, and assigns them adaptive sample-wise radii instead of using the fixed clipping radius. These radii are derived from a prescribed tail profile, which specifies desired target values for the selected extreme ratios. The resulting objective remains first-order and requires only minibatch sorting and a modified policy loss. Experiments conducted on MuJoCo and Atari benchmarks demonstrate that this simple modification achieves strong performance relative to established baselines.

Figures & tables

Figure 1: Overview of OPO . A minibatch of importance ratios is ordered, and the empirical tails are assigned teacher targets induced by a prescribed generalized Pareto tail profile. This yields adaptive sample-wise proximal targets, thereby replacing scalar clipping with order-based tail control. In contrast, PPO imposes the same scalar clipping boundary across ratios.

Figure 2: Ratio distribution induced by the two policies in the LQR case study at a representative state. Dashed lines mark

1\pm\kappa

with

\kappa=0.2

Figure 3: LQR case study. Top: heavier ratio tails emerge as

k

increases. Bottom: repeated estimates of

\widehat{\mathcal{L}}_{n}

become more dispersed as

k

approaches and exceeds

1/2

Figure 4: Overview of the OPO algorithm. OPO generates the target

r_{t}^{\star}

from the GPD teacher profile, then assigns each target to the corresponding order statistic: the

t

-th ordered ratio

r_{(t)}

is assigned the corresponding

r_{t}^{\star}

. This process encourages the empirical tail ratios to follow the prescribed shape.

Figure 5: Aggregate MuJoCo performance under the architecture-selected protocol. We report the median, interquartile mean (IQM), mean, and optimality gap across tasks. Higher is better for the first three metrics, and lower is better for the optimality gap.

Figure 6: Effect of the generalized Pareto shape parameter on the empirical upper tail of the policy ratio. Left and Middle: At a representative time step, we plot the empirical survival function

\widehat{\bar{F}}(x)=B^{-1}\sum_{t=1}^{B}\mathbf{1}\{r_{t}>x\}

of the raw policy ratio

r_{t}

for three prescribed GPD shape parameters

k\in\{0.33,0.49,0.70\}

. Larger

k

leads to slower tail decay. Right: Evolution of the estimated tail shape

\hat{k}

across updates, with dashed lines indicating the corresponding prescribed values of

k

Figure 7: Relative improvement over PPO on Atari environments. Each bar reports the relative improvement observed over the final 10% of validation runs for OPO and SPO. Bars exceeding the visible range are truncated for readability, and this truncation is indicated in the plot.

Figure 8: Representative Atari learning curves. Left: RoadRunner. Middle: NameThisGame. Right: BeamRider. OPO reaches higher returns than PPO and SPO on all tasks.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	$\hat{k}$	QQ correlation
Ant	0.963	0.871
HalfCheetah	0.465	0.980
Hopper	0.522	0.978
Humanoid	0.665	0.906
HumanoidStandup	1.002	0.817
Walker2d	0.513	0.961

Appendix

Table 1: Estimated GPD shape parameters and QQ correlations.

Figure 9: QQ correlation plot with the GPD.

Moment order $m$	$k$	Boundary for $\sigma_{\bm{\theta}}^{2}$	Boundary for $\sigma_{\bm{\theta}}$	$\mathcal{L}(\bm{\theta})$
$2$	$0.500$	$0.3186$	$0.5645$	$-0.1816$
$3$	$0.333$	$0.2380$	$0.4878$	$-0.0889$
$1/0.7\approx 1.429$	$0.700$	$0.5315$	$0.7290$	$-0.4262$
$1/0.9\approx 1.111$	$0.900$	$1.5989$	$1.2645$	$-1.6530$

Appendix

Table 2: Variance boundary values and corresponding population target values for fixed

K_{\bm{\theta}}=-0.45

. Here

K_{\mathrm{old}}=-0.5

\sigma_{0}=0.4

\sigma_{s}\simeq 0.52

D\simeq 0.048

, and

\Lambda\simeq 2.299

\mathcal{L}_{\mathrm{OPO}}\leftarrow-\frac{1}{|\mathcal{B}|}\sum_{t\in\mathcal{B}}\left\{\frac{\pi_{\bm{\theta}}(a_{t}\mid s_{t})}{\pi_{\bm{\theta}_{\mathrm{old}}}(a_{t}\mid s_{t})}\cdot\widehat{A}(s_{t},a_{t})-\frac{|\widehat{A}(s_{t},a_{t})|}{2\varepsilon_{t}}\cdot\left[\frac{\pi_{\bm{\theta}}(a_{t}\mid s_{t})}{\pi_{\bm{\theta}_{\mathrm{old}}}(a_{t}\mid s_{t})}-1\right]^{2}\right\},

Appendix

Algorithm 1 Ordered Policy Optimization (OPO)

Notation	Role
$\kappa$	Nominal bulk radius and upper bound on the empirical tail boundary
$\sigma^{\star}$	Scale of the prescribed teacher tail above $u_{\eta}$
$k^{\star}$	Shape parameter governing tail heaviness and the associated moment regime
$Q_{z}$	Teacher target assigned to the $z$ -th retained order statistic

Appendix

Table 3: Main quantities defining the OPO tail construction.

	Prescribed shape $k^{\star}$
Environment	$0.33$	$0.49$	$0.70$
Ant-v4	0.243	0.387	0.559
HalfCheetah-v4	0.241	0.323	0.507
Hopper-v4	0.217	0.350	0.538
Humanoid-v4	0.338	0.461	0.658
HumanoidStandup-v4	0.342	0.460	0.594

Appendix

Table 4: Prescribed and observed tail shapes under OPO. Each entry is the mean GPD shape fitted to the realized relevant likelihood ratio tail over nine training updates.

Tail	Main $\widehat{k}$ range	Exceedances	Trajectory correlation	$\max\|\Delta\widehat{k}\|$
$r_{t}$ , $\widehat{A}_{t}\geq 0$	0.219–0.519	84–125	0.975	0.067
$1/r_{t}$ , $\widehat{A}_{t}<0$	0.340–0.755	52–106	0.923	0.129

Appendix

Table 5: Sensitivity of the fitted tail-shape trajectory to the empirical threshold. The nearby construction retains

20\%

more exceedances than the default rule.

Hyperparameters	Atari 2600	MuJoCo
Optimizer	Adam	Adam
Number of workers	8	8
Total steps	10M	10M
Horizon	128	256
Batch size	1024	2048
Number of minibatches	4	4

Appendix

Table 6: Detailed hyperparameters used.

Figure 10: MuJoCo-v4 learning curves under the architecture-selected protocol. Solid curves denote averages over seeds, and shaded regions indicate across-seed variability.

Environment	OPO	PPO	SPO
Ant-v4	$1.91\pm 0.18$	$1.80\pm 0.03$	$2.12\pm 0.22$
HalfCheetah-v4	$2.37\pm 0.67$	$1.65\pm 0.01$	$2.43\pm 0.34$
Hopper-v4	$1.79\pm 0.10$	$1.60\pm 0.02$	$2.04\pm 0.29$
Humanoid-v4	$2.44\pm 0.36$	$1.92\pm 0.07$	$2.39\pm 0.21$
HumanoidStandup-v4	$2.65\pm 0.10$	$2.37\pm 0.07$	$2.17\pm 0.05$
Walker2d-v4	$2.03\pm 0.12$	$1.61\pm 0.05$	$1.83\pm 0.04$

Appendix

Table 7: Representative MuJoCo wall-clock training time in hours. Entries report mean

\pm

sample standard deviation over seeds. The last row averages per-environment means.

$k$	Ant-v4	HalfCheetah-v4	Hopper-v4	Humanoid-v4	Humanoid Standup-v4	Walker2d-v4
$0.33$	4821.96 $\pm$ 841.13	5239.74 $\pm$ 1970.91	1567.71 $\pm$ 165.72	4521.40 $\pm$ 741.32	187300.73 $\pm$ 54819.22	3412.71 $\pm$ 550.91
$0.49$	4318.59 $\pm$ 937.35	3563.51 $\pm$ 2144.21	1498.09 $\pm$ 177.21	4248.31 $\pm$ 271.17	216661.41 $\pm$ 44528.67	3171.71 $\pm$ 773.86
$0.70$	4678.31 $\pm$ 916.41	2005.49 $\pm$ 67.46	1543.86 $\pm$ 115.41	3717.99 $\pm$ 573.71	177072.27 $\pm$ 26268.78	2999.32 $\pm$ 718.49

Appendix

Table 8: Ablation over the GPD shape parameter

k

on MuJoCo. Each entry reports the mean on the first line and the standard deviation on the second line. Higher is better.

Figure 11: MuJoCo-v4 learning curves under different policy-network architectures. Solid curves correspond to seven-layer policies, while dotted curves correspond to three-layer policies. Colors denote different methods. Curves show averages over seeds, and shaded regions indicate across-seed variability.

Setting	Ant-v4	HalfCheetah-v4	Humanoid Standup-v4	Walker2d-v4
$\kappa=0.2,\ \sigma^{\star}=0.3$	4902.29 $\pm$ 59.21	3836.51 $\pm$ 1371.68	192545.05 $\pm$ 21638.89	3212.46 $\pm$ 196.34
$\kappa=0.2,\ \sigma^{\star}=0.1$	4821.96 $\pm$ 841.13	5239.74 $\pm$ 1970.91	187300.73 $\pm$ 54819.22	3412.71 $\pm$ 550.91
$\kappa=0.1,\ \sigma^{\star}=0.3$	4230.80 $\pm$ 443.05	4480.91 $\pm$ 1935.52	177288.03 $\pm$ 14523.83	3259.16 $\pm$ 720.85
$\kappa=0.1,\ \sigma^{\star}=0.1$	3618.48 $\pm$ 1009.94	3586.43 $\pm$ 1928.31	158855.81 $\pm$ 2707.94	3040.67 $\pm$ 475.03

Appendix

Table 9: Ablation over

\kappa

and GPD scale

\sigma^{\star}

on MuJoCo. Each entry reports the mean on the first line and the standard deviation on the second line. Higher is better.

Setting	Ant-v4	HalfCheetah-v4	Hopper-v4	Humanoid-v4	Humanoid Standup-v4	Walker2d-v4
$2$ minibatches	4479.57 $\pm$ 528.75	2778.80 $\pm$ 955.79	1527.27 $\pm$ 77.03	3465.25 $\pm$ 267.99	189587.57 $\pm$ 43327.78	3326.54 $\pm$ 261.21
$4$ minibatches	4821.96 $\pm$ 841.13	5239.74 $\pm$ 1970.91	1567.71 $\pm$ 165.72	4521.40 $\pm$ 741.32	187300.73 $\pm$ 54819.22	3412.71 $\pm$ 550.91
$6$ minibatches	4330.91 $\pm$ 1155.30	5764.36 $\pm$ 1056.23	1746.36 $\pm$ 258.82	4589.90 $\pm$ 200.05	156140.88 $\pm$ 3714.68	3757.89 $\pm$ 215.99
$8$ minibatches	4568.11 $\pm$ 422.11	4407.45 $\pm$ 1727.63	1365.53 $\pm$ 119.48	4336.45 $\pm$ 487.01	165167.27 $\pm$ 9843.86	2695.99 $\pm$ 342.23

Appendix

Table 10: Ablation over the number of minibatches on MuJoCo. Each entry reports the mean on the first line and the standard deviation on the second line. Higher is better.

Method	Mean	Min	Q10	Q25	Median	Q75	Q90	Max
OPO	318.37	-49.21	-1.66	17.77	62.58	225.54	859.02	4150.75
SPO	165.45	-70.47	-3.29	16.41	48.95	162.89	365.28	1169.36

Appendix

Table 11: Relative improvement over PPO on Atari2600. Each row summarizes the distribution of environment-wise relative improvements across environments. Higher is better.

Figure 12: Learning curves on Atari 2600. The mean and standard deviation are shown across 3 random seeds.

Policy Optimization

Policy optimization develops algorithms for improving decision rules in reinforcement learning, language models, and sequential decision systems.

See all 233 papers

Ratio-Variance Regularized Policy Optimization
May 26, 2026Yu Luo, Shuo Han, Yihan Hu +5RL for Language Model Reasoning Policy Optimization
Department of Foundation Model, 2012 Labs, Huawei Shanghai · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Department of Computer Science and Technology, Tsinghua University +1
Bounded Ratio Reinforcement Learning
Apr 20, 2026Yunke Ao, Le Chen, Bruce D. Lee +5Reinforcement Learning Group Relative Policy Optimization
ETH Zurich · MPI for Intelligent Systems · University of Alberta +2
Beyond Importance Sampling: Rejection-Gated Policy Optimization
Apr 16, 2026Ziwu Sun, Zhen Gao, Jiyong Zhang +1Importance Sampling Policy Optimization

May 26, 2026J/K move · Enter open · S save

Ratio-Variance Regularized Policy Optimization

Yu Luo, Shuo Han, Yihan Hu, Lei Lv+4

Department of Foundation Model, 2012 Labs, Huawei Shanghai · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Department of Computer Science and Technology, Tsinghua University+1

RL for Language Model Reasoning Policy Optimization Robotic RL Proximal Policy Optimization Sample-Efficient RL Regularized RL

Open paper AskPDF arXiv

Standard on-policy reinforcement learning relies on heuristic clipping to enforce trust regions, but this mechanism imposes a severe cost by indiscriminately truncating high-return yet high-divergence updates. We demonstrate that explicitly constraining the policy ratio variance provides a principled local approximation to trust-region constraints, eliminating the need for binary hard clipping. By acting as a distributional ``soft brake'', this approach preserves critical gradient signals from novel discoveries while naturally down-weighting and enabling the reuse of stale, off-policy data. We introduce

{\bf R}^2{\bf VPO}

(Ratio-Variance Regularized Policy Optimization), which implements this constraint via a primal-dual optimization framework. Extensive evaluations across

7

LLM scales, spanning both fast and slow reasoning paradigms, and

10

robotic control tasks demonstrate the generality of the proposed approach. R

^2

VPO achieves substantial performance gains on mathematical reasoning benchmarks, with particularly pronounced improvements on smaller models, while significantly improving sample efficiency. Furthermore, it consistently outperforms PPO baselines in continuous control domains, particularly in sparse-reward and dynamic environments. Together, these findings establish ratio-variance regularization as a principled foundation for stable and data-efficient policy optimization.

Abstract

Policy Optimization

Explore similar work

Ratio-Variance Regularized Policy Optimization

Bounded Ratio Reinforcement Learning

Beyond Importance Sampling: Rejection-Gated Policy Optimization