Authors: Jan Kobiolka, Christian Frey, Gresa Shala, Arlind Kadra, Erind Bedalli, Josif Grabocka
Organizations: Department of Computer Science University of Technology Nuremberg Germany · Department of Computer Science Albert Ludwig University of Freiburg Germany · Faculty of Natural Sciences University of Elbasan Albania
Gradient-based optimizers are highly sensitive to design choices in their adaptive learning rate mechanisms. To address this limitation, we introduce POP, a meta-learned Reinforcement Learning (RL) policy that predicts adaptive learning rates for gradient descent, conditioned on the contextual information provided in the optimization trajectory. Our method introduces a novel RL reward formulation, a new function-scaling strategy for in-distribution generalization, and a novel prior that is used to sample millions of synthetic optimization problems. We evaluate POP on an established benchmark including 43 optimization functions of various complexity, where it significantly outperforms gradient-based methods. Our evaluation demonstrates strong generalization capabilities without task-specific tuning.
Figures & tables
Figure 1: POP (blue) adapts its learning rate based on the optimization landscape, enabling rapid convergence, escape from local minima, and improved global optimization compared to Adam (red). Yellow diamond represents the global minima, the white cross represents the start state, and the square represents the end state.
Figure 2Figure 3
Dim.
POP
Random
Adam
GD
L-BFGS
Lion
Sophia-G
16D
0.2911
0.1177
0.2887
0.2510
0.1728
0.2409
0.2579
32D
0.4966
0.1669
0.5029
0.4003
0.4171
0.4245
0.4820
Table 1: Mean performance across different dimensionalities.
Figure 6: The average normalized regret for all methods on VLSE. Solid lines represent the mean value.
Figure 7: Performance of POP compared across different model categories and representative models.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol / Quantity
Distribution / Value
D
2
Domain
[−50,50]D
Input shift
U(−40,40)D
Vertical offset b
0
M
1000
Quadratic mixing weight α3
U(0.05,0.4) for bowl-dominated instances
Appendix
Table 3: Main hyperparameter values of the proposed prior.
Figure 8: Exemplary functions sampled from the prior distribution (cf. Section 2.4 )
Component
Value
Shared backbone (Transformer)
Embedding dimension
64
# Transformer blocks
8
# Attention heads
8 ( dhead=8 )
Dropout
0.05
Pooling
Last-token pooling
Appendix
Table 4: Key architecture hyperparameters for the shared Transformer backbone and actor/critic heads.
Component
Value
Actor lr
5×10−5
Critic lr
5×10−5
Optimizer
AdamWScheduleFree [ 8 ]
Actor weight decay
1×10−4
Critic weight decay
1×10−4
Actor warmup
300
Appendix
Table 5: PPO training hyperparameters and optimizer settings.
Figure 9: Learning rate sweep on the in-distribution validation set.
Figure 10: Hyperparameter sweep for the evolutionary methods on the in-distribution validation set.
Figure 11: Trajectory of POP’s on in-distribution functions. The algorithm progresses from the initial state (white cross) terminating at the final state (square).
Figure 12: Trajectory of POP’s on in-distribution functions. The algorithm progresses from the initial state (white cross) terminating at the final state (square).
Figure 13: In-distribution test set performance vs. all baselines. Mean normalized improvement over steps; shading indicates 95% CIs. Dashed line marks the context/optimization boundary.
Figure 14: In-distribution test set performance vs. all baselines. Mean normalized improvement over steps; shading indicates 95% CIs. Dashed line marks the context/optimization boundary.
Dim.
POP
Random
Adam
GD
L-BFGS
Lion
Sophia-G
TPE
GP
HEBO
CMA-ES
DE
GA
VELO
16D
0.2911
0.1177
0.2887
0.2510
0.1728
0.2409
0.2579
0.2107
0.2692
0.2702
0.1153
0.1421
0.2145
0.3100
32D
0.4966
0.1669
0.5029
0.4003
0.4171
0.4245
0.4820
0.3176
0.4807
0.4368
0.0572
0.1953
0.3691
0.5225
Appendix
Table 6: Mean performance across different dimensionalities.
Figure 15: The average normalized regret for all methods on VLSE. Solid lines represent the mean value.
Figure 16: Performance of POP compared to individual competitors from different model categories on diverse function classes
Figure 17: Performance of POP compared across different model categories and representative models.
We propose a new perspective on policy optimization: rather than reweighting all samples by their importance ratios, an optimizer should select which samples are trustworthy enough to drive a policy update. Building on this view, we introduce Rejection-Gated Policy Optimization (RGPO), which replaces the importance sampling ratio r_theta = pi_theta / pi_old with a smooth, differentiable acceptance gate alpha_theta(s, a) = g(r_theta(s, a)) in the range [0, 1]. Unlike prior work that applies rejection sampling as a data-level heuristic before training, RGPO elevates rejection to an optimization principle: the gate participates directly in gradient computation and is implicitly updated alongside the policy. RGPO provides a unified framework: the policy gradients of TRPO, PPO, and REINFORCE all correspond to specific choices of the effective gradient weight w(r) = g'(r) * r. We prove that RGPO guarantees finite, bounded gradient variance even when importance sampling ratios are heavy-tailed (where IS variance diverges). We further show that RGPO incurs only a bounded, controllable bias and provides an approximate monotonic policy improvement guarantee analogous to TRPO. RGPO matches PPO in computational cost, requires no second-order optimization, and extends naturally to RLHF-style preference alignment. In online preference fine-tuning of Qwen2.5-1.5B-Instruct on Anthropic HH-RLHF (n = 3 seeds), RGPO uses a dual-ratio gate that anchors learning to both the previous policy and the reference model, achieving a Pareto-dominant outcome: the highest reward among online RL methods (+14.8% vs. PPO-RLHF) and the lowest KL divergence to the reference model (-16.0% vs. PPO-RLHF, -53.1% vs. GRPO).
Group relative policy optimization for reinforcement learning with verifiable rewards (RLVR) typically uses a fixed importance-sampling (IS) ratio clipping boundary across all rollouts. We identify a key limitation: rare correct rollouts on harder problems and abundant correct rollouts on easier problems are clipped at comparable rates, despite contributing very different learning signals. Rollouts with low group success exhibit larger IS ratios and carry stronger gradient signal for exploration and solving new problems, yet are disproportionately suppressed by fixed clipping. To address this, we propose Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage. GAPO is motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update headroom. GAPO requires no reward shaping and preserves the standard PPO/GSPO surrogate while adapting only the clipping threshold. Across Qwen and Llama models, GAPO consistently improves both Pass@1 and Pass@k over fixed clipping and advantage-shaping baselines on math reasoning and coding benchmarks where the pass rates by the base model are relatively low.
Policy-gradient methods usually optimize expected return, but many real world applications care about distributional properties of returns: tail risk, outlier robustness, or best-of-K discovery. We introduce OrderGrad, a family of likelihood-ratio and reparameterization gradient estimators for order-statistic objectives. OrderGrad optimizes finite-sample L-statistics, i.e., weighted averages of sorted rewards or costs, recovering objectives such as VaR, CVaR, trimmed means, medians, and top-m/best-of-K criteria by changing only the rank weights. For any fixed sample size and rank-weight vector, OrderGrad provides an unbiased gradient estimator for the corresponding order-statistic objective. The method is implemented as a simple reward transformation that can then be used in an otherwise standard policy-gradient or reparameterized update. We study the resulting estimator's variance behavior and evaluate it on tasks where mean optimization is mismatched to the deployment objective, including LLM math post-training and other tasks. OrderGrad provides a unified, plug-and-play route to risk-averse, robust, and exploratory learning. Code: https://github.com/paavo5/ordergrad