cs.LGFeb 24, 2026

Group Orthogonalized Policy Optimization:Group Policy Optimization as Orthogonal Projection in Hilbert Space

Authors: Wang Zixian

Organizations: China Mobile Communications Group Shandong Co., Ltd. Tai’an Branch

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present Group Orthogonalized Policy Optimization (GOPO), a new alignment algorithm for large language models derived from the geometry of Hilbert function spaces. Instead of optimizing on the probability simplex and inheriting the exponential curvature of Kullback-Leibler divergence, GOPO lifts alignment into the Hilbert space L2(pi_k) of square-integrable functions with respect to the reference policy. Within this space, the simplex constraint reduces to a linear orthogonality condition <v, 1> = 0, defining a codimension-one subspace H0. Minimizing distance to an unconstrained target u_star yields the work-dissipation functional J(v) = <g, v> - (mu / 2) ||v||^2, whose maximizer follows directly from the Hilbert projection theorem. Enforcing the boundary v >= -1 produces a bounded Hilbert projection that induces exact sparsity, assigning zero probability to catastrophically poor actions through a closed-form threshold. To connect this functional theory with practice, GOPO projects from infinite-dimensional L2(pi_k) to a finite empirical subspace induced by group sampling. Because group-normalized advantages sum to zero, the Lagrange multiplier enforcing probability conservation vanishes exactly, reducing the constrained projection to an unconstrained empirical loss. The resulting objective has constant Hessian curvature mu I, non-saturating linear gradients, and an intrinsic dead-zone mechanism without heuristic clipping. Experiments on mathematical reasoning benchmarks show that GOPO achieves competitive generalization while maintaining stable gradient dynamics and entropy preservation in regimes where clipping-based methods plateau.

Figures & tables

Figure 1: Geometric interpretation of GOPO. The theory operates in L2(πk)L^{2}(\pi_{k}) . The reference policy πk\pi_{k} sits at the origin ( v=0v=0 ). Valid policies must reside in H0\mathcal{H}_{0} (gray plane) and satisfy non-negativity, restricting them to the feasible polytope K\mathcal{K} (green). The unconstrained target u∗=gα/μu^{*}=g_{\alpha}/\mu (red) is first projected vertically onto H0\mathcal{H}_{0} by subtracting the chemical potential λ∗\lambda^{*} , then truncated along the plane onto the K\mathcal{K} boundary (dead zone), yielding the final bounded GOPO update v∗v^{*} (blue).
Figure 3: GOPO flowchart. The core operation (green) implements the empirical orthogonal projection: the advantage signal provides the linear driving force, while the quadratic dissipation term provides the constant-curvature regularization.
AlgorithmMean Reward ↑\uparrowVal Acc (L4) ↑\uparrowGrad NormEntropy
GRPO0.54444%0.6740.115
DAPO0.54844%0.2130.126
GSPO0.55348%0.6230.128
OPO0.55848%1.2790.126
GOPO (Ours)0.55547%1.0290.134
Table 1: Comparison of policy optimization algorithms on MATH benchmarks. All methods use Qwen3-1.7B trained on ∼\sim 10% of MATH Level 3, validated on 100 MATH Level 4 problems. Reward is averaged over the entire training process.
Figure 4: Training dynamics comparison. (a) Training reward: OPO and GOPO achieve the highest mean rewards over the training process. (b) Validation accuracy on MATH Level 4: OPO and GSPO lead at 48%, with GOPO at 47% and monotonically improving. (c) Gradient norm: OPO/GOPO maintain healthy norms throughout, while DAPO exhibits severe gradient saturation. (d) Policy entropy: GOPO preserves the most diversity, preventing premature mode collapse.
More in this topic

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

Explore similar work

CardsList
May 12, 2026

Holder Policy Optimisation

Group Relative Policy Optimisation (GRPO) enhances large language models by estimating advantages across a group of sampled trajectories. However, mapping these trajectory-level advantages to policy updates requires aggregating token-level probabilities within each sequence. Relying on a fixed aggregation mechanism for this step fundamentally limits the algorithm's adaptability. Empirically, we observe a critical trade-off: certain fixed aggregations frequently suffer from training collapse, while others fail to yield satisfactory performance. To resolve this, we propose HölderPO, a generalised policy optimisation framework unifying token-level probability aggregation via the Hölder mean. By explicitly modulating the parameter pp, our framework provides continuous control over the trade-off between gradient concentration and variance bounds. Theoretically, we prove that a larger pp concentrates the gradient to amplify sparse learning signals, whereas a smaller pp strictly bounds gradient variance. Because no static configuration can universally resolve this concentration-stability trade-off, we instantiate the framework with a dynamic annealing algorithm that progressively schedules pp across the training lifecycle. Extensive evaluations demonstrate superior stability and convergence over existing baselines. Specifically, our approach achieves a state-of-the-art average accuracy of 54.9%54.9\% across multiple mathematical benchmarks, yielding a substantial 7.2%7.2\% relative gain over standard GRPO and secures an exceptional 93.8%93.8\% success rate on ALFWorld.

Yuxiang Chen, Dingli Liang, Yihang Chen +8

University College London, London, United Kingdom. · 3The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · 2Shanghai Jiao Tong University, Shanghai, China.

RL for Language Model ReasoningGroup Relative Policy OptimizationPolicy Optimization
AskPDF
Aug 18, 2026

GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models

Group Relative Policy Optimization (GRPO) has become a widely used approach for post-training Large Language Models (LLMs) for reasoning. In GRPO, the group gradients induced by different queries within the same mini-batch are directly averaged to form the policy update. However, these group gradients can point in conflicting directions. Our empirical analysis suggests that group-gradient conflicts tend to be associated with less effective policy updates, motivating the need for a reliable aggregated update direction under such conflicts. Standard GRPO aggregation treats the realized group gradients as deterministic contributions and does not account for differences in their reliability during aggregation. To address this issue, we propose Gradient Uncertainty-Aware Policy Optimization (GUPO), which models each group gradient as a random variable under a Bayesian formulation and estimates its probability distribution. GUPO then derives gradient uncertainty using a Dirichlet-based formulation and uses it to calibrate the contribution of each group gradient during aggregation. Extensive experiments on multiple benchmarks demonstrate the effectiveness of GUPO.

Peizheng Guo, Jianqi Zhang, Xingyu Zhang +4

Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences · National Key Laboratory of Space Integrated Information System +1

RL Post-TrainingGroup Relative Policy OptimizationGradient Interference
AskPDF
Feb 22, 2026

Soft Sequence Policy Optimization

A significant portion of recent research on Large Language Model (LLM) alignment focuses on developing new policy optimization methods based on Group Relative Policy Optimization (GRPO). Two prominent directions have emerged: (i) a shift toward sequence-level importance sampling weights that better align with the sequence-level rewards used in many tasks, and (ii) alternatives to the PPO-style clipping that aim to avoid the associated loss of training signal and entropy collapse. We introduce Soft Sequence Policy Optimization, an off-policy reinforcement learning objective that incorporates soft gating functions over token-level probability ratios within sequence-level importance weights. We provide theoretical motivation for SSPO and investigate practical modifications to improve optimization behavior. Empirically, we demonstrate that SSPO improves training stability and performance both in mathematical reasoning and coding tasks.

Svetlana Glazyrina, Maksim Kryzhanovskiy, Roman Ischenko

Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia

LLM AlignmentRL Post-TrainingRL for Language Model ReasoningOff-Policy RLImportance Sampling
AskPDF