Orthogonality constraints are ubiquitous in robust and probabilistic machine learning. Unfortunately, current optimizers are computationally expensive and do not scale to problems with hundreds or thousands of constraints. One notable exception is the Landing algorithm (Ablin et al., 2024) which, however comes at the expense of temporarily relaxing orthogonality. In this work, we revisit and improve on the ideas behind Landing, enabling the inclusion of modern adaptive optimizers while ensuring that orthogonal constraints are effectively met. Remarkably, these improvements come at little to no cost, and reduce the number of required hyperparemeters. Our algorithm POGO is fast and GPU-friendly, consisting of only 5 matrix products, and in practice maintains orthogonality at all times. On several challenging benchmarks, POGO greatly outperforms recent optimizers and shows it can optimize problems with thousands of orthogonal matrices in minutes while alternatives would take hours. As such, POGO sets a milestone to finally exploit orthogonality constraints in ML at scale. A PyTorch implementation of POGO is publicly available at https://github.com/adrianjav/pogo.
Figures & tables
Figure 1: POGO optimizes thousands of orthogonal matrices orders of magnitude faster than retraction methods while achieving performance comparable to unconstrained optimizers , as shown in this CIFAR-10 ( Krizhevsky et al., 2009 ) classification problem with a tailored CNN ( Jordan, 2024 ) parameterized with orthogonal filters or kernels. While RSDM ( Han et al., 2025 ) takes 17 hours to train on average, POGO trains in 3 minutes .
Figure 2: Illustration of the landing algorithm, adapted from ( Ablin et al., 2024 ) . Landing combines two orthogonal gradients at each iteration and adapts the learning rate η to keep X1 within ε -distance from the Stiefel manifold.
Figure 3: Illustration of the POGO algorithm, see Section 3 . Computing the distance w.r.t. the intermediate point M , POGO can calculate the exact λ to stay within the manifold.
Figure 4: POGO reduces the optimality gap the fastest across all baselines while staying on the manifold . Results are averaged over 10 independent runs and the orthogonal matrices are of size 1500×2000 for PCA and 2000×2000 for Procrustes.
Figure 5: While all methods obtain similar test accuracies with an O-ViT ( Fei et al., 2022 ) on CIFAR-10 ( Krizhevsky et al., 2009 ) , POGO is the fastest method to complete 10 epochs without leaving the manifold . Results show average and 95% confidence intervals over 5 independent runs.
Figure 6: Normalized distance and test-time accuracy on the CNN experiment from Section 5.2 . We observe that in all instances POGO obtains similar accuracy to the unconstrained baseline while staying on the Stiefel manifold .
Figure 7: Evolution of the accuracy for the CNN experiment with orthogonal kernels. POGO can learn at the same pace as the unconstrained Adam baseline despite orthogonality constraints.
Figure 8: Training curves for validation bits-per-dimension and manifold distance for the squared unitary PC experiment. POGO obtains state-of-the-art results for this class of models, while converging the fastest and not leaving the Stiefel manifold.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure D.1 : Ablation on online PCA with a subset of methods exploring the trade-off between multiplication speed and numerical precision.
Department of Mathematics Linköping University, Sweden · Academy for Advanced Interdisciplinary Studies Peking University, Beijing, People’s Republic of China · Department of Industrial and Systems Engineering University of Minnesota, USA