Stiefel Manifold Optimization

Ask about this topic

Overview

The Stiefel manifold is the set of matrices with orthonormal columns, and optimization on it is useful when models or statistical methods require transformations that preserve lengths or orthogonality. Common approaches include Riemannian gradient and Adam-style updates, spectral methods such as Muon, tangent-space projection with polar retractions, and retraction-free corrections; derivative-free search is another option for black-box objectives. Applications include transformer attention and weight optimization, low-rank adaptation, and classical tasks such as principal component analysis and orthogonal Procrustes. A central tension is whether the geometric constraint improves results beyond its effects on scaling and preconditioning, while methods must balance reliable feasibility, adaptable spectra, and computational cost.

Latest papers

CardsList

13 papers0 papers in the last four weeks, down 100% on the four weeks before. 0.0% of all new papers.

  1. Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks

    Sep 17, 2026Emanuele Zangrando, Marco Sutti, Francesco TudiscoDeep Learning OptimizationRiemannian Optimization

    Gran Sasso Science Institute, L’Aquila, Italy · University of Edinburgh, Edinburgh, UK

  2. Directions That Don't Drift: Stiefel Manifold Routing for Transformer Attention

    Sep 16, 2026Rubén Darío GuerreroStiefel Manifold OptimizationRiemannian Optimization

    NeuroTechNet S.A.S., 1108831, Bogotá, Colombia

  3. Geometry-Aware Bayesian Parameter-Efficient Fine-Tuning on the Stiefel Manifold via Stein Variational Gradient Descent

    Sep 8, 2026Quang-Duy Tran, Trung Le, Bao Duong +2Stiefel Manifold OptimizationUncertainty Quantification

    Deakin Applied Artificial Intelligence Initiative, Deakin University, Geelong, Australia · Department of Computer Science, Aalto University, Espoo, Finland · Department of Data Science & AI, Monash University, Melbourne, Australia

  4. Muon on the Stiefel Manifold Admits an Exact Closed-Form Update

    Aug 6, 2026Mikhail Solonko, Molozhavenko Alexander, Maxim RakhubaStiefel Manifold OptimizationRiemannian Optimization

    HSE University, Moscow, Russia.

  5. Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning

    Jul 28, 2026Yuan Zhang, Jiang Hu, Zhijian Lai +2Stiefel Manifold OptimizationRiemannian Optimization

    Center for Data Science, Peking University, Beijing 100871, China · Yau Mathematical Sciences Center, Tsinghua University, Beijing 100084, China · Beijing International Center for Mathematical Research, Peking University, Beijing 100871, China +2

  6. ManifoldFlow: Spectrum-Learnable Stiefel Optimization via SPD Relaxation

    Jul 5, 2026Haiwen Yi, Xinyuan SongStiefel Manifold Optimization

    University of Toronto · Emory University

  7. Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization

    Jun 11, 2026Kirato YoshiharaStiefel Manifold OptimizationNeural Network Optimization

    School of Engineering Science, The University of Osaka, Osaka, Japan.

  8. Routing on the Stiefel Manifold: When Does Adaptive Subspace Selection Help for Cross-Domain EEG Decoding?

    May 29, 2026Isabella Costa Maia, Pedro L. C. Rodrigues, Salem Said +1Cross-Subject EEG DecodingStiefel Manifold Optimization

    GIPSA-lab, University Grenoble Alpes, CNRS, Grenoble-INP · Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK · Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK

  9. Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds

    May 10, 2026Yibang Li, Bihari Lal Pandey, Ravi Sah +4Stiefel Manifold OptimizationRiemannian Optimization

    Nanyang Technological University, Singapore. · Indian Institute of Technology Bombay, India. · University of Sydney, Australia. +1

  10. A second-order method landing on the Stiefel manifold via Newton\unicodex2013\unicode{x2013}Schulz iteration

    May 4, 2026Xinhui Xiong, Bin Gao, P. -A. AbsilNewton-Schulz IterationStiefel Manifold Optimization

    State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, and University of Chinese Academy of Sciences, China · State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, China · ICTEAM Institute, UCLouvain, Belgium

  11. BOOOM: Loss-Function-Agnostic Black-Box Optimization over Orthonormal Manifolds for Machine Learning and Statistical Inference

    Apr 23, 2026Beomchang Kim, Subhrajyoty Roy, Priyam DasStiefel Manifold OptimizationBlack-Box Optimization

    Department of Biostatistics, Virginia Commonwealth University, Richmond, VA, USA · Department of Statistics and Data Science, Washington University in St. Louis,[math.OC] St. Louis, MO, USA

  12. Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning

    Mar 10, 2026Yechen Zhang, Shuhao Xing, Junhao Huang +5Deep Learning OptimizationMuon Optimizer

    Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University

  13. Generalizing Adam to Manifolds for Efficiently Training Transformers

    May 26, 2023Benedikt BrantnerStiefel Manifold OptimizationRiemannian Optimization

    Max-Planck-Institut für Plasmaphysik Boltzmannstraße 2, 85748 Garching, Deutschland · Technische Universität München, Zentrum Mathematik Boltzmannstraße 3, 85748 Garching, Deutschland