cs.LGJul 5, 2026

ManifoldFlow: SPD-Relaxed Stiefel Layers with Learnable Singular Spectrum

Authors: Haiwen Yi, Xinyuan Song

Organizations: University of Toronto · Emory University

Abstract

Orthogonal and Stiefel layers give neural weights exact spectral control, but they also impose a strong modeling constraint: all represented singular values are fixed at one. Many settings that benefit from an orthonormal basis still need direction-dependent attenuation or amplification. We introduce ManifoldFlow, a minimal relaxation of a fixed-spectrum Stiefel layer that keeps the basis on the Stiefel manifold while learning a bounded positive spectrum through W = Q S^{1/2}, with Q^T Q = I and S positive definite. Since W^T W = S, the eigenvalues of S are exactly the squared singular values of the realized weight, making eigenvalue clipping a direct singular-value control mechanism. Across paired sequence, tabular, and image experiments, the learnable SPD spectrum improves the fixed-spectrum Stiefel counterpart in the reported settings where the Stiefel prior is useful, with the largest gains in recurrent language-model projections. Boundary cases in convolutional classifier heads clarify the intended scope: ManifoldFlow is not a universal dense-layer replacement, but a spectrum-learnable Stiefel relaxation for settings where an orthonormal basis is a useful prior. When the basis should be orthonormal, its spectrum need not be frozen. Code available at https://github.com/Hik289/manifold_flow

Explore similar work

CardsList
Jul 28, 2026cs.LG

Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning

Optimization over the Stiefel manifold plays a significant role in various machine learning tasks. Existing methods either use the retraction operators, requiring costly orthonormalization for large-scale matrices, or employ landing methods that rely on careful step size selection and penalty parameter tuning. To address these challenges, we propose a retraction-free and penalty parameter-free algorithm that directly lands on the manifold. By leveraging the strongly-convex-like property of the quadratic penalty function and the proximal smoothness of the Stiefel manifold, we establish global convergence guarantees with the best-known iteration complexities under both constant and diminishing step sizes. Then, we reformulate the low-rank adaptation (LoRA) fine-tuning problem for large language models as a manifold optimization problem, introducing Manifold-LoRA for geometry-accelerated adaptation. This approach employs the proposed landing technique and a carefully designed step size strategy to accelerate the training process. Numerical experiments on benchmark datasets demonstrate the efficiency and strong downstream performance of the proposed method.
Yuan Zhang, Jiang Hu, Zhijian Lai +2
Riemannian OptimizationManifoldsQuadratic Slack Penalty
AskPDF
May 4, 2026math.OC

A second-order method landing on the Stiefel manifold via Newton\unicode{x2013}Schulz iteration

Retraction-free approaches offer attractive low-cost alternatives to Riemannian methods on the Stiefel manifold, but they are often first-order, which may limit the efficiency under high-accuracy requirements. To this end, we propose a second-order method landing on the Stiefel manifold without invoking retractions, which is proved to enjoy local quadratic (or superlinear for its inexact variant) convergence. The update consists of the sum of (i) a component tangent to the level set of the constraint-defining function that aims to reduce the objective and (ii) a component normal to the same level set that reduces the infeasibility. Specifically, we construct the normal component via Newton\unicodex2013\unicode{x2013}Schulz, a fixed-point iteration for orthogonalization. Moreover, we establish a geometric connection between the Newton\unicodex2013\unicode{x2013}Schulz iteration and Stiefel manifolds, in which Newton\unicodex2013\unicode{x2013}Schulz moves along the normal space. For the tangent component, we formulate a modified Newton equation that incorporates Newton\unicodex2013\unicode{x2013}Schulz. Numerical experiments on the orthogonal Procrustes problem, principal component analysis, and real-data independent component analysis illustrate that the proposed method performs better than the existing methods.
Xinhui Xiong, Bin Gao, P. -A. Absil
NewtonRiemannian ManifoldsManifoldsIndependent Component Analysis
AskPDF
Aug 6, 2026math.OC

Muon on the Stiefel Manifold Admits an Exact Closed-Form Update

We study Muon, a recently proposed matrix-aware optimization method, in the context of the Stiefel manifold. This manifold consists of matrices with orthonormal columns and is ubiquitous in machine learning and scientific computing. Existing extensions of Muon to this manifold rely on heuristic, approximate, or iterative updates with varying computational efficiency. We show that the corresponding Stiefel Muon update admits an exact closed-form solution and use this result to develop Skewon, a practical algorithm for orthogonality-constrained optimization with an efficient implementation. We further establish first-order convergence guarantees for Skewon in the smooth non-convex setting.
Mikhail Solonko, Molozhavenko Alexander, Maxim Rakhuba
MuonManifoldsLabel Skew
AskPDF