Overview

Riemannian optimization studies how to minimize objectives when parameters lie on curved spaces such as spheres, Stiefel manifolds, and spaces of low-rank matrices. Common methods project gradients onto tangent spaces and take feasible steps with retractions or geodesics; natural-gradient, mirror-descent, and second-order methods adapt updates to the chosen metric. These techniques are used for orthogonality-constrained learning, low-rank model adaptation, covariance and probability optimization, tensor networks, and robotics. A central tension is balancing geometric fidelity and convergence guarantees against the cost of computing geometric updates, while empirical gains over well-tuned Euclidean optimizers are not consistent across settings.

Latest papers

85 papers8 papers in the last four weeks, down 50% on the four weeks before. 0.1% of all new papers.

Nov 7, 2026

Zeroth-Order Riemannian Optimization on Fixed-Rank Manifolds for LLM Fine-Tuning

Zeroth-order (ZO) optimization estimates descent directions from function values alone, which removes the activation storage required by backpropagation and makes it attractive for memory-efficient fine-tuning of large language models (LLMs). Existing low-rank ZO methods either parameterize the additive update through Euclidean factors with a non-unique representation, or restrict each finite-difference direction to be low-rank. In this work, we propose to constrain each additive update block to a fixed-rank matrix manifold while keeping the pretrained weights frozen, so the learned update remains rank-constrained throughout fine-tuning. On this geometry, we build a Riemannian ZO gradient estimator that projects a random matrix onto the tangent space, normalizes it to correct the scale mismatch induced by projection, and further propose Zeroth-order Riemannian Gradient Descent (ZO-RGD) and its smoothed variant, ZO-RGD-S. We show that the ambient finite difference matches its retracted counterpart, the normalized estimator recovers the Riemannian gradient in expectation, and ZO-RGD converges under trajectory-regularity conditions. Experiments on LLM fine-tuning tasks show that the proposed algorithms are competitive with strong forward-only baselines across model families and tasks, with gains in several settings.
Nov 6, 2026

Fixed-point acceleration on Neural Wasserstein manifolds

We propose a fixed-point acceleration method for optimization over probability measures, applicable to a variety of machine learning problems. We represent probability measures as pushforwards of a reference distribution through a neural network map and equip the parameter space with the induced pullback Wasserstein metric. This geometric structure allows natural gradient descent to be interpreted as a fixed-point iteration on a Riemannian manifold. We accelerate this iteration using Riemannian Anderson mixing method, which exploits historical residuals to capture approximate second-order information without forming Hessians. In the present parametric setting, the manifold is represented by Euclidean coordinates endowed with a point-dependent metric, simplifying retraction and vector transport. We evaluate the method on multimodal sampling, PDE-based inverse problems, aggregation dynamics, Bayesian logistic regression, and generative modeling. Across the tested tasks, Anderson mixing reduces the number of natural-gradient iterations and achieves competitive performance relative to SGD and Adam in terms of both loss-gradient evaluations and wall-clock time. These results suggest that fixed-point acceleration is an efficient tool for improving the computational efficiency of training on parametric Wasserstein methods.
Oct 31, 2026

Self-Normalized Inference from Riemannian Stochastic Gradient Paths

Riemannian stochastic gradient paths estimate constrained parameters, but confidence regions usually require estimating the Hessian and gradient-noise covariance. We develop self-normalized inference from one path by expressing retained iterates in a common random tangent space. Their average and centered partial sums share an unknown covariance factor that cancels in a Brownian bridge pivot. Under suitable regularity conditions, a joint expansion of the averaged estimator and integrated path controls nonlinear coordinate errors using conditional second moments. The procedure uses fixed-dimensional memory and extends to nondegenerate smooth functionals and alternative local coordinates. Numerical studies show competitive coverage relative to plug-in Wald inference while avoiding Hessian and residual-density estimation.
Oct 31, 2026

Riemannian Newton Methods with Statistical Inference

We develop online statistical inference for Polyak-Ruppert averages of inexact Newton iterates on Riemannian manifolds. A local asymptotic representation separates observation noise, solver randomization, and accumulated implementation bias, and yields one-pass covariance estimators and asymptotically valid Wald regions under local regularity and centering conditions. Independent fresh solves add a positive semidefinite term to the score-sandwich covariance; averaging solver replicas reduces this term in inverse proportion to their number. Slowly varying predictable gains instead cancel from the first order law. Product torus simulations show how Newton scaling shortens an ill-conditioned transient and how retaining solver variation restores coverage. Intrinsic means and principal subspaces provide geometric applications, with a streaming data illustration.
Oct 31, 2026

Bootstrap Inference for Riemannian Optimization Problems

We develop an online multiplier bootstrap for manifold-valued parameters estimated by averaging the iterates of Riemannian stochastic approximation. The method runs weighted iterations alongside the original algorithm, with each path using the same observations while updating its own state and intrinsic average. The weighted averages provide uncertainty assessments without a plug-in asymptotic covariance estimator. Under local stability and regularity conditions, we establish bootstrap consistency and show that a fixed number of paths suffices for asymptotically valid Student intervals and intrinsic confidence regions. The theory accommodates samplewise nonsmooth scores and identifies a critical oracle-bias regime in which the estimator's limiting distribution shifts, but the centered bootstrap distribution does not. Applications include spherical means, positive-definite matrix means and medians, and principal-subspace estimation. Simulations show coverage close to nominal in regular settings, and an analysis of North Atlantic storm locations illustrates descriptive online inference for a spherical center.
Oct 30, 2026

Riemannian Optimization on Unknown Manifolds via Stochastic Proximal Refinement

We study Riemannian optimization on an unknown manifold accessed only through feasible samples. The central challenge is to turn feasible samples into optimization steps, i.e., the samples are drawn from the unknown feasible manifold, but do not directly provide the tangent spaces, projections, or retractions required by classic Riemannian methods. Moreover, regions with low objective values may be sparsely represented in the data, so directly sampling from a distribution fitted to these observations can require many draws to find better feasible solutions. We introduce Proximal Refinement on Implicit Smooth Manifolds (PRISM), which turns feasible-sample access into an iterative optimization procedure by repeatedly sampling an objective-aware proximal distribution conditioned on the current iterate. Theoretically, we show that PRISM recovers the effect of Riemannian-gradient refinement, leading to provable objective descent, feasibility (manifold adherence), and convergence toward first-order stationarity. We demonstrate the effectiveness of our approach through synthetic experiments illustrating its geometric properties and applications to data-driven building control and diverse real world optimization tasks, highlighting its potential as a general framework for data-driven optimization across scientific and engineering domains.
Oct 24, 2026

Online Covariance Estimation for Averaged Stochastic Gradient Descent on Riemannian Manifolds

We study online inference for averaged Riemannian stochastic gradient descent (RSGD) when the unknown parameter lies on a manifold. Although the averaged RSGD iterate is asymptotically normal, practical inference requires estimating the unknown covariance operator in its limiting distribution. Existing online batch-means methods cannot be applied directly because a general Riemannian manifold lacks an addition operation, and the tangent space containing the Riemannian gradient varies across iterations. Through vector transport, we construct overlapping and non-overlapping covariance estimators that adaptively combine information from successive batches while accounting for the changing tangent spaces. The proposed estimators are online, require neither Riemannian Hessian computations nor a prespecified terminal sample size, and attain the same convergence rates as their Euclidean counterparts (Zhu et al., 2023). Together with asymptotic normality, these estimators enable online statistical inference for RSGD on manifolds. We illustrate the proposed framework through a detailed study of streaming principal component analysis on Grassmann manifolds.
Oct 19, 2026

Corrected Block Sparse Spectral Optimization: Exact Models, Finite Identification, and Local Dynamics

The block generalized-power method pairs threshold-based support selection with inexpensive Stiefel updates, but the standard block formulation conflates two nonequivalent objectives and leaves inactive components, threshold ties, and post-selection refitting ill-specified. We rebuild the model from the primal. Eliminating each loading column from the normalized loading-ball problem gives a sum of soft-thresholded Euclidean norms (A-SUM); the squared-threshold objective in common use arises from a quadratic lift (A-SQ). We prove both identities, exhibit a two-component instance whose maximizers disagree, and specify explicit conventions for zero-response columns, ties, rank deficiency, and mask-before-refit. For A-SQ we separate the historical full-polar iteration from a safeguarded Riemannian ascent and prove, for the latter, whole-sequence convergence via a sufficient-ascent lemma, a relative-error lemma, and the Kurdyka-Lojasiewicz property, which applies because the soft-threshold objective and the Stiefel constraint are semi-algebraic. A strict threshold margin then forces finite identification of the support and sign pattern. The post-identification map is ordinary power iteration on the active covariance for a rank-one hard threshold, and the Frechet derivative of a coupled polar map for component-specific supports; a radius-based safe-screening rule follows from the same local analysis. Planted-spike experiments match the predicted identification time and local rates. On modern-AI panels (GPT-2 residual stream, ViT CLS stream, Pitprops) the thresholded block-sparse components match dense PCA on reconstruction and improve on it in monosemanticity and class specialisation. On three native-label scRNA datasets (Zheng17 FACS-sorted PBMCs, 68k donor-A with FACS-correlated labels, Tabula Muris) A-SQ beats dense PCA on cluster-recovery AMI by +0.023 to +0.044 at matched sparsity.
Oct 6, 2026

A Riemannian Geometry for Low-rank Adaptation

Low-rank adaptation (LoRA) is widely used as a parameter-efficient fine-tuning technique for pre-trained deep neural networks, which approximates the weight update via full fine-tuning by a low-rank matrix BA⊤BA^\top. This parameterization leads to the equivalence relation (B,A)∼(BG−1,AG⊤)(B, A) \sim (BG^{-1}, AG^\top) for any invertible matrix GG because BA⊤=BG−1(AG⊤)⊤BA^\top = BG^{-1}(AG^\top)^\top and thus both pairs yield the same loss value. This relation induces a quotient manifold where matrices (BG−1,AG⊤)(BG^{-1}, AG^\top) for all GG are identified, eliminating redundant directions along which the loss value remains unchanged. To respect the geometry of this manifold, the original search space is endowed with a Riemannian metric that is invariant under the equivalence relation. Such a metric induces preconditioning at each gradient step and ensures that each weight update via LoRA changes the loss value, leading to efficient optimization. In this paper, we propose a new Riemannian metric that is specifically tailored to LoRA to close the gap to full fine-tuning at the weight level. We theoretically show that LoRA with our preconditioning induced by this metric satisfies the following two properties at each iteration: (i) The weight update follows the direction closest to the gradient of full fine-tuning within the subspace of first-order weight changes allowed by the LoRA parameterization. (ii) The updated weight matrix is closer in Frobenius norm to that of full fine-tuning than the updated weight matrices of LoRA with conventional preconditioning and without preconditioning. These theoretical insights suggest that our preconditioning makes LoRA better approximate full fine-tuning, thereby leading to more efficient optimization. Experiments show the effectiveness and efficiency of our preconditioning for LoRA on fine-tuning tasks with language and vision domains.
Oct 5, 2026

How Inefficient Is Natural Gradient Descent? From Exact Optimality to Θ( \sqrt{ \log d } ) Divergence

Natural gradient descent (NGD) underlies common methods in ML. For dually flat families, idealized NGD on the forward Kullback--Leibler objective follows the mixture geodesic which is often longer than the shortest Fisher--Rao path. We quantify this overhead by the inefficiency ratio R≥1R \ge 1, the Fisher length of the mixture geodesic divided by the Fisher--Rao distance, and bound its supremum over endpoint pairs as a function of the parameter dimension dd. A tensor criterion identifies the regime (I) families, with R=1R=1 everywhere: exactly those with quadratic potential or dimension one, such as fixed-covariance Gaussians. For non-quadratic families, we prove two further regimes: (II) bounded third-order skewness plus finite Fisher--Rao diameter yields a dimension-independent bound; and (III) for products of scale families---including Gaussian covariances and Gamma rates---RR grows as Θ(log⁡d)Θ(\sqrt{\log d}), unbounded in dd. Under a per-step Fisher-chord budget, RR translates to a practical computational cost: NGD requires asymptotically at least RR times as many steps as an optimizer following the Fisher--Rao geodesic. Experiments confirm all three regimes: R=1R=1 to machine precision for quadratic-potential families (I), the categorical bound π/(22)π/(2\sqrt{2}) is approached but not attained (II), and sampled scale-product RR grows with dd, reaching R≈1.5R \approx 1.5 for long, high-dimensional moves (III).
Oct 5, 2026

The Implicit Bias of Hyperbolic Representation Learning for Multiclass Data: A Busemann Risk Perspective

We study the implicit bias of Riemannian gradient flow for hyperbolic multiclass classification with fixed class prototypes in hyperbolic space Hn\mathbb H^n. Our framework accommodates general permutation invariant relative margin (PERM) losses, a class that includes cross entropy and other standard multiclass losses. Our analysis is based on a decomposition: at large radius, the distance to each prototype splits into a radial term and a direction dependent term described by the Busemann function. This yields two main results. First, we prove a radial dichotomy: the sign of a drift coefficient μμ determines whether the radius is pushed toward the ideal boundary or back toward the interior; if the positive drift persists, then r(t)=12log⁡t+O(1)r(t)=\tfrac12\log t+O(1), while persistent negative drift returns the trajectory to the large radius threshold in finite time. Second, we show that large-radius directions can approach critical points of the Busemann risk on ∂Hn\partial\mathbb H^n, even during radial return. These results provide a rigorous large-radius perspective on two phenomena we refer to as boundary saturation and near-boundary clustering in hyperbolic representation learning.
Oct 5, 2026

Hyperbolic Graph Representation Learning: Embed in One Metric, Optimize with Another

Hierarchical graphs embed in hyperbolic space with lower distortion than in Euclidean space owing to its negative curvature. However, their gradient-based learning is hampered at large radii, where the Poincaré ball and the Lorentz hyperboloid models fail numerically. Polar coordinates avoid this problem, but the hyperbolic metric scales the angular step by the hyperbolic sine of the radius, freezing angular motion. We observe that this factor is a choice, silently fixed by existing implementations: the Euclidean tangent parametrization, for instance, uses the radius itself. We show that other choices are not only possible but preferable. They are endpoints of a one-parameter family of optimization preconditioners with curvatures from −1-1 to 00, while the embedding remains at curvature −1-1. We show that since the Euclidean preconditioner rearranges a layout but refines it poorly, while an intermediate one refines far better once a layout is in place, combining them in two stages reduces the loss on real-world trees by 46-74% over the best single curvature.
Oct 4, 2026

Hamiltonian Metric Learning and Energy-Based Training: A Dissipative Geometric Framework for Optimization

Optimization in machine learning is usually expressed through iterative rules that update model parameters using information from the loss landscape. In this work, we study an alternative viewpoint in which parameter optimization is treated as the evolution of a dissipative dynamical system. The model parameters are regarded as generalized coordinates, the loss acts as a potential energy, and a positive-definite metric defines the local kinetic geometry of parameter space. Starting from a variational formulation, we derive the corresponding Hamiltonian dynamics, the geometric force generated by a position-dependent metric, and a metric-compatible Rayleigh dissipation law. The resulting continuous system satisfies a monotonic energy-dissipation relation, while its discrete form allows the influence of curvature on the optimization trajectory to be studied directly. We illustrate the framework using a controlled CIFAR-10 image-reconstruction problem for which the optimum is known analytically. With the image Hessian used as the metric, the curvature dependence of the quadratic modal dynamics is removed. We then reparameterize the same image as a matrix product state, producing a genuinely position-dependent metric and a nonzero geometric force. Poincaré return maps provide a complementary phase-space view of the resulting contraction dynamics. These examples establish HAMLET as a geometric, energy-based framework for studying optimization as dissipative motion in parameter space. On a five-seed MNIST MLP benchmark, HAMLET attains the highest mean test accuracy (97.95%) and the lowest mean test negative log-likelihood (0.0696) among the three evaluated optimizers.
Sep 28, 2026

Riemannian Difference-of-Convex Optimization for K-Means Clustering

K-means is a widely adopted clustering approach in signal processing and machine learning. In this paper, we study K-means clustering through a cardinality-constrained formulation on a compact embedded submanifold. We replace the cardinality constraint with a difference-of-convex (DC) penalty and establish a global error bound to prove that the penalized and constrained formulations share the same global minimizers whenever the penalty parameter exceeds a finite threshold. To solve the resulting nonsmooth Riemannian DC problem, we reformulate it as a minimax problem and propose RADA-DC, a Riemannian alternating descent ascent method combining dual regularization with DC linearization. Under standard assumptions and suitable parameter choices, RADA-DC finds an εε-Riemannian critical point within O(ε−3)O(ε^{-3}) iterations. We conduct experiments on synthetic and real-world datasets to demonstrate that the proposed method outperforms the tested baselines, including K-means++, in solution quality at competitive computational cost when the number of clusters is large.
Sep 28, 2026

Rotated Manifold Optimization for Low-Rank Adaptation

We propose a novel optimizer for low-rank adaptation (LoRA) that explicitly incorporates the gauge symmetry of low-rank factorization. Our optimizer extends recent matrix optimizers for full-parameter training to the manifold of fixed-rank matrices by interpreting them as normalization under a rotated basis. We show how rotation and normalization can be integrated with the fixed-rank manifold efficiently. Our optimizer converges faster to lower held-out loss and achieves better or comparable downstream performance on both supervised finetuning and reinforcement learning tasks.
Sep 25, 2026

Retraction-Based Gradient Projection Algorithms on Manifolds

We introduce a framework for retraction-based convex optimization on Riemannian manifolds, which includes a notion of retraction-specific convex sets and retraction-based gradient projection algorithms. The standard theory of gradient projection algorithms generalizes easily to this framework. Within this framework, we establish convergence results for retraction-based gradient projection algorithms with various stepsize rules. As an application, we use our framework to study the weighted low-rank approximation. We also provide numerical validation of our convergence results on the image completion task.
Sep 23, 2026

Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices

In this paper, we are concerned with matrices formed by block-diagonal factors interleaved with fixed permutations -- a flexible family of structured matrices. This class has recently drawn interest in deep learning architectures for its balanced expressivity-efficiency trade-off, yet efficient computational strategies for working with it remain to be found. We approach this problem through Riemannian geometry and examine under what conditions this class admits a smooth manifold structure. For the practically important case of orthogonal two-factor matrices, we derive the essential Riemannian tools and propose efficient algorithms for their implementation. The algorithms leverage automatic differentiation, support parameter sharing within each factor, and avoid explicit dense matrix construction. We test them within the Riemannian optimization framework on the best matrix approximation problem and for parameter-efficient fine-tuning of large language models. Beyond the two-factor setting, we study the geometric and matrix-theoretic properties of factorizations with a larger number of block-diagonal factors.
Sep 21, 2026

Guaranteed Low-Rank Tensor Recovery from Modewise Measurements via Normalized Block-Weighted Riemannian Gradient Descent

We consider the recovery of low-multilinear-rank tensors from linear measurements and propose an adaptive block-weighted modewise Riemannian gradient descent method. The method combines memory-efficient modewise measurements with a normalized adaptive weighting strategy for the core and factor components of the Riemannian gradient. The weighting improves convergence without increasing the multilinear-rank bound of the search direction or the size of the reduced core used for retraction. Under the tensor restricted isometry property and a suitable initialization, we establish local linear convergence and derive sampling guarantees for sub-Gaussian and subsampled orthogonal with random sign (SORS) measurements. Numerical experiments on synthetic low-Tucker-rank tensors show that the proposed method reduces iteration counts and computational time while maintaining reliable recovery performance, especially near the recovery threshold and for structured SORS measurements.
Sep 18, 2026

Locally Private Inference for Riemannian Stochastic Optimization

We develop inference for manifold-valued population minimizers when each observation belongs to a different participant and only locally private messages reach the analyst. The method releases randomized tangent gradients and combines them through Riemannian stochastic approximation and Polyak-Ruppert averaging. Directly inserting a private data surrogate into a nonlinear loss can shift its population target, whereas conditional centring of the released gradient preserves the first-order equation. We introduce symmetric-pair regression (SPR) to estimate the asymptotic variance from the same private messages used for point estimation, without holding out participants or requesting a second release. We prove the central limit theorem and consistency of the fully transcript-based sandwich covariance and intrinsic Wald region under local differential privacy. Simulations across various statistical problems and manifolds support the predicted decrease in estimation error and near-nominal coverage under moderate privacy. An application to NHANES anthropometric data illustrates private estimation of a leading body-size direction and its uncertainty.
Sep 17, 2026

Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks

A pervasive structural pattern in modern deep learning is the linear factorization block: a submodule of the form W=BAW = BA in which two parameter matrices are multiplied directly, with no intervening nonlinearity. Such blocks appear in LoRA adapters, low-rank compressed layers, query-key products of self-attention, and share a common pathology: the factorization is non-unique, which can destabilize training and limit usable learning rates. Despite this, factorization blocks are typically optimized with standard Euclidean methods that ignore the underlying geometry. We introduce Stiefel-AdamW, a near drop-in replacement for AdamW for use wherever such blocks appear. By constraining one factor on the Stiefel manifold while leaving the other Euclidean, Stiefel-AdamW relaxes the full GL(Rr)\mathrm{GL}(\mathbb{R}^r) gauge symmetry to a compact orthogonal symmetry, ruling out factor blow-up while retaining the coordinate-wise diagonal preconditioning that gives AdamW its practical strength. Moment estimation is performed in the ambient Euclidean space, with geometry entering only through a tangent-space projection and a manifold retraction. The implementation overhead over AdamW is minimal, and we show that the resulting optimizer inherits both the stability benefits of Riemannian methods and standard convergence guarantees. We validate Stiefel-AdamW on LoRA-style fine-tuning of GPT2, ViT, and Mistral 7B and on full pretraining of GPT2 on OpenWebText, showing consistent improvements over strong baselines at essentially no additional cost over AdamW.
Sep 16, 2026

Directions That Don't Drift: Stiefel Manifold Routing for Transformer Attention

The query and key projections \WQ,\WK\WQ,\WK in attention are almost always trained by Euclidean optimizers with no geometric constraint. We constrain them to the Stiefel manifold and optimize with a Riemannian Adam carrying one scalar second moment per frame---the form of \citet{becigneul2019}, here extended to the compact, non-Hadamard \St(d,r)\St(d,r) with a tangent projector, step-norm cap, and polar retraction. Four propositions prove steepest descent in the embedded metric, gradient-scale independence, well-conditioning, and exact O(d)\mathrm{O}(d)-equivariance. A fifth records that weight decay has identically zero Riemannian gradient on \St(d,r)\St(d,r) (W=WIrW{=}WI_r lies in the normal space), so decay cannot act on the constrained frames. On a CIFAR-10 patch benchmark at n=10kn{=}10\mathrm{k} this rule gains +6.79\mathbf{+6.79},pp over AdamW across 12 paired starts (t=38.33t{=}38.33, 12/1212/12); earlier fixed-step Riemannian SGD gains +1.97+1.97,pp, of which +1.69+1.69,pp comes from frozen orthonormal initialization alone. The corrected Adam's lead grows with data: +1.9+1.9,pp at n=1kn{=}1\mathrm{k} to +6.7+6.7,pp at n=50kn{=}50\mathrm{k}. A 12-seed ablation credits all gain to the scale-free step (+4.63+4.63,pp, 12/1212/12), nothing to the projector or equivariance; a targeted ε\varepsilon-sweep causally confirms the mechanism (−2.6-2.6,pp at ε=0.1\varepsilon{=}0.1, p<0.001p{<}0.001). Two five-seed grokking studies confirm the constrained arm does not grok better than the baseline (p=0.019p{=}0.019, A2 wins): the weight-decay exemption has no grokking consequence. A single-seed pilot exploiting this localization achieves the first stable grokking under slingshot conditions---Stiefel + targeted circuit regularization keeps routing-frame isometry error 106×10^6\times lower than the unconstrained ablation through every collapse.
Sep 15, 2026

Optimization over covariance matrices with a parameterized metric

The choice of Riemannian metric can strongly influence the convergence of gradient-based optimization over covariance matrices. Euclidean, Bures-Wasserstein and affine-invariant metrics are common choices, but their relative effectiveness depends on the objective. We introduce a two-parameter family defined by XpLXq+XqLXp=UX^{p}LX^{q}+X^{q}LX^{p}=U, solved for LL at each tangent vector UU, that contains all three as exact members, at (0,0)(0,0), (1,0)(1,0) and (1,1)(1,1), and extends past them. We treat the choice of member as a particular way of preconditioning for a given problem. To this end, we analyze the conditioning of the Riemannian Hessian at the solution. We show that it obeys a lower bound that depends on (p,q)(p,q) only through the exponent r=p+qr=p+q. When the Euclidean Hessian is a pure power that mixes no eigendirections, the member p=q=r/2p=q=r/2 attains that bound, and a closed-form criterion identifies the other members that do. We discuss ways to tune rr for a given problem. Experiments on real covariance data confirm the predicted conditioning and the benefit of tuning rr. A task covariance example shows a further gain from tuning the shape.
Sep 15, 2026

Near-Optimal Nonconvex Matrix Completion

We study nonconvex methods for matrix completion, the problem of recovering a low-rank matrix from a subset of its entries. Convex methods achieve sample complexity linear in the matrix dimension and the rank, up to logarithmic factors, whereas global guarantees for commonly used nonconvex methods require a higher polynomial dependence on the rank. We close this gap by analyzing Riemannian gradient descent (RGD) and Riemannian Gauss--Newton (RGN) methods. For an n×nn\times n matrix of rank rr with incoherence parameter μμ and condition number κκ, the two methods achieve exact recovery with high probability from O(μnrlog⁡nlog⁡(nκ))O(μnr\log n\log(nκ)) and O(μnrlog⁡nlog⁡(2μrκ))O(μnr\log n\log(2μrκ)) observations, respectively. The methods use a multiscale residual initialization, while the analysis simultaneously controls the spectral error and incoherence. The resulting RGD iterates converge linearly, whereas RGN eventually converges Q-quadratically.
Sep 14, 2026

Information-Induced Training Geometry: Exact Reduction, Canonical Completion, and Structured Expressivity

Training data constrains optimizer geometry through the covectors visible to a declared information channel. We study how such partial information determines a full positive cometric relative to a reference and which degrees of freedom remain unidentified. Our central result resolves full-column-rank positive-definite compression under affine-invariant Riemannian geometry. The compression map is a split-Hadamard metric submetry and admits an explicit unique completion that is the affine-invariant nearest full geometry realizing a visible target and yields exact full-to-visible variational reduction. When the channel moves, the completions form a gauge-invariant rank stratification of the positive-definite cone. Its closed-form pullback pair metric separates visible-metric motion from subspace rotation through a reference-mismatch weight, yields an explicit positive-semidefinite multi-direction Gram matrix, and exposes the precise singularity of reference-valued modes. The mechanism is explained by a metric theorem equating ball submetry, attained fiber distance, and lossless reduction of every monotone radial visible decision problem. A smooth split-Hadamard theorem supplies coherent information sheets, proximal commutation, and solution-wise gradient-flow lifting. The positive-definite realization also gives closed-form prior-data shrinkage. Diagonal and block optimizer families reduce to relative-interior conic image tests with valid facial certificates, while deterministic and finite-sample bounds quantify recovery of the visible geometry and its subspace. Together these results characterize exact reduction, reference-dependent completion, and structured expressivity for the stated finite-dimensional affine-invariant model.
Sep 12, 2026

Riemannian ascent--descent for nonconvex nonconcave minimax landscapes: convergence to basin saddle points and applications to distributionally robust optimization

We study a class of distributionally robust optimization (DRO) problems for the statistical risk problem, formulated as minimax problems over the product of a Euclidean space and a Riemannian manifold. Because the resulting minimax landscape is nonconvex nonconcave in general, no globally convergent first order method is known to be available. We instead introduce the notion of a basin saddle point, a Nash equilibrium defined locally on the Cartesian product of a δδ basin around a connected component of the local minima critical set and a geodesic ball on the measure manifold. We develop an abstract convergence framework for a Riemannian gradient ascent multistep descent iteration to a basin saddle point under a local Łojasiewicz type growth condition, with exponent β∈(1,2]β\in (1,2], in the δδ basin around connected components of the local minima critical sets. Under Lipschitz regularity of critical sets we establish linear convergence for β=2β= 2 and polynomial convergence for β∈(1,2)β\in (1,2) to a basin saddle point, with explicit dependence on the sectional curvature of the manifold. We then instantiate this framework for the statistical risk DRO problem over Gaussian measures, where the ambiguity set is naturally modeled as the product of Euclidean space and the Bures Wasserstein manifold of covariance matrices, which we relax to a penalized DRO formulation. We derive nonasymptotic Hessian estimates for the resulting Lagrangian, establish existence and local uniqueness of its maximizer, and prove that an alternating Riemannian gradient scheme converges to a basin saddle point of the penalized DRO problem, recovering the linear and polynomial rates of the abstract theory with all constants explicit in terms of data dimension, loss moments, and the reference covariance.
Sep 12, 2026

Curvature-Independent Regret Bounds for Distributed Online Optimization on Hadamard Manifolds

This work addresses decentralized online Riemannian optimization on Hadamard manifolds. Prior work under geodesic convexity (g-convexity) may require curvature information in the optimization analysis, typically through a finite lower bound on the sectional curvature. Curvature may also enter the step size or contraction factor of tangent-space Riemannian consensus schemes. In this work, we relax the curvature dependence for a narrower class of horospherical convex (h-convex) functions. We study Distributed Riemannian Online Gradient Descent (D-ROGD), which combines local Riemannian h-subgradient updates with an implicit Fréchet-mean consensus. For h-convex and strongly h-convex local objectives, we establish O(T)O(\sqrt{T}) and O(log⁡T)O(\log T) static regret, respectively, matching the corresponding Euclidean rates with respect to TT, with network dependence governed solely by the spectral gap. To our knowledge, these are the first curvature-independent regret guarantees for decentralized online optimization on Hadamard manifolds. Experiments on hyperbolic embeddings corroborate the predicted rates, with no observable degradation due to curvature.
Sep 7, 2026

Riemannian Optimization for Multi-Player Quantum Games on Product Unitary Manifolds

Quantum game theory is an extension of classical game theory that uses quantum principles in game theory. The Eisert-Wilkens-Lewenstein (EWL) quantum game is an early example of the two-player classical Prisoner's Dilemma transformed into a quantum Prisoner's Dilemma. In the EWL game, the players choose pure quantum strategies represented by unitary matrices. This extension can resolve the classical dilemma by enabling cooperative equilibrium with higher payoff. In this paper, we first discuss the Extended EWL (EEWL) for multiplayer quantum games with mixed strategies. In EEWL, each player controls a set of unitary operators as quantum actions and uses a classical mixed strategy over these actions. The payoffs are defined as expectation values of Hermitian reward operators acting on a shared quantum state, which is generated and measured according to the EEWL protocol. We then propose the Unitary Strategy Matrix Exponential Algorithm (USMEA), a geometry-aware sequential algorithm for the EEWL mixed-strategy setting, in which each player jointly learns a trainable set of local unitary actions and the associated classical mixing probabilities. Thereby it acts as a learning-and-control layer for multi-agent quantum decision systems. We analyze the convergence properties of USMEA under standard smoothness and step-size conditions and validate the theory with numerical experiments. These results show how classical optimization methods can be systematically integrated into the design and analysis of engineered quantum strategic interactions.
Sep 3, 2026

Projected Riemannian Gradient Descent for the Bures-Wasserstein Barycenter: Dimension-Independent Linear Convergence at Unit Step Size

The computation of the Bures-Wasserstein (BW) barycenter of an ensemble of positive definite matrices arises throughout machine learning, optimal transport, and quantum information. Riemannian gradient descent (RGD) at unit step size -- the fixed-point iteration used in practice -- converges rapidly, yet existing analyses present a dichotomy: unit-step guarantees carry worst-case exponential dependence on the dimension, while dimension-independent guarantees require small step sizes that forfeit the empirical speed. We resolve this dichotomy, not by improving the guarantees for unit-step RGD, but by proposing a Projected RGD algorithm that achieves dimension-independent linear convergence at unit step size. The achieved rate, (1−κ−3/2)(1 - κ^{-3/2}), where κκ is the condition number of the ensemble, also polynomially improves on the best small-step guarantee (κ3/2κ^{3/2} versus κ5/2κ^{5/2} iteration complexity). The crux is a novel Projection Lemma: clipping the eigenvalues of a positive matrix to an interval [α,β][α, β] is the closed-form, non-expansive (1-Lipschitz) BW-metric projection onto the set {S:αI≤S≤βI}\{S : αI \leq S \leq βI\} -- a statement which, unlike its known one-sided counterpart, does not follow from convexity. The projection is moreover free: it reuses an eigendecomposition the next iteration must perform in any case, so the projected and unprojected iterations cost the same per step. The same analysis covers the invariant matrix projection problem of Brahmachari et al. (2025), whose fixed-point algorithm we identify as unit-step RGD on a totally geodesic submanifold, thereby extending the dimension-independent guarantee to that setting verbatim.
Sep 2, 2026

LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates

Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization. The step avoids expensive operations on full weight matrices, and its retraction is up to 2.8×2.8\times cheaper than the truncated-SVD retraction used by prior manifold methods. We prove that the Frobenius-norm version of our surrogate recovers LoRA-Pro, and we identify the tangent-projected gradient, the Riemannian gradient of the manifold, as the stationarity measure natural to LoRA training and computable from the factor gradients alone. Under this measure we give the first global convergence guarantees for both LoRA-Pro and LoRA-TSD, with rates that drive the factor-gradient norms to zero. Across six commonsense and natural-language-inference benchmarks with Llama-3.2-1B, Llama-3.1-8B and Qwen3-32B, LoRA-TSD outperforms every competing LoRA optimizer and stays robust to the adapter rank. Code is available at https://github.com/brain-lab-research/LoRA-TSD.
Sep 1, 2026

Stochastic Optimization of Tree Tensor Networks

Tensor networks, originally developed for quantum many-body physics, are promising models for machine learning. We derive stochastic Riemannian optimizers for tree tensor networks (TTNs) on both their parameter and quotient manifolds, including adaptive and learning-rate-free schemes suitable for minibatch training. Using a hybrid CNN-TTN architecture, we evaluate the methods on Fashion-MNIST, CIFAR10, and Imagenette. The proposed optimizers achieve predictive performance comparable to unconstrained optimization while enabling numerically stable downstream compression.