Mirror Descent on Riemannian Manifolds

Jiaxin Jiang; Lei Shi; Jiyuan Tan

stat.MLMar 18, 2026

Mirror Descent on Riemannian Manifolds

Authors: Jiaxin Jiang, Lei Shi, Jiyuan Tan

Organizations: School of Mathematical Sciences, Fudan University, Shanghai 200433, China · Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai 200433, China

Abstract

Mirror Descent (MD) is a scalable first-order method widely used in large-scale optimization, with applications in image processing, policy optimization, and neural network training. This paper generalizes MD to optimization on Riemannian manifolds. In particular, we develop a Riemannian Mirror Descent (RMD) framework via reparameterization and further propose a stochastic variant of RMD. We also establish non-asymptotic convergence guarantees for both RMD and stochastic RMD. As an application to the Stiefel manifold, our RMD framework reduces to the Curvilinear Gradient Descent (CGD) method proposed in [26]. Moreover, when specializing the stochastic RMD framework to the Stiefel setting, we obtain a stochastic extension of CGD, which effectively addresses large-scale manifold optimization problems.

Figures & tables

Algorithm	$n$	1000	2000	5000
CGD	Time	1.41	7.10	64.67
CGD	Error	1.20e-06	9.28e-07	5.88e-06
SCGD	Time	0.72	4.56	38.10
SCGD	Error	7.08e-08	9.54e-07	8.09e-06

Table 1: Results of the two algorithms. When

K=1

the step size is

10^{-3}

. When

K>1

the step size is

10^{-2}

. We take

p=10

. The number of block

K

in Algorithm 2 is chosen to be

\lfloor n/300\rfloor

. The results are the average of 5 runs.

Algorithm	$n$	1000	2000	5000
CGD	Time	11.34	37.68	341.41
CGD	Error	7.55e-2	7.30e-2	1.13e-1
SCGD	Time	11.58	38.49	211.09
SCGD	Error	2.05e-2	4.41e-2	9.76e-2

Table 2: Results of the two algorithms. When

K=1

the step size is

10^{-3}

. When

K>1

the step size is

10^{-2}

. We take

p=10

. The number of block

K

in Algorithm 2 is chosen to be

\lfloor n/300\rfloor

Riemannian Optimization

Riemannian optimization develops algorithms for minimizing objectives on curved spaces while respecting geometric structure and constraints.

See all 76 papers

Inversion-Free Natural Gradient Descent on Riemannian Manifolds
Apr 3, 2026Dario Draca, Takuo Matsubara, Minh-Ngoc TranNatural Gradient Methods Riemannian Optimization
School of Mathematics and Statistics, University of Sydney, Australia · University of Sydney Business School, Australia
Riemannian Stochastic Optimization for Sufficient Dimension Reduction
May 29, 2026Thibault Pautrel, François PortierStochastic Optimization Riemannian Optimization
Laboratoire des Signaux et Systèmes (L2S), CentraleSupélec, Université Paris-Saclay, Gif-sur-Yvette, France · ENSAI, CREST, Bruz, France
Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning
Jul 28, 2026Yuan Zhang, Jiang Hu, Zhijian Lai +2Stiefel Manifold Optimization Riemannian Optimization
Center for Data Science, Peking University, Beijing 100871, China · Yau Mathematical Sciences Center, Tsinghua University, Beijing 100084, China · Beijing International Center for Mathematical Research, Peking University, Beijing 100871, China +2

Apr 3, 2026J/K move · Enter open · S save

Inversion-Free Natural Gradient Descent on Riemannian Manifolds

Dario Draca, Takuo Matsubara, Minh-Ngoc Tran

School of Mathematics and Statistics, University of Sydney, Australia · University of Sydney Business School, Australia

Natural Gradient Methods Riemannian Optimization

Open paper AskPDF arXiv

The natural gradient method is a central tool for statistical optimisation, but its broader application is hindered by the assumption of a Euclidean parameter space, the repeated estimation of the Fisher information matrix (FIM), and the computational cost of its subsequent inversion. This paper proposes an intrinsic, inversion-free natural gradient method for statistical models whose parameters lie on general Riemannian manifolds. Formulating statistical optimisation in this non-Euclidean setting allows for the natural enforcement of parameter constraints, the elimination of non-identifiable parameters, and the exploitation of geodesic convexity. Our algorithm is based on a moving approximation of the inverse FIM, which is maintained directly on the manifold. This approximation is efficiently updated with new score vectors using low-rank matrix identities. We prove almost-sure convergence rates of

O(\log s / s^α)

for the sequence of iterates, and a similar rate for the approximate FIM. A limited-memory variant with sub-quadratic storage complexity is further proposed for large-scale applications. We demonstrate the efficacy of our method on variational Bayes within the Bures-Wasserstein manifold, normalising flows on the Stiefel manifold, and reduced-rank logistic regression.

Abstract

Riemannian Optimization

Explore similar work

Inversion-Free Natural Gradient Descent on Riemannian Manifolds

Riemannian Stochastic Optimization for Sufficient Dimension Reduction

Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning