Riemannian Lyapunov Framework: Optimization as Closed Loop Control on Riemannian Manifolds

Yixuan Wang; Omkar Sudhir Patil; Warren E. Dixon

cs.LGJan 29, 2026

Riemannian Lyapunov Framework: Optimization as Closed Loop Control on Riemannian Manifolds

Authors: Yixuan Wang, Omkar Sudhir Patil, Warren E. Dixon

Organizations: Department of Mechanical and Aerospace Engineering University of Florida

Abstract

We introduce the Riemannian Lyapunov Framework (RLF), a unified framework that represents first order optimization as a closed loop controlled dynamical system on a Riemannian parameter manifold. Within this framework, optimizers are specified by a quadruple $(M, g, Φ, η)$ consisting of a Riemannian metric, an internal state transition map, a direction field generator, and a lifting parameter. Classical and modern methods including SGD, Adam, AdamW, Lion and Shampoo are recovered as specific instantiations of this quadruple. The central structural object of the framework is a time indexed family of target submanifolds in the extended state space of parameters, velocities, and internal memory, which we call the Normally Attracting Invariant Manifold (NAIM) family. These submanifolds organize training dynamics into two timescales: a fast contraction of the velocity state onto the current target submanifold, followed by a slow descent along it. Using Riemannian backstepping, we construct a strict Lyapunov function on the extended state space and prove Uniform Ultimate Boundedness (UUB) of the optimization trajectory under a descent alignment assumption and bounded stochastic disturbances. Under the additional Polyak-Lojasiewicz (PL) condition, the Lyapunov function converges linearly to a bounded neighborhood of the optimum, defined as the drift of the target direction field between consecutive iterates after parallel transport. To validate the framework's generativity, we derive three concrete instantiations of the quadruple, each exercising a distinct design axis, and show on geometric diagnostics, image classification benchmarks, and language modeling that their empirical behavior matches the theoretical predictions. The framework casts optimizer design as controller synthesis, complementing heuristic perspectives by providing a constructive template with provable guarantees.

Figures & tables

Figure 1: Geometric intuition of the NAIM Lyapunov framework. The orange surface represents the NAIM embedded in the extended state space $\theta,v$ . Blue streamlines illustrate the fast dynamics: the velocity state $v$ rapidly contracts onto $\Lambda$ at a rate governed by the lifting parameter $\eta$ , regardless of initial conditions. Orange arrows show the slow dynamics: once the trajectory reaches $\Lambda$ , the system evolves along the direction field $\Phi$ toward the target optimum at the center.

$\Phi$	$h$	Best val. (%)	Final val. (%)	Tail $\\|z_{k}\\|_{2}$	Tail $\\|\delta_{k}\\|_{2}$	Mean $\Delta\widehat{V}_{k}$	Tail $\widehat{V}_{k}$ (max)
$\tanh$	$3\times 10^{-4}$	$48.37\pm 1.04$	$48.35\pm 1.03$	$0.84$	$0.96$	$-3.8\times 10^{-5}$	$1.44\;(1.47)$
$\tanh$	$10^{-3}$	$65.01\pm 0.51$	$64.97\pm 0.47$	$0.84$	$0.95$	$-3.1\times 10^{-5}$	$1.22\;(1.26)$
$\tanh$	$3\times 10^{-3}$	$71.09\pm 1.25$	$59.24\pm 0.69$	$0.88$	$1.00$	$+2.5\times 10^{-5}$	$1.84\;(1.86)$
$\tanh$	$10^{-2}$	$72.52\pm 3.25$	$67.71\pm 3.42$	$0.86$	$0.97$	$+1.7\times 10^{-5}$	$1.45\;(1.48)$
$\operatorname{sign}$	$3\times 10^{-4}$	$91.69\pm 0.01$	$91.35\pm 0.32$	$3339$	$3801$	$-4.2\times 10^{-5}$	$1.10\;(1.45)$
$\operatorname{sign}$	$10^{-3}$	$91.37\pm 0.23$	$90.70\pm 0.11$	$2227$	$2534$	$-6.0\times 10^{-5}$	$0.73\;(4.46)$

Table 1: Mechanism diagnostics on CIFAR 10 with ResNet 18. Validation accuracy is reported as mean

\pm

standard deviation over three seeds.

Metric	Form	LR	Best Acc (%)	Final Acc (%)
RMSProp	Diagonal	$3\times 10^{-3}$	$94.38\pm 0.07$	$94.38\pm 0.07$
AdaGrad	Diagonal cumulative	$1\times 10^{-2}$	$93.96\pm 0.13$	$93.96\pm 0.13$
Adam	Diagonal debiased	$1\times 10^{-3}$	$93.86\pm 0.15$	$93.84\pm 0.13$
Layer-wise	Block diagonal	$3\times 10^{-3}$	$91.57\pm 0.10$	$91.53\pm 0.10$
Euclidean	Identity	$3\times 10^{-2}$	$81.72\pm 0.17$	$81.72\pm 0.17$

Table 2: Metric ablation on CIFAR-10 with ResNet-18, all framework components held fixed except the Riemannian metric. Each variant is evaluated at its individually selected peak learning rate. Best Acc and Final Acc are reported as mean

\pm

standard deviation across three seeds.

Optimizer	Best (%)	Avg (%)	Std
AdamW	93.87	93.77	0.13
Lion	94.03	93.88	0.13
Adan	93.86	93.78	0.09
Shampoo	94.92	94.83	0.09
RLO- $\Lambda$	93.90	93.89	0.02
RLO-Lifted	94.01	93.91	0.14

Table 3: Baseline comparison on CIFAR-10 with ResNet-18.

Optimizer	ResNet-50	ViT-S/16	ViT-B/16
AdamW	73.87	75.18	71.42
Lion	73.36	75.14	76.27
RLO	73.45	75.38	76.00
RLO- $\Lambda$	73.98	76.18	76.47
RLO-Lifted	73.98	71.43	76.33

Table 4: ImageNet Top-1 accuracy (%). All models trained from scratch for 90 epochs.

Optimizer	Val Loss	Best PPL	Final PPL	Train Loss	Generalization Gap
AdamW	3.1476	23.29	23.28	3.1432	0.0044
Lion	3.2012	24.53	24.56	3.1975	0.0037
RLO	3.2019	24.57	24.58	3.1983	0.0036
RLO- $\Lambda$	3.0708	21.56	21.56	3.0658	0.0049
RLO-Lifted	3.1614	23.61	23.60	3.1579	0.0035

Table 5: GPT-2 Small on OpenWebText,

10\,000

gradient steps. PPL denotes perplexity.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
Geometry and Manifold Structure
$\mathcal{M}$	The $d$ dimensional smooth parameter manifold.
$T_{\theta}\mathcal{M}$	The tangent space at $\theta\in\mathcal{M}$ .
$g_{\theta}(\cdot,\cdot)$	Riemannian metric, an inner product on $T_{\theta}\mathcal{M}$ .
$\\|\xi\\|_{g_{\theta}}$	Riemannian norm induced by $g$ , that is $\sqrt{g_{\theta}(\xi,\xi)}$ .
$\operatorname{grad}f(\theta)$	Riemannian gradient of $f$ at $\theta$ .

Appendix

Table 6: Summary of Notation

Optimizer	Metric $g$	State $y_{k}$	Target Field $\Phi$	Lift $\eta$	Manifold $\Lambda$
SGD	Euclidean $I$	$\emptyset$	$\hat{g}_{k}$ (Gradient)	$1$	Gradient Graph
Momentum	Euclidean $I$	Velocity $m_{k}$	$\hat{g}_{k}$ (Gradient)	$(0,1)$	Gradient Graph
AdamW	Adaptive $\text{diag}(\sqrt{s})^{-1}$	Moments $m,s$	$m_{k}$ (Momentum)	$(0,1)$	Precond. Momentum
Lion	Euclidean $I$	Momentum $m$	$\text{sign}(m)$	$1$	Hypercube Corners
RLO	Euclidean $I$	Momentum $m$	$\text{sign}(m)+\text{belief}$	$1$	Shifted Hypercube
RLO $\Lambda$	Euclidean $I$	Moments $m,s$	$\frac{\tanh(\gamma m)}{\sqrt{s}+\epsilon}$	$1$	Smooth Saturation

Appendix

Table 7: Unification of optimizers under the RLO framework

(\mathcal{M},g,\Phi,\eta)

$\eta$	GN	Optimal LR	Best Acc	Final Acc
$0.7$	✓	$3\times 10^{-5}$	91.59%	91.21%
$0.7$	$\times$	$3\times 10^{-3}$	89.76%	89.42%
$1$	✓	$3\times 10^{-5}$	91.51%	91.13%
$1$	$\times$	$3\times 10^{-3}$	89.02%	88.67%

Appendix

Table 8: Factorial ablation of RLO-Lifted on CIFAR-10 with ResNet-18 trained for 50 epochs. Each variant is evaluated at its optimal learning rate determined by grid search (see Section D.2 ). The Nolifted variants set

\eta=1

, eliminating the explicit velocity state so that

v_{k}=d_{k}

at every step. GN denotes global normalization , LR denotes the learning rate and Acc denotes accuracy in this and all subsequent tables.

$\Phi$	GN	LR	Best Acc	Final Acc	Loss Std
$\tanh(\gamma c)$	✓	$10^{-4}$	91.69	91.49	0.249
$\text{sign}(c)$	✓	$10^{-4}$	91.91	91.91	0.239
$\tanh(\gamma c)$	$\times$	$3\times 10^{-3}$	90.74	90.74	0.360
$\text{sign}(c)$	$\times$	$3\times 10^{-3}$	87.15	74.43	1.610

Appendix

Table 9: Effect of direction field smoothness on CIFAR-10. The smooth tanh mapping significantly outperforms the discontinuous sign function when global normalization is disabled.

Figure 2: Hyperparameter sensitivity heatmaps for the four factorial variants. Color intensity indicates test accuracy (%). The global normalization enabled variants (left column) achieve peak performance at small learning rates ( $\sim 3\times 10^{-5}$ ), while the without global normalization variants (right column) require large learning rates ( $\sim 3\times 10^{-3}$ ). The optimal regions differ by approximately two orders of magnitude in the learning rate axis.

$\eta$	Best Acc (%)	Final Acc (%)	$\bar{r}$	$\overline{\cos(v,d)}$	$\bar{q}^{\perp}$
0.1	90.87	90.52	0.83	0.55	0.94
0.2	91.12	90.78	0.87	0.50	0.95
0.3	91.38	91.02	0.90	0.47	0.96
0.5	91.54	91.21	0.96	0.42	0.96
0.7	91.69	91.49	1.03	0.37	0.96
1.0	91.75	91.64	0.00	1.00	0.88

Appendix

Table 10: Effect of lifting parameter

\eta

on test accuracy and NAIM diagnostics. Metrics

\bar{r}

and

\overline{\cos(v,d)}

are averaged over all training steps.

Figure 3: $\eta$ ablation test. (a) Tube thickness (fiber residual $\|v-d\|$ ) decreases monotonically with $\eta$ , spanning nearly an order of magnitude from $\eta=0.1$ ( $\sim$ 4000) to $\eta=0.9$ ( $\sim$ 400). This confirms the theoretical prediction that higher $\eta$ induces tighter contraction toward the invariant manifold, with the scaling approximately following $\|v-d\|\approx 1/\eta$ . (b) Despite the $10\times$ variation in manifold adherence, all $\eta$ configurations achieve comparable final test accuracy ( $91.2\%-92.0\%$ ), demonstrating the robustness of the NAIM framework. Notably, $\eta=0.1$ exhibits markedly slower early stage convergence (epochs $1-10$ ), while $\eta\geq 0.3$ configurations show nearly identical learning dynamics. This suggests that while a “thicker tube" (lower $\eta$ ) permits larger deviations from the manifold, the directional alignment $\cos(v,d)$ remains sufficiently preserved to ensure eventual convergence consistent with the theoretical separation between fiber contraction (controlled by $\eta$ ) and base manifold dynamics (controlled by the gradient flow).

Figure 4: Validation accuracy curves for ImageNet classification using ResNet-50 (left), ViT-B/16 (center), and ViT-S/16 (right). RLO- $\Lambda$ consistently achieves the highest final accuracy across all architectures (73.98%, 76.47%, 76.18% respectively). Notably, sign-based optimizers (Lion, RLO variants) demonstrate substantially faster convergence and higher final accuracy than AdamW on Vision Transformers, suggesting that the NAIM-guided updates are particularly effective for attention-based architectures.

Peak learning rate	Warmup steps	Validation loss	PPL
$10^{-4}$	$2{,}000$	$3.5215$	$33.83$
$10^{-4}$	$4{,}000$	$3.5412$	$34.51$
$3\times 10^{-4}$	$2{,}000$	$3.2382$	$25.49$
$3\times 10^{-4}$	$4{,}000$	$3.2564$	$25.96$
$6\times 10^{-4}$	$2{,}000$	$\mathbf{3.1432}$	$\mathbf{23.18}$
$6\times 10^{-4}$	$4{,}000$	$3.1543$	$23.44$

Appendix

Table 11: Additional AdamW search on GPT 2 Small with OpenWebText over

10{,}000

optimizer steps. PPL denotes validation perplexity. Best and final validation loss and perplexity coincide for all six configurations.

Hyperparameter	Default Value
Momentum decay $\beta_{1}$	0.9
EMA decay $\beta_{2}$	0.99
Weight decay	0.1
Belief coefficient $\lambda_{b}$	0.2
Tanh scaling $\gamma$	5.0
Lifting parameter $\eta$	0.7 (Full) or 1.0 (Nolifted)

Appendix

Table 16

Parameter	Value
Training epochs	90
Global batch size	1024
Per-GPU batch size	128
Warmup epochs	5
Warmup schedule	Linear
Learning rate schedule	Cosine decay

Appendix

Table 12: Common training hyperparameters for ImageNet classification.

Model	Optimizer	LR	WD
ResNet-50	AdamW	$1\times 10^{-3}$	0.05
	Lion	$1\times 10^{-4}$	0.5
	RLO	$1\times 10^{-4}$	0.5
	RLO- $\Lambda$	$1\times 10^{-4}$	0.5
	RLO-Lifted	$1\times 10^{-4}$	0.5
ViT-S/16	AdamW	$3\times 10^{-3}$	0.1

Appendix

Table 13: Optimizer hyperparameters for ImageNet classification experiments. LR denotes peak learning rate, WD denotes weight decay coefficient.

Epoch	AdamW	Lion	RLO	RLO- $\Lambda$	RLO-Lifted
10	33.46	51.78	50.58	53.41	49.95
20	38.26	51.30	55.45	57.82	55.52
30	41.98	57.89	58.08	60.65	58.24
40	42.57	63.06	61.44	63.92	61.15
50	49.70	66.64	64.62	67.22	65.03
60	55.39	69.91	68.64	70.36	68.86

Appendix

Table 14: ViT-B/16 validation accuracy (%) at selected epochs during training.

Optimizer	Final Loss	Min Loss	Loss at Best Val
AdamW	2.21	2.21	2.21
Lion	1.80	1.80	1.80
RLO	1.83	1.83	1.83
RLO- $\Lambda$	1.76	1.76	1.76
RLO-Lifted	1.81	1.81	1.81

Appendix

Table 15: Training loss statistics for ViT-B/16. Final loss is measured at epoch 90.

$\eta$	Best Accuracy (%)	Final Accuracy (%)
0.3	69.87	69.54
0.5	70.65	70.32
0.7	71.43	71.12
0.9	73.89	73.67
1.0	75.21	74.98

Appendix

Table 16: RLO-Lifted accuracy on ViT-S/16 with different lifting parameters.

Stochastic Optimization Convergence

Stochastic optimization convergence studies how noisy, sampled, or delayed updates approach solutions and how fast they do so.

See all 212 papers

Feb 26, 2026

MSINO: Curvature-Aware Sobolev Optimization for Manifold Neural Networks

We introduce Manifold Sobolev Informed Neural Optimization (MSINO), a curvature aware training framework for neural networks defined on Riemannian manifolds. The method replaces standard Euclidean derivative supervision with a covariant Sobolev loss that aligns gradients using parallel transport and improves stability via a Laplace Beltrami smoothness regularization term. Building on classical results in Riemannian optimization and Sobolev theory on manifolds, we derive geometry dependent constants that yield (i) a Descent Lemma with a manifold Sobolev smoothness constant, (ii) a Sobolev Polyak Lojasiewicz inequality giving linear convergence guarantees for Riemannian gradient descent and stochastic gradient descent under explicit step size bounds, and (iii) a two step Newton Sobolev method with local quadratic contraction in curvature controlled neighborhoods. Unlike prior Sobolev training in Euclidean space, MSINO provides training time guarantees that explicitly track curvature and transported Jacobians. Applications include surface imaging, physics informed learning settings, and robotics on Lie groups such as SO(3) and SE(3). The framework unifies value and gradient based learning with curvature aware convergence guarantees for neural training on manifolds.

Suresan Pareth

Riemannian Neural Networks Neural Network Optimization Curvature-Aware Optimization Riemannian Optimization

Ask PDF

Mar 18, 2026

Mirror Descent on Riemannian Manifolds

Mirror Descent (MD) is a scalable first-order method widely used in large-scale optimization, with applications in image processing, policy optimization, and neural network training. This paper generalizes MD to optimization on Riemannian manifolds. In particular, we develop a Riemannian Mirror Descent (RMD) framework via reparameterization and further propose a stochastic variant of RMD. We also establish non-asymptotic convergence guarantees for both RMD and stochastic RMD. As an application to the Stiefel manifold, our RMD framework reduces to the Curvilinear Gradient Descent (CGD) method proposed in [26]. Moreover, when specializing the stochastic RMD framework to the Stiefel setting, we obtain a stochastic extension of CGD, which effectively addresses large-scale manifold optimization problems.

Jiaxin Jiang, Lei Shi, Jiyuan Tan

School of Mathematical Sciences, Fudan University, Shanghai 200433, China · Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai 200433, China

Riemannian Optimization Stochastic Optimization Convergence

Ask PDF

May 8, 2026

A Unified Lyapunov-IQC Framework for Uniform Stability of Smooth Quadratic First-Order Accelerated Optimizers

We develop a unified Lyapunov-integral quadratic constraint (IQC) framework for establishing uniform stability of first-order accelerated optimization algorithms in the

β

-smooth and

γ

-strongly convex regime. Classical analyses of uniform stability, such as the work of Hardt, Recht, and Singer for stochastic gradient descent (SGD), rely on direct coupling arguments and case-by-case control of iterate differences under random sampling. Extending such arguments to accelerated methods, such as Nesterov Accelerated Gradient (NAG), is complicated by the presence of higher-order state dynamics induced by momentum. We first extend this classical approach with the use of Lyapunov functions to provide a uniform stability bound for smooth quadratic NAG, and supplement this result with small-scale numerical experiments. We then extend this framework by modeling first-order accelerated optimizers as Lur'e-type feedback interconnections between a linear dynamical system and a (non-linear) gradient operator.

β

-Smoothness and

γ

-strong convexity are encoded a sector IQC inequality. Under this representation, uniform stability is certified via the existence of a quadratic Lyapunov function satisfying a finite-dimensional linear matrix inequality (LMI) in the form of a feasibility problem, which can be solved via semi-definite programming (SDP). We instantiate this framework for NAG and show how classical uniform stability bounds can be recovered via this framework. These results underscore a structural connection between optimization dynamics and robust control theory, providing a modular methodology for reliable and reproducible numerical certification of uniform stability and generalization behavior of first-order methods via convex optimization tools that is adaptable to increasingly complex optimization algorithms.

Don Li, Dacian Daescu

Department of Mathematics & Statistics, Portland State University, Portland, OR 97201, USA

Lyapunov Stability Algorithmic Stability Gradient Descent Accelerated Gradient Methods

Ask PDF

Abstract

Stochastic Optimization Convergence

Explore similar work

MSINO: Curvature-Aware Sobolev Optimization for Manifold Neural Networks

Mirror Descent on Riemannian Manifolds

A Unified Lyapunov-IQC Framework for Uniform Stability of Smooth Quadratic First-Order Accelerated Optimizers