Riemannian Optimization

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 38

All topics

A Riemannian Geometry for Low-rank Adaptation
Oct 6, 2026Shoichiro Takeda, Shin'ya Yamaguchi, Satoshi Suzuki +1Riemannian Optimization
NTT, Inc.
Minkowski Attractor Networks: Closed-Form Hyperbolic Flows for Visual Representations
Sep 29, 2026Zhongping JiHyperbolic Learning Hyperbolic Space
Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices
Sep 23, 2026Ali Aliev, Maxim RakhubaRiemannian Optimization Orthogonality
HSE University
Locally Private Inference for Riemannian Stochastic Optimization
Sep 18, 2026Xiaotian Chang, Yangdi Jiang, Qirui HuRiemannian Optimization Covariance
School of Physical and Mathematical Sciences, Nanyang Technological University · School of Statistics and Data Science, Shanghai University of Finance and Economics.
Directions That Don't Drift: Stiefel Manifold Routing for Transformer Attention
Sep 16, 2026Rubén Darío GuerreroRiemannian Optimization Adam
NeuroTechNet S.A.S., 1108831, Bogotá, Colombia
Optimization over covariance matrices with a parameterized metric
Sep 15, 2026Yibang Li, Bamdev Mishra, Pratik Jawanpuria +1Riemannian Optimization Hessian
Nanyang Technological University, Singapore · Microsoft India · Indian Institute of Technology Bombay, India
Riemannian Optimization for Multi-Player Quantum Games on Product Unitary Manifolds
Sep 7, 2026Alireza Habibi, Setareh MaghsudiImperfect-Information Games Riemannian Optimization
Faculty of Electrical Engineering and Information Technology, Ruhr University Bochum, Bochum, Germany
Projected Riemannian Gradient Descent for the Bures-Wasserstein Barycenter: Dimension-Independent Linear Convergence at Unit Step Size
Sep 3, 2026A. AfhamRiemannian Optimization Wasserstein Distance
Centre for Quantum Technologies, National University of Singapore, Singapore
Stochastic Optimization of Tree Tensor Networks
Sep 1, 2026Marius Willner, Maximilian Scharf, André Uschmajew +2Tensor Networks Neural Network Optimization
Institute of Mathematics, University of Augsburg, 86159 Augsburg, Germany · Centre for Advanced Analytics and Predictive Sciences, University of Augsburg, 86159 Augsburg, Germany · Tensor AI Solutions GmbH, 89284 Pfaffenhofen an der Roth, Germany +1
Controlling Refusal Behavior of LLMs via Stiefel-Constrained Rotation Steering
Aug 31, 2026Kirill Bunin, Dmitry Bylinkin, Vladimir Aletov +3Linear Activation Steering Steering
1Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · 2Kandinsky Lab · 3Innopolis University
Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning
Jul 28, 2026Yuan Zhang, Jiang Hu, Zhijian Lai +2Riemannian Optimization Model Fine-Tuning
Center for Data Science, Peking University, Beijing 100871, China · Yau Mathematical Sciences Center, Tsinghua University, Beijing 100084, China · Beijing International Center for Mathematical Research, Peking University, Beijing 100871, China +2
Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning
Jul 26, 2026Minh Vu, Konstantinos SlavakisOffline Reinforcement Learning Value Functions
Institute of Science Tokyo, Department of Information and Communications Engineering, Yokohama, Japan.
Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solvers
Jul 24, 2026Zhangyong Liang, Huanhuan GaoNeural Partial Differential Equation Solvers Riemannian Optimization
National Center for Applied Mathematics Tianjin University Tianjin, 300072, China · School of Mechanical and Aerospace Engineering Jilin University Changchun, 130025, China
Decentralized Online Riemannian Optimization for Strongly Geodesically Convex Functions
Jul 22, 2026Zhanyuan Cai, Emre Sahinoglu, Shahin ShahrampourRiemannian Optimization Convex Loss
Department of Mechanical & Industrial Engineering at Northeastern University, Boston, MA 02115, USA
Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization
Jul 11, 2026Zhicheng Cai, Xinyuan Guo, Hanlin Wu +4Large Language Model Reinforcement Learning Gradient Clipping
Institute for AI Industry Research (AIR), Tsinghua University · 2ByteDance Seed 3SIA-Lab of Tsinghua AIR and ByteDance Seed.
Learning Adaptive Solvers for Distributed Factor Graph Optimization on Matrix Lie Groups
Jul 9, 2026Jaeho Shin, Maani Ghaffari, Yulun TianRiemannian Optimization Decentralized Optimization
University of Michigan
Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamics
Jun 30, 2026Yudong Wei, Liang Zhang, Bingcong Li +1Low-Rank Structure Riemannian Optimization
Department of Mathematics, ETH Zurich, Zurich, Switzerland · Department of Computer Science, ETH Zurich, Zurich, Switzerland
Decentralized Pose Graph Riemannian Optimization for Object-based Multi-Robot SLAM
Jun 23, 2026Yixian Zhao, Yan Huang, Yang Xu +2Multi-Robot Systems Simultaneous Localization And Mapping
College of Control Science and Engineering, Zhejiang University, Hangzhou, 310027, China · Division of Decision and Control Systems, School of EECS, KTH Royal Institute of Technology, Stockholm, SE-100 44, Sweden · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai, 201210, China
A Riemannian Approach to Low-Rank Optimal Transport
Jun 10, 2026Pratik Jawanpuria, Bamdev MishraLow-Rank Structure Riemannian Optimization
Centre for Machine Intelligence and Data Science, IIT Bombay · Microsoft India
Generalized-CVO: Fast and Correspondence-Free Local Point Cloud Registration with Second Order Riemannian Optimization
Jun 8, 2026Ray Zhang, Marcus Greiff, Thomas Lew +1Point Cloud Registration Point Clouds
Toyota Research Institute, 4440 El Camino Real, Los Altos, California, USA
Riemannian Gradient Descent for Low-Rank Architectures
Jun 1, 2026Nicholas KnightRiemannian Optimization Multi-Head Attention
NVIDIA
Riemannian Optimization for Hadamard Products of Low-Rank Matrices
May 31, 2026Pratik Jawanpuria, Ankish Chandresh, Bamdev MishraRiemannian Optimization Low-Rank Structure
Centre for Machine Intelligence and Data Science, Indian Institute of Technology Bombay, India · Microsoft India
Riemannian Stochastic Optimization for Sufficient Dimension Reduction
May 29, 2026Thibault Pautrel, François PortierDimensionality Reduction Riemannian Optimization
Laboratoire des Signaux et Systèmes (L2S), CentraleSupélec, Université Paris-Saclay, Gif-sur-Yvette, France · ENSAI, CREST, Bruz, France
Foundation-Preserving Optimization in Generalized Eigenspace
May 28, 2026Dongjun Kim, Adrian de Wynter, Huancheng Chen +2Parameter-Efficient Adaptation Rapid Adaptation
The University of Texas at Austin · Microsoft · Microsoft AI +1
Dynamic Elliptical Graph Factor Models via Riemannian Optimization with Geodesic Temporal Regularization
May 18, 2026Chuansen Peng, Xiaojing ShenRiemannian Manifolds Factor Graphs
School of Mathematics, Sichuan University, Chengdu, Sichuan, 610064, China
RL4RLA: Teaching ML to Discover Randomized Linear Algebra Algorithms Through Curriculum Design and Graph-Based Search
May 18, 2026Jinglong Xiong, Xiaotian Liu, Ruoxin Wang +4Solver Iterations Low-Rank Structure
Pratt School of Engineering, Duke University, Durham, NC, USA · Department of Computer Science, Dartmouth College, Hanover, NH, USA.
Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds
May 10, 2026Yibang Li, Bihari Lal Pandey, Ravi Sah +4Muon Riemannian Optimization
Nanyang Technological University, Singapore. · Indian Institute of Technology Bombay, India. · University of Sydney, Australia. +1
Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
May 7, 2026Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri +5Low-Rank Structure Spectral Representation Method
1Keio University · 2Stanford University · 3Midwestern University +3
Demystifying Manifold Constraints in LLM Pre-training
May 6, 2026Kang An, Jiaxiang Li, Donald Goldfarb +1Large Language Model Pretraining Riemannian Optimization
Rice University · Independent Researcher · Columbia University
Foundations of Riemannian Geometry for Riemannian Optimization: A Monograph with Detailed Derivations
May 4, 2026Benyamin GhojoghRiemannian Optimization Riemannian Manifolds
Waterloo, Ontario, Canada
Model Compression with Exact Budget Constraints via Riemannian Manifolds
May 1, 2026Michael Helcig, Dan AlistarhRiemannian Optimization Token Budget Allocation
ETH Zürich · IST Austria
BOOOM: Loss-Function-Agnostic Black-Box Optimization over Orthonormal Manifolds for Machine Learning and Statistical Inference
Apr 23, 2026Beomchang Kim, Subhrajyoty Roy, Priyam DasRiemannian Optimization Black-Box Optimization
Department of Biostatistics, Virginia Commonwealth University, Richmond, VA, USA · Department of Statistics and Data Science, Washington University in St. Louis,[math.OC] St. Louis, MO, USA
Safety-Critical Contextual Control via Online Riemannian Optimization with World Models
Apr 21, 2026Tongxin LiModel Predictive Control Safety-Critical Scenarios
Member, IEEE
Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization
Apr 16, 2026Mathias DusRiemannian Optimization Wasserstein Distance
IRMA
An operator splitting analysis of Wasserstein--Fisher--Rao gradient flows
Nov 22, 2025Francesca Romana Crucinio, Sahani PathirajaWasserstein Gradient Flows Wasserstein Distance
ESOMAS, University of Turin, Italy & Collegio Carlo Alberto, Turin, Italy · School of Mathematics & Statistics, UNSW Sydney, Australia
Smooth Quasar-Convex Optimization with Constraints
Oct 2, 2025David Martínez-RubioNonconvex Riemannian Optimization
IMDEA Software Institute · Madrid, Spain
Riemannian Optimization on Tree Tensor Networks with Application in Machine Learning
Jul 29, 2025Marius Willner, Marco Trenti, Dirk LebiedzTensor Networks Riemannian Optimization
Chair of Mathematical Data Science, University of Augsburg, Augsburg, Germany · Tensor AI Solutions GmbH, Pfaffenhofen a.d. Roth, Germany · Institute for Numerical Mathematics, University of Ulm, Ulm, Germany
Optimization on the Oblique Manifold for Sparse Simplex Constraints via Multiplicative Updates
Mar 31, 2025Flavia Esposito, Andersen AngRiemannian Optimization Low-Rank Structure
Department of Mathematics, University of Bari Aldo Moro, Bari, 70125, Italy. · School of Electronics and Computer Science, University of Southampton, Southampton, SO17 1BJ, United Kingdom.

Oct 6, 2026, cs.LGJ/K move · Enter open · S save

A Riemannian Geometry for Low-rank Adaptation

Shoichiro Takeda, Shin'ya Yamaguchi, Satoshi Suzuki, Yasunori Akagi

NTT, Inc.

Riemannian Optimization

Open paper AskPDF arXiv

Low-rank adaptation (LoRA) is widely used as a parameter-efficient fine-tuning technique for pre-trained deep neural networks, which approximates the weight update via full fine-tuning by a low-rank matrix

BA^\top

. This parameterization leads to the equivalence relation

(B, A) \sim (BG^{-1}, AG^\top)

for any invertible matrix

G

because

BA^\top = BG^{-1}(AG^\top)^\top

and thus both pairs yield the same loss value. This relation induces a quotient manifold where matrices

(BG^{-1}, AG^\top)

for all

G

are identified, eliminating redundant directions along which the loss value remains unchanged. To respect the geometry of this manifold, the original search space is endowed with a Riemannian metric that is invariant under the equivalence relation. Such a metric induces preconditioning at each gradient step and ensures that each weight update via LoRA changes the loss value, leading to efficient optimization. In this paper, we propose a new Riemannian metric that is specifically tailored to LoRA to close the gap to full fine-tuning at the weight level. We theoretically show that LoRA with our preconditioning induced by this metric satisfies the following two properties at each iteration: (i) The weight update follows the direction closest to the gradient of full fine-tuning within the subspace of first-order weight changes allowed by the LoRA parameterization. (ii) The updated weight matrix is closer in Frobenius norm to that of full fine-tuning than the updated weight matrices of LoRA with conventional preconditioning and without preconditioning. These theoretical insights suggest that our preconditioning makes LoRA better approximate full fine-tuning, thereby leading to more efficient optimization. Experiments show the effectiveness and efficiency of our preconditioning for LoRA on fine-tuning tasks with language and vision domains.

Momentum

Related topics

Latest papers 38