Information Geometry

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 75% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 38

All topics

How Inefficient Is Natural Gradient Descent? From Exact Optimality to Θ( \sqrt{ \log d } ) Divergence
Oct 5, 2026Guni Sharon, Alan KuhnleInformation Geometry Riemannian Optimization
Department of Computer Science and Engineering, Texas A&M University, College Station, TX 77843, USA.
Technical note on: Zero-Training Feature-Space Alignment via Information Geometry
Sep 29, 2026Behraj Khan, Tahir Qasim Syed, Syed Ahmad Chan BukhariDistribution Shift Robustness Information Geometry
School of Mathematics and Computer Science, Institute of Business Administration Karachi, Pakistan · Division of Computer Science, Mathematics and Science, St. John’s University, USA
Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models
Sep 29, 2026Abdullah All Tanvir, Xin ZhongInformation Geometry Neural Representation Geometry
Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA
Information Geometric Self-Organization at the Edge of Stability in High-Capacity Kernel Associative Memories
Sep 15, 2026Akira TamamoriInformation Geometry Edge of Stability
Faculty of Information Science, Aichi Institute of Technology, Aichi, 470-0392, Japan
Geometry of learning dynamics: Gradient descent versus natural gradient on the ridge of optimization
Sep 15, 2026Akira TamamoriInformation Geometry Gradient Descent
Faculty of Information Science, Aichi Institute of Technology 1247 Yachigusa, Yakusa-cho, Toyota-shi, Aichi 470-0392, Japan
Information-Induced Training Geometry: Exact Reduction, Canonical Completion, and Structured Expressivity
Sep 14, 2026Zavier LiInformation Geometry Riemannian Optimization
Xidian University Xi’an, China
The information geometry of large language models is shared, learned, and controllable
Sep 11, 2026Dario PicozziInformation Geometry Language Model Steering
Department of Physics and Astronomy, University College London (UCL), Gower Street, London WC1E 6BT, United Kingdom · London Centre for Nanotechnology, 19 Gordon St, London WC1H 0AH, United Kingdom
A Generalization of Amari's Bayesian Duality
Sep 8, 2026Mohammad Emtiyaz Khan, Thomas MöllenhoffInformation Geometry Bayesian Inference
RIKEN Center for Advanced Intelligence Project, 1-4-1 Nihonbashi, Chuo-ku, Tokyo, 103-0027, Tokyo, Japan. · Department of Computer Science, Technische Universität Darmstadt, Hochschulstraße 10, Darmstadt, 64289, Hessen, Germany. · The Hessian Center for Artificial Intelligence, Landwehrstraße 50a, Darmstadt, 64293, Hessen, Germany.
The Dually Flat Geometry of Planning as Inference
Sep 3, 2026Nikola Milosevic, Asaki Kataoka, Nicolas Hinrichs +2Information Geometry
Neural Data Science and Statistical Computing Group Max Planck Institute for Human Cognitive and Brain Sciences Leipzig, 04103 Germany · Neural Computation Unit Okinawa Institute of Science and Technology Okinawa, 904-0495 Japan
Informative Label Missingness in Multiclass Classification Information Geometry and Excess Risk
Aug 31, 2026Fariborz Setoudehtazang, Geoffrey J. McLachlanInformation Geometry Learning with Missing Data
Dipartimento di Scienze Statistiche, Universit`a di Padova, Padova, Italy · School of Mathematics and Physics, The University of Queensland, Australia
The Advective Fisher-Rao Geometry of Deterministic Measure Transport
Aug 12, 2026Benjamin Gess, Johannes MüllerInformation Geometry Riemannian Optimization
Institut für Mathematik, Technische Universität Berlin & Max-Planck-Institut für Mathematik in den Naturwissenschaften · Institut für Mathematik, Technische Universität Berlin
Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvature
Aug 10, 2026Tong CheGauss-Newton Optimization Information Geometry
NVIDIA Research
Readout-Rank Laws for Isotropic Quantum Tangents
Aug 7, 2026Marwan Ait HaddouInformation Geometry Variational Quantum Circuits
Independent Researcher, Morocco
Information-Geometric Forward Policy Training in GFlowNets
Aug 4, 2026Yordan Raykov, Rodrigo VeigaInformation Geometry Natural Gradient Methods
School of Mathematical Sciences, University of Nottingham, Nottingham, UK
Interventional Score Geometry for Causal Inference
Jul 24, 2026Mojtaba EslamiInformation Geometry Structural Causal Models
Post-doctoral Researcher, University of Calgary.
Fisher Widths: Local Learning Geometry and Anisotropic Recovery
Jul 22, 2026Vu Khac KyInformation Geometry Sparse Recovery
Department of Mathematics, FPT University, Vietnam
Statistically Meaningful Geometry (SMG) Beyond the Euclidean Paradigm, with Application to Generative AI
Jul 3, 2026Bing Cheng, Yi-Shuai Niu, Howell Tong +1Neural Network Generalization Information Geometry
Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, China · AMSS Center for Forecasting Science, Chinese Academy of Sciences, Beijing, China · Beijing Institute of Mathematical Sciences and Applications (BIMSA), Beijing, China +4
CuBAS: Information Geometric Curvature-Based Adaptive Sampling for Supervised Classification
Jul 3, 2026Alexandre L. M. LevadaInformation Geometry Adaptive Sampling
Federal University of São Carlos 13565-905, São Carlos-SP, Brazil
An Information-Geometric Justification for Composite Coherence in Event-Based Narrative Extraction
Jun 28, 2026Brian Keith-NorambuenaInformation Geometry
Department of Computing & Systems Engineering, Universidad Católica del Norte, Antofagasta, Chile.
Spectral Perturbation of the Empirical Fisher Information Matrix under Weight Quantization
Jun 26, 2026Rahid Zahid Alekberli, Hikmat KarimovInformation Geometry Weight-Only Quantization
Institute of Defense Technologies and Cybersecurity, Azerbaijan Technical University, Baku, Azerbaijan
A Transport-Based Geometry of Belief-Cost
Jun 19, 2026Laurent CaraffaInformation Geometry Optimal Transport
Université Gustave Eiffel, LASTIG, IGN-ENSG
Fisher-Geometric Sharpness and the Implicit Bias of SGD toward Flat Minima
Jun 18, 2026Md Sakir Ahmed, Kumaresh Sarmah, Hemen DuttaInformation Geometry Implicit Bias
Department of Electronics and Communication Technology Gauhati University Guwahati, Assam, India · Department of Mathematics Gauhati University Guwahati, Assam, India
Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale
Jun 17, 2026Tejas Pradeep Shirodkar, P. J. NarayananInformation Geometry Transformer
IIIT, Hyderabad
Fisher Width: A Geometric Measure of Complexity on Statistical Manifolds
Jun 16, 2026Vu Khac KyInformation Geometry Riemannian Geometry
Department of Mathematics, FPT University, Vietnam
Beyond IGO-Flow: Toward Convergence Analysis of IGO in Continuous Spaces
Jun 16, 2026Ryosuke Kimura, Youhei AkimotoInformation Geometry Optimization Convergence Analysis
University of Tsukuba, Tsukuba, Japan · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
Dead Directions: Geometric Singular Learning
Jun 4, 2026Tejas Pradeep ShirodkarInformation Geometry Singular Learning Theory
IIIT, Hyderabad
Information-Geometric Optimization on Spheres
May 27, 2026Vladimir Ja\' cimovićInformation Geometry Riemannian Optimization
Faculty of Natural Sciences and Mathematics University of Montenegro Cetinjski put bb., 81000 Podgorica Montenegro
FishBack: Pullback Fisher Geometry for Optimal Activation Steering in Transformers
May 17, 2026Sihan Wang, Jiayi ZhaoInformation Geometry Language Model Steering
South China University of Technology
Information-geometric adaptive sampling for graph diffusion
Apr 30, 2026Yuhui Lu, Wenjing Liu, Kun ZhanInformation Geometry Graph Generation
School of Information Science and Engineering, Lanzhou University, Gansu, China.
Exponential families from a single KL identity
Apr 30, 2026Marc DymetmanInformation Geometry
Scientific Consultant

Oct 5, 2026, stat.MLJ/K move · Enter open · S save

How Inefficient Is Natural Gradient Descent? From Exact Optimality to Θ( \sqrt{ \log d } ) Divergence

Guni Sharon, Alan Kuhnle

Department of Computer Science and Engineering, Texas A&M University, College Station, TX 77843, USA.

Information Geometry Riemannian Optimization Natural Gradient Methods

Open paper AskPDF arXiv

Natural gradient descent (NGD) underlies common methods in ML. For dually flat families, idealized NGD on the forward Kullback--Leibler objective follows the mixture geodesic which is often longer than the shortest Fisher--Rao path. We quantify this overhead by the inefficiency ratio

R \ge 1

, the Fisher length of the mixture geodesic divided by the Fisher--Rao distance, and bound its supremum over endpoint pairs as a function of the parameter dimension

d

. A tensor criterion identifies the regime (I) families, with

R=1

everywhere: exactly those with quadratic potential or dimension one, such as fixed-covariance Gaussians. For non-quadratic families, we prove two further regimes: (II) bounded third-order skewness plus finite Fisher--Rao diameter yields a dimension-independent bound; and (III) for products of scale families---including Gaussian covariances and Gamma rates---

R

grows as

Θ(\sqrt{\log d})

, unbounded in

d

. Under a per-step Fisher-chord budget,

R

translates to a practical computational cost: NGD requires asymptotically at least

R

times as many steps as an optimizer following the Fisher--Rao geodesic. Experiments confirm all three regimes:

R=1

to machine precision for quadratic-potential families (I), the categorical bound

π/(2\sqrt{2})

is approached but not attained (II), and sampled scale-product

R

grows with

d

, reaching

R \approx 1.5

for long, high-dimensional moves (III).

Momentum

Related topics

Latest papers 38