Overview

Natural gradient methods study how to use the geometry of a model’s probability distributions, commonly represented by the Fisher information matrix, to choose updates that are less dependent on parameterization. They matter when ordinary gradients are poorly scaled or constrained, including neural-network training, policy optimization in reinforcement learning, variational inference, and optimization on manifolds. Approaches range from exact Fisher-based updates to Kronecker-factored, low-rank, and matrix-free approximations, alongside Riemannian methods and convergence analyses for policy learning and other structured problems. A central tension is that curvature-aware updates can improve convergence, but estimating and applying them can be costly or noisy, and useful step sizes and reliable scaling remain challenging.

Latest papers

CardsList

33 papers3 papers in the last four weeks, down 40% on the four weeks before. 0.0% of all new papers.

  1. Natural Gradient Learning in Predictive Coding through Lateral Circuits

    Nov 12, 2026Lhea Beumer, Sander de Haan, Niclas Pokel +2Predictive CodingLocal Learning

  2. Fixed-point acceleration on Neural Wasserstein manifolds

    Nov 6, 2026Vitalii Aksenov, Sebastian Gutierrez HernandezInference AccelerationRiemannian Optimization

  3. Variational Learning with Lie Group Exponential Families

    Nov 3, 2026Eren Mehmet Kıral, Thomas Möllenhoff, Koichi Tojo +2Variational InferenceExponential Family

  4. How Inefficient Is Natural Gradient Descent? From Exact Optimality to Θ( \sqrt{ \log d } ) Divergence

    Oct 5, 2026Guni Sharon, Alan KuhnleInformation GeometryRiemannian Optimization

  5. Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization

    Oct 3, 2026Mahalakshmi Sabanayagam, Simon LuceyGradient Descent DynamicsSpectral Bias

  6. The cost of useful natural gradient updates

    Sep 27, 2026Subhransu S. Bhattacharjee, Dylan Campbell, Rahul ShomeSample ComplexityNatural Gradient Methods

  7. Newton Matching for Generative Modeling: A Unified Framework for Fine-Tuning and Sampling

    Sep 4, 2026Zeyang Li, Yunan Wang, Paolo Giaretta +1Generative ModelingRL for Generative Models

  8. Integer Natural Evolution Strategies

    Aug 24, 2026Jacob de Nobel, Diederick Vermetten, Hao Wang +3Evolution StrategiesNatural Gradient Methods

  9. Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic

    Aug 20, 2026Zhiqiang TanPolicy OptimizationActor-Critic Methods

  10. Efficient Hypergradient Descent for Inverse Reinforcement Learning

    Aug 11, 2026Nikita Sevriukov, Anna Barabanova, Uliana Gagarina +4Reinforcement LearningGradient Descent

  11. Fisher8: Stabilizing Neural Heteroscedastic Regression via Output-Layer Fisher Geometry

    Aug 11, 2026Sumedh Vemuganti, Nickvash KaniNatural Gradient MethodsHeteroscedastic Regression

  12. Information-Geometric Forward Policy Training in GFlowNets

    Aug 4, 2026Yordan Raykov, Rodrigo VeigaInformation GeometryNatural Gradient Methods

  13. Finite-Time Analysis of the Natural Policy Gradient in Finite-Horizon Markov Decision Processes

    Jul 25, 2026Asha Barua, Sajad KhodadadianPolicy GradientMarkov Decision Processes

  14. Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solvers

    Jul 24, 2026Zhangyong Liang, Huanhuan GaoNeural PDE SolversRiemannian Optimization

  15. Differentially Private Natural Gradient Descent

    Jul 7, 2026Pan Li, Kai Chen, Shuai Chang +3Gradient DescentNatural Gradient Methods

  16. On the Optimizer Dependence of Neural Scaling Laws

    May 28, 2026Vansh Ramani, Shourya Vir JainNatural Gradient MethodsNeural Network Optimization

  17. Adaptive Stochastic Natural Gradient Method for Safe Optimization on Binary Space

    May 18, 2026Kento Uchida, Ryoki Hamano, Masahiro Nomura +1Black-Box OptimizationNatural Gradient Methods

  18. Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework

    May 11, 2026Phalguni Nanda, Zaiwei ChenPolicy GradientReinforcement Learning

  19. Natural gradient descent with momentum

    Apr 16, 2026Anthony Nouy, Agustín SomacalDeep Learning OptimizationRiemannian Optimization

  20. Natural Gradient Gaussian Approximation Filter on Lie Groups for Robot State Estimation

    Apr 11, 2026Tianyi Zhang, Wenhan Cao, Chang Liu +2Robot State EstimationBayesian Filtering

  21. Natural Riemannian gradient for learning functional tensor networks

    Apr 10, 2026Nikolas Klug, Michael Ulbrich, André Uschmajew +1Tensor NetworksRiemannian Optimization

  22. Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks

    Apr 6, 2026Anas Jnini, Elham Kiyani, Khemraj Shukla +5Physics-Informed MLNeural PDE Solvers

  23. Inversion-Free Natural Gradient Descent on Riemannian Manifolds

    Apr 3, 2026Dario Draca, Takuo Matsubara, Minh-Ngoc TranNatural Gradient MethodsRiemannian Optimization

  24. Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method

    Apr 1, 2026Samuel Bright-Thonney, Thomas R. Harvey, Andre Lukas +1Deep Learning OptimizationNatural Gradient Methods

  25. Inverse-Free Sparse Variational Gaussian Processes

    Apr 1, 2026Stefano Cortinovis, Laurence Aitchison, Stefanos Eleftheriadis +1Sparse Gaussian ProcessesGaussian Processes

  26. Natural Gradient Descent for Online Continual Learning

    Mar 21, 2026Joe Khawand, David ColliauxKronecker-Factored ApproximationOnline Continual Learning

  27. Riemannian Optimization in Modular Systems

    Mar 4, 2026Christian Pehle, Jean-Jacques SlotineContraction TheoryRiemannian Optimization