Overview

Natural gradient methods study how to use the geometry of a model’s probability distributions, commonly represented by the Fisher information matrix, to choose updates that are less dependent on parameterization. They matter when ordinary gradients are poorly scaled or constrained, including neural-network training, policy optimization in reinforcement learning, variational inference, and optimization on manifolds. Approaches range from exact Fisher-based updates to Kronecker-factored, low-rank, and matrix-free approximations, alongside Riemannian methods and convergence analyses for policy learning and other structured problems. A central tension is that curvature-aware updates can improve convergence, but estimating and applying them can be costly or noisy, and useful step sizes and reliable scaling remain challenging.

Latest papers

CardsList

33 papers3 papers in the last four weeks, down 40% on the four weeks before. 0.0% of all new papers.

  1. TENG-BC: Time-Evolving Natural Gradient for High-Accuracy Neural PDE Solvers with General Boundary Conditions

    Feb 28, 2026Hongjie Jiang, Di LuoNeural PDE SolversPDE Solving

  2. Correlations Are Ruining Your Gradient Descent

    Jul 15, 2024Joshua Offergeld, Nasir AhmadNeural Network OptimizationNatural Gradient Methods

  3. Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients

    Mar 28, 2024Johannes Müller, Semih Çaycı, Guido MontúfarPolicy GradientOptimization Convergence Analysis