cs.LGMar 13, 2025

Langevin Monte-Carlo Provably Learns Depth Two Neural Nets at Any Size and Data

Authors: Dibyakanti Kumar, Samyak Jha, Anirbit Mukherjee

Organizations: Department of Computer Science, The University of Manchester, UK · Department of Mathematics, Indian Institute of Technology, Bombay, India

Abstract

In this note we prove that the Langevin Monte Carlo (LMC) algorithm can minimize the population risk of appropriately regularized depth-2 nets with no restrictions on the size of the net and for any choice of training data --- and we give non-asymptotic convergence rates for it. We achieve this via showing that in q-Renyi divergence, the iterates of LMC converge to the Gibbs distribution of certain Frobenius norm regularized losses for these nets --- in both classification and regression settings. The key technicality of the work is two-fold (a) to keep the amount of regularization needed be independent of the size of the net and (b) to obtain convergence rates that scale with just the Poincare constant of the corresponding Gibbs measure. This work synthesizes several recent observations about isoperimetry conditions under which LMC converges and that two-layer neural loss functions can always be regularized by a constant amount such that they satisfy the Villani conditions, and thus their Gibbs measures satisfy a Poincare inequality.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Global Convergence in Deep Networks via the Second Law of Thermodynamics

    Oct 17, 2026Matan Tsipory, Ofir Gaash, Itamar Harel +2Neural Network Training DynamicsStochastic Optimization Convergence

  2. Controlled Langevin Dynamics for Sampling of Feedforward Neural Networks Trained with Minibatches

    Mar 16, 2026Alessandro Zambon, Francesca Caruso, Riccardo Zecchina +1Stochastic Gradient Langevin DynamicsNeural Network Training Dynamics