cs.LGOct 30, 2026

Infinite ResNets are Gaussian Mean ODEs

Authors: Louis-Pierre Chaintron, Lénaïc Chizat, Javier Maass

Organizations: Institute of Mathematics, École Polytechnique Fédérale de Lausanne (EPFL), Switzerland · Department of Mathematics, National University of Singapore, Singapore

Abstract

Understanding the training dynamics of deep neural networks in large-scale limits is a question both of foundational and of practical importance. Prior works have obtained such descriptions through dynamical mean field theory (DMFT), yielding a system of non-Markovian dynamics that are challenging to analyze and that obscure the optimization structure of the original dynamics. In this paper, we obtain instead a simple description of the large-scale limit training dynamics for ResNets in standard shape scalings and in the 1/depth1/\text{depth} branch scaling that allows maximal (local) feature updates. For a model trained with gradient descent (GD) from a random initialization, the limit dynamics is simply GD on a Neural Mean ODE with Gaussian initial weights. Our analysis covers population or empirical risk GD in the joint limit of large width, depth and embedding, input, and output dimensions. We obtain approximation scaling laws that depend on the shape of the ResNet and the covariance spectrum of the data. We also illustrate the flexibility of our approach by discussing Mixture of Expert models, where likewise the limit admits an intuitive (nested) Mean ODE description.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit

    Mar 18, 2026Louis-Pierre Chaintron, Lénaïc Chizat, Javier MaassDynamical Mean-Field TheoryNeural Network Training Dynamics

  2. Early learning in large residual networks

    Oct 22, 2026Vladimir Narovlansky, Boris HaninNeural Network Training DynamicsResidual Learning

  3. Width and Depth Limits Commute During Training in Linear Residual Networks

    Oct 22, 2026Ruijia Zhang, Soufiane HayouNeural Network Training DynamicsDeep Linear Networks