Deep Learning Theory through the Lens of Statistical Physics: A Topical Review
Organizations: RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan · The Institute of Statistical Mathematics, Tokyo, Japan · Hefei National Research Center for Physical Sciences at the Microscale and School of Physical Sciences, University of Science and Technology of China, Hefei 230026, China · Hefei National Laboratory, University of Science and Technology of China, Hefei 230088, China ·
Abstract
The recent success of foundation models and deep learning has far outpaced our theoretical understanding of how large-scale neural networks learn and generalize. Statistical physics provides a natural language for describing such high-dimensional, nonlinear, and strongly coupled systems, offering concepts such as mean-field theory, phase transitions, energy landscapes, and stochastic dynamics. In this topical review, we adopt a statistical-physics way of thinking -- from concepts including phases and criticality, and free-energy viewpoints, to methods such as kernel and Gaussian-process limits, mean-field and dynamical mean-field theories, random-matrix tools, and Fokker-Planck formalisms, and to a mindset emphasizing universality, scaling, and tractable limits with explicit validity regimes. We survey recent progress on understanding modern deep learning through this lens, including mean-field analyses of wide networks, spectral and geometric perspectives on loss landscapes, stochastic-gradient-based training viewed as a dynamical process, and phase-transition analogies that organize phenomena such as overparameterization, generalization, and representation change. Our goal is to synthesize these threads into a coherent framework that clarifies what each approximation explains, when it is expected to hold, and how apparently competing narratives relate or diverge. Together, these perspectives form a unified physical framework that views representation learning as a collective phenomenon in high-dimensional systems, bridging deterministic and stochastic regimes of gradient-based optimization.
Figures & tables
| Deep Learning Concept | Statistical Physics Analogy | Physical Interpretation & Role |
|---|---|---|
| Loss function | Energy | The potential landscape governing the system’s dynamics. |
| Global minima correspond to ground states. | ||
| Parameters | Microscopic degrees of freedom | The interacting variables (e.g., spins in a spin glass) |
| that define the system’s microscopic configuration. | ||
| SGD noise | Effective temperature | Stochastic fluctuations regulating exploration–exploitation. |
| Flat minima | High-entropy basin ( term) | Solutions occupying large volume in parameter space, |
Statistical Physics of Learning
Statistical physics of learning uses tools from physics to explain how data, model structure, and training dynamics shape learning and generalization.
See all 60 papers