Deep Learning Theory through the Lens of Statistical Physics: A Topical Review
Authors: Wei Huang, Youjin Deng
Organizations: RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan · The Institute of Statistical Mathematics, Tokyo, Japan · Hefei National Research Center for Physical Sciences at the Microscale and School of Physical Sciences, University of Science and Technology of China, Hefei 230026, China · Hefei National Laboratory, University of Science and Technology of China, Hefei 230088, China · Shanghai Research Center for Quantum Science and CAS Center for Excellence in Quantum Information and Quantum Physics, University of Science and Technology of China, Shanghai 201315, China
The recent success of foundation models and deep learning has far outpaced our theoretical understanding of how large-scale neural networks learn and generalize. Statistical physics provides a natural language for describing such high-dimensional, nonlinear, and strongly coupled systems, offering concepts such as mean-field theory, phase transitions, energy landscapes, and stochastic dynamics. In this topical review, we adopt a statistical-physics way of thinking -- from concepts including phases and criticality, and free-energy viewpoints, to methods such as kernel and Gaussian-process limits, mean-field and dynamical mean-field theories, random-matrix tools, and Fokker-Planck formalisms, and to a mindset emphasizing universality, scaling, and tractable limits with explicit validity regimes. We survey recent progress on understanding modern deep learning through this lens, including mean-field analyses of wide networks, spectral and geometric perspectives on loss landscapes, stochastic-gradient-based training viewed as a dynamical process, and phase-transition analogies that organize phenomena such as overparameterization, generalization, and representation change. Our goal is to synthesize these threads into a coherent framework that clarifies what each approximation explains, when it is expected to hold, and how apparently competing narratives relate or diverge. Together, these perspectives form a unified physical framework that views representation learning as a collective phenomenon in high-dimensional systems, bridging deterministic and stochastic regimes of gradient-based optimization.
Figures & tables
Figure 1: Relationship map of the review. Three theoretical pillars—loss-landscape geometry (Sec. II ), infinite-width mean-field limits (Sec. III ), and stochastic learning dynamics (Sec. IV )—are connected by shared statistical-physics concepts: spectral universality, mean-field stochastic equations, and noise-driven escape from metastable minima. Abbreviations are listed in the sidebar; see also Table 1 for the full statistical-physics dictionary.
Deep Learning Concept
Statistical Physics Analogy
Physical Interpretation & Role
Loss function L(θ)
Energy H(σ)
The potential landscape governing the system’s dynamics.
Global minima correspond to ground states.
Parameters θ
Microscopic degrees of freedom
The interacting variables (e.g., spins in a spin glass)
that define the system’s microscopic configuration.
Solutions occupying large volume in parameter space,
Table 1: A statistical-physics dictionary for deep learning. This table summarizes the correspondence between key concepts in deep learning and their analogies in statistical mechanics. These mappings provide the conceptual basis for understanding phenomena such as flat minima selection, phase transitions, and non-equilibrium dynamics in overparameterized networks.
Figure 2: Spectral evolution of the Hessian matrix at critical points. The plots show the spectral density ρ(λ) for different parameter-to-data ratios ϕ and energy levels ϵ , derived using random matrix theory (reproduced from [ 48 ] ). As the energy ϵ increases, the spectrum transitions from a bulk with positive outliers (resembling the Marchenko–Pastur distribution, associated with stable minima) to a broader, semi-circular distribution spanning negative eigenvalues (associated with unstable saddle points). This illustrates the landscape’s transition from an ordered phase to a disordered phase in the high-dimensional parameter space.
Figure 3: Schematic visualization of mode connectivity in the loss landscape. The plot illustrates a 2D projection of the loss surface containing two independent solutions, θ1 and θ2 . A linear interpolation between them (white dashed line) traverses a region of high loss (energy barrier). However, consistent with the phenomenon of mode connectivity [ 51 , 52 ] , a non-linear path exists (solid red curve) along which the loss remains consistently low. This topology suggests that distinct minima belong to a single connected manifold of solutions, facilitating exploration by stochastic optimization algorithms.
Figure 4: Comparison of optimization trajectories under mean-field and NTK parameterizations in a two-feature teacher model. The first-layer weights are projected onto the teacher w1∗ and w2∗ . Under mean-field training, neurons move far from initialization and align with the teacher directions, indicating feature learning and specialization. Under NTK training, the parameters remain close to initialization, consistent with lazy training and limited representation change. This plot is adapted from [ 127 ] .
Figure 5: Schematic illustration of temperature-controlled exploration on a non-convex landscape using overdamped Langevin dynamics. Both panels use the same step size, while the effective temperature differs. At higher temperature, the trajectory explores a broader region of the landscape, reflecting enhanced barrier crossing and faster escape from metastable neighborhoods. At lower temperature, the motion remains more localized and stays within the same basin for a longer time. This figure is intended as a qualitative illustration of metastability and first-passage dynamics rather than a faithful simulation of mini-batch SGD.
Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investigate the key features and surprises of deep learning from a physics-informed perspective, taking care to point out and justify where possible the many choices inherent in constructing a deep learning model. In particular, we review the phenomenon of neural scaling laws and discuss their interplay with the constraints and inductive biases which may be present when applying machine learning to problems in physics.
Itay Lavie, Noam Levi, Yonatan Kahn
John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · Tel Aviv University, Tel Aviv, Israel · Department of Physics, University of Toronto and Vector Institute, Toronto, ON, Canada
These lecture notes introduce some topics of classical statistical physics, particularly those that are relevant for neural networks and deep learning. Statistical physics is treated as a branch of probability theory or statistics, with the goal of making concepts such as phase transitions and the renormalization group accessible to readers without prior knowledge of physics. We introduce the Boltzmann-Gibbs distribution and the thermodynamic potentials on a finite configuration space, notably for Ising spins and spin-glass models on a lattice, and then define phase transitions as discontinuities that arise in the limit that the number of lattice points goes to infinity. We further introduce Hopfield networks and Boltzmann machines, which are governed by the same energy function as spin-glass models, and discuss the learning algorithm for restricted Boltzmann machines. In this algorithm hidden neurons are integrated out as in the renormalization group. Finally, modern deep learning is introduced, whose early developments were in part motivated by restricted Boltzmann machines in that they carry many layers of hidden neurons. A description of large language models is given.
Olaf Hohm
Institute for Physics, Humboldt University Berlin, Zum Großen Windkanal 2, D-12489 Berlin, Germany
In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: (a) solvable idealized settings that provide intuition for learning dynamics in realistic systems; (b) tractable limits that reveal insights into fundamental learning phenomena; (c) simple mathematical laws that capture important macroscopic observables; (d) theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems behind; and (e) universal behaviors shared across systems and settings which clarify which phenomena call for explanation. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic relationship between learning mechanics and mechanistic interpretability. We also review and address common arguments that fundamental theory will not be possible or is not important. We conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further introductory materials, perspectives, and open questions at learningmechanics.pub.
Jamie Simon, Daniel Kunin, Alexander Atanasov +11
UC Berkeley and Imbue · UC Berkeley · Harvard University +6