A statistical physics framework for optimal learning

Francesca Mignacco; Francesco Mori

cond-mat.dis-nnJul 10, 2025

A statistical physics framework for optimal learning

Authors: Francesca Mignacco, Francesco Mori

Organizations: 1Joseph Henry Laboratories of Physics, Princeton University, Princeton, NJ 08544, USA · 2Graduate Center, City University of New York, New York, NY 10016, USA · Center of Mathematical Sciences and Applications, Harvard University, Cambridge, MA 02138, USA

Abstract

Learning is a complex dynamical process shaped by a range of interconnected decisions. Careful design of hyperparameter schedules for artificial neural networks or efficient allocation of cognitive resources by biological learners can dramatically affect performance. Yet, theoretical understanding of optimal learning strategies remains sparse, especially due to the intricate interplay between evolving metaparameters and nonlinear learning dynamics. The search for optimal protocols is further hindered by the high dimensionality of the learning space, often resulting in predominantly heuristic, difficult to interpret, and computationally demanding solutions. Here, we combine statistical physics with control theory in a unified theoretical framework to identify optimal learning protocols in prototypical neural network models. In the high-dimensional limit, we derive closed-form ordinary differential equations that track online stochastic gradient descent through low-dimensional order parameters. We formulate the design of learning protocols as an optimal control problem directly on the dynamics of the order parameters with the goal of minimizing the generalization error. This formulation encompasses a variety of learning scenarios, optimization constraints, and control budgets. We apply it to representative cases, including optimal curricula, adaptive dropout regularization and noise schedules in denoising autoencoders. We find nontrivial yet interpretable strategies highlighting how optimal protocols mediate learning trade-offs. Our results establish a principled foundation for understanding and designing optimal protocols and suggest a path toward a theory of meta-learning grounded in statistical physics.

Explore similar work

May 19, 2026cs.LG

Training Neural Networks with Optimal Double-Bayesian Learning

Backpropagation with gradient descent is a common optimization strategy employed by most neural network architectures in machine learning. However, finding optimal hyperparameters to guide training has proven challenging. While it is widely acknowledged that selecting appropriate parameters is crucial for avoiding overfitting and achieving unbiased outcomes, this choice remains largely based on empirical experiments and experience. This paper presents a new probabilistic framework for the learning rate, a key parameter in stochastic gradient descent. The framework develops classic Bayesian statistics into a double-Bayesian decision mechanism involving two antagonistic Bayesian processes. A theoretically optimal learning rate can be derived from these two processes and used for stochastic gradient descent. Experiments across various classification, segmentation, and detection tasks corroborate the practical significance of the theoretically derived learning rate. The paper also discusses the ramifications of the proposed double-Bayesian framework for network training and model performance.

Vy Bui, Hang Yu, Karthik Kantipudi +2

Learning Rates Stochastic Gradient Descent Backpropagation Bayesian Neural Networks Bayesian Neural Network

Ask PDF

Sep 10, 2026cs.AI

A Function-Space Approach to the Statistical Mechanics of Learning Dynamics

Deep neural networks exhibit regular macroscopic behavior despite highly nonlinear dynamics in vast parameter spaces. We develop a statistical-mechanical description of learning directly in function space, treating parameter configurations as microscopic realizations and functions with their dynamical operators as macroscopic variables. For mean-squared loss, the exact error dynamics are governed by the learning operator

M=JJ^\ast

. Combining the dynamical Boltzmann weight of the conditional stochastic dynamics with the parameter-space density of states, whose local curvature defines a statistical operator

B

, and integrating over local fluctuations yields

\Phi_{\mathrm{fluc}}(M;B)=\frac{\sigma_\xi^2}{2}\log\det(M^{-1}+B)+\mathrm{const}.

At fixed spectrum, this term is rotationally stationary when

[M,B]=0

, is minimized by pairing large eigenvalues of

M

with small eigenvalues of

B

, and generates a local restoring contribution against rotational mismatch. For ReLU-type function spaces under mild stable statistical conditions,

B=\sigma_\xi^2L^\ast\mathcal K L

, where

L

measures coarse-grained second-order structure. Thus the low-

B

sector corresponds, up to bounded anisotropy of

\mathcal K

, to low structural curvature, implying a preference for faster relaxation along smooth, data-adaptive directions. These results identify function space as a natural macroscopic level for studying stable collective organization in learning.

Yizhou Zhang, Weichen Wu, Lun Du +1

Learning Dynamics Statistical Mechanics Weight-Space Boltzmann Distributions

Ask PDF

Sep 14, 2026cond-mat.stat-mech

Bridging Control, Inference, Transport, and Thermodynamics: From Theory to Applications in Learning

The last decade has seen the development of powerful methods for learning complex structure from high-dimensional data. These advances have brought to the foreground fundamental connections between subdisciplines of physics, applied mathematics, and machine learning. In this review, we bring together some of these ideas, often expressed in different languages, to highlight a conceptual thread that links five distinct fields: control theory, optimal transport, probabilistic inference, non-equilibrium thermodynamics, and machine learning. A common theme is the optimization of free-energy-like functionals under dynamical or statistical constraints. We offer a guided tour through this thread and present selected applications in reinforcement learning, variational inference, and generative modeling. The review does not assume prior familiarity with these topics, and begins with principles originating from physics.

Emmy Blumenthal, Nikolas Claussen, Benjamin Eysenbach +4

Statistical Mechanics Free Energy Principle Thermodynamics Variational Inference Learning Dynamics Optimal Transport+3

Ask PDF