Statistical Physics of Learning

Follow this topic Ask about this topic

The field studies learning systems as high-dimensional statistical-mechanical models, asking how data and optimization produce generalization, memorization, and distinct training regimes. Common approaches include teacher–student models, replica and dynamical mean-field methods, Gibbs distributions, and order parameters that track quantities such as overlap, loss, and representation structure. Researchers use these tools to analyze phase transitions in settings including perceptrons, associative memories, energy-based models, attention, and neural networks, alongside finite-size tests on phenomena such as grokking and double descent. A central challenge is determining when phase-transition explanations and equilibrium theories predict behavior in realistic networks trained for finite time, and which observables can reliably distinguish mechanisms.

Latest papers

53 papers9 papers in the last four weeks, up 29% on the four weeks before. 0.1% of all new papers.

Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking
Oct 26, 2026Min Chen, Junqi Wang, Zeguan Wu +2Grokking Phase Transitions in Neural Networks
Department of Computer Science, University of Pittsburgh, Pittsburgh, PA, USA
Generalization in Learning: Eyring Formula and Predator-prey Model
Oct 25, 2026S. V. Kozyrev, I. A. Lopatin, Alexander PechenNeural Network Generalization Statistical Physics of Learning
Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia · Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, Russia
Generalization properties of the token perceptron: the role of embedding and vocabulary
Oct 23, 2026Dario Bocchi, Matteo Negri, Flavio Nicoletti +1Language Model Generalization Representation Geometry in Language Models
Physics Department, University of Rome Sapienza, Rome, Italy · Institute of Nanotechnology, CNR-NANOTEC, Rome, Italy · Laboratoire de Physique Théorique et Modélisation, CY Cergy Paris Université +1
Phase Transitions in Learning Driven by Extremal Data
Oct 21, 2026Ethan Levien, Trevor GrandPreMultiple Instance Learning Phase Transitions in Neural Networks
Department of Mathematics, Dartmouth College, Hanover, NH 03755, USA · Department of Physics, Washington University in St. Louis, St. Louis, MO 63130, USA · National Institute for Theory and Mathematics in Biology, Northwestern University and The University of Chicago, Chicago, IL, USA
Sample-efficient learning with Dense Associative Memories
Oct 19, 2026Andrew Lucas, Dmitry KrotovAssociative Memory Data-Efficient Learning
Department of Physics and Center for Theory of Quantum Matter, University of Colorado, Boulder, CO 80309, USA · Dynamical Mind, Cambridge, MA 02141, USA
Mesoscopic learning dynamics of extensive-width neural networks
Oct 16, 2026Shu Zhou, Jean Barbier, Sebastian GoldtNeural Network Training Dynamics Representation Learning
International School for Advanced Studies (SISSA) Via Bonomea 265, 34136 Trieste, Italy · The Abdus Salam International Centre for Theoretical Physics (ICTP) Strada Costiera 11, 34151 Trieste, Italy
Attention over noisy examples: generalization and memorization in the exponential Hopfield network
Oct 15, 2026Francesco AlemannoHopfield Networks Neural Network Memorization
On the Statistical Mechanics of Attention
Oct 12, 2026Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su +1Transformer Attention Attention Mechanisms
Center for Foundation Models and Generative AI, Northwestern University Department of Computer Science, Northwestern University · Department of Computer Science, College of Connected Computing, Vanderbilt University · Department of Physics, National Taiwan University +1
On the Theoretical Limits of Learning Unconstrained Interatomic Potentials
Oct 10, 2026Prajwal Ananth, Shuwen YueMachine Learning Interatomic Potentials Molecular Dynamics Simulation
Center for Applied Mathematics, Cornell University, Ithaca, NY · R. F. Smith School of Chemical and Biomolecular Engineering, Cornell University, Ithaca, NY
Asymptotic Analysis of Empirical Risk Minimization on Entry-wise i.i.d. Heavy-Tailed Data
Oct 6, 2026Kaito Takanami, Takashi Takahashi, Yoshiyuki KabashimaStatistical Physics of Learning High-Dimensional Asymptotics
Graduate School of Science, The University of Tokyo · Institute for Physics of Intelligence, The University of Tokyo · Trans-Scale Quantum Science Institute, The University of Tokyo
A Flatness-Generalization Relation in the Teacher-Student Tree-Committee Machine
Sep 25, 2026Brandon Livio Annesi, Davide Straziota, Enrico Maria MalatestaFlat Minima Neural Network Generalization
Department of Quantitative Life Sciences, ICTP Trieste, Italy
An Analytical Theory of Auxiliary Learning
Sep 24, 2026Federico Milanesio, Alessandro Ingrosso, Matteo OsellaStatistical Physics of Learning Neural Network Generalization
Department of Physics and INFN University of Turin Via Giuria 1, 10125 Turin, Italy · Donders Centre for Neuroscience Radboud University Heyendaalseweg 135, 6525 AJ Nijmegen, The Netherlands
An Exploratory Replica-Overlap Probe of the Grokking Transition
Sep 22, 2026A. C. Opus, J. Q. LuStatistical Physics of Learning Neural Network Training Dynamics
Department of Physics, University of Puerto Rico, Mayagüez, PR 00680, USA
Do Quantum Models Scale Like LLMs?
Sep 17, 2026David S. Berman, Ying-Jer Kao, Roger G. Melko +1Neural Scaling Laws Transformer
Centre for Theoretical Physics, Queen Mary University of London, Mile End Road, London E1 4NS, United Kingdom · Department of Physics, National Taiwan University, Taipei 10607, Taiwan · Department of Physics and Astronomy, University of Waterloo, Waterloo, Ontario N2L 3G1, Canada +1
Double descent is the principle of least action
Sep 16, 2026Congzhou M ShaDouble Descent Statistical Physics of Learning
Penn Medicine Doylestown Hospital, 595 W State St, Doylestown, PA 18901, United States
A Function-Space Approach to the Statistical Mechanics of Learning Dynamics
Sep 9, 2026Yizhou Zhang, Weichen Wu, Lun Du +1Statistical Physics of Learning Representation Learning
Variational AI · The Voleon Group · Independent Researcher +1
Memory as an Energy Landscape---Hopfield
Sep 2, 2026Nima DehghaniHopfield Networks Associative Memory
McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, USA · The NSF AI Institute for Artificial Intelligence and Fundamental Interactions (IAIFI), Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, USA
Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance
Aug 29, 2026Kazuyuki Hara, Hideitsu HinoKnowledge Distillation Model Misspecification
Nihon University · The Institute of Statistical Mathematics
Landau theory of quenched criticality in linear in-context learning
Aug 28, 2026Daesik Kim, Sumin Choi, Hyojae Jeon +1In-Context Learning Double Descent
Department of Physics, Sungkyunkwan University, Suwon 16419, South Korea · Center for Quantum Dynamics of Angular Momentum, Pohang University of Science and Technology, Pohang 37673, Korea
Dataset Complexity Shapes Finite-Distance Loss Geometry in Neural Networks
Aug 23, 2026Jaeyong Bae, Hawoong JeongLoss Landscape Loss Landscape Geometry
Department of Physics, Korea Advanced Institute of Science and Technology, Daejeon, Korea · Center of Complex Systems, Korea Advanced Institute of Science and Technology, Daejeon, Korea
Learning reshapes power-law anisotropy in internal representations
Aug 15, 2026Asahi Nakamuta, Jun-nosuke TeramaeNeural Representation Geometry Representation Learning
Graduate School of Informatics, Kyoto University
Thermodynamics of Learning: A Typed Four-Component Accounting of Memory, Fit, and Value
Aug 13, 2026Akihito SudoStatistical Physics of Learning Distribution Shift
ZeroStruct Inc.
Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networks
Aug 6, 2026Björn Ladewig, Ibrahim Talha Ersoy, Karoline WiesnerStatistical Physics of Learning Representation Learning
Institute of Physics and Astronomy, University of Potsdam
Variational Bounds for Perceptron Learning from Structured Data
Aug 5, 2026Francesco Camilli, Pierluigi Contucci, Federica Gerace +1Statistical Physics of Learning Statistical Learning Theory
Department of Mathematics, Alma Mater Studiorum – Universit`a di Bologna, Piazza di Porta San Donato 5, 40126 Bologna, Italy
Statistical Mechanics of Learning on Product Wasserstein Manifolds
Aug 2, 2026Srinivasa Rao P Vangmayi P ReddyStatistical Physics of Learning Wasserstein Gradient Flows
Curlvee Technolabs, India; Former Scientist, C-DAC.
Broken Ergodicity and the Violation of the Fluctuation-Dissipation Theorem Lead to Generalization Beyond Overfitting in Machine Learning
Jul 5, 2026Chan Li, Nigel GoldenfeldDynamical Mean-Field Theory Statistical Physics of Learning
Department of Physics, University of California San Diego, 9500 Gilman Drive, La Jolla, California 92093, USA. · Halıcıoğlu Data Science Institute, University of California San Diego, 9500 Gilman Drive, La Jolla, California 92093, USA
Explaining Machine Learning and Memorization with Statistical Mechanics
Jun 30, 2026Robin TheriaultStatistical Physics of Learning Neural Network Memorization
Faculty of Science
Informational Frustration in Neural Manifolds: Shannon Bottlenecks and the Limits of Learnability
Jun 29, 2026Srinivasa Rao P., Vangmayi P ReddyStatistical Physics of Learning Neural Network Generalization
Curlvee Technolabs, India, Former Scientist, C-DAC · Former Scientist, C-DAC · Indian Institute of Technology Madras, India
Data-Driven Energy-Based Learning via Gibbs Measures on Hierarchical Structures
Jun 29, 2026L. U. Abdullaev, F. Herrera, U. A. Rozikov +1Statistical Physics of Learning Phase Transitions
National University of Singapore, Singapore · Department of Computer Science and Artificial Intelligence, University of Granada, E-18071 Granada, Spain · V.I. Romanovskiy Institute of Mathematics, Uzbekistan Academy of Sciences, 9, Universitet str., 100174, Tashkent, Uzbekistan +3
Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networks
Jun 15, 2026Ibrahim Talha Ersoy, Karoline WiesnerStatistical Physics of Learning Neural Network Generalization
Complexity Science Group, Institute of Physics and Astronomy, University of Potsdam, Potsdam, Germany

Oct 26, 2026J/K move · Enter open · S save

Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking

Min Chen, Junqi Wang, Zeguan Wu, Wenhao Huang+1

Department of Computer Science, University of Pittsburgh, Pittsburgh, PA, USA

Grokking Phase Transitions in Neural Networks Neural Network Training Dynamics Statistical Physics of Learning

Open paper AskPDF arXiv

Grokking is a delayed transition from memorization to generalization in deep neural networks (NNs). We examine whether scaling and renormalization group (RG) theories can describe this dynamics, establishing a quantitative physical framework for neural generalization transitions. For modular addition on a two-layer NN, an ensemble of trained models defines an order parameter (the squared weight norm) and fluctuation proxies for the heat capacity and the susceptibility, with training epoch and weight decay serving as thermal and field control coordinates. To bridge neural architectures to real-space RG, we construct an exact coupling matrix via a second-order expansion of the loss with respect to per-neuron norms and use its Fiedler vector to define an effective geometry. This spectral ordering yields a monotonic spatial decay of coupling strength that is absent under random permutations. We then execute real-space RG by block-spin aggregating adjacent neurons and parameterizing the coupling matrix into off-diagonal band strengths. Linearizing this local RG map yields relevant scaling exponents and five derived critical exponents, with direct empirical estimates matching RG predictions within 1-10%. Overall, our framework provides a consistent effective description of grokking, suggesting the scaling and renormalization group formalism as a tool to investigate NN dynamics.