Statistical Properties of Training & Generalization

Itay Lavie; Noam Levi; Yonatan Kahn

stat.MLJun 18, 2026

Statistical Properties of Training & Generalization

Authors: Itay Lavie, Noam Levi, Yonatan Kahn

Organizations: John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · Tel Aviv University, Tel Aviv, Israel · Department of Physics, University of Toronto and Vector Institute, Toronto, ON, Canada

Abstract

Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investigate the key features and surprises of deep learning from a physics-informed perspective, taking care to point out and justify where possible the many choices inherent in constructing a deep learning model. In particular, we review the phenomenon of neural scaling laws and discuss their interplay with the constraints and inductive biases which may be present when applying machine learning to problems in physics.

Explore similar work

On the Invariance and Generality of Neural Scaling Laws
May 8, 2026Xing Han, Ziyin Liu, Suchi Saria +1Scaling Laws Invariance
Lecture Notes on Statistical Physics and Neural Networks
May 7, 2026Olaf HohmStatistical Mechanics Two-Dimensional Ising Model
Interpretability and Generalization Bounds for Learning Spatial Physics
Jun 18, 2025Alejandro Francisco Queiruga, Theo Gutman-Solo, Shuai JiangPhysics-Informed Machine Learning Generalization Bounds

May 8, 2026 · cs.LGJ/K move · Enter open · S save

On the Invariance and Generality of Neural Scaling Laws

Xing Han, Ziyin Liu, Suchi Saria, Paul Pu Liang

Johns Hopkins University · Baltimore, MD 21218, USA · NTT Research, MIT · Cambridge, MA 02139, USA · Johns Hopkins University, Bayesian Health

Neural scaling laws establish a predictable relationship between model performance and data or compute, offering crucial guidance for resource allocation in new domains and tasks. Yet such laws are most needed precisely where they are hardest to obtain: fitting one for a new model task pair demands expensive sweeps that typically exhaust the very compute budget the law is meant to economize. This paper poses the research question of how to develop generalizable scaling laws: laws fit once on a well-resourced source domain and reliably transported to new domains where running a full sweep is infeasible, which requires a fundamental understanding of when and why scaling properties change. We address this by identifying the right invariants: scaling laws are preserved under bijective (information-preserving) transformations of the data and modified in predictable, information-theoretically grounded ways under non-bijective transformations that lower its information resolution

ρ

: a single axis along which a law fit in one domain can be transported to another. We validate this across language, vision, and speech, and demonstrate two cross-domain applications: predicting scaling for language models trained on electronic health records from laws fit on general text, and predicting time-series classification scaling under varying levels of noise injection, recovering the data-scaling exponents to within

3\%

error.

Scaling Laws Invariance Exponents

Open paper Ask about it PDF

Abstract

Explore similar work

On the Invariance and Generality of Neural Scaling Laws

Lecture Notes on Statistical Physics and Neural Networks

Interpretability and Generalization Bounds for Learning Spatial Physics