AI Scaling Laws

Follow this topic Ask about this topic

AI scaling laws study how increasing resources changes model performance and help predict what to train when compute, data, or memory is limited. Researchers fit power-law relationships and compute-optimal rules, extending Kaplan- and Chinchilla-style methods to mixture-of-experts models, multimodal data, and domain-specific systems. Results show that the best balance of model size, data, and compute can vary with architecture and task, while data quality, repetition, and cross-modal interference also affect gains. A central open question is how reliably training loss and small-scale experiments predict downstream capabilities and the best resource allocation in deployment.

Latest papers

27 papers5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Data Density Scaling Laws for Image Self-Distillation
Oct 13, 2026Alvard Barseghyan, Ani Vanyan, Hakob Tamazyan +1AI Scaling Laws Self-Supervised Knowledge Distillation
YerevaNN and Yerevan State University, Yerevan, Armenia
The Singularity Has a Distribution: Exact Timing Laws and Detection Limits for a Stochastic Intelligence Explosion
Oct 13, 2026Xiaoyu Li, Zheng Gao, Chiwun Yang +1AI Scaling Laws
University of New South Wales · City University of Hong Kong · University of Texas at Austin
Scaling Laws for Multimodal Data Mixtures
Oct 10, 2026Aditi Khandelwal, Ayush K. Tarun, Yixuan Xu +5Multimodal Pretraining AI Scaling Laws
MILA - Quebec AI Institute, McGill University · EPFL · ETH +1
Training and Scaling Compute-Optimal Physiological Waveform Foundation Models
Oct 5, 2026Pingzhi Li, Jie Peng, Shuqing Luo +2Time-Series Foundation Models AI Scaling Laws
University of North Carolina at Chapel Hill · Foci Labs
Scaling Laws for Looped Mixture of Experts
Sep 30, 2026Yanbei Chen, Anirudh Goyal, Raghuraman KrishnamoorthiMixture-of-Experts Language Models Language Model Scaling Laws
Meta AI
The Capability Manifold and ML Scaling Laws
Sep 23, 2026Syed Ali Raza Zaidi, Maryam HafeezScaling Laws Neural Scaling Laws
SPRINT Lab University of Leeds Leeds, LS2 9JT, UK.
When does a scaling result justify a different allocation? A critical review of resource-allocation evidence for AI systems
Sep 13, 2026Seyed Morteza EmadiAI Agent Evaluation Inference-Time Compute Allocation
Kenan-Flagler Business School University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
Scaling an Autoregressive Transformer for Single-Cell Generation
Aug 3, 2026Aleksandr Sharipov, Yusif Mukhtarov, Igor MolybogTransformer Autoregressive Generation
HawAII, University of Hawai’i at Manoa
The Scaling Paradox in Human-AI Collaboration
Aug 1, 2026Anyan Qi, Mengxin WangHuman-AI Decision Making Human-AI Collaboration
Naveen Jindal School of Management, University of Texas at Dallas, USA
Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
Jul 29, 2026Alexi Gladstone, Heng Ji, Yilun DuImage Generation Generative Modeling
1UIUC · 2Harvard
Predict before you train: Scaling Laws for particle physics foundation models
Jul 25, 2026Jan-Lucas Uslu, Benjamin Nachman, Christopher ReHigh-Energy Physics AI Scaling Laws
Department of Applied Physics, Stanford University, Stanford, CA 94305, USA · Fundamental Physics Directorate, SLAC National Accelerator Laboratory, Menlo Park, CA 94025, USA · Department of Physics, Stanford University, Stanford, CA 94305, USA +1
TESSERA v2: Scaling Pixel-wise Earth Foundation Models
Jul 4, 2026Zhengpeng Feng, Sadiq Jaffer, Ira Shokar +10Geospatial Foundation Models AI Scaling Laws
University of Cambridge · 2NVIDIA · 3dClimate Labs
Two AI Metrics Diverged: Will it Make All the Difference?
Jul 1, 2026Alex Fogelson, Zachary A. Brown, Hans Gundlach +2Language Model Scaling Laws AI Governance
MIT FutureTech, CSAIL, Cambridge, MA, USA.
An Empirical Study of Data Scale, Model Complexity, and Input Modalities in Visual Generalization
Jun 3, 2026Yidi ZhouluoNeural Network Generalization AI Scaling Laws
School of Medical Information and Artificial Intelligence, Shandong First Medical University, Taian 271000, Shandong, China
MobileMoE: Scaling On-Device Mixture of Experts
May 26, 2026Yanbei Chen, Hanxian Huang, Ernie Chang +5Mixture-of-Experts Language Models On-Device Language Model Inference
Meta AI
Pretraining Strategies and Scaling for ECG Foundation Models: A Systematic Study
May 12, 2026M A Al-Masud, Nils StrodthoffElectrocardiogram Foundation Models Self-Supervised Time Series Representation Learning
AI4Health Division Carl von Ossietzky Universität Oldenburg Oldenburg, Lower Saxony, Germany
Slicing and Dicing: Configuring Optimal Mixtures of Experts
May 12, 2026Margaret Li, Sneha Kudugunta, Danielle Rothermel +1Mixture-of-Experts Language Models Mixture of Experts
Paul G Allen School of Computer Science University of Washington Seattle, USA · Courant School of Data Science New York University New York, USA
A Limit Theory of Foundation Models: A Mathematical Approach to Understanding Emergent Intelligence and Scaling Laws
Apr 27, 2026Jun Shu, Junxiong Jia, Deyu Meng +1Neural Scaling Laws AI Scaling Laws
School of Mathematics and Statistics, Xi’an Jiaotong University.
Generalization and Scaling Laws for Mixture-of-Experts Transformers
Apr 10, 2026Mansour Zoubeirou a MayakiMixture-of-Experts Models AI Scaling Laws
Univ Lyon, Lyon 2, LIRIS UMR 5205 F-69676, Lyon, France
Data Scaling as Progressive Coverage of a Predictive Contribution Spectrum
Apr 5, 2026Zihui Song, Shihao Ji, Hongxi Li +2Language Model Scaling Laws AI Scaling Laws
Continued AI Scaling Requires Repeated Efficiency Doublings
Mar 30, 2026Chien-Ping LuAI Scaling Laws
Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
Mar 23, 2026Weilin Wan, Jingtao Han, Debing Zhang +2Mixture-of-Experts Models AI Scaling Laws
College of Computer Science and Artificial Intelligence, Fudan University · Hi Lab, Xiaohongshu Inc. · Project Numina +1
Modernizing Amdahl's Law: How AI Scaling Laws Shape Computer Architecture
Mar 21, 2026Chien-Ping LuHeterogeneous Computing AI Scaling Laws
Punctuated Equilibria in Artificial Intelligence: The Institutional Scaling Law and the Speciation of Sovereign AI
Mar 15, 2026Mark Baciak, Thomas A. Cellucci, Deanna M. FalkowskiAI Scaling Laws
Ekta Inc. · Georgetown University
Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
Mar 11, 2026Junzhuo Li, Peijie Jiang, Changxin Tian +3Mixture-of-Experts Models AI Scaling Laws
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · Ant Group
Scaling Laws of Global Weather Models
Feb 26, 2026Yuejiang Yu, Langwen Huang, Alexandru Calotoiu +1Scaling Laws Weather Forecasting
Department of Computer Science, ETH Zurich, Zurich, Switzerland.
Neural Scaling Laws for Boosted Jet Tagging
Feb 17, 2026Matthias Vigl, Nicole Hartman, Michael Kagan +1Neural Scaling Laws AI Scaling Laws
Technical University of Munich · SLAC National Accelerator Laboratory

Oct 13, 2026J/K move · Enter open · S save

Data Density Scaling Laws for Image Self-Distillation

Alvard Barseghyan, Ani Vanyan, Hakob Tamazyan, Hrant Khachatrian

YerevaNN and Yerevan State University, Yerevan, Armenia

AI Scaling Laws Self-Supervised Knowledge Distillation

Open paper AskPDF arXiv

Current scaling law formulations suggest that increasing the number of unique samples in a dataset improves model performance, while repeated exposure to the same samples provides diminishing value. In this work, we examine this assumption for image self-distillation models. Through systematic experiments across diverse datasets, namely Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M, we demonstrate that the effects of sample repetition on downstream performance are non-trivial and dataset-dependent. We identify settings where repeatedly training on a smaller random subset can outperform training on a larger set of unique samples. This finding highlights a limitation of existing scaling laws. To address this gap, we propose a modified version of a Chinchilla-style scaling law that explicitly accounts for two factors: the number of unique samples and repetition per sample. This formulation provides a framework for understanding data-compute trade-offs and improves prediction in some regimes where repetition plays a central role, although its benefits are not consistent across datasets. Finally, we examine how subset selection and augmentation affect downstream performance.