Language Model Scaling Laws

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 94

All topics

Toward Alignment Scaling Laws: A Framework and First Preregistered Measurements
Oct 6, 2026Jeremy CanaleLLM Alignment Language Model Safety Evaluation
Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models
Oct 5, 2026Joe DwyerLanguage Model Scaling Laws Efficient Language Model Training
ECPI University
What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining
Oct 4, 2026Yekun Chai, Haoyi XiongLanguage Model Pretraining Language Model Scaling Laws
FloatAI · ETH Zurich · Independent Researcher
Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining
Oct 4, 2026Ziyue WANG, T. KanamoriLanguage Model Pretraining Language Model Scaling Laws
Institute of Science Tokyo
A Systematic Analysis of the Predictive Power of LM Surprisal in Reading Chinese
Oct 4, 2026Hongao Zhu, Muxiaoqiao Xu, Yikang Liu +4Language Model Scaling Laws Language Modeling
Department of Linguistics, UC San Diego · School of Foreign Languages, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University +3
Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference
Oct 1, 2026Xinye Zhao, Yunkai Dang, Yunchen Wu +1Vision-Language Models Efficient VLM Inference
School of Intelligence Science and Technology, Nanjing University
Scaling Laws for Looped Mixture of Experts
Sep 30, 2026Yanbei Chen, Anirudh Goyal, Raghuraman KrishnamoorthiMixture-of-Experts Language Models Language Model Scaling Laws
Meta AI
How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text
Sep 30, 2026Jenna Russell, Ben Glickenhaus, Katherine Thai +4Language Model Pretraining Synthetic Data Pretraining
University of Maryland · Pangram Labs
From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes
Sep 30, 2026Yichen Wang, Fanghui Liu, Yudong ChenLanguage Model Pretraining Deep Learning Optimization
Department of Computer Sciences, University of Wisconsin–Madison, USA. · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.
ScAn-Bench: Evaluating Scaling Analysis Methodology
Sep 28, 2026Artin Sermaxhaj, Nastaran Alipour, Donat Sinani +4VLM Evaluation LLM Evaluation
University of Freiburg · Zuse School ELIZA · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)
X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depths
Sep 28, 2026Bowen Dong, Yilong Fan, Tengyu Pan +6Language Model Scaling Laws Mixture-of-Experts Models
Tsinghua University · Tianjin University
Training and Inference Dynamics of PLDR-LLMs: Row-Map Collapse, Renormalization, and Predictive Reduction
Sep 28, 2026Burc GokdenLLM Inference Language Model Scaling Laws
Fromthesky Research Labs LLC, Oregon, USA
Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters
Sep 23, 2026Egor Romanyukov, Timofey Novikov, Timur Shokarov +3Language Model Scaling Laws Hyperparameter Transfer
Faculty of Computer Science HSE University
Scaling of Capability and Efficiency at Inference Time in Large Reasoning Models
Sep 22, 2026Moritz Laber, Zohair Shafi, Germans Savcisens +6LLM Inference Efficiency Language Model Scaling Laws
Northeastern University, Boston, Massachusetts, USA · Complexity Science Hub Vienna, Vienna, Austria · Santa Fe Institute, Santa Fe, New Mexico, USA
How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
Sep 16, 2026Zixi Chen, Akshay Vegesna, Samip Dahal +1Language Model Scaling Laws Recurrent Transformers
New York University · Q Labs
Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
Sep 14, 2026Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz +4Decoder-Only Language Models Language Model Scaling Laws
University of Washington, Seattle · Work done at Meta FAIR · Meta FAIR
Hyperparameter Scaling Laws Across MoE Sparsity
Sep 8, 2026Changxin Tian, Kunlong Chen, Jia Liu +3Language Model Scaling Laws Sparse Mixture-of-Experts
Ling Team, Ant Group
Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
Sep 1, 2026Zhiliang Chen, Sebastian Ament, David Eriksson +4Bayesian Optimization Language Model Scaling Laws
Meta · National University of Singapore · Atomic Machines; work done at Meta
Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss
Aug 28, 2026Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4Language Model Pretraining Learning Rate Scheduling
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +6
Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090
Aug 27, 2026Kairong Luo, Jiarui Cui, Yaorui Yin +8Language Model Pretraining Language Model Scaling Laws
Tsinghua University · Pengcheng Laboratory
Small-Scale Experiments: Are We There Yet?
Aug 12, 2026Nicholas Lourie, Kyunghyun Cho, Karen Ullrich +1Language Model Scaling Laws Hyperparameter Optimization
1FAIR at MSL Meta · 2New York University
Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
Aug 11, 2026Soumajyoti Sarkar, Yuxin Tang, Sheng ZhaMixture-of-Experts Language Models Language Model Scaling Laws
Amazon AGI Foundations
Skaling: Chinchilla's Exponents Meet Kaplan's Coupling
Aug 7, 2026Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz +1Language Model Scaling Laws Neural Scaling Laws
1FAIR at Meta
Small Foundation Models of Human Cognition and Behaviour
Aug 5, 2026Nick Oh, Fernand GobetComputational Cognitive Modeling LLM Fine-Tuning
socius labs · London School of Economics and Political Science
LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
Aug 4, 2026Fengqi Zhu, Shaoxuan Xu, Jingyang Ou +11Mixture-of-Experts Language Models Language Model Scaling Laws
Gaoling School of AI, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +2
Towards joint scaling laws with optimal batch size schedules
Jul 30, 2026Jiaxiang Li, Zhiqi Bu, Shiyun XuDeep Learning Optimization Language Model Scaling Laws
Meta · Independent researcher
Bridging Compute- and Data-Optimal Pretraining
Jul 28, 2026Tian Qin, Kimia Hamidieh, David Alvarez-MelisLanguage Model Pretraining Language Model Scaling Laws
Harvard University · MIT CSAIL
Lomekwi: Resource-Bounded Tool Discovery in LLM Agents
Jul 18, 2026Roshan Klein-Seetharaman, Daniel Wang, Andrew XuLanguage Model Scaling Laws LLM Agent Evaluation
Sea12 Technologies · Yale University
Domain-Aware Scaling Laws Uncover Data Synergy
Jul 13, 2026Kimia Hamidieh, Lester Mackey, David Alvarez-MelisLanguage Model Pretraining Language Model Scaling Laws
MIT CSAIL · Microsoft Research · Harvard University
Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
Jul 9, 2026Aiwei Liu, Cheng Shi, Chuhan Wu +45Language Model Scaling Laws Transformer Attention
Will Scaling Improve Social Simulation with LLMs?
Jul 2, 2026Caleb Ziems, William Held, Su Doga Karaca +3Opinion Dynamics Language Model Scaling Laws
Stanford University · Open Athena
When Does Generating More Help? Disentangling Fixed-Source Synthesis from Source Expansion in Synthetic Data Scaling
Jul 2, 2026Xu Guo, Jian Tong, Zhihui Lu +1Synthetic Data Generation Language Model Scaling Laws
Shanghai AI Laboratory · Shanghai Innovation Institute · Fudan University
How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
Jul 1, 2026Fabian SchaippLanguage Model Scaling Laws Scaling Laws
Inria, ´Ecole Normale Sup´erieure, PSL Research University, Paris
Two AI Metrics Diverged: Will it Make All the Difference?
Jul 1, 2026Alex Fogelson, Zachary A. Brown, Hans Gundlach +2Language Model Scaling Laws AI Governance
MIT FutureTech, CSAIL, Cambridge, MA, USA.
Smooth Scaling Laws Hide Stepwise Token Learning
Jun 29, 2026Pingjie Wang, Zechen Hu, Peiru Yang +2Language Model Pretraining Language Model Scaling Laws
Dots Studio, Xiaohongshu Inc. · Shanghai Jiao Tong University · Tsinghua university
Representational Depth of Evaluation Awareness Shifts With Scale in Open-Weight Language Models
Jun 28, 2026Archit ManekLLM Interpretability Language Model Scaling Laws
Independent Researcher.
On the Nonlinearity of Learning Rate Scaling for LLM Training
Jun 28, 2026Zaiwen Yang, Huaqing Zhang, Jing Xu +1Language Model Scaling Laws Hyperparameter Transfer
Tsinghua University · Tsinghua University, Shanghai Qi Zhi Institute
Scaling limit of the Random Language Model
Jun 26, 2026Eric De GiuliLanguage Model Scaling Laws Language Modeling
Department of Physics, Toronto Metropolitan University, M5B 2K3, Toronto, Canada
Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
Jun 23, 2026Vatsal Baherwani, Zixi Chen, Shikai Qiu +2Language Model Scaling Laws Transformer Attention
New York University
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Jun 23, 2026Yizhou Liu, Jeff GoreLanguage Model Scaling Laws Neural Scaling Laws
Massachusetts Institute of Technology Cambridge, MA 02139
Can Scale Save Us From Plasticity Loss in Large Language Models?
Jun 23, 2026J. Fernando Hernandez-Garcia, Tomás Figliolia, Beren MillidgeContinual Learning for LLMs Loss of Plasticity
Zyphra San Francisco, CA
Scaling Laws for Task-Specific LLM Distillation
Jun 23, 2026Lavinia Ghita, Dhruv Desai, Ioana BoierLLM Pruning LLM Compression
NVIDIA
Internal Data Repetition Destroys Language Models
Jun 23, 2026Jessica Chudnovsky, Joshua Kazdan, Noam Levi +6Memorization in Language Models Language Model Scaling Laws
1Stanford Computer Science. · 2Stanford Statistics. · 3Tel Aviv University. +1
On the Smallness of the Large Language Models Scaling Exponents
Jun 23, 2026Sauro Succi, Peter V. Coveney, Alex HansenEnergy-Efficient ML Language Model Scaling Laws
Italian Institute of Technology, Viale Regina Elena, 291, 00161, Rome, Italy · PoreLab, Physics Department, Norwegian University of Science and Technology, 7491 Trondheim, Norway · Centre for Computational Science, Chemistry Department, University College of London, 20 Gordon Street, WC1H 0AJ, London, United Kingdom
The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model
Jun 22, 2026Mansour Zoubeirou a MayakiEnergy-Efficient ML Transformer
Université Lumière Lyon 2, CNRS, Ecole Centrale de Lyon, INSA Lyon, Universite Claude Bernard Lyon 1, LIRIS, UMR5205, 69007 Lyon, France
Human vs Machine Mathematical Difficulty on Project Euler: An Experimental Analysis
Jun 20, 2026David Holmes, Johannes SchmittMathematical Reasoning Benchmarks Language Model Scaling Laws
Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
Jun 5, 2026Zhiwei Xu, Shihao Wu, Hanseul Cho +2Language Model Pretraining Language Model Scaling Laws
University of Michigan · KAIST AI
Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness
Jun 4, 2026Victor De Marez, Luna De Bruyne, Walter DaelemansLLM Evaluation LLM Sycophancy
Centre for Computational Linguistics, Psycholinguistics and Sociolinguistics University of Antwerp Antwerp, Belgium
Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives
Jun 4, 2026Karolina Drożdż, Micha HeilbronLLM Evaluation Language Model Scaling Laws
IDEAS Research Institute; Warsaw, Poland · Max Planck Institute for Psycholinguistics; Nijmegen, Netherlands · University of Amsterdam, Amsterdam Brain and Cognition; Amsterdam, Netherlands
Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training
Jun 4, 2026Yongwei Zhou, Juncheng Diao, Junlin Shang +2Language Model Scaling Laws Hyperparameter Transfer
MeiTuan · University of Chinese Academy of Sciences · Harbin Institute of Technology
Structure and Scale in Simplicial Sequence Modelling
May 31, 2026Matthew Farrugia-RobertsRepresentation Geometry Transformer
Department of Computer Science, University of Oxford
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
May 31, 2026Boqian Wu, Qiao Xiao, Patrik Okanovic +6Language Model Scaling Laws Efficient Language Model Training
University of Luxembourg · University of Twente · Eindhoven University of Technology +1
Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation
May 29, 2026Sang Truong, Yuheng Tu, Rylan Schaeffer +1Language Model Scaling Laws Test-Time Scaling
1Stanford University · University of California, Los Angeles
How LoRA Remembers? A Parametric Memory Law for LLM Finetuning
May 28, 2026Ziwen Xu, Haiwen Hong, Linsong Yu +4Memorization in Language Models LLM Fine-Tuning
Zhejiang University · Alibaba Group
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
May 28, 2026Jing Huang, Daniel Wurgaft, Rachit Bansal +6Long-Tail Learning Language Model Scaling Laws
Stanford University · Kempner Institute at Harvard University · MIT +1
The Curse of Helpfulness: Inverse Scaling Law in Robustness to Distractor Instructions via DistractionIF
May 28, 2026Zeli Su, Zhankai Xu, Tianlei Chen +4Language Model Scaling Laws Language Model Robustness
Minzu University of China, Beijing, China · Ant Group, Hangzhou, China · Renmin University of China, Beijing, China +1
SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling
May 27, 2026Quanen Sun, Changxin Tian, Ke Shi +6LLM Evaluation Language Model Scaling Laws
Ant Group
Law of Neural Interaction: Depth-Width Shape, Interaction Efficiency, and Generalization
May 27, 2026Wenjie Sun, Jinning Yang, Shuai Zhang +1Neural Network Generalization Language Model Scaling Laws
The Chinese University of Hong Kong, Shenzhen · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · New Jersey Institute of Technology
Unified Neural Scaling Laws
May 25, 2026Ethan Caballero, Priyank Jaini, David Krueger +1Language Model Scaling Laws Scaling Laws
Mila, University of Montreal · Google DeepMind
How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws
May 25, 2026Zhitao Zhu, Xili Wang, Shizhe Wu +2Language Model Scaling Laws LLM Training
Peking University · Meituan

Oct 6, 2026, cs.AIJ/K move · Enter open · S save

Toward Alignment Scaling Laws: A Framework and First Preregistered Measurements

Jeremy Canale

LLM Alignment Language Model Safety Evaluation Language Model Scaling Laws

Open paper AskPDF arXiv

Whether alignment gets easier or harder as models grow is often argued from isolated findings, as if alignment were one property. We treat it as a family of measurable scaling relations: for each risk category r, the alignment burden needed to hold a fixed safety target is modeled as B_r(N)=a_rN^alpha_r, with N a capability proxy; against a budget proportional to N, scaling helps if alpha_r<1, keeps pace if alpha_r~1, and accumulates alignment debt if alpha_r>1. We give three operationalizations of burden and distinguish observed, audited and true alignment. A toy model, in which corrections consume capability headroom, makes the consequences explicit. We prove that the largest exponent among corrected risks, not an average, sets the long-run regime; that above 1 any policy holding headroom above a floor must grow super-exponentially; that, for burdens that are positive mixtures of power laws, fits on small models underestimate large-scale exponents; and that an audit that uncovers hidden failures without false positives never underestimates true alignment. We propose a pre-registrable protocol and apply reduced versions of it twice. A preregistered reanalysis of public adversarial-training data for Pythia classifiers finds that the compute needed to bring attack success under 10% grows as N^0.60. A preregistered pilot on Qwen2.5 0.5B-72B finds exponents of -0.05 for truthfulness and 0.48 for stated dispositions (both scaling helps under its reduced rule, though local slopes approach 1 at the top; replicated on Qwen3 0.6B-14B), while sycophancy (0.89, or 0.83 with two seeds added at 72B) and a planted backdoor are undetermined: the backdoor is removed quickly when its trigger is known but survives blind safety training at four of five sizes. We release four browser games that play these laws (www.aisafety.fun). We make no claim about which regime holds for current frontier models.

Momentum

Related topics

Latest papers 94