Language Model Scaling Laws

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 95

All topics

How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws
May 25, 2026Zhitao Zhu, Xili Wang, Shizhe Wu +2Language Model Scaling Laws LLM Training
Peking University · Meituan
LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws
May 22, 2026Xu Ouyang, Deyi Liu, Yuhang Cai +5Language Model Scaling Laws
1ByteDance Seed · University of Virginia · University of California, Berkeley
Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most
May 21, 2026Nick Merrill, Jaeho Lee, Ezra KargerLanguage Model Scaling Laws Forecasting Benchmarks
Forecasting Research Institute · UC Berkeley
Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws
May 20, 2026Nandan Kumar Jha, Brandon ReagenDeep Learning Optimization Language Model Scaling Laws
New York University
Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate
May 20, 2026Dayal Singh Kalra, Maissam BarkeshliLanguage Model Scaling Laws Hyperparameter Transfer
Department of Computer Science, University of Maryland, College Park · Department of Physics, University of Maryland, College Park · Joint Quantum Institute, University of Maryland, College Park +1
Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency
May 18, 2026Matthew L. Smith, Jonathan P. Shock, Samuel T. Segun +2LLM Evaluation Language Model Scaling Laws
International Development Research Centre Canada · University of Cape Town South Africa · Global Center on AI Governance Canada +2
Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
May 18, 2026Zhihan Yang, Wei Guo, Shuibai Zhang +5Language Model Scaling Laws Diffusion Language Models
NVIDIA & Cornell · NVIDIA & Georgia Tech · UW-Madison +3
Scale-Dependent Collective Adaptation in Self-Amending LLM Societies: A Cross-Family Study of Emergent Governance
May 17, 2026Kazuya Horibe, Masaomi Hatakeyama, Gen Masumoto +2Multi-Agent LLM Systems Multi-Agent Coordination
Center for Brain Science, RIKEN. · Department of Evolutionary Biology and Environmental Studies, University of Zurich. · Information R&D and Strategy Headquarters. +3
The Scaling Laws of Skills in LLM Agent Systems
May 15, 2026Charles Chen, Qiming Yu, Yuhang Gu +12Language Model Scaling Laws LLM Agent Reliability
Evolvent AI Team
A Hierarchical Language Model with Predictable Scaling Laws and Provable Benefits of Reasoning
May 13, 2026Jason Gaitonde, Frederic Koehler, Elchanan Mossel +2Long-Context Language Modeling Language Model Scaling Laws
Duke University · University of Chicago · Massachusetts Institute of Technology +1
When is Warmstarting Effective for Scaling Language Models?
May 13, 2026Neeratyoy Mallik, Maciej Janowski, Johannes Hog +4Language Model Scaling Laws Language Model Scaling
University of Freiburg · Zuse School ELIZA · University of Technology Nuremberg +3
Lying Is Just a Phase: The Hidden Alignment Transition in Language Model Scaling
May 13, 2026Adil AminLLM Alignment LLM Interpretability
ZEHEN Labs
The Efficiency Gap in Byte Modeling
May 13, 2026Celine Lee, Jing Nathan Yan, Chen Liang +9Language Model Scaling Laws Autoregressive Language Modeling
1Google DeepMind · 2Work done while at Google DeepMind · Department of Computer Science, Cornell University.
Scaling Laws for Mixture Pretraining Under Data Constraints
May 12, 2026Anastasiia Sedova, Skyler Seto, Natalie Schluter +1Language Model Pretraining Language Model Scaling Laws
Apple · ITU
Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World
May 9, 2026Christopher M. Bryant, Hao LiuLanguage Model Scaling Laws Neural Scaling Laws
Arena Physica
Predicting Large Model Test Losses with a Noisy Quadratic System
May 9, 2026Chuning Li, Chris J. MaddisonLanguage Model Scaling Laws Efficient Language Model Training
University of Toronto · Vector Institute
Where Larger Models Excel: The Primacy of Constraint-Guided Reasoning
May 9, 2026Guan-Yi Lin, Hen-Hsen HuangLanguage Model Scaling Laws LLM Reasoning
National Chengchi University Taipei, Taiwan · Academia Sinica Taipei, Taiwan
Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation
May 8, 2026Joshua Shay Kricheli, Alexander Lawrence Reid, Soumajyoti Sarkar +2Parameter Estimation Language Model Scaling Laws
Syracuse University · Amazon AGI Foundations
On the Invariance and Generality of Neural Scaling Laws
May 8, 2026Xing Han, Liu Ziyin, Suchi Saria +1Language Model Scaling Laws Cross-Domain Generalization
Johns Hopkins University Baltimore, MD 21218, USA · NTT Research, MIT Cambridge, MA 02139, USA · Johns Hopkins University, Bayesian Health Baltimore, MD 21218, USA +1
Limits of Reliability and Scaling in Language Models
May 8, 2026Subhabrata MajumdarLanguage Model Scaling Laws Autoregressive Language Modeling
Indian Institute of Management Bangalore Bengaluru, India
Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key
May 7, 2026Tianle Wang, Zhaoyang Wang, Guangchen Lan +4Logical Reasoning RL for Language Model Reasoning
Purdue University · UNC Chapel Hill · Georgia Tech +1
Safety and accuracy follow different scaling laws in clinical large language models
May 5, 2026Sebastian Wind, Tri-Thien Nguyen, Jeta Sopa +9LLM Safety Benchmarks Retrieval-Augmented Generation
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Institute of Radiology, University Hospital Erlangen, Erlangen, Germany · Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany +3
InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition
May 4, 2026Fengze Liu, Weidong Zhou, Binbin Liu +7Language Model Pretraining Language Model Scaling Laws
1ByteDance · 2UC Santa Cruz
Finite-Size Gradient Transport in Large Language Model Pretraining: From Cascade Size to Intensive Transport Efficiency
May 3, 2026Ping Wang, Yan-Qi DuLanguage Model Pretraining Language Model Scaling Laws
Institute of High Energy Physics, Chinese Academy of Sciences, Beijing 100049, China · Beijing Shenzhou Aerospace Software Technology Co., Ltd., Beijing 100094, China
Prescriptive Scaling Laws for Data Constrained Training
May 2, 2026Justin Lovelace, Christian Belardi, Srivatsa Kundurthy +2Weight Decay Language Model Scaling Laws
Department of Computer Science Cornell University Ithaca, NY
Compute Optimal Tokenization
May 2, 2026Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer +6Language Model Scaling Laws Efficient Language Model Training
1FAIR at Meta · University of Washington
Scaling Properties of Continuous Diffusion Spoken Language Models
Apr 27, 2026Jason Ramapuram, Eeshan Gunesh Dhekane, Amitis Shidani +6Language Model Scaling Laws Speech Language Models
Apple
How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
Apr 22, 2026Kristian Schwethelm, Daniel Rueckert, Georgios KaissisLanguage Model Scaling Laws Recurrent Neural Networks
Chair for AI in Healthcare and Medicine, Technical University of Munich, Germany · Department of Computing, Imperial College London, UK · Munich Center for Machine Learning (MCML), Germany +1
Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling
Apr 21, 2026Weijie Zhao, Mingquan Liu, Bolun Wang +4Transformer Language Model Scaling Laws
Luxi Tech, Shenzhen, China · The School of Computer Science, China University of Geosciences, 388 Lumo Road, Wuhan, 430074, Hubei, China.
Stabilizing Native Low-Rank LLM Pretraining
Feb 12, 2026Paul Janson, Edouard Oyallon, Eugene BelilovskyLanguage Model Pretraining Low-Rank Matrix Decomposition
Concordia University, Montreal, Canada · Mila Quebec AI Institute, Montreal, Canada · Sorbonne University, CNRS, Paris, France

May 25, 2026, cs.LGJ/K move · Enter open · S save

How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws

Zhitao Zhu, Xili Wang, Shizhe Wu, Jiawei Fu+1

Peking University · Meituan

Language Model Scaling Laws LLM Training

Open paper AskPDF arXiv

High-quality data is scarce in large language model (LLM) training, yet how to schedule its use jointly with training dynamics lacks theoretical guidance. We extend functional scaling laws by incorporating a data-quality dimension, and solve the joint data-quality and batch-size scheduling problem in asymptotic closed form. The solution reveals two regimes and a dual role of high-quality data. In the noise-limited regime, high-quality data should be used as a signal amplifier: lowering the batch size converts cleaner data into more signal without amplifying noise. In the signal-limited regime, it should be used as a noise suppressor: late placement reduces terminal noise without sacrificing signal accumulation. Existing curriculum-style pipelines primarily exploit the second role by placing cleaner data late, but miss the first role because conventional decay schedules reduce update intensity exactly when high-quality data becomes available. Guided by this, we propose Drop-Stable-Rampup for LLM midtraining: upon the quality transition, drop the batch size, hold it stable to accumulate signal, then ramp up to suppress terminal noise. On a 15B Mixture-of-Experts model midtrained on 108B tokens, Drop-Stable-Rampup improves average accuracy over Warmup-Stable-Decay (WSD) by +1.70 and over Cosine-decay by +2.98, with particularly large gains on mathematical reasoning benchmarks such as GSM8K (+4.23) and MATH (+2.80).

Momentum

Related topics

Latest papers 95