Language Model Scaling Laws

Overview

Language model scaling laws study how predictive loss and downstream abilities change with parameter count, training tokens, compute, and test-time effort, helping forecast performance and allocate costly resources. Classic power-law and Chinchilla-style fits guide compute-optimal model and data trade-offs, while newer mixture-aware and information-theoretic laws account for data quality, repetition, batch size, tokenization, and architectures such as Mixture-of-Experts and recurrent models. Researchers use controlled IsoFLOP experiments and smaller proxy runs to fit these relationships, then test their predictions on held-out loss and capability benchmarks. A central open tension is whether laws extrapolate beyond measured scales: data scarcity and duplication can shift optimal allocations, and better loss or greater scale does not guarantee uniformly better, safer, or more reliable task performance.

Latest papers

CardsList

151 papers8 papers in the last four weeks, down 73% on the four weeks before. 0.1% of all new papers.

  1. Latent Feedback Is a Compute and Data Amplifier

    Nov 16, 2026Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma +2Language Model PretrainingRecurrent Transformers

  2. Condensing Law of Security Information

    Nov 13, 2026Qi Tan, Tianyu Cui, Qi Li +1LLMs for CybersecurityLanguage Model Scaling Laws

  3. A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling

    Nov 2, 2026Zhen Yang, Yidi Miao, Blake BordelonLanguage Model Scaling LawsLLM Training

  4. Hinted Scaling Laws

    Nov 1, 2026Suze van Adrichem, Luke Bailey, Rohith Kuditipudi +6Language Model Performance PredictionLanguage Model Scaling Laws

  5. Improving Data-Constrained Scaling with Hyperparameter Transfer and Ensembling

    Oct 27, 2026Ethan Baron, Shmuel Berman, Anirudh Satheesh +3Language Model Scaling LawsHyperparameter Transfer

  6. Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

    Oct 22, 2026Nicholas Roberts, Ahmed Ahmed, Suhas Kotha +4Synthetic Data PretrainingLanguage Model Scaling Laws

  7. Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law

    Oct 21, 2026Ziqing Qiao, Xiaoyue Xu, Sikui Zhang +4Long-Context Language ModelingContinued Pretraining

  8. Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation

    Oct 19, 2026Xuyuan Xiong, Siyu Chen, Zihan Shao +5Truncated Backpropagation Through TimeEfficient Language Model Training

  9. Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation

    Oct 14, 2026Joel Dag, Nikit Srivastava, Hamada M. Zahera +1Multilingual Language ModelsMixture-of-Experts Language Models

  10. Articulation Upper Bounds for Language Models

    Oct 14, 2026Alexander Spangher, Daniel E. Ho, Diyi Yang +2LLM EvaluationLanguage Model Scaling Laws

  11. Exact Scaling Laws for a Random-Features Bigram Model

    Oct 13, 2026Julian Enoch, Elvis DohmatobZipf's LawRandom Feature Methods

  12. Mixed Precision Scaling Laws for Quantized Pretraining

    Oct 13, 2026Jinuk Kim, Rajarshi Saha, Muhammed Fatih Balin +1Mixed-Precision QuantizationLLM Quantization

  13. SLDBench v2: Active Scaling Law Discovery

    Oct 12, 2026Lingbei Meng, Rui Huang, Zhirong Shen +5AI Agents for Scientific DiscoveryLanguage Model Scaling Laws

  14. Compute-Optimal Scaling Laws for Spiking Language Models

    Oct 10, 2026Dan Saunders, Benjamin Ren, Catherine HeSpiking Language ModelsLanguage Model Scaling Laws

  15. Scaling Laws for Multimodal Data Mixtures

    Oct 10, 2026Aditi Khandelwal, Ayush K. Tarun, Yixuan Xu +5Multimodal PretrainingAI Scaling Laws

  16. Neurograf v1: An End-to-End Romanian-First Language Model Family

    Oct 9, 2026Ionut Adrian RotaruLanguage ModelingLLM Training

  17. How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies

    Oct 8, 2026Bowei He, Yankai Chen, Xiaokun Zhang +4Agentic RLSynthetic Data Generation for LLM Agents

  18. Fault-tolerant foundation models

    Oct 7, 2026Trevor McCourt, Ila R. Fiete, Isaac L. ChuangNeural Network Fault ToleranceLLM Training

  19. OpenEBFM: Energy-based Foundation Models Are Scalable Architecture

    Oct 7, 2026Yuan Pu, Yudong Lu, Ting Wang +6Language ModelingLanguage Model Pretraining

  20. A Systematic Analysis of the Predictive Power of LM Surprisal in Reading Chinese

    Oct 4, 2026Hongao Zhu, Muxiaoqiao Xu, Yikang Liu +4Language Model Scaling LawsLanguage Modeling

  21. Capability Scaling-Down Laws for LLM Compression

    Oct 1, 2026Xueqi Cheng, Liang Wu, Kelly Wan +2LLM CompressionLLM Pruning

  22. Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference

    Oct 1, 2026Xinye Zhao, Yunkai Dang, Yunchen Wu +1Vision-Language ModelsEfficient VLM Inference

  23. Scaling Laws for Looped Mixture of Experts

    Sep 30, 2026Yanbei Chen, Anirudh Goyal, Raghuraman KrishnamoorthiMixture-of-Experts Language ModelsLanguage Model Scaling Laws

  24. How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text

    Sep 30, 2026Jenna Russell, Ben Glickenhaus, Katherine Thai +4Language Model PretrainingSynthetic Data Pretraining

  25. From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

    Sep 30, 2026Yichen Wang, Fanghui Liu, Yudong ChenLanguage Model PretrainingDeep Learning Optimization