Language Model Scaling Laws

Ask about this topic

Momentum

15 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 95

All topics
CardsList

  1. Fault-tolerant foundation models

    Oct 7, 2026Trevor McCourt, Ila R. Fiete, Isaac L. ChuangLLM TrainingLanguage Model Scaling Laws

    Department of Electrical Engineering and Computer Science, MIT · Department of Brain and Cognitive Sciences, McGovern Institute, MIT · Department of Physics, MIT

  2. Toward Alignment Scaling Laws: A Framework and First Preregistered Measurements

    Oct 6, 2026Jeremy CanaleLLM AlignmentLanguage Model Safety Evaluation

  3. Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models

    Oct 5, 2026Joe DwyerLanguage Model Scaling LawsEfficient Language Model Training

    ECPI University

  4. What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining

    Oct 4, 2026Yekun Chai, Haoyi XiongLanguage Model PretrainingLanguage Model Scaling Laws

    FloatAI · ETH Zurich · Independent Researcher

  5. Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining

    Oct 4, 2026Ziyue WANG, T. KanamoriLanguage Model PretrainingLanguage Model Scaling Laws

    Institute of Science Tokyo

  6. A Systematic Analysis of the Predictive Power of LM Surprisal in Reading Chinese

    Oct 4, 2026Hongao Zhu, Muxiaoqiao Xu, Yikang Liu +4Language Model Scaling LawsLanguage Modeling

    Department of Linguistics, UC San Diego · School of Foreign Languages, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University +3

  7. Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference

    Oct 1, 2026Xinye Zhao, Yunkai Dang, Yunchen Wu +1Vision-Language ModelsEfficient VLM Inference

    School of Intelligence Science and Technology, Nanjing University

  8. Scaling Laws for Looped Mixture of Experts

    Sep 30, 2026Yanbei Chen, Anirudh Goyal, Raghuraman KrishnamoorthiMixture-of-Experts Language ModelsLanguage Model Scaling Laws

    Meta AI

  9. How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text

    Sep 30, 2026Jenna Russell, Ben Glickenhaus, Katherine Thai +4Language Model PretrainingSynthetic Data Pretraining

    University of Maryland · Pangram Labs

  10. From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

    Sep 30, 2026Yichen Wang, Fanghui Liu, Yudong ChenLanguage Model PretrainingDeep Learning Optimization

    Department of Computer Sciences, University of Wisconsin–Madison, USA. · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.

  11. ScAn-Bench: Evaluating Scaling Analysis Methodology

    Sep 28, 2026Artin Sermaxhaj, Nastaran Alipour, Donat Sinani +4VLM EvaluationLLM Evaluation

    University of Freiburg · Zuse School ELIZA · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)

  12. X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depths

    Sep 28, 2026Bowen Dong, Yilong Fan, Tengyu Pan +6Language Model Scaling LawsMixture-of-Experts Models

    Tsinghua University · Tianjin University

  13. Training and Inference Dynamics of PLDR-LLMs: Row-Map Collapse, Renormalization, and Predictive Reduction

    Sep 28, 2026Burc GokdenLLM InferenceLanguage Model Scaling Laws

    Fromthesky Research Labs LLC, Oregon, USA

  14. Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters

    Sep 23, 2026Egor Romanyukov, Timofey Novikov, Timur Shokarov +3Language Model Scaling LawsHyperparameter Transfer

    Faculty of Computer Science HSE University

  15. Scaling of Capability and Efficiency at Inference Time in Large Reasoning Models

    Sep 22, 2026Moritz Laber, Zohair Shafi, Germans Savcisens +6LLM Inference EfficiencyLanguage Model Scaling Laws

    Northeastern University, Boston, Massachusetts, USA · Complexity Science Hub Vienna, Vienna, Austria · Santa Fe Institute, Santa Fe, New Mexico, USA

  16. How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents

    Sep 16, 2026Zixi Chen, Akshay Vegesna, Samip Dahal +1Language Model Scaling LawsRecurrent Transformers

    New York University · Q Labs

  17. Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

    Sep 14, 2026Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz +4Decoder-Only Language ModelsLanguage Model Scaling Laws

    University of Washington, Seattle · Work done at Meta FAIR · Meta FAIR

  18. Hyperparameter Scaling Laws Across MoE Sparsity

    Sep 8, 2026Changxin Tian, Kunlong Chen, Jia Liu +3Language Model Scaling LawsSparse Mixture-of-Experts

    Ling Team, Ant Group

  19. Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search

    Sep 1, 2026Zhiliang Chen, Sebastian Ament, David Eriksson +4Bayesian OptimizationLanguage Model Scaling Laws

    Meta · National University of Singapore · Atomic Machines; work done at Meta

  20. Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss

    Aug 28, 2026Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4Language Model PretrainingLearning Rate Scheduling

    ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +6

  21. Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

    Aug 27, 2026Kairong Luo, Jiarui Cui, Yaorui Yin +8Language Model PretrainingLanguage Model Scaling Laws

    Tsinghua University · Pengcheng Laboratory

  22. Small-Scale Experiments: Are We There Yet?

    Aug 12, 2026Nicholas Lourie, Kyunghyun Cho, Karen Ullrich +1Language Model Scaling LawsHyperparameter Optimization

    1FAIR at MSL Meta · 2New York University

  23. Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

    Aug 11, 2026Soumajyoti Sarkar, Yuxin Tang, Sheng ZhaMixture-of-Experts Language ModelsLanguage Model Scaling Laws

    Amazon AGI Foundations

  24. Skaling: Chinchilla's Exponents Meet Kaplan's Coupling

    Aug 7, 2026Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz +1Language Model Scaling LawsNeural Scaling Laws

    1FAIR at Meta

  25. Small Foundation Models of Human Cognition and Behaviour

    Aug 5, 2026Nick Oh, Fernand GobetComputational Cognitive ModelingLLM Fine-Tuning

    socius labs · London School of Economics and Political Science

  26. LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

    Aug 4, 2026Fengqi Zhu, Shaoxuan Xu, Jingyang Ou +11Mixture-of-Experts Language ModelsLanguage Model Scaling Laws

    Gaoling School of AI, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +2

  27. Towards joint scaling laws with optimal batch size schedules

    Jul 30, 2026Jiaxiang Li, Zhiqi Bu, Shiyun XuDeep Learning OptimizationLanguage Model Scaling Laws

    Meta · Independent researcher

  28. Bridging Compute- and Data-Optimal Pretraining

    Jul 28, 2026Tian Qin, Kimia Hamidieh, David Alvarez-MelisLanguage Model PretrainingLanguage Model Scaling Laws

    Harvard University · MIT CSAIL

  29. Lomekwi: Resource-Bounded Tool Discovery in LLM Agents

    Jul 18, 2026Roshan Klein-Seetharaman, Daniel Wang, Andrew XuLanguage Model Scaling LawsLLM Agent Evaluation

    Sea12 Technologies · Yale University

  30. Domain-Aware Scaling Laws Uncover Data Synergy

    Jul 13, 2026Kimia Hamidieh, Lester Mackey, David Alvarez-MelisLanguage Model PretrainingLanguage Model Scaling Laws

    MIT CSAIL · Microsoft Research · Harvard University