Language Model Scaling Laws

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 95

All topics

Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
Jul 9, 2026Aiwei Liu, Cheng Shi, Chuhan Wu +45Language Model Scaling Laws Transformer Attention
Will Scaling Improve Social Simulation with LLMs?
Jul 2, 2026Caleb Ziems, William Held, Su Doga Karaca +3Opinion Dynamics Language Model Scaling Laws
Stanford University · Open Athena
When Does Generating More Help? Disentangling Fixed-Source Synthesis from Source Expansion in Synthetic Data Scaling
Jul 2, 2026Xu Guo, Jian Tong, Zhihui Lu +1Synthetic Data Generation Language Model Scaling Laws
Shanghai AI Laboratory · Shanghai Innovation Institute · Fudan University
How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
Jul 1, 2026Fabian SchaippLanguage Model Scaling Laws Scaling Laws
Inria, ´Ecole Normale Sup´erieure, PSL Research University, Paris
Two AI Metrics Diverged: Will it Make All the Difference?
Jul 1, 2026Alex Fogelson, Zachary A. Brown, Hans Gundlach +2Language Model Scaling Laws AI Governance
MIT FutureTech, CSAIL, Cambridge, MA, USA.
Smooth Scaling Laws Hide Stepwise Token Learning
Jun 29, 2026Pingjie Wang, Zechen Hu, Peiru Yang +2Language Model Pretraining Language Model Scaling Laws
Dots Studio, Xiaohongshu Inc. · Shanghai Jiao Tong University · Tsinghua university
Representational Depth of Evaluation Awareness Shifts With Scale in Open-Weight Language Models
Jun 28, 2026Archit ManekLLM Interpretability Language Model Scaling Laws
Independent Researcher.
On the Nonlinearity of Learning Rate Scaling for LLM Training
Jun 28, 2026Zaiwen Yang, Huaqing Zhang, Jing Xu +1Language Model Scaling Laws Hyperparameter Transfer
Tsinghua University · Tsinghua University, Shanghai Qi Zhi Institute
Scaling limit of the Random Language Model
Jun 26, 2026Eric De GiuliLanguage Model Scaling Laws Language Modeling
Department of Physics, Toronto Metropolitan University, M5B 2K3, Toronto, Canada
Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
Jun 23, 2026Vatsal Baherwani, Zixi Chen, Shikai Qiu +2Language Model Scaling Laws Transformer Attention
New York University
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Jun 23, 2026Yizhou Liu, Jeff GoreLanguage Model Scaling Laws Neural Scaling Laws
Massachusetts Institute of Technology Cambridge, MA 02139
Can Scale Save Us From Plasticity Loss in Large Language Models?
Jun 23, 2026J. Fernando Hernandez-Garcia, Tomás Figliolia, Beren MillidgeContinual Learning for LLMs Loss of Plasticity
Zyphra San Francisco, CA
Scaling Laws for Task-Specific LLM Distillation
Jun 23, 2026Lavinia Ghita, Dhruv Desai, Ioana BoierLLM Pruning LLM Compression
NVIDIA
Internal Data Repetition Destroys Language Models
Jun 23, 2026Jessica Chudnovsky, Joshua Kazdan, Noam Levi +6Memorization in Language Models Language Model Scaling Laws
1Stanford Computer Science. · 2Stanford Statistics. · 3Tel Aviv University. +1
On the Smallness of the Large Language Models Scaling Exponents
Jun 23, 2026Sauro Succi, Peter V. Coveney, Alex HansenEnergy-Efficient ML Language Model Scaling Laws
Italian Institute of Technology, Viale Regina Elena, 291, 00161, Rome, Italy · PoreLab, Physics Department, Norwegian University of Science and Technology, 7491 Trondheim, Norway · Centre for Computational Science, Chemistry Department, University College of London, 20 Gordon Street, WC1H 0AJ, London, United Kingdom
The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model
Jun 22, 2026Mansour Zoubeirou a MayakiEnergy-Efficient ML Transformer
Université Lumière Lyon 2, CNRS, Ecole Centrale de Lyon, INSA Lyon, Universite Claude Bernard Lyon 1, LIRIS, UMR5205, 69007 Lyon, France
Human vs Machine Mathematical Difficulty on Project Euler: An Experimental Analysis
Jun 20, 2026David Holmes, Johannes SchmittMathematical Reasoning Benchmarks Language Model Scaling Laws
Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
Jun 5, 2026Zhiwei Xu, Shihao Wu, Hanseul Cho +2Language Model Pretraining Language Model Scaling Laws
University of Michigan · KAIST AI
Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness
Jun 4, 2026Victor De Marez, Luna De Bruyne, Walter DaelemansLLM Evaluation LLM Sycophancy
Centre for Computational Linguistics, Psycholinguistics and Sociolinguistics University of Antwerp Antwerp, Belgium
Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives
Jun 4, 2026Karolina Drożdż, Micha HeilbronLLM Evaluation Language Model Scaling Laws
IDEAS Research Institute; Warsaw, Poland · Max Planck Institute for Psycholinguistics; Nijmegen, Netherlands · University of Amsterdam, Amsterdam Brain and Cognition; Amsterdam, Netherlands
Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training
Jun 4, 2026Yongwei Zhou, Juncheng Diao, Junlin Shang +2Language Model Scaling Laws Hyperparameter Transfer
MeiTuan · University of Chinese Academy of Sciences · Harbin Institute of Technology
Structure and Scale in Simplicial Sequence Modelling
May 31, 2026Matthew Farrugia-RobertsRepresentation Geometry Transformer
Department of Computer Science, University of Oxford
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
May 31, 2026Boqian Wu, Qiao Xiao, Patrik Okanovic +6Language Model Scaling Laws Efficient Language Model Training
University of Luxembourg · University of Twente · Eindhoven University of Technology +1
Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation
May 29, 2026Sang Truong, Yuheng Tu, Rylan Schaeffer +1Language Model Scaling Laws Test-Time Scaling
1Stanford University · University of California, Los Angeles
How LoRA Remembers? A Parametric Memory Law for LLM Finetuning
May 28, 2026Ziwen Xu, Haiwen Hong, Linsong Yu +4Memorization in Language Models LLM Fine-Tuning
Zhejiang University · Alibaba Group
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
May 28, 2026Jing Huang, Daniel Wurgaft, Rachit Bansal +6Long-Tail Learning Language Model Scaling Laws
Stanford University · Kempner Institute at Harvard University · MIT +1
The Curse of Helpfulness: Inverse Scaling Law in Robustness to Distractor Instructions via DistractionIF
May 28, 2026Zeli Su, Zhankai Xu, Tianlei Chen +4Language Model Scaling Laws Language Model Robustness
Minzu University of China, Beijing, China · Ant Group, Hangzhou, China · Renmin University of China, Beijing, China +1
SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling
May 27, 2026Quanen Sun, Changxin Tian, Ke Shi +6LLM Evaluation Language Model Scaling Laws
Ant Group
Law of Neural Interaction: Depth-Width Shape, Interaction Efficiency, and Generalization
May 27, 2026Wenjie Sun, Jinning Yang, Shuai Zhang +1Neural Network Generalization Language Model Scaling Laws
The Chinese University of Hong Kong, Shenzhen · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · New Jersey Institute of Technology
Unified Neural Scaling Laws
May 25, 2026Ethan Caballero, Priyank Jaini, David Krueger +1Language Model Scaling Laws Scaling Laws
Mila, University of Montreal · Google DeepMind

Jul 9, 2026, cs.CLJ/K move · Enter open · S save

Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

Aiwei Liu, Cheng Shi, Chuhan Wu, Ci Lei+44

Language Model Scaling Laws Transformer Attention Language Model Scaling LLM Training

Open paper AskPDF arXiv

Scaling Large Language Models (LLMs) has been driven mainly by enlarging the Transformer backbone, but for an already-strong model this requires another round of costly pretraining. We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed. Depth-recurrent (looped) Transformers pursue this goal but are hard to scale, because looped computation does not fit naturally with the pipeline parallelism used to train the largest models. We add computation along the sequence-length dimension, where the extra computation is simply a longer input and stays compatible with standard large-model training. We propose Hidden Decoding, a sequence-length scaling method applied during continued pretraining (CPT). It expands each token into n streams with independent embedding tables and keeps the intermediate streams' key-value cache as context, so each token performs more internal computation without adding or widening Transformer layers. To keep this affordable at scale, we introduce Stream-Factorized Attention, in which most layers attend only within each stream and only a few layers mix across streams, reducing the attention cost from quadratic to roughly linear in n. Experiments support two scaling results. At frontier scale, we train WeLM-HD4-80B and WeLM-HD4-617B at n=4 and improve their matched non-HD baselines, making Hidden Decoding the first demonstrated sequence-length scaling method at the 100B+ MoE scale. Across expansion factors, the gains grow as n increases, showing that sequence-length expansion is a practical fixed-backbone scaling path for frontier-scale LLMs.

Momentum

Related topics

Latest papers 95