Language Model Pretraining

Follow this topic Ask about this topic

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

448 papers72 papers in the last four weeks, up 57% on the four weeks before. 0.4% of all new papers.

Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
Jun 5, 2026Zhiwei Xu, Shihao Wu, Hanseul Cho +2Language Model Pretraining Language Model Scaling Laws
University of Michigan · KAIST AI
PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training
Jun 4, 2026Senmiao Wang, Tiantian Fang, Haoran Zhang +4Language Model Pretraining Transformer
The Chinese University of Hong Kong, Shenzhen, China · Google LLC, Mountain View, CA, United States · Shenzhen International Center for Industrial and Applied Mathematics, Shenzhen2026 Research Institute of Big Data +1
Arithmetic Pedagogy for Language Models
Jun 3, 2026Andhika Bernard Lumbantobing, Hokky SitungkirNumerical Reasoning in Language Models Language Model Pretraining
Bandung Fe Institute & Adjunct Science Fellow in InaAI · Adjunct Science Fellow in InaAI · AI Research Center IT Del +1
RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
Jun 2, 2026Rachit Bansal, Clara Mohri, Tian Qin +2Language Model Pretraining Supervised Fine-Tuning
Harvard University
q0: Primitives for Hyper-Epoch Pretraining
Jun 2, 2026Bishwas Mandal, Shmuel Berman, Akshay Vegesna +1Language Model Pretraining Efficient Language Model Training
Q Labs · Princeton University
KletterMix: Climbing Toward High-Quality German Pretraining Data
Jun 2, 2026Maurice Kraus, Ruben Härle, Sebastian Sztwiertnia +4Language Model Pretraining Synthetic Data Pretraining
AI & ML Group, TU Darmstadt · Lab1141 · Fraunhofer IAIS +3
Regret Pre-training: Bridging Prior and Posterior Views for Enhanced Knowledge Grounding
Jun 2, 2026Mingkuan Zhao, Xiayu Sun, Wentao Hu +5Language Model Pretraining LLM Grounding
The School of Computer Science and Technology, Xi’an Jiaotong University, China.
Pretraining Language Models on Historical Text
Jun 2, 2026Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber +5Language Model Pretraining LLM Evaluation
University of Waterloo · Vector Institute · AIML, Adelaide University +3
EURO-5K: When Does Domain Pretraining Matter? Benchmarking Transformers for EU Reporting Obligation Extraction
Jun 2, 2026Marios Koniaris, Vasileios Kotronis, Eugenia Giannini +1Language Model Pretraining Legal NLP
Division of Computer Science, School of Electrical and Computer Engineering, National Technical University of Athens, Iroon Polytechniou 9, Zographou Campus, 15780, Athens, Greece. · Department of Humanities Social Sciences and Law, School of Applied Mathematical and Physical Sciences, National Technical University of Athens, Iroon Polytechniou 9, Zographou Campus, 15780, Athens, Greece.
When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures
Jun 1, 2026Yongzhong XuLanguage Model Pretraining Transformer Interpretability
PortBERT: Navigating the Depths of Portuguese Language Models
Jun 1, 2026Raphael Scheible-Schmitt, Henry He, Armando B. MendesLanguage Model Pretraining Portuguese Language Models
School of Computation, Information and Technology, Technical University of Munich, Munich, Germany · IS2E - Intelligent Systems, Science and Engineering, LIACC polo on Azores University, Ponta Delgada, Portugal
KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts
Jun 1, 2026Christian Autenried, Cosimo PersiaLanguage Model Pretraining Domain-Adaptive Pretraining
Helse Vest ICT
Child-directed speech facilitates production, not comprehension, in BabyLMs
May 31, 2026Bastian Bunzeck, Sina ZarrießLanguage Model Pretraining LLM Evaluation
Computational Linguistics, Department of Linguistics Bielefeld University, Germany
Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
May 30, 2026Qiao Xiao, Boqian Wu, Patrik Okanovic +6Language Model Pretraining Efficient Language Model Training
Eindhoven University of Technology · University of Luxembourg · University of Twente +1
A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
May 29, 2026Sherin Muckatira, Namrata Shivagunde, Vijeta Deshpande +1Language Model Pretraining Neural Network Generalization
University of Massachusetts Lowell
Towards Efficient LLMs Annealing with Principled Sample Selection
May 29, 2026Yuanjian Xu, Jianing Hao, Wanbo Zhang +2Language Model Pretraining Efficient Language Model Training
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Microsoft Research Asia, Beijing, China · Fudan University, Shanghai, China
Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them
May 29, 2026Kevin Zhou, Lisa Alazraki, Kris Cao +1Language Model Pretraining Data Mixture Optimization
Imperial College London · Cohere
Demystifying Data Organization for Enhanced LLM Training
May 28, 2026Yalun Dai, Yangyu Huang, Tongshen Yang +8Language Model Pretraining Training Data Curation
Nanyang Technological University · Microsoft Research · The Hong Kong University of Science and Technology
Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
May 28, 2026Rohan ShravanLanguage Model Pretraining LLM Compression
The School of AI Bengaluru, India
Parallax: Parameterized Local Linear Attention for Language Modeling
May 27, 2026Yifei Zuo, Dhruv Pai, Zhichen Zeng +3Language Model Pretraining Self-Attention
Northwestern University · Tilde Research · University of Washington
Efficient Pre-Training of LLMs through Truncated SVD Representations
May 27, 2026Kaivan Kamali, Kajetan Schweighofer, Hormoz Shahrzad +3Language Model Pretraining Low-Rank Matrix Decomposition
Cognizant AI Lab
DunbaaBERT: From Sacrifice to Semantics
May 26, 2026Iffat Maab, Waleed Jamil, Raphael SchmittLanguage Model Pretraining Language Modeling
Research and Development Center for Large Language Models (LLMC), National Institute of Informatics, Tokyo · Independent Researcher, Edinburgh, United Kingdom · School of Computation, Information and Technology, Technical University of Munich, Germany +1
Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
May 26, 2026Mingze Wang, Shuchen Zhu, Yuxin Fang +3Language Model Pretraining Weight Decay
ByteDance Seed · Peking University
MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
May 26, 2026Jiacheng Li, Jianchao Tan, Hongtao Xu +5Language Model Pretraining Gradient Descent
Meituan, Beijing, China
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
May 26, 2026Zeyi Huang, Xuehai He, LiLiang Ren +8Language Model Pretraining Transformer Inference
Microsoft · University of Washington · University of Wisconsin-Madison
The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training
May 26, 2026Hongtao Zhang, Wenjie Zhou, Chenxi Jia +2Language Model Pretraining Deep Learning Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China +1
Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
May 26, 2026Wenjie Zhou, Bohan Wang, Hongtao Zhang +3Language Model Pretraining Neural Network Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · Alibaba Group, China +2
On the Limits of Model Merging for Multilinguality in Pre-Training
May 25, 2026Seth Aycock, Fedor Vitiugin, Aleksandr Umnov +2Language Model Pretraining Multilingual Language Models
University of Amsterdam
PowLU: An Activation Function for Stable Pre-Training of LLMs
May 25, 2026Peijie Jiang, Yuqi Feng, Cunyin Peng +5Language Model Pretraining Neural Network Activation Functions
Ling Team, Ant Group
Mimir: Large-scale Multilingual Concept Modeling
May 24, 2026Elio Musacchio, Lucia Siciliani, Pierpaolo BasileLanguage Model Pretraining Multilingual Language Models
Department of Computer Science University of Bari Aldo Moro Bari, Italy

Jun 5, 2026J/K move · Enter open · S save

Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws

Zhiwei Xu, Shihao Wu, Hanseul Cho, Wei Hu+1

University of Michigan · KAIST AI

Language Model Pretraining Language Model Scaling Laws Scaling Laws Data Repetition in Language Model Training

Open paper AskPDF arXiv

Classical scaling laws for language model pretraining balance model size against training dataset size under a fixed compute budget, assuming abundant data and a single pass over the corpus. As training compute grows faster than the supply of natural language data, pretraining is likely to enter a data-constrained, compute-rich regime where models train for multiple epochs over a finite dataset. We study data-constrained pretraining along two axes, regularization and scaling. For regularization, we study masked-input regularization (MIR), an auxiliary next-token prediction loss on randomly masked inputs. MIR tests whether the random masking central to diffusion language models can benefit autoregressive pretraining without architectural changes or inference overhead. Across 72M to 1.4B parameter models, we find that MIR added on top of strong weight decay improves validation loss over autoregressive strong-weight-decay-only models, with downstream gains at 1.4B. For scaling, we propose SoftQ, a scaling law that couples model size and data size to capture their interaction under repeated data. Classical alternatives such as the Chinchilla law use an additive form that decouples these terms, making them misspecified in the data-constrained regime. We find that SoftQ fits data-constrained experiments substantially better than these alternatives, and estimates MIR's gains as equivalent to roughly 1.3 times as much unique training data. We release our code at https://github.com/yixinw-lab/dc_pretrain.