Language Model Pretraining

Follow this topic Ask about this topic

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

448 papers72 papers in the last four weeks, up 57% on the four weeks before. 0.4% of all new papers.

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning
May 12, 2026Lawrence Feng, Gaurav R. Ghosal, Jacob Mitchell Springer +2Language Model Pretraining Continual Learning for LLMs
Carnegie Mellon University
Solve the Loop: Attractor Models for Language and Reasoning
May 12, 2026Jacob Fein-Ashley, Paria RashidinejadLanguage Model Pretraining Recurrent Transformers
University of Southern California
A Causal Language Modeling Detour Improves Encoder Continued Pretraining
May 12, 2026Rian Touchent, Eric de la ClergerieLanguage Model Pretraining Masked Language Modeling
Sorbonne Université / INRIA Paris · ALMAnaCH Team · INRIA Paris
Pretraining Exposure Explains Popularity Judgments in Large Language Models
May 12, 2026Jamshid Mozafari, Bhawna Piryani, Adam JatowtLanguage Model Pretraining Language Model Bias Evaluation
University of Innsbruck Innsbruck, Austria
A Study on Hidden Layer Distillation for Large Language Model Pre-Training
May 12, 2026Maxime Guigon, Lucas Dixon, Michaël E. SanderLanguage Model Pretraining Decoder-Only Language Models
Google DeepMind
ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload
May 11, 2026Ziyue Liu, Zhengyang Wang, Ruijie Zhang +7Language Model Pretraining Fault-Tolerant Distributed Training
University of California at Santa Barbara · Argonne National Laboratory
Muown: Row-Norm Control for Muon Optimization
May 11, 2026Kai Lion, Florian Hübler, Bingcong Li +2Language Model Pretraining Muon Optimizer
Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany
A Single-Layer Model Can Do Language Modeling
May 11, 2026Zanmin WangLanguage Model Pretraining Recurrent Language Models
Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
May 11, 2026Jinchang Zhu, Jindong Li, Yuwen Hao +3Language Model Pretraining Self-Attention
The Hong Kong University of Science and Technology (Guangzhou)
Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
May 11, 2026Xu Guo, Runyu Peng, Jian Tong +4Language Model Pretraining Synthetic Data Pretraining
Shanghai AI Laboratory · Fudan University · Shanghai Innovation Institute
Pretraining large language models with MXFP4 on Native FP4 Hardware
May 11, 2026Musa Cim, Poovaiah Palangappa, Miro Hodak +3Language Model Pretraining LLM Quantization
1The Pennsylvania State University · 2Advanced Micro Devices, Inc.
Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes
May 10, 2026A. BochkovLanguage Model Pretraining Language Modeling
Emergent Semantic Role Understanding in Language Models
May 9, 2026Carla Griffiths, Mirco MusolesiLanguage Model Pretraining
University College London · University of Bologna
LLiMba: Sardinian on a Single GPU -- Adapting a 3B Language Model to a Vanishing Romance Language
May 9, 2026Luca BalloreLanguage Model Pretraining Adapter Tuning
SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization
May 9, 2026Yan Sun, Guoxia Wang, Jinle Zeng +6Language Model Pretraining Contrastive Learning
Baidu Inc. · Nanyang Technological University
Is She Even Relevant? When BERT Ignores Explicit Gender Cues
May 8, 2026Jonas Klein, Chiara Manna, Eva VanmassenhoveLanguage Model Pretraining Gender Bias in Language Models
Department of Computational Cognitive Science, Tilburg University, Warandelaan 2, 5037 AB Tilburg, Netherlands
ProteinJEPA: Latent prediction improves protein language model pretraining
May 8, 2026Dan Ofer, Dafna Shahaf, Michal LinialLanguage Model Pretraining Masked Language Modeling
Department of Biological Chemistry The Hebrew University of Jerusalem · School of Computer Science and Engineering The Hebrew University of Jerusalem
EMO: Pretraining Mixture of Experts for Emergent Modularity
May 7, 2026Ryan Wang, Akshita Bhagia, Sewon MinLanguage Model Pretraining Mixture-of-Experts Language Models
University of California, Berkeley · Allen Institute for AI
Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleAttention Mechanisms Self-Attention
Nous Research
Efficient Pre-Training with Token Superposition
May 7, 2026Bowen Peng, Théo Gigant, Jeffrey QuesnelleLanguage Model Pretraining Efficient Language Model Training
Nous Research
Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization
May 7, 2026Ruotong Sun, Ermin WeiLanguage Model Pretraining Deep Learning Optimization
†Department of Industrial Engineering & Management Sciences, Northwestern University · ∗Department of Electrical & Computer Engineering, Northwestern University
On the Blessing of Pre-training in Weak-to-Strong Generalization
May 7, 2026Wei Yao, Wang Zhaoyang, Gengze Xu +5Language Model Pretraining Neural Network Generalization
Renmin University of China · Shanghai Jiao Tong University · Tongji University +1
Demystifying Manifold Constraints in LLM Pre-training
May 6, 2026Kang An, Jiaxiang Li, Donald Goldfarb +1Language Model Pretraining Riemannian Optimization
Rice University · Independent Researcher · Columbia University
ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
May 5, 2026Jiaxi Li, Lu Yin, Li Shen +5Language Model Pretraining Activation Sparsity
University of Surrey · Sun Yat-sen University · Bytedance +2
InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition
May 4, 2026Fengze Liu, Weidong Zhou, Binbin Liu +7Language Model Pretraining Language Model Scaling Laws
1ByteDance · 2UC Santa Cruz
Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting
May 4, 2026Ishaan Watts, Catherine Li, Sachin Goyal +2Language Model Pretraining Sharpness-Aware Minimization
Carnegie Mellon University
Finite-Size Gradient Transport in Large Language Model Pretraining: From Cascade Size to Intensive Transport Efficiency
May 3, 2026Ping Wang, Yan-Qi DuLanguage Model Pretraining Language Model Scaling Laws
Institute of High Energy Physics, Chinese Academy of Sciences, Beijing 100049, China · Beijing Shenzhou Aerospace Software Technology Co., Ltd., Beijing 100094, China
The Pre-Training Study of Expanded-SPLADE Models on Web Document Titles
May 2, 2026Hiun Kim, Tae Kwan Lee, Taeryun WonLanguage Model Pretraining Masked Language Modeling
Naver Seongnam, South Korea
NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus
Apr 30, 2026Enzo S. N. Silva, Pablo B. Costa, Raphael C. Vlasman +8Language Model Pretraining Portuguese Language Models
Itaú Unibanco, São Paulo, SP, Brasil · ICTi, São Paulo, SP, Brasil
Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling
Apr 30, 2026Ansar Aynetdinov, Patrick Haller, Alan AkbikLanguage Model Pretraining Efficient Language Model Training
Humboldt-Universität zu Berlin

May 12, 2026J/K move · Enter open · S save

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

Lawrence Feng, Gaurav R. Ghosal, Jacob Mitchell Springer, Ziqian Zhong+1

Carnegie Mellon University

Language Model Pretraining Continual Learning for LLMs LLM Fine-Tuning Language Model Robustness Catastrophic Forgetting

Open paper AskPDF arXiv

How can we train models whose post-trained capabilities survive subsequent fine-tuning? Rather than focusing on downstream interventions to mitigate forgetting of upstream capabilities, we study how upstream training choices - that is, the manner in which a capability is acquired - shape how robustly that capability is retained. We investigate this question in a controlled three-stage language-model pipeline: pretraining, post-training to acquire a target capability, and downstream fine-tuning on a new objective. Across 135M and 1B models, two post-training domains, and two downstream fine-tuning tasks, we find that immediate post-training performance does not reliably predict retention after subsequent fine-tuning: training recipes that look equivalent immediately after post-training can retain the target capability very differently after subsequent fine-tuning. In particular, early exposure - mixing post-training data into pretraining - consistently improves the frontier between retained upstream performance and downstream performance. In compute-matched experiments, where the target data must be allocated between pretraining and post-training, we find that the optimum lies at neither extreme. Together with our other empirical and theoretical findings, this supports the view that post-training drives immediate specialization while early exposure improves robustness to later forgetting. Replay and dropout, typically used to mitigate forgetting as it occurs during fine-tuning, provide complementary gains to early exposure when applied during post-training. Our findings suggest that robustness to subsequent fine-tuning should be treated as a first-class objective of upstream training, addressed preventatively through choices like early exposure rather than reactively during fine-tuning itself.