Language Model Pretraining

Ask about this topic

Overview

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

CardsList

448 papers72 papers in the last four weeks, up 57% on the four weeks before. 0.4% of all new papers.

  1. NITP: Next Implicit Token Prediction for LLM Pre-training

    May 24, 2026Xiangdong Zhang, Debing Zhang, Shaofeng Zhang +3Language Model PretrainingRepresentation Geometry

    School of AI, Shanghai Jiao Tong University · Dots Studio, Xiaohongshu Inc. · University of Science and Technology of China +1

  2. Temporal Concept Drift in Legal Judgment Prediction: Neural Baselines Across Three Epochs of Ukrainian Court Decisions

    May 23, 2026Volodymyr OvcharovLanguage Model PretrainingLegal NLP

  3. Multilingual Knowledge Transfer under Data Constraints via Lexical Interventions

    May 22, 2026Anastasiia Sedova, Natalie Schluter, Skyler Seto +1Language Model PretrainingMultilingual Language Models

    1Apple

  4. Strong Teacher Not Needed? On Distillation in LLM Pretraining

    May 22, 2026Taiming Lu, Zhuang LiuLanguage Model PretrainingLanguage Model Distillation

    Princeton University

  5. Memorization Dynamics of Fill-in-the-Middle Pretraining

    May 21, 2026Tobias von Arx, Tanguy DieudonnéLanguage Model PretrainingMemorization in Language Models

    Department of Computer Science, ETH Zurich, Zurich, Switzerland.

  6. Understanding Data Temporality Impact on Large Language Models Pre-training

    May 21, 2026Hippolyte Pilchen, Romain Fabre, Franck Signe Talla +2Language Model PretrainingTemporal Reasoning in Language Models

    Kyutai, Paris

  7. Is a Document Educational or Just Wikipedia-Style? -- Pitfalls of Classifier-Based Quality Filtering

    May 21, 2026Mateusz Klimaszewski, Piotr AndruszkiewiczLanguage Model PretrainingTraining Data Curation

    Warsaw University of Technology · 2IDEAS Research Institute

  8. SymbolicLight V1: Spike-Gated Dual-Path Language Modeling at High Encoder Spike Sparsity

    May 20, 2026Ting LiuLanguage Model PretrainingSpiking Neural Networks

    SymbolicLight Research Foshan, Guangdong, China

  9. Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory

    May 20, 2026Runxi Cheng, Yuchen Guan, Yongxian Wei +7Language Model PretrainingMemory-Augmented Language Models

    Tsinghua University · Microsoft Reasearch Asia

  10. HRM-Text: Efficient Pretraining Beyond Scaling

    May 20, 2026Guan Wang, Changling Liu, Chenyu Wang +6Language Model PretrainingRecurrent Neural Networks

    1Sapient Intelligence · 2MIT

  11. LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series

    May 19, 2026Alexis Roger, Prateek Humane, Zhenghan Tai +4Cross-Modal LearningLanguage Model Pretraining

    Université de Montréal Mila - Quebec AI Institute

  12. m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encoder

    May 19, 2026Yaoxiang Wang, Simiao Zuo, Qingguo Hu +4Language Model PretrainingRepresentation Learning

    Xiamen University Xiamen, China · Microsoft · Microsoft Redmond, USA +2

  13. Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages

    May 19, 2026Brandon Cui, Ximing Lu, Jaehun Jung +7Language Model PretrainingEfficient Language Model Training

    1NVIDIA · University of Washington · 3Carnegie Mellon University +1

  14. A Bitter Lesson for Data Filtering

    May 19, 2026Christopher Mohri, John Duchi, Tatsunori HashimotoLanguage Model PretrainingTraining Data Curation

    Department of Computer Science Stanford University · Departments of Statistics and Electrical Engineering Stanford University

  15. ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models

    May 18, 2026Aaron DefazioLanguage Model PretrainingLearning Rate Scheduling

    FAIR at Meta Super-Intelligence Labs

  16. Forecasting Downstream Performance of LLMs With Proxy Metrics

    May 18, 2026Arkil Patel, Siva Reddy, Marius Mosbach +1Language Model PretrainingLLM Evaluation

    Mila – Quebec AI Institute & McGill University Canada · ServiceNow Research · Periodic Labs

  17. Generating Pretraining Tokens from Organic Data for Data-Bound Scaling

    May 18, 2026Zichun Yu, Chenyan XiongLanguage Model PretrainingSynthetic Data Pretraining

    Language Technologies Institute, Carnegie Mellon University · Xlue

  18. AMO: Operator-level Adaptive Muon Orthogonalization

    May 18, 2026Xinlin Zhuang, Panyi Ouyang, Yichen Li +7Language Model PretrainingNewton-Schulz Iteration

    The Chinese University of Hong Kong · Shopee · MBZUAI +3

  19. Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates

    May 18, 2026Athanasios Glentis, Dawei Li, Chung-Yiu Yau +1Language Model PretrainingGradient Clipping

    University of Minnesota

  20. MiniGPT: Rebuilding GPT from First Principles

    May 17, 2026Jibin JosephLanguage Model PretrainingDecoder-Only Language Models

    Department of Computer Science The University of Texas at Austin Austin, TX 78712

  21. Scale Determines Whether Language Models Organize Representation Geometry for Prediction

    May 16, 2026Weilun XuLanguage Model PretrainingNeural Representation Geometry

    School of Computer and Communication Sciences ´Ecole Polytechnique F´ed´erale de Lausanne

  22. Language Acquisition Device in Large Language Models

    May 16, 2026Masato Mita, Taiga Someya, Ryo Yoshida +1Language Model PretrainingSynthetic Data Pretraining

    The University of Tokyo

  23. The Geometry of Learning: Read-Write Asymmetry Across Training

    May 15, 2026Valeria Ruscio, Keiran ThompsonLanguage Model PretrainingTransformer Interpretability

    Intuition Machines

  24. Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matching

    May 13, 2026Abdalrahman WaelLanguage Model PretrainingMixture-of-Experts Language Models

    Independent Researcher

  25. Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training

    May 13, 2026Namrata Shivagunde, Vijeta Deshpande, Sherin Muckatira +1Language Model PretrainingLoss Landscape

    University of Massachusetts Lowell

  26. Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP

    May 13, 2026Ruan Visser, Trienko Grobler, Marcel DunaiskiLanguage Model PretrainingLow-Resource Language Processing

    Department of Computer Science, Stellenbosch University, Stellenbosch, South Africa

  27. Tracing Persona Vectors Through LLM Pretraining

    May 13, 2026Viktor Moskvoretskii, Dominik Glandorf, Jorge Medina Moreira +2Language Model PretrainingLLM Interpretability

    EPFL

  28. Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

    May 13, 2026Paul Jeha, Anastasiia Sedova, Louis Béthune +4Language Model PretrainingMultilingual Language Models

    1Apple · 2DTU

  29. Always Learning, Always Mixing: Efficient and Simple Data Mixing All the Time

    May 13, 2026Michael Y. Hu, Apurva Gandhi, Kyunghyun Cho +2Language Model PretrainingContinual Learning for LLMs

    New York University · Carnegie Mellon University · Microsoft

  30. Scaling Laws for Mixture Pretraining Under Data Constraints

    May 12, 2026Anastasiia Sedova, Skyler Seto, Natalie Schluter +1Language Model PretrainingLanguage Model Scaling Laws

    Apple · ITU