Language Model Pretraining

Ask about this topic

Overview

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

CardsList

448 papers72 papers in the last four weeks, up 57% on the four weeks before. 0.4% of all new papers.

  1. Rethinking Pre-pretraining

    Oct 29, 2026Alexander Stern, Tim Rocktäschel, Acyr Locatelli +1Language Model PretrainingSynthetic Data Pretraining

    MIT · AI Centre, UCL

  2. Multi-Token Prediction Done Right

    Oct 28, 2026Jayden TeohMulti-Token PredictionLanguage Model Pretraining

    MIT

  3. The Role of Quantization in Loss Spikes

    Oct 28, 2026Zaiqian Chen, Aditya Somasundaram, Sophie Li +4LLM TrainingLanguage Model Pretraining

    Columbia University · Mohamed bin Zayed University of Artificial Intelligence · University of Maryland +1

  4. Bi-Phase Training: Learning Efficiently From Scratch

    Oct 27, 2026Abel Gurung, Joseph CampbellEfficient Language Model TrainingLanguage Model Pretraining

    Purdue University

  5. MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

    Oct 27, 2026Athanasios Glentis, Dawei Li, Mingyi HongMuon OptimizerLanguage Model Pretraining

    University of Minnesota

  6. What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models

    Oct 27, 2026Chang Su, Bin Wu, Emine Yilmaz +1Language Model PretrainingFactual Knowledge in Language Models

    University of Pennsylvania · Centre for Artificial Intelligence, University College London

  7. Agentic Backdoors via Pretraining Poisoning

    Oct 27, 2026Xinyan Hu, Pingbang Hu, Chloe Loughridge +5LLM Backdoor AttacksData Poisoning Attacks

    UC Berkeley · Anthropic Fellows Program · UIUC +3

  8. What Carries Forward? Pretraining Disclosure Across Language Model Releases

    Oct 27, 2026Md Sakib Ul Islam, S M Mehedi ZamanLLM AuditingLanguage Model Pretraining

    Victorian Institute of Technology, Australia · Rutgers University, USA

  9. Does LLM Pre-Training Typically Occur at the Edge of Stability?

    Oct 27, 2026Yuhang Cai, Haodong Wen, Haofeng Huang +3Edge of StabilityNeural Network Training Dynamics

    UC Berkeley · Tsinghua University · ByteDance Seed

  10. Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters

    Oct 26, 2026Hafsteinn EinarssonLanguage Model PretrainingLow-Resource Language Processing

    University of Iceland

  11. Auto-Research Arena: Benchmarking and Advancing Auto-Research in Language Modeling

    Oct 26, 2026Xi Chen, Chongjian Ge, Yiran Xu +5Automated Machine LearningEfficient Language Model Training

    University of California San Diego · Adobe Research

  12. TUON: A Tensor-Aware Optimizer Utilizing Tensor-SVD and Polarization for Large Language Model Pretraining

    Oct 26, 2026Anda Tang, Xuanlin Yang, Junjie Wang +1Language Model Pretraining

    the State Key Lab of General AI School of Intelligence Science and Technology Peking University Beijing, China · the Pazhou Laboratory (Huangpu) the State Key Lab of General AI School of Intelligence Science and Technology the Institute for Artificial Intelligence Peking University Beijing, China

  13. Identically Trained Language Models Break on Different Jailbreaks

    Oct 25, 2026Chi-Wei Chang, Mu-Tien Kuo, Kuan-Wei Lee +1Language Model Safety EvaluationLLM Jailbreak Attacks

    Cerulean Labs · National Taiwan University, Taipei, Taiwan · National Yang Ming Chiao Tung University, Hsinchu, Taiwan

  14. A Comprehensive Framework for All Stages of Large-Model Training: From Data and Tokenization, through Pretraining, to Post-Training Alignment

    Oct 24, 2026Xiangjing An, Zhibin Jiao, Aiting ChenLLM TrainingLanguage Model Post-Training

    Independent Researcher Team

  15. Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting

    Oct 23, 2026Jiazheng Li, Huaqing Zhang, Jing Xu +2Language Model PretrainingAdaptive Loss Weighting

    Tsinghua University · ByteDance Seed

  16. Kolibri: A Sovereign European Model on the Pareto Frontier

    Oct 22, 2026Aleph Alpha, :, Moritz Althaus +65Language ModelingLanguage Model Pretraining

  17. Nested Sparsity for Scalable Pretraining and Continual Learning

    Oct 22, 2026Tianzhe Chu, Druv Pai, Yi MaConditional ComputationSparse Attention

    University of Hong Kong · UC Berkeley

  18. LAMB: A Modern Contextual Language Model for Latin

    Oct 22, 2026Aidan McCarthyLanguage ModelingLanguage Model Pretraining

  19. Tracing Subliminal Learning Through Pretraining

    Oct 22, 2026Angela Tang, Vishesh Anand, Gonçalo PauloLanguage Model DistillationOn-Policy Distillation

    EleutherAI

  20. AQL-Lite: Enabling Efficient Knowledge Acquisition for Arabic LLMs via Data-Centric Continual Pretraining

    Oct 21, 2026Houssam Eddine Boukhalfa, Mourad Bouache, Mourad DahmaneArabic Language ModelsLanguage Model Pretraining

    National School of Artificial Intelligence (ENSIA), Algiers, Algeria · Nojoom AI

  21. How to Use Parallel Data for Multilingual Pretraining

    Oct 20, 2026Zhihong Huang, Yao Lu, Derek F. Wong +2Language Model PretrainingMultilingual Language Models

    University of Macau · University College London · UNSW Sydney +1

  22. Africa-Papers: A 10-Billion-Token Multi-Domain Academic Corpus for African Foundation Language Models

    Oct 20, 2026Mich-Seth OwusuLanguage Model PretrainingLLM Training

    AfriSpeech

  23. Benefit and Cost of Random-Token Replacement under Repeated Data: A Known-Entropy Decomposition and Evidence from Ten Real Corpora

    Oct 20, 2026Wonsup Shin, Jingu KimLanguage Model PretrainingData Repetition in Language Model Training

    Flitto

  24. Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs

    Oct 19, 2026Abhinav Khattar, Aishwarya Chhabra, Ameya Velingker +24Language ModelingLanguage Model Pretraining

    NVIDIA · AI4Bharat

  25. Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining

    Oct 19, 2026Xinyi Wang, Danqi ChenRL for Language Model ReasoningLanguage Model Pretraining

    Princeton Language and Intelligence

  26. T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training

    Oct 18, 2026Dongyang Li, Mingsong Yan, Zheng Zhang +1Muon OptimizerLanguage Model Pretraining

    University of California, Santa Barbara Santa Barbara, CA 93106, USA

  27. What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes

    Oct 17, 2026Anjulie Agrusa, Andrei Panferov, Elizabeth Wei +7Low-Precision TrainingLanguage Model Pretraining

    NVIDIA · Institute of Science and Technology Austria · Red Hat AI

  28. Pearl-Chat: Engineering a native Luganda language model from first principles in JAX and Flax NNX

    Oct 15, 2026Wesley Kambale, Roland SankaraLanguage ModelingLow-Resource Language Processing

    AI for Uganda Initiative · ISBAT University

  29. In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model

    Oct 15, 2026Francesco Cagnetta, Casper L. ChristensenIn-Context LearningLanguage Model Pretraining

    Simplex · Simplex, Astera Institute

  30. Data Strategies for Large Language Models: A Unified Perspective on the Training Lifecycle

    Oct 15, 2026Yuanjian Xu, Jianing Hao, Tianze Sun +7LLM TrainingData Mixture Optimization

    The Hong Kong University of Science and Technology (Guangzhou) · Harbin Institute of Technology · OpenCSG +4