Language Model Pretraining

Ask about this topic

Overview

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

CardsList

448 papers72 papers in the last four weeks, up 57% on the four weeks before. 0.4% of all new papers.

  1. From Zero to Hero: An Open LLM Ecosystem for Armenian

    Sep 3, 2026Erik Arakelyan, Khatun Avetisyan, Meri Davtyan +5Language Model PretrainingLanguage Modeling

    1NVIDIA · 2COPA

  2. UE5M3 FP4 Block Scaling for Stable Language Model Pretraining

    Sep 2, 2026Robert Hu, Carlo Luschi, Paul BalancaStochastic RoundingLanguage Model Pretraining

    GRAPHCORE RESEARCH

  3. Polish ModernBERT: The Long and Short of Polish Language Understanding

    Sep 1, 2026Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata +1Language Model PretrainingLong-Context Language Model Evaluation

    National Information Processing Institute, Warsaw, Poland

  4. Probing Factual Knowledge Transfer with Training Data Interventions

    Sep 1, 2026Romina Oji, Marc Braun, Marcel Bollmann +2Language Model PretrainingMultilingual Language Model Evaluation

    Department of Computer and Information Science Linköping University

  5. Pre-carved Niches: The Formation Dynamics of Modular Task Partitions in Early LLM Training

    Sep 1, 2026Guangqi Li, Yongxin LiLanguage Model PretrainingModular Neural Networks

    Zaozhuang University, Zaozhuang, Shandong, China

  6. Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretraining

    Sep 1, 2026Francois MeyerLanguage Model PretrainingAutoregressive Language Modeling

    Department of Computer Science University of Cape Town

  7. REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation

    Aug 31, 2026Haoran Que, Jiajun Shi, Ting Huang +7Language Model PretrainingSynthetic Data Pretraining

    Peking University · ByteDance Seed · Work done at ByteDance Seed +1

  8. KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Training

    Aug 31, 2026Meghanadh Pulivarthi, Kushagra Bhushan, Vineet Kumar +5Language Model PretrainingKnowledge Augmentation for Language Models

    IBM · Work done while at IBM. Currently at Amazon Books

  9. Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization

    Aug 31, 2026Junjie Yao, Liangkai Hang, Zhi-Qin John XuLanguage Model PretrainingWord Embeddings

    School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China · Institute of Natural Sciences, School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China

  10. Manacá-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation

    Aug 31, 2026Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio Andre Machado PortoLanguage Model PretrainingPortuguese Language Models

    Laboratório Nacional de Computação Científica (LNCC), Petrópolis, Brazil

  11. Arkios: An Open Bilingual English-Nepali Language Model Trained From Scratch, with a Devanagari-Aware Tokenizer

    Aug 30, 2026Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami PunLanguage Model PretrainingMultilingual Language Models

    Karela Technologies Inc.

  12. Learning Representations through Token Prediction: Geometry, Approximation, and Downstream Guarantees

    Aug 30, 2026Shulei WangLanguage Model PretrainingNext-Token Prediction

    Department of Statistics, University of Illinois at Urbana-Champaign, 605 E. Springfield Ave., Champaign, IL 61820

  13. Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining

    Aug 28, 2026Shuchen Zhu, Yuxin Fang, Mingze Wang +1Curvature-Aware OptimizationMomentum Methods

    ByteDance Seed · Peking University

  14. Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss

    Aug 28, 2026Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4Language Model PretrainingLearning Rate Scheduling

    ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +6

  15. Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

    Aug 27, 2026Kairong Luo, Jiarui Cui, Yaorui Yin +8Language Model PretrainingLanguage Model Scaling Laws

    Tsinghua University · Pengcheng Laboratory

  16. Squeezing More from Limited Data with Recursive Transformers

    Aug 27, 2026Serdar Gülbahar, Lukas Edman, Alexander FraserLanguage Model PretrainingLanguage Model Scaling Laws

    Technical University of Munich

  17. TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages

    Aug 27, 2026Ireddi Rakshitha, Devavarapu Yashwanth, Ntakirutimana PierreLanguage Model PretrainingLanguage Modeling

    Barclays · Carnegie Mellon University

  18. Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations

    Aug 27, 2026Anjishnu Mukherjee, Ziwei Zhu, Antonios AnastasopoulosLanguage Model PretrainingMultilingual Language Models

    George Mason University

  19. SPT: Skills as Pre-Training Data for Agentic Language Models

    Aug 27, 2026Yufei Sun, Yudong Li, Yiming ChengLLM Agent TrainingTool-Augmented Language Model Agents

    Beijing University of Posts and Telecommunications · Tsinghua University

  20. Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

    Aug 26, 2026Xiaodong Wu, Wenyi Yu, Chao Zhang +1Deep Learning OptimizationMuon Optimizer

    Department of Engineering University of Cambridge · Department of Electronic Engineering Tsinghua University

  21. One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography

    Aug 26, 2026Muge Zhang, Aaron Jencks, Krishna Badikela +2Language Model PretrainingMultilingual Language Models

    Department of Computer Science and Engineering, Ohio State University · Paul G. Allen School of Computer Science & Engineering, University of Washington

  22. MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum

    Aug 26, 2026Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk GozayLanguage Model PretrainingLanguage Modeling

  23. Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

    Aug 25, 2026Zihan Liu, Ruiheng Zheng, Shaobo Zhang +4Neural Network Training DynamicsLanguage Model Pretraining

    Peking University · Ant Group

  24. Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining

    Aug 24, 2026Yicheng Mao, Hongru DuData Mixture OptimizationLanguage Model Pretraining

    Department of Mathematics and Statistics, University of Calgary, University Drive NW, Calgary, T2N 1N4, Canada · Department of Systems and Information Engineering, University of Virginia, Charlottesville, VA, USA

  25. The Geometry of Low-Resource Language Representations

    Aug 24, 2026Francois Meyer, Jan BuysLow-Resource Language ProcessingRepresentation Geometry in Language Models

    Department of Computer Science University of Cape Town

  26. HelaBERT: Enhancing Sinhala Language Understanding with Dual Pooling Classification Head

    Aug 24, 2026Thisen Ekanayake, Nisansa de SilvaLanguage Model PretrainingLarge Language Model-Based Text Classification

    Department of Computer Science & Engineering University of Moratuwa

  27. Learning When Not to Listen: Selective Anti-Interference Pretraining for Language Models

    Aug 22, 2026Jinchang Zhu, Haowei He, Yi Ding +5Language Model PretrainingLanguage Model Robustness

    The Hong Kong University of Science and Technology (Guangzhou) · Institute of Artificial Intelligence (TeleAI), China Telecom · University of Macau

  28. Learned, Then Lost: A Measured Single-Example Counterfactual in Pre-training

    Aug 19, 2026Zachary Speck, Asa ShepardLanguage Model PretrainingMemorization in Language Models

    Arizona State University · Williams College

  29. Beyond the pale: Assessing prevalence and contents of extremist speech in LLM training data

    Aug 14, 2026Dmitry Nikolaev, Ashley A. MattheisLanguage Model PretrainingTraining Data Curation

    University of Manchester

  30. Scaling Domain Data Repetition in LLM Pretraining

    Aug 14, 2026Jingwei Li, Xinran Gu, Rui Dai +5LLM TrainingData Repetition in Language Model Training

    Tsinghua University · ByteDance Seed