Language Model Pretraining

Ask about this topic

Overview

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

CardsList

448 papers72 papers in the last four weeks, up 57% on the four weeks before. 0.4% of all new papers.

  1. Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling

    Sep 26, 2026Zailin Ma, Quzhe Huang, Yujun Li +2Language Model Pretraining

    Peking University · Wizard Intelligence Learning Lab · Tsinghua University

  2. A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents

    Sep 26, 2026Indranil Halder, Rastri Dey, Cengiz PehlevanData Poisoning AttacksLanguage Model Pretraining

    John A. Paulson School of Engineering and Applied Sciences at Harvard University · PACCAR Inc. · Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University +1

  3. IndustryLLM: Failure-Driven LLM Training for Industrial Procurement

    Sep 25, 2026Liang Ding, Zhiang Xu, Yuyang Sheng +13LLM Fine-TuningDomain Adaptation for LLMs

    Project Lead · Multimodal and Industrial AI Team†∗ Alibaba

  4. Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence

    Sep 24, 2026Dries Rooryck, Alex Cai, Yonatan Belinkov +2Multilingual Language ModelsLanguage Model Pretraining

    Kempner Institute, Harvard University, Cambridge, MA, USA · Technion – Israel Institute of Technology, Haifa, Israel

  5. Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretraining

    Sep 21, 2026Adam Ousherovitch, Yixin WangLanguage Model PretrainingLearning Rate Scheduling

    Department of Statistics, University of Michigan

  6. PermuFormer: Multi-Task Pretraining for Permutation Representation in Algebraic Combinatorics

    Sep 21, 2026Henry KvingeLanguage Model PretrainingMulti-Task Learning

    Pacific Northwest National Laboratory · University of Washington

  7. Beetle: A Bilingual Model Suite for Modelling Second-Language Processing

    Sep 18, 2026Suchir Salhan, Catherine Arnett, James Michaelov +1Multilingual Language ModelsLanguage Model Pretraining

    University of Cambridge · EleutherAI · MIT

  8. ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratch

    Sep 18, 2026Sait Furkan TekeLanguage ModelingLanguage Model Pretraining

    ufak AI

  9. Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining

    Sep 18, 2026Nghia Hieu Nguyen, Thai Bao Huynh, Binh-An Dinh-Le +4TokenizationLanguage Model Pretraining

    University of Information Technology, Vietnam National University, Ho Chi Minh city, Viet Nam · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE, Abu Dhabi, UEA

  10. Relational Attention for Data-Efficient Language Modeling

    Sep 17, 2026Adrian Brasoveanu, Ece Takmaz, Jakub DotlačilLanguage Model PretrainingDecoder-Only Language Models

    UC Santa Cruz · Utrecht University

  11. AutoData: Agentic Search for Pre-training Data Selection

    Sep 17, 2026Yan Meng, Dhruv Srikanth, Bingchen Zhao +2Language Model PretrainingAutomated Algorithm Discovery

    University of Amsterdam · Weco AI

  12. QVAC Genesis III: A Large-Scale, High-Quality Open Synthetic STEM Corpus for Efficient Language Model Pre-Training

    Sep 17, 2026Davide Vitabile, N. Ranjan, Akshay Nambiar +2Language Model PretrainingSynthetic Data Pretraining

    Tether Data, S.A. de C.V. d.b.a. Tether AI Research

  13. Why Pretraining Fails to Share Cross-Lingual Knowledge

    Sep 16, 2026Adam Gaber, Uriel Dolev, Elisabeth Fittschen +3Language Model PretrainingMultilingual Language Models

    Weizmann Institute of Science · Bar-Ilan University · Johns Hopkins University +4

  14. Size Matters: Foundation Model for Czech HTML documents

    Sep 16, 2026Martin Dvořák, Vít Tlustoš, Artyom Voronin +4Language Model PretrainingEfficient Language Model Training

    Seznam.cz, Prague, Czech Republic

  15. MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training

    Sep 14, 2026Keunyoung Kim, Nojun KwakLanguage Model PretrainingLow-Rank Approximation

    Seoul National University

  16. Towards a knowledge-enhanced single-cell foundation model

    Sep 14, 2026Hanqing Zhang, Jie Bao, Mei Ma +7Language Model PretrainingSingle-Cell Transcriptomics

    College of Animal Science and Technology, China Agricultural University, Beijing 100193, China · College of Information and Electrical Engineering, China Agricultural University, Beijing 100193, China

  17. Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M

    Sep 13, 2026Dushyant Rajput, Nirdesh Chauhan, Siddharth KosarajuLanguage Model PretrainingSmall Language Models

    AltSlate Labs LLP

  18. POSPAN: Position-Constrained Span Masking for Language Model Pre-training

    Sep 13, 2026Zhenyu Zhang, Lei Shen, Yuming Zhao +2Language Model PretrainingMasked Language Modeling

    JD AI Research China

  19. URCHIN: A Horizontal Spiking Language Model for Data-Constrained Pretraining

    Sep 12, 2026Po-Han ChiangSpiking Language ModelsLanguage Modeling

    National Yang Ming Chiao Tung University

  20. Structural priors for data-efficient language learning

    Sep 10, 2026Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager +1Language Model PretrainingNeural Network Initialization

    University of Göttingen, Germany

  21. NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

    Sep 9, 2026The Intern-NCP Team, :, Jiaqi Cao +26Language Model PretrainingAutoregressive Language Modeling

  22. Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling

    Sep 9, 2026Tingshuo Fan, Hongtao Mu, Tianyu Zhou +2Language Model PretrainingSmall Language Models

    College of Foreign Languages and Literature, Fudan University

  23. Edu-QuRating: Multi-Dimensional Educational Data Curation with Distilled Pairwise Judgements

    Sep 8, 2026Oliver G. B. Garrod, Robin A. A. Ince, Meng Liu +6Pairwise Preference EvaluationPairwise Preference Learning

    Fab AI

  24. Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding

    Sep 8, 2026Fengxiang Bie, Yuqing Jian, Yifan Yu +7Language Model PretrainingSpeculative Decoding

    Together AI · The University of Sydney, Australia · University of Illinois Urbana-Champaign, USA +1

  25. Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

    Sep 8, 2026Sohir Maskey, Philipp Scholl, Jonas Knupp +2Language Model PretrainingNeural Network Optimization

    Aleph Alpha

  26. Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Web

    Sep 7, 2026Gonçalo Vinagre, Rui Pedro Guerra, Pedro Gomes +7Language Model PretrainingLanguage Identification

    NOVA School of Science and Technology · NOVA LINCS · Fundação para a Ciência e Tecnologia +2

  27. Format-Aware Fusion for Fast FP4 Pretraining

    Sep 4, 2026Robert HuLow-Precision TrainingLLM Quantization

    GRAPHCORE RESEARCH

  28. Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views

    Sep 3, 2026Joseph Lee, Yidi Huang, Dokyoon Kim +2Language Model PretrainingData Augmentation

    University of Pennsylvania, Philadelphia, PA, USA

  29. Almost Free State Prediction Separation

    Sep 3, 2026John Langford, Nathan Godey, Giovanni Monea +5Language Model PretrainingEfficient Language Model Training

    Microsoft · Cornell University

  30. ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models

    Sep 3, 2026Quang Hoang Trung, Quang Huu Hieu, Nguyen Van Hoang Phuc +1Language Model PretrainingAutoregressive Language Modeling

    VJ Technologies, Da Nang, Vietnam · AJ Technologies, Nagoya, Japan · Vietnam National University, Ho Chi Minh City, Vietnam +1