Language Model Pretraining

Overview

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

CardsList

480 papers52 papers in the last four weeks, down 27% on the four weeks before. 0.4% of all new papers.

  1. Structure--Residual Inductive Biases for Language Model Pretraining

    Nov 17, 2026Hai Huang, Bartosz Żuk, Rassul Magauin +2Language Model PretrainingAutoregressive Language Modeling

  2. Latent Feedback Is a Compute and Data Amplifier

    Nov 16, 2026Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma +2Language Model PretrainingRecurrent Transformers

  3. Bioformer-RetroMAE: retrieval-oriented pretraining of a compact biomedical language model for literature search

    Nov 15, 2026Jialu Li, Li FangBiomedical IRDense Retrieval

  4. Data Attribution Should Steer Pretraining

    Nov 15, 2026Glenn Matlin, Chandreyi Chakraborty, Mohammed Rehan Parwani +8Training Data SelectionTraining Data Curation

  5. Understanding the Role of Reasoning Data in Pretraining

    Nov 14, 2026Jianhao Huang, Allan Zhang, Ali Payani +1Language Model PretrainingStructured Reasoning in Language Models

  6. Geometrically Disentangling Concept Learning from the Language Modeling Loss

    Nov 13, 2026Yupei Wang, Neil Mallinar, Mikhail Belkin +1LLM InterpretabilityConcept Learning

  7. LECAT: Measuring LLM Category Learning from Pretraining Data

    Nov 13, 2026Jimin Mun, Maarten Sap, Yanai ElazarLanguage Model PretrainingLanguage Modeling

  8. BLiMP-RO: A Benchmark of Linguistic Minimal Pairs for Romanian

    Nov 13, 2026Radu-Stefan Popa, Valentin Feroiu, Adina Camelia Bleotu +1LLM EvaluationLanguage Model Pretraining

  9. Evaluating Pretrained Language Semantics for Structured EHR Prediction Tasks

    Nov 12, 2026Ashvin Gupta, Brendan Delaney, Alessandra RussoLongitudinal EHR ModelingEHR Foundation Models

  10. Pretraining Data Statistics Shape the Phases of Learning Entity Comparison in Language Models

    Nov 12, 2026Yik Siu Chan, Jing Huang, Yanai Elazar +1Language Model PretrainingShortcut Learning in Language Models

  11. Beyond Passive Imitation: On-Policy Mid-Training Distillation for Language Models

    Nov 10, 2026Ruize Zheng, Xuyang Liu, Siyu Ren +5Mid-Training for Language ModelsLanguage Model Distillation

  12. An Analytical Model of Compute-limited Multistage Training Pipelines

    Nov 9, 2026Nishil Patel, Jin Hwa Lee, Basile Confavreux +1RL Fine-TuningSupervised Fine-Tuning

  13. Perturbing Inputs Improves Pretraining Distillation

    Nov 8, 2026Ziyang Cai, Yihe Dong, Dimitris Papailiopoulos +1Language Model DistillationKnowledge Distillation

  14. Pruning DNA Sequence

    Nov 7, 2026Faias Satter, Shaker Islam, Aminul IslamDataset PruningEfficient Language Model Training

  15. Compact Greek Language Models from Scratch: Pretraining 500M- and 1B-Parameter Decoder-Only Transformers

    Nov 6, 2026Konstantinos Perifanos, Anne Laure Poite, Giannis Tsilikas +3Language ModelingLanguage Model Pretraining

  16. Different Corpora, Different Skills? A Paired Study of Small Language Models

    Nov 5, 2026Edoardo Vaira, Georgios Chavales, Pierre NuguesLanguage Model PretrainingSmall Language Models

  17. A Night at the LLM Zoo: Data Conditioned Flow Matching for Generating Pretrained LLM Weights

    Nov 4, 2026Scott Biggs, Bedionita Soro, Saumya Gupta +2Weight-Space LearningLanguage Model Pretraining

  18. Loop Before You Leap: What Recurrent Depth Can and Cannot Buy in Data-Limited Pretraining

    Nov 4, 2026Daichi Hotta, Yohei Kobashi, Takeshi Kojima +2Language Model PretrainingRecurrent Transformers

  19. Adapting Qwen3 to Tajik from 0.6B to 8B: Lower Text Loss Did Not Mean Better Understanding

    Nov 4, 2026Tohir Saidzoda, Madina Abdusalomzoda, Nasimjon Rajabov +7LLM Fine-TuningLanguage Model Pretraining

  20. ARLoop: Fully-Open Pre-trained Looped Transformers with Attention Residuals

    Nov 3, 2026Yuekun Yao, Yupei Du, Dawei Zhu +1Recurrent TransformersAttention Residual Connections

  21. DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction

    Nov 2, 2026Xiaohan Qin, Xiangdong Zhang, Yu Wang +4KV CachingLanguage Model Pretraining

  22. Language Models Can Benefit from Domain-Specific Inductive Biases

    Nov 2, 2026Damien Teney, Liangze Jiang, Zachary Shinnick +2Language Model PretrainingEfficient Language Model Training

  23. Accelerating LLM Pretraining with Zeroth-Order Curvature on Idle Serving Compute

    Nov 2, 2026Mingchuan Ma, Dongqi Cai, Daqian Yang +8Language Model PretrainingZeroth-Order Optimization

  24. Saidzoda-Tajik-LFM-2.6B: Adapting a 2.6B Hybrid Model to Tajik Tool Calling, and What Continued Pretraining Takes Away

    Oct 30, 2026Tohir Saidzoda, Nekruz Maxudov, Bedil Karimov +5LLM Tool UseFine-Tuning Language Models

  25. FARO: Function-Space Adaptive Return-to-Risk Optimization via Cross-Block Equal-Risk-Contribution Budget Allocation

    Oct 29, 2026Xiaojun Wu, Cehao Yang, Xuhui Jiang +3Deep Learning OptimizationLLM Training

  26. Rethinking Pre-pretraining

    Oct 29, 2026Alexander Stern, Tim Rocktäschel, Acyr Locatelli +1Language Model PretrainingSynthetic Data Pretraining