Language Model Pretraining

Ask about this topic

Overview

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

CardsList

448 papers72 papers in the last four weeks, up 57% on the four weeks before. 0.4% of all new papers.

  1. To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval

    Apr 1, 2026Karan Singh, Michael Yu, Varun Gangal +4Retrieval-Augmented GenerationKnowledge Augmentation for Language Models

    Stanford University · †DegenAI Labs · Independent Researcher +3

  2. Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models

    Mar 31, 2026Christine Zhang, Dan Jurafsky, Chen ShaniLanguage ModelingLLM Alignment

    Stanford University

  3. On the limited utility of parallel data for learning shared multilingual representations

    Mar 30, 2026Julius Leino, Jörg TiedemannCross-Lingual Representation AlignmentMultilingual Language Models

    University of Helsinki

  4. daVinci-LLM:Towards the Science of Pretraining

    Mar 28, 2026Yiwei Qin, Yixiu Liu, Tiantian Mi +12Language Model PretrainingLLM Training

    SII · SJTU · GAIR

  5. Data Mixing for Large Language Models Pretraining: A Survey and Outlook

    Mar 25, 2026Zhuo Chen, Yuxuan Miao, Supryadi +1Data Mixture OptimizationLanguage Model Pretraining

    The International Joint Institute of Tianjin University, Tianjin University, Tianjin 300072, China · TJUNLP Lab, School of Computer Science and Technology, Tianjin University, Tianjin 300350, China

  6. Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge

    Mar 21, 2026Bhavya Vasudeva, Puneesh Deora, Alberto Bietti +2In-Context LearningImplicit Reasoning in Language Models

    University of Southern California · University of British Columbia · Flatiron Institute

  7. SozKZ: Training Efficient Small Language Models for Kazakh from Scratch

    Mar 21, 2026Saken TukenovLanguage Model PretrainingSmall Language Models

    Independent Researcher

  8. MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages

    Mar 21, 2026Anri Lombard, Simbarashe Mawere, Temi Aina +5African Language ProcessingMultilingual Language Models

    University of Cape Town, Cape Town, South Africa

  9. Data-efficient pre-training by scaling synthetic megadocs

    Mar 19, 2026Konwoo Kim, Suhas Kotha, Yejin Choi +3Language Model PretrainingSynthetic Data Pretraining

    Stanford University

  10. Humans and transformer LMs: Abstraction drives language learning

    Mar 18, 2026Jasper Jian, Christopher D. ManningLLM TrainingLanguage Model Pretraining

    Stanford University

  11. Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model

    Mar 17, 2026Ryo Kishino, Riku Shiomi, Hiroaki Yamagiwa +2Language Model PretrainingLLM Alignment

    1Kyoto University · 2RIKEN

  12. Fanar 2.0: Arabic Generative AI Stack

    Mar 17, 2026FANAR TEAM, Ummar Abbas, Mohammad Shahmeer Ahmad +34Arabic Language ModelsArabic NLP

    Qatar Computing Research Institute (QCRI), Hamad Bin Khalifa University

  13. PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Development

    Mar 17, 2026Hanif RahmanLow-Resource Language ProcessingLanguage Model Pretraining

    Independent Researcher

  14. The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data

    Mar 17, 2026Christina Baek, Ricardo Pio Monti, David Schwab +31Language Model PretrainingDomain-Adaptive Pretraining

    DatologyAI

  15. NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling

    Mar 17, 2026Zhengzheng TangSpiking Language ModelsSpiking Neural Networks

    Boston University

  16. SIA: A Synthesize-Inject-Align Framework for Knowledge-Grounded and Secure E-commerce Search LLMs with Industrial Deployment

    Mar 17, 2026Zhouwei Zhai, Mengxiang Chen, Anmeng ZhangE-Commerce SearchKnowledge Augmentation for Language Models

    JD.com Beijing, China

  17. Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning

    Mar 17, 2026Kazuki Yano, Shun Kiyono, Sosuke Kobayashi +2Learning Rate SchedulingLanguage Model Pretraining

    Tohoku University · SB Intuitions

  18. A Family of LLMs Liberated from Static Vocabularies

    Mar 16, 2026Aleph Alpha, :, Adnen Abdessaied +35Language ModelingByte-Level Language Model

    Aleph Alpha Research

  19. Attention Residuals

    Mar 16, 2026Kimi Team, Guangyu Chen, Yu Zhang +34Attention Residual ConnectionsResidual Learning

  20. Pretraining and Benchmarking Modern Encoders for Latvian

    Mar 16, 2026Arturs ZnotinsLanguage Model PretrainingLow-Resource Language Processing

    Institute of Mathematics and Computer Science, University of Latvia, 29 Raina bulv., Riga, LV-1459, Latvia

  21. Data Darwinism Part II: DataEvolve -- AI can Autonomously Evolve Pretraining Data Curation

    Mar 15, 2026Tiantian Mi, Dongming Shan, Zhen Huang +6Training Data CurationLanguage Model Pretraining

    SII · FDU · GAIR +2

  22. Enhancing LLM Training via Spectral Clipping

    Mar 15, 2026Xiaowen Jiang, Andrei Semenov, Sebastian U. StichLLM Input OptimizationSpectral Regularization

    CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Universität des Saarlandes, Saarbrücken, Germany · EPFL

  23. FLUX: Data Worth Training On

    Mar 14, 2026Gowtham, Sai Rupesh, Sanjay Kumar +2Training Data CurationLLM Training

  24. BERTology of Molecular Property Prediction

    Mar 13, 2026Mohammad Mostafanejad, Paul Saxe, T. Daniel CrawfordChemical Language ModelsMolecular Property Prediction

    Department of Chemistry, Virginia Tech, Blacksburg, Virginia 24061, USA · Molecular Sciences Software Institute, Blacksburg, Virginia 24060, USA

  25. Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study

    Mar 13, 2026Liel Binyamin, Elior SulemLanguage Model PretrainingLanguage Modeling

    Faculty of Computer and Information Science, Institute for Applied AI Research Data Science Research Center Ben-Gurion University of the Negev

  26. Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stages

    Mar 13, 2026Hongyang Chen, Zhongwu Sun, Hongfei Ye +2Continual Learning for LLMsCatastrophic Forgetting

    Zhejiang Lab, Hangzhou 311100, China

  27. Long-Context Encoder Models for Polish Language Understanding

    Mar 12, 2026Sławomir Dadas, Rafał Poświata, Marek Kozłowski +4Long-Context Language ModelingLong-Context Modeling

    National Information Processing Institute, Warsaw, Poland · PKO Bank Polski, Warsaw, Poland

  28. DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining

    Mar 12, 2026Yutong Yan, Raphael Tang, Zhenyu Gao +2Look-Ahead Bias in Language ModelsLanguage Model Pretraining

    Department of Finance, CUHK Business School, The Chinese University of Hong Kong · Centre for Artificial Intelligence, University College London

  29. H2LooP Spark Preview: Continual Pretraining of Large Language Models for Low-Level Embedded Systems Code

    Mar 11, 2026Amit Singh, Vedant Nipane, Pulkit Agrawal +2Code Language ModelsLLM Code Generation

  30. Sabiá-4 Technical Report

    Mar 10, 2026Thiago Laitz, Thales Sales Almeida, Hugo Abonizio +6Portuguese Language ModelsLanguage Modeling

    Maritaca AI · Jusbrasil