Language Model Pretraining

Overview

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

CardsList

480 papers52 papers in the last four weeks, down 27% on the four weeks before. 0.4% of all new papers.

  1. Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-training

    Oct 14, 2026JiWoo Hwang, Lukas Edman, Alexander FraserLanguage Model PretrainingMultilingual Language Models

  2. CAST: History-Adaptive Spectral Clipping for LLM Pretraining

    Oct 14, 2026Jiacheng Li, Jianchao Tan, Kefeng Zhang +2Language Model Pretraining

  3. Qwenfant: Train Big Character-level BabyLMs on Concise Parallel Text

    Oct 13, 2026Lukas Edman, Alexander FraserLanguage ModelingLanguage Model Pretraining

  4. Copy-Paste Language Models

    Oct 13, 2026Nathan Godey, Yoav ArtziLanguage Model PretrainingEfficient Language Model Training

  5. Does Pretraining Teach Chemical Language Models Chemistry?

    Oct 13, 2026Leif Sieben, Yoel ZimmermannChemical Language ModelsLanguage Model Pretraining

  6. The Geometry of Language Model Logits: Generic Structure and Learned Departures

    Oct 13, 2026Swathi Shree Narashiman, Aravind Ramana Venkatramanan, Nambi SeshadriAutoregressive Language ModelingRepresentation Geometry in Language Models

  7. Mixed Precision Scaling Laws for Quantized Pretraining

    Oct 13, 2026Jinuk Kim, Rajarshi Saha, Muhammed Fatih Balin +1Mixed-Precision QuantizationLLM Quantization

  8. Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions

    Oct 12, 2026Ju-Chieh Chou, Jiawei Zhou, Karen LivescuSpeech Language ModelsLanguage Model Distillation

  9. Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining

    Oct 12, 2026Alexander Miasnikov, Ruslan Kabirov, Gleb Molodtsov +7Muon OptimizerLLM Training

  10. Latent Taxonomies: How Property Induction Develops in Language Model Pre-training

    Oct 11, 2026Jan Sobotka, Ekaterina Goliakova, Marcin Detyniecki +2Inductive Reasoning in Language ModelsInductive Inference

  11. A Sovereign, Indic-First, Reasoning Model

    Oct 11, 2026Sutanshu Saksena Raj, Prasanjith Pasunuti, Parijat Chatterjee +4Language ModelingCross-Lingual Reasoning in Language Models

  12. UltraBERT: Beyond the Limit of BERT Pre-Training

    Oct 10, 2026Jungwoo Park, Taewhoo Lee, Jaewoo KangLanguage Model PretrainingMasked Language Modeling

  13. OpenEBFM: Energy-based Foundation Models Are Scalable Architecture

    Oct 7, 2026Yuan Pu, Yudong Lu, Ting Wang +6Language ModelingLanguage Model Pretraining

  14. Sequential Pretraining Favors Large Models

    Oct 7, 2026Mohnish Harwani, Yujia ZhengLanguage Model PretrainingContinued Pretraining

  15. Domain adaptation of Russian ModernBERT for long legal documents

    Oct 5, 2026I. Litvak, D. Gvozdetsky, F. Lashkin +7Language Model PretrainingLegal NLP

  16. LightMTP: Lightweight Latent Multi-Token Prediction

    Oct 5, 2026Tamara Czinczoll, Julie Kallini, Gerard de Melo +1Language Model PretrainingMulti-Token Prediction

  17. Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention

    Oct 5, 2026Hong Ha Le, Jackie Lok, Atsushi Nitanda +1Language Model PretrainingSoftmax Attention

  18. Learning to Learn a Language

    Oct 5, 2026Lennart Carstens-Behrens, Holger FröhlichLanguage Model PretrainingMeta-Learning