Language Model Pretraining

Follow this topic Ask about this topic

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

448 papers72 papers in the last four weeks, up 57% on the four weeks before. 0.4% of all new papers.

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
Jan 29, 2026Ajay Patel, Colin Raffel, Chris Callison-BurchLanguage Model Pretraining Synthetic Data Pretraining
University of Pennsylvania · University of Toronto · Vector Institute +1
MAPLE: Metadata Conditioned LLM Pretraining for Locale-Aware Question Answering
Jan 21, 2026Anjishnu Mukherjee, Ziwei Zhu, Antonios AnastasopoulosLanguage Model Pretraining LLM Evaluation
George Mason University
Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model Pretraining
Jan 21, 2026Jingru Li, Yibo Fan, Huan LiStochastic Optimization Language Model Pretraining
College of Artificial Intelligence, Nankai University
AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
Jan 10, 2026Hao Yu, Tianyi Xu, Michael A. Hedderich +3Language Model Pretraining Multilingual Language Models
McGill University, Canada · Mila-Quebec AI Institute, Canada · LMU Munich & Munich Center for Machine Learning, Germany +2
Output Embedding Centering for Stable LLM Pretraining
Jan 5, 2026Felix Stollenwerk, Anna Lokrantz, Niclas HertzbergLanguage Model Pretraining Neural Representation Geometry
AI Sweden
TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish
Dec 28, 2025Melikşah Türker, A. Ebrar Kızıloğlu, Onur Güngör +1Language Model Pretraining Long-Context Language Modeling
Department of Computer Engineering, Boğaziçi University, Türkiye · VNGRS, Istanbul, Türkiye · Technical University of Munich (TUM), Munich, Germany
Predicting the Emergence of Induction Heads in Language Model Pretraining
Nov 21, 2025Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan SchneiderLanguage Model Pretraining In-Context Learning
Department of Linguistics, Georgetown University.
Scaling Latent Reasoning via Looped Language Models
Oct 29, 2025Rui-Jie Zhu, Zixuan Wang, Kai Hua +30Language Model Pretraining Implicit Reasoning in Language Models
ByteDance Seed · UC Santa Cruz · Princeton University +8
Geometrically Principled Randomized Optimization for Efficient LLM Training
Oct 2, 2025Sahar Rajabi, Nayeema Nonta, Sirisha RambhatlaLanguage Model Pretraining Deep Learning Optimization
CriticalML Lab, University of Waterloo
How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off
Oct 1, 2025Waïss Azizian, Ali HasanLanguage Model Pretraining OOD Generalization
Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LJK, 38000 Grenoble, France · Work done during an internship at Morgan Stanley Machine Learning Research. · Machine Learning Research, Morgan Stanley, New York, USA
Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining
Oct 1, 2025Thiziri Nait Saada, Louis Bethune, Michal Klein +3Language Model Pretraining Training Data Curation
Work done as an intern at Apple · University of Oxford · Apple
Predicting LLM Reasoning Performance with Small Proxy Model
Sep 25, 2025Woosung Koh, Juyoung Suk, Sungjun Han +2Language Model Pretraining LLM Evaluation
Trillion Labs · KAIST AI · ▽Trillion Labs, ♠KAIST AI
Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining
Sep 12, 2025Rupert Mitchell, Kristian KerstingLanguage Model Pretraining Softmax Attention
Department of Computer Science, TU Darmstadt, Darmstadt, Germany · Hessian Center for AI (hessian.AI), Darmstadt, Germany · German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany +1
LMEnt: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations
Sep 3, 2025Daniela Gottesman, Alon Gilae-Dotan, Ido Cohen +4Language Model Pretraining Knowledge Editing
Tel Aviv University · Mila – Quebec AI Institute & McGill University
Fantastic Pretraining Optimizers and Where to Find Them
Sep 2, 2025Kaiyue Wen, David Hall, Tengyu Ma +1Deep Learning Optimization Language Model Pretraining
Stanford University
Dataset Construction for Training LLM to Learn Analog Circuit Knowledge
Aug 14, 2025Zihao Chen, Ji Zhuang, Jinyi Shen +11Language Model Pretraining Knowledge Augmentation for Language Models
Fudan University, Shanghai, China
SinLlama -- A Large Language Model for Sinhala
Aug 12, 2025H. W. K. Aravinda, Rashad Sirajudeen, Samith Karunathilake +3Language Model Pretraining Multilingual Language Models
Dept. of Computer Science & Engineering, University of Moratuwa, Sri Lanka. · School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand. · Central Scientific Instruments Organisation, Academy of Scientific and Innovative Research, Chandigarh, India. +1
RooseBERT: A New Deal For Political Language Modelling
Aug 5, 2025Deborah Dore, Elena Cabrio, Serena VillataLanguage Model Pretraining Argument Mining
Universit´e Cˆote d’Azur CNRS, INRIA, I3S Sophia Antipolis, France
Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens
Jun 30, 2025Salahuddin Salahuddin, Ahmed Hussain, Jussi Löppönen +1Language Model Pretraining Domain-Adaptive Pretraining
SSH Communications Security, Helsinki, Finland · Aalto University, Espoo, Finland · Networked Systems Security (NSS) Group, KTH Royal Institute of Technology, Stockholm, Sweden
Curriculum-Guided Layer Scaling for Language Model Pretraining
Jun 13, 2025Karanpartap Singh, Neil Band, Ehsan AdeliLanguage Model Pretraining Efficient Language Model Training
1Stanford University.
Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training
Jun 2, 2025Pierre-Carl Langlais, Pavel Chizhov, Catherine Arnett +7Language Model Pretraining Training Data Curation
PleIAs, Paris, France · CAIRO, Technical University of Applied Sciences Würzburg-Schweinfurt, Germany · EleutherAI
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
May 26, 2025Melis Ilayda Bal, Volkan Cevher, Michael MuehlebachLanguage Model Pretraining Efficient Language Model Training
Max Planck Institute for Intelligent Systems, Tübingen, Germany
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
May 19, 2025Jitai Hao, Qiang Huang, Hao Liu +3Language Model Pretraining LLM Pruning
School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen · Baidu Inc. · Leiden University +1
Foundations of Large Language Models
Jan 16, 2025Tong Xiao, Jingbo ZhuLanguage Model Pretraining LLM Alignment
NLP Lab, Northeastern University & NiuTrans Research
All Entities are Not Created Equal: Examining the Long Tail for Ultra-Fine Entity Typing
Oct 22, 2024Advait Deshmukh, Ashwin Umadi, Dananjay Srinivas +1Language Model Pretraining Long-Tail Learning
University of Colorado Boulder
Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian
May 22, 2024Aleksandr Nikolich, Konstantin Korolev, Sergei Bratchikov +2Language Model Pretraining Multilingual Language Models
ITMO University · HSE University · Vikhr +2
SynGhost: Invisible and Universal Task-agnostic Backdoor Attack via Syntactic Transfer
Feb 29, 2024Pengzhou Cheng, Wei Du, Zongru Wu +4Language Model Pretraining LLM Backdoor Attacks
School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University · Department of Computer Science and Engineering Southern University of Science and Technology
DA-Cramming: Enhancing Cost-Effective Language Model Pretraining with Dependency Agreement Integration
Nov 8, 2023Martin Kuo, Jianyi Zhang, Dongting Li +1Language Model Pretraining Efficient Language Model Training
Center for Computational Evolutionary Intelligence, Duke University

Jan 29, 2026J/K move · Enter open · S save

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Ajay Patel, Colin Raffel, Chris Callison-Burch

University of Pennsylvania · University of Toronto · Vector Institute+1

Language Model Pretraining Synthetic Data Pretraining Synthetic Data Generation Synthetic Instruction Generation Instruction Tuning Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .