Language Model Pretraining

Follow this topic Ask about this topic

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

448 papers72 papers in the last four weeks, up 57% on the four weeks before. 0.4% of all new papers.

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
Apr 29, 2026Théo Gigant, Bowen Peng, Jeffrey QuesnelleLanguage Model Pretraining Efficient Language Model Training
Nous Research
Staged Factorial Screening for Budget-Constrained Micro-Pretraining
Apr 27, 2026Felipe Chavarro PolaniaLanguage Model Pretraining Hyperparameter Optimization
Hewlett Packard Enterprise
GEM: Geometric Entropy Mixing for Optimal LLM Data Curation
Apr 27, 2026Yue Min, Ziyun Qiao, Ruining Chen +1Language Model Pretraining Training Data Curation
Wizard Quant, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Peking University, Beijing, China +1
PermaFrost-Attack: Stealth Pretraining Seeding(SPS) for planting Logic Landmines During LLM Training
Apr 23, 2026Harsh Kumar, Rahul Maity, Tanmay Joshi +4Language Model Pretraining LLM Backdoor Attacks
Manipal University Jaipur · NIT, Karnataka · Pragya Lab, BITS Pilani Goa, India +2
Decoupled DiLoCo for Resilient Distributed Pre-training
Apr 23, 2026Arthur Douillard, Keith Rush, Yani Donchev +14Language Model Pretraining Distributed Optimization
1Google DeepMind · 2Google Research
Listen and Chant Before You Read: The Ladder of Beauty in LM Pre-Training
Apr 23, 2026Yoshinori NomuraLanguage Model Pretraining Audio-Language Models
Mirage Mountain Technologies Inc.
Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection
Apr 22, 2026Yassine Turki, Vinko Sabolčec, Bettina Messmer +1Language Model Pretraining Multilingual Language Models
Machine Learning Optimization Lab Ecole Polytechnique Fédérale de Lausanne (EPFL) Lausanne, Switzerland
Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?
Apr 20, 2026H S V N S Kowndinya Renduchintala, Sumit BhatiaLanguage Model Pretraining Language Modeling
Media and Data Science Research Adobe Inc., India
Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
Apr 19, 2026Felicia Körner, Maria Matveev, Florian Eichin +3Language Model Pretraining Multilingual Language Models
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML) · Department of Mathematics, LMU Munich, Germany +2
Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods
Apr 19, 2026Wanru Zhao, Yihong Chen, Yuzhi Tang +6Language Model Pretraining LLM Fine-Tuning
University of Cambridge · OATML, University of Oxford · University of Toronto +2
In Search of Lost DNA Sequence Pretraining
Apr 17, 2026Zhijiang Tang, Jiaxin Qi, Yan Cui +3Language Model Pretraining Masked Language Modeling
Computer Network Information Center, Chinese Academy of Sciences, Beijing, China · Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Zhejiang, China
Target-Oriented Pretraining Data Selection via Neuron-Activated Graph
Apr 17, 2026Zijun Wang, Haoqin Tu, Weidong Zhou +7Language Model Pretraining Training Data Selection
1ByteDance · 2UC Santa Cruz · 3UNC-Chapel Hill
KoCo: Conditioning Language Model Pre-training on Knowledge Coordinates
Apr 14, 2026Yudong Li, Jiawei Cai, Linlin ShenLanguage Model Pretraining Knowledge Augmentation for Language Models
School of Computer Science and Software Engineering, Shenzhen University · Department of Electrical Engineering, Tsinghua University · School of Artificial Intelligence, Shenzhen University +1
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Apr 14, 2026NVIDIA, :, Aakshita Chandiramani +544Hybrid Transformer-State Space Model Architectures Mixture-of-Experts Language Models
ks-pret-5m: a 5 million word, 12 million token kashmiri pretraining dataset
Apr 13, 2026Haq Nawaz Malik, Nahfid NissarLow-Resource Language Processing Language Model Pretraining
Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series
Apr 12, 2026Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas +2Language Modeling Subword Tokenization
SpeakLeash · Azurro · ACK Cyfronet AGH +2
Bridging Linguistic Gaps: Cross-Lingual Mapping in Pre-Training and Dataset for Enhanced Multilingual LLM Performance
Apr 12, 2026Weihua Zheng, Chang Liu, Zhengyuan Liu +5Multilingual Language Models Cross-Lingual Representation Alignment
Singapore University of Technology and Design, Singapore · ByteDance, Singapore · Agency for Science, Technology and Research, Singapore
Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima
Apr 10, 2026Huanran Chen, Huaqing Zhang, Xiao Li +3Language Model Pretraining Neural Network Generalization
Tsinghua University · ByteDance Seed
HiFloat4 Format for Language Model Pre-training on Ascend NPUs
Apr 9, 2026Mehran Taghian, Yunke Peng, Xing Huang +22Low-Precision Training LLM Quantization
Huawei
Static Program Slicing Using Language Models With Dataflow-Aware Pretraining and Constrained Decoding
Apr 9, 2026Pengfei He, Shaowei Wang, Tse-Hsun Chen +1Code Language Models Language Model Pretraining
Department of Computer Science, University of Manitoba, Winnipeg, Canada · School of Engineering and Computer Science, Concordia University, Montreal, Canada · School of Computer Science, University of Windsor, Windsor, Canada
What do Language Models Learn and When? The Implicit Curriculum Hypothesis
Apr 9, 2026Emmy Liu, Kaiser Sun, Millicent Li +4Emergent Abilities in Language Models Language Model Pretraining
Language Technologies Institute, Carnegie Mellon University · Department of Computer Science, Data Science and AI Institute, Johns Hopkins University · Khoury College of Computer Science, Northeastern University +1
Improving DNS Exfiltration Detection via Transformer Pretraining
Apr 9, 2026Miloš Tomić, Aleksa Cvetanović, Predrag TadićNetwork Intrusion Detection Domain-Adaptive Pretraining
School of Electrical Engineering, University of Belgrade, Serbia
Soro: A Lightweight Foundation Model and Chatbot for Tajik
Apr 9, 2026Stanislav Liashkov, Haitz Sáez de Ocáriz Borde, Azizjon Azimi +3Low-Resource Language Processing LLM Fine-Tuning
SAGE: Sign-Adaptive Gradient for Memory-Efficient LLM Optimization
Apr 9, 2026Wooin Lee, Hyun-Tae KimLLM Input Optimization Memory-Efficient Optimization
London, United Kingdom · Seoul, Republic of Korea
Learning is Forgetting: LLM Training As Lossy Compression
Apr 8, 2026Henry C. Conklin, Tom Hosking, Tan Yi-Chern +5Language Model Pretraining LLM Compression
Princeton University · Cohere
How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipeline
Apr 5, 2026Mir Tafseer Nayeem, Davood RafieiLanguage Model Pretraining LLM Auditing
Department of Computing Science University of Alberta
Olmo Hybrid: From Theory to Practice and Back
Apr 3, 2026William Merrill, Yanhong Li, Tyler Romero +19Hybrid Language Models Hybrid Transformer-State Space Model Architectures
Allen Institute for AI · ETH Zürich · Lambda ML Team +2
Test-Time Scaling Makes Overtraining Compute-Optimal
Apr 1, 2026Nicholas Roberts, Sungjun Cho, Zhiqi Gao +7Scaling Laws Language Model Scaling Laws
University of Wisconsin-Madison · Stanford University
GPT-NL Public Corpus: A Permissively Licensed, Dutch-First Dataset for LLM Pre-training
Apr 1, 2026Jesse van Oort, Frank Brinkkemper, Erik de Graaf +2Language Model Pretraining Low-Resource Language Processing
TNO, The Netherlands · GPT-NL, The Netherlands · Instituut voor de Nederlandse Taal, The Netherlands +1
Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer
Apr 1, 2026Dharma Teja Vooturi, Dhiraj Kalamkar, Dipankar Das +1Language Model Pretraining LLM Training
Parallel Computing Lab (India) Intel Corporation

Apr 29, 2026J/K move · Enter open · S save

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation

Théo Gigant, Bowen Peng, Jeffrey Quesnelle

Nous Research

Language Model Pretraining Efficient Language Model Training Language Modeling Subword Tokenization

Open paper AskPDF arXiv

Subword tokenization is an essential part of modern large language models (LLMs), yet its specific contributions to training efficiency and model performance remain poorly understood. In this work, we decouple the effects of subword tokenization by isolating them within a controlled byte-level pretraining pipeline. We formulate and test hypotheses across various dimensions, including sample throughput, vocabulary scaling, and the linguistic prior of subword boundaries. By simulating these effects in a byte-level setting, we refine our understanding of why subword models outperform raw byte models and offer insights to improve the pretraining of future byte-level and subword models. Specifically, our experiments highlight the critical role of increased training throughput and the integration of subword boundaries as either explicit priors or inductive biases.