Language Model Pretraining

Follow this topic Ask about this topic

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

448 papers72 papers in the last four weeks, up 57% on the four weeks before. 0.4% of all new papers.

DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training
Jun 27, 2026Dong Wang, Wenwu Tang, Yun Cheng +1Language Model Pretraining LLM Compression
Graz University of Technology, Austria · Swiss Data Science Center, Switzerland
LA4VLA: Learning to Act without Seeing via Language-Action Pretraining
Jun 25, 2026Tao Lin, Yuxin Du, Yiran Mao +13Language Model Pretraining Visuomotor Policy Learning
School of AI, Shanghai Jiao Tong University · Alibaba Group · KAUST +1
Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining
Jun 24, 2026Juliana Li, Diya SreedharLanguage Model Pretraining LLM Unlearning
Harvard University, Cambridge, MA, USA.
Improved Large Language Diffusion Models
Jun 24, 2026Shen Nie, Qiyang Min, Shaoxuan Xu +7Language Model Pretraining Masked Diffusion Models
Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +1
Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models
Jun 23, 2026Ahmad Pouramini, Hesham FailiLanguage Model Pretraining Fine-Tuning
School of Electrical and Computer Engineering, College of Engineering, University of Tehran, North Kargar Street, Tehran, 515-14395, Tehran, Iran.
Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning
Jun 23, 2026Chenhao Dang, Jing Ma, Mingjie LiaoLanguage Model Pretraining
China Electronics Technology Group Corporation 15th Research Institute · Renmin University of China BRAIN Beijing, China · Alibaba Group Beijing, China
moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT
Jun 21, 2026Thiago Laitz, Thales Sales Almeida, João Guilherme Alves Santos +1Language Model Pretraining Portuguese Language Models
UNICAMP, Campinas, Brazil · Tropic AI · Maritaca AI
L20-Edu-135M: An Auditable Single-GPU Study of Data-Efficient Small Language Modeling
Jun 20, 2026Yin LiLanguage Model Pretraining Small Language Models
University of Birmingham
Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining
Jun 20, 2026Bofeng Huang, Jacques Sun, Diane Bouchacourt +2Language Model Pretraining Masked Language Modeling
Doctolib · Work done while at Doctolib.
Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
Jun 18, 2026Qian Zhao, Kunlong Chen, Changxin Tian +9Language Model Pretraining LLM Quantization
Ling Team, Ant Group
IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources
Jun 18, 2026Arash Ghafouri, Mahdi Firouzmandi, Hossein Saberi +1Language Model Pretraining Language Modeling
Department of Artificial Intelligence and Cognitive Science, Imam Hossein Comprehensive University, Tehran, Iran.
Characterizing Narrative Content in Web-scale LLM Pretraining Data
Jun 17, 2026Teagan Johnson, Elliott Ash, Andrew Piper +1Document Understanding Language Model Pretraining
University of Colorado Boulder · ETH Zürich · McGill University
Dango: A Strictly L1-Only Large Language Model for Studying Second Language Acquisition
Jun 17, 2026Shiho Matta, Yin Jou Huang, Fei Cheng +3Language Model Pretraining Data Contamination in Language Models
Kyoto University · NII-LLMC
Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
Jun 17, 2026Jinhan Li, Kexian Tang, Yihan Xu +2Language Model Pretraining LLM Alignment
Institute for Interdisciplinary Information Sciences, Tsinghua University
Sumi: Open Uniform Diffusion Language Model from Scratch
Jun 17, 2026Mengyu Ye, Keito Kudo, Wataru Ikeda +3Language Model Pretraining Diffusion Language Models
1Tohoku University
RegMix-D: Dynamic Data Mixing via Proxy Training Trajectories
Jun 17, 2026Kaiyan Zhao, Zhongtao Miao, Akiko Aizawa +1Language Model Pretraining Training Data Selection
The University of Tokyo · National Institute of Informatics
Small Initialization Matters for Large Language Models
Jun 16, 2026Liangkai Hang, Junjie Yao, Zhiyu Li +3Language Model Pretraining Neural Network Initialization
School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China. · MemTensor (Shanghai) Technology Co., Ltd.. for Advanced Algorithms Research, Shanghai, China. · Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China.
Nothing from Something: Can a Language Model Discover 0?
Jun 15, 2026Phoebe Zeng, Thomas L. Griffiths, Brenden M. LakeNumerical Reasoning in Language Models Language Model Pretraining
Department of Computer Science, Princeton University
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
Jun 15, 2026Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2Language Model Pretraining Weight Decay
Stanford University. · Princeton University. · Tsinghua University.
CraBERT: Efficient Phoneme Encoder Pre-Training via Cascade Fusion of Subword Representations for Text-to-Speech
Jun 15, 2026Dong Yang, Yuki Saito, Wataru Nakata +1Language Model Pretraining Representation Learning
The University of Tokyo, Japan
Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining
Jun 15, 2026Michael K. Chen, Xikun Zhang, Fan Bai +2Language Model Pretraining Data Augmentation
1UC San Diego · 2RMIT University · 3Bloomberg AI
Spokes: Optimizing for Diverse Pretraining Data Selection
Jun 13, 2026Clarence Lee, Yejin Choi, Luke Zettlemoyer +2Language Model Pretraining Diversity-Aware Sampling
DSO National Laboratories · Stanford University · University of Washington
FastMix: Fast Data Mixture Optimization via Gradient Descent
Jun 12, 2026Haoru Tan, Sitong Wu, Yanfeng Chen +5Language Model Pretraining Data Mixture Optimization
University of Hong Kong · 2Hunyuan LLM, Tencent · 3Chinese University of Hong Kong
Small LLMs: Pruning vs. Training from Scratch
Jun 12, 2026Yufeng Xu, Taiming Lu, Kunjun Li +3Language Model Pretraining LLM Pruning
Princeton University · New York University · Carnegie Mellon University
Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality
Jun 9, 2026Vedant Badoni, Danqi Chen, Xinyi WangLanguage Model Pretraining
Princeton Language and Intelligence
Small Experiments, Cheaper Decisions: A Case Study in Staged Promotion for Micro-Pretraining
Jun 9, 2026Felipe Chavarro PolaniaLanguage Model Pretraining Hyperparameter Optimization
Hewlett Packard Enterprise
When Probing Accuracy Saturates, Fragility Resolves: A Complementary Metric for LLM Pre-Training Analysis
Jun 9, 2026Orion Reblitz-RichardsonLanguage Model Pretraining Noise Robustness
Distiller Labs.
Unifying Local Communications and Local Updates for LLM Pretraining
Jun 9, 2026Pietro Cagnasso, Eugene Belilovsky, Edouard OyallonLanguage Model Pretraining Communication-Efficient Distributed Training
Concordia University · Mila · CNRS, Sorbonne University
Multi-Hop Knowledge Composition is Bound by Pretraining Exposure
Jun 8, 2026Yannis Karmim, Luis Marti, Djamé Seddah +1Language Model Pretraining Implicit Reasoning in Language Models
Inria, Paris, France · Inria Chile, Las Condes, Chile · Dept. of Computer Science, Univ. de Chile +1
Unlocking Latent Value: Taxonomy-Guided Recovery of High-Performing Data from Low-Tier Web Corpora
Jun 5, 2026Neeraj Varshney, Sanket Lokegaonkar, Nasser Zalmout +3Language Model Pretraining Training Data Curation
Amazon

Jun 27, 2026J/K move · Enter open · S save

DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training

Dong Wang, Wenwu Tang, Yun Cheng, Olga Saukh

Graz University of Technology, Austria · Swiss Data Science Center, Switzerland

Language Model Pretraining LLM Compression Low-Rank Matrix Decomposition Low-Rank Compression Matrix Factorization Low-Rank Representation Learning

Open paper AskPDF arXiv

Large language models have driven recent progress in language and multimodal AI, yet pre-training them at scale is prohibitively expensive. Low-rank pre-training, which factorizes each weight matrix into a rank-r product to reduce both parameters and FLOPs, is a promising response but typically lags full-rank training in quality. We propose Duplicated Latent Residual (DLR), a training-only, parameter-free, foldable plug-in for low-rank pre-training. DLR augments the standard low-rank output Bz with a fixed structured residual alpha/sqrt(K) * Expand_K(z) that replicates each latent coordinate K = ceil(d_out/r) times across the output. With alpha fixed, DLR adds zero learnable parameters per layer; after training, it is absorbed into the up-projection in closed form, B* = B + alpha/sqrt(K) R^T, so deployment parameter count, FLOPs and memory match the underlying low-rank backbone exactly. Across LLaMA models from 60M to 7B parameters, DLR strengthens low-rank pre-training on C4 validation perplexity in most settings, with the clearest gains at 130M and above; folded checkpoints transfer cleanly to supervised fine-tuning on standard benchmarks.