Language Model Pretraining

Follow this topic Ask about this topic

Language model pretraining builds general-purpose models by learning from large collections of text, commonly through next-token prediction or masked-token prediction. Researchers study how data curation, filtering, mixing, repetition, and multilingual or domain-specific corpora affect what models learn, alongside changes to tokenization, architecture, training objectives, and optimizers. Work also aims to make training cheaper through methods such as sparse attention, low-precision computation, parameter sharing, and more efficient data selection. A central tension is that lower training loss or greater data volume does not always bring better downstream abilities: data quality, diversity, exposure, and evaluation choices all matter, and their effects can vary with model scale and task.

Latest papers

448 papers72 papers in the last four weeks, up 57% on the four weeks before. 0.4% of all new papers.

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
Jul 17, 2026Haodong Wen, Yiran Zhang, Yingfa Chen +1Language Model Pretraining Rotary Positional Embeddings
Tsinghua University
Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya
Jul 16, 2026Hailay Kidu Teklehaymanot, Debela Desalegn Yadeta, Wolfgang NejdlLanguage Model Pretraining Multilingual Language Models
L3S Research Center, Leibniz University Hannover, Germany · Addis Ababa University, Ethiopia
xHC: Expanded Hyper-Connections
Jul 16, 2026Xiangdong Zhang, Xiaohan Qin, Sunan Zou +10Language Model Pretraining Transformer
School of AI, Shanghai Jiao Tong University · Dots Studio, Xiaohongshu Inc. · School of CS, Peking University +3
Tracing LLM Behavior to the Training Data with Empirical Next-Token Distributions
Jul 15, 2026Zachary IzzoLanguage Model Pretraining Transformer Interpretability
NEC Labs America
Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling
Jul 14, 2026Roi Cohen, Yvan Carré, Nick Lechtenbörger +5Language Model Pretraining LLM Grounding
HPI / University of Potsdam · African Institute for Mathematical Sciences · Polytechnique Montréal +2
Domain-Aware Scaling Laws Uncover Data Synergy
Jul 13, 2026Kimia Hamidieh, Lester Mackey, David Alvarez-MelisLanguage Model Pretraining Language Model Scaling Laws
MIT CSAIL · Microsoft Research · Harvard University
The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinese
Jul 12, 2026Siyuan Song, Zhiheng Qian, Yunhao Zhang +11Language Model Pretraining LLM Evaluation
Princeton University · Shanghai Jiao Tong University · Chinese Academy of Sciences +5
LeRoPE: Learnable RoPE Frequencies Improve Language Modeling
Jul 11, 2026Petros Karypis, Sean O'Brien, Shreyas Kadekodi +2Language Model Pretraining Rotary Positional Embeddings
UC San Diego · Independent Researcher
Index SLM Technical Report
Jul 10, 2026Tianjiao Li, Lusheng Zhang, Shien He +11Language Model Pretraining Small Language Models
Bilibili Index LLM Team
Scalable Visual Pretraining for Language Intelligence
Jul 10, 2026Yiming Zhang, Zhonghan Zhao, Wenwei Zhang +13Language Model Pretraining Self-Supervised Pre-Training
Shanghai Artificial Intelligence Laboratory · University of Science and Technology of China · Zhejiang University +1
A Sovereign, Open-Source Foundation Model for German and English
Jul 10, 2026The Soofi-Team, :, Benedikt Droste +29Language Model Pretraining Multilingual Language Models
ellamind · Fraunhofer IAIS · Lamarr +8
Complexity-Guided Component-wise Initialization for Language Model Pretraining
Jul 10, 2026Konstantin Garbers, Nicholas OhLanguage Model Pretraining Neural Network Initialization
Peking University Beijing, China
UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing
Jul 9, 2026Xinlong Zhao, Dongsheng Liu, Hengyu Zhao +9Language Model Pretraining
1Peking University · 2ModelBest Inc. · 3Tsinghua University +1
TheBioCollection: Unified Pre-Training Scale LLM Corpus for Biology
Jul 9, 2026Hyunjin Seo, Hyeon Hwang, Gyubok Lee +7Language Model Pretraining Bioinformatics
Trillion Labs · KAIST · SK Biopharmaceuticals Co., Ltd. +2
Co-LMLM: Continuous-Query Limited Memory Language Models
Jul 8, 2026Yair Feldman, Linxi Zhao, Nathan Godey +5Language Model Pretraining Memory-Augmented Language Models
Department of Computer Science Cornell University
DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation
Jul 8, 2026Jordan Painter, Dipankar Srirag, Adarsh Kappiyath +3Language Modeling LLM Alignment
Institute for People-Centred AI, University of Surrey, Surrey, United Kingdom · University of New South Wales, Sydney, Australia
GeoFP8: Geometry-Aware FP8 Gradient Compression for Distributed LLM Training
Jul 8, 2026Jieying Wang, Zizhong Wang, Fangru Linghu +3Language Model Pretraining Communication-Efficient Distributed Training
Department of Computer Science, Rutgers University, Piscataway, NJ, USA · Department of Electrical and Computer Engineering, Rutgers University, Piscataway, NJ, USA
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Jul 5, 2026Siyu Ding, Mingchuan Ma, Jiabo Tong +3Language Model Pretraining Mixed-Precision Quantization
Institute of Automation, Chinese Academy of Sciences · Sichuan University · Beijing Zhongguancun Academy +3
HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures
Jul 2, 2026Ziyun Qiao, Yue Min, Ruining Chen +1Language Model Pretraining Training Data Curation
Wizard Quant, Beijing, China · University of Science and Technology of China, Hefei, China
BamiBERT: A New BERT-based Language Model for Vietnamese
Jul 2, 2026Dat Quoc Nguyen, Thinh Pham, Chi Tran +1Language Model Pretraining Transformer
Qualcomm AI Research · Virginia Tech
Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning
Jul 2, 2026Congrui Du, Yang Zhang, Kaizhi Qian +1Language Model Pretraining Instruction Following
University of California, Santa Barbara, USA · MIT-IBM Computing Research Lab, IBM Research, USA
Probing Chemical Language Models: Effects of Pre-training and Fine-tuning
Jul 2, 2026Anna Karnysheva, Dietrich Klakow, Ji-Ung LeeLanguage Model Pretraining Chemical Language Models
1RTG Neuroexplicit Models · 2Spoken Language Systems · Saarland University +1
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
Jul 2, 2026Mingkai Zheng, Junlin Chen, Haotian Xie +1Language Model Pretraining Communication-Efficient Distributed Training
Rutgers University
Ember: A Kilobyte-State Optimizer for Token Interfaces
Jul 1, 2026Kathan ShahLanguage Model Pretraining LLM Fine-Tuning
Preprint.
The State-Prediction Separation Hypothesis
Jul 1, 2026Giovanni Monea, Nathan Godey, Kianté Brantley +1Language Model Pretraining Transformer
†Cornell University · ⋄Harvard University
MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages
Jul 1, 2026Maximilian Idahl, Jörg Tiedemann, Sampo Pyysalo +19Language Model Pretraining Synthetic Data Pretraining
ellamind · Leibniz University Hannover · University of Helsinki +14
DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks
Jun 29, 2026Romain Karpinsky, Julien Mozziconacci, Mickaël DelceyLanguage Model Pretraining Fine-Tuning
CNRS, LORIA · Nancy, 54000, France · CNRS +1
Smooth Scaling Laws Hide Stepwise Token Learning
Jun 29, 2026Pingjie Wang, Zechen Hu, Peiru Yang +2Language Model Pretraining Language Model Scaling Laws
Dots Studio, Xiaohongshu Inc. · Shanghai Jiao Tong University · Tsinghua university
GLIP: Graph and LLM Joint Pretraining for Graph-Level Tasks
Jun 29, 2026Haoxin Sun, Yiqing Lin, Yajun Huang +3Language Model Pretraining Graph SSL
Fudan University Shanghai, China · ByteDance Shanghai, China
BERTomelo: Your Portuguese Encoder Best Friend
Jun 27, 2026Rennê Ruan Alves Oliveira, Gustavo Cordeiro Galvão Van Erven, Luís Paulo Faina GarciaLanguage Model Pretraining Portuguese Language Models
Department of Computer Science University of Brasília Brasília - DF, Brazil

Jul 17, 2026J/K move · Enter open · S save

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

Haodong Wen, Yiran Zhang, Yingfa Chen, Kaifeng Lyu

Tsinghua University

Language Model Pretraining Rotary Positional Embeddings Memorization in Language Models Positional Encoding

Open paper AskPDF arXiv

While large language models (LLMs) can solve advanced reasoning problems in seconds, we show that even frontier models fail to perform a much simpler operation: exactly copying an input string that lies well within their context windows. We attribute this failure to positional encodings in Transformer architectures, whose inductive bias favors copying through a shortcut based on matching local contexts rather than carefully locating the corresponding input positions. To address this issue, we introduce 2D-RoPE, which organizes text into a 2D grid rather than a 1D sequence and assigns each token a row ID and a column ID. Under this view, copying becomes simply retrieving input tokens at a fixed column offset, which makes the task easy to learn. In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at input lengths hundreds of times longer than those seen during training, whereas standard positional encodings fall far behind. We further show that the advantage of 2D-RoPE language models on copy tasks consistently holds in large-scale pretraining on DCLM with model sizes up to 1.4B parameters. Overall, our results suggest that viewing text in 2D can benefit language modeling, and we hope this encourages future work to further explore the potential of 2D positional encodings.