Data Repetition in Language Model Training

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 25

All topics

What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining
Oct 4, 2026Yekun Chai, Haoyi XiongLanguage Model Pretraining Language Model Scaling Laws
FloatAI · ETH Zurich · Independent Researcher
Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining
Oct 4, 2026Ziyue WANG, T. KanamoriLanguage Model Pretraining Language Model Scaling Laws
Institute of Science Tokyo
No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse
Oct 1, 2026Lewis MitchellLLM Fine-Tuning Model Collapse
Adelaide Data Science Centre School of Mathematical Sciences Adelaide University Adelaide SA 5005, Australia
Forking: Sudden Overfitting Under Replay
Sep 30, 2026Shanbin Yu, Shaoyang Guo, Haoran Zhao +2Neural Network Generalization LLM Training
MetaCircle (元环智能) · Tsinghua University · Peking University +1
Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models
Sep 28, 2026Florian Eichin, Philipp Mondorf, Andrei Mircea +3Memorization in Language Models Data Repetition in Language Model Training
MaiNLP, Center for Information and Language Processing, LMU Munich & Munich Center for Machine Learning (MCML), Germany · Mila – Quebec AI Institute & University of Montreal, Canada · Saarland University, Germany
Fine Until Fine-Tuned: Repeated Solutions Make Reasoning Fragile
Sep 27, 2026Ely SheikhContinual Learning for LLMs Data Repetition in Language Model Training
Aix-Marseille University
Data-Efficient Language Modeling: From Frontier Advancement to Principle-Guided Model Improvement
Sep 12, 2026Shuxing Yang, Kaihao Zhu, Junjie Yang +13Efficient Language Model Training Language Modeling
College of Information Science and Electronic Engineering, Zhejiang University · Qiushi Engine Team
Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Sep 10, 2026Atindra Jha, Margaret Li, Jure Leskovec +2Mixture of Experts Sparse Mixture-of-Experts
Stanford University · Paul G. Allen School of Computer Science, University of Washington
Break Step: Recursive Training Resonates with Replayed Sampling Noise
Sep 10, 2026Yangze Liu, Zhongyi HanSelf-Training for Language Models Model Collapse
Shandong University
Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models
Sep 9, 2026Arman Nik KhahMemorization in Language Models LLM Auditing
The University of Texas at Dallas
Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views
Sep 3, 2026Joseph Lee, Yidi Huang, Dokyoon Kim +2Language Model Pretraining Data Augmentation
University of Pennsylvania, Philadelphia, PA, USA
Rewriting or Reweighting? A Geometric Account in Language Models
Aug 3, 2026Juntong Wang, Shengkun Yang, Xiyuan Wang +1Language Model Post-Training Representation Geometry in Language Models
Institute for Artificial Intelligence, Peking University, Beijing, China · School of Intelligence Science and Technology, Peking University, Beijing, China · School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China
Temperature-driven inversion and nonlinear dynamics in ChatGPT-like AIs
Aug 2, 2026Neil F. Johnson, Frank Yingjie Huo, Bella Xinrui LiLanguage Model Decoding Autoregressive Generation
Department of Physics, The George Washington University Washington, DC 20052, USA
Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning
Jul 21, 2026Lizhe Fang, Weizhou Shen, Tianyi Tang +1LLM Grounding Reward Shaping
Peking University · Qwen Team of Alibaba Group · Alibaba Group ATH
ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
Jul 14, 2026Qingyu Zhang, Qianhao Yuan, Hongyu Lin +7Open-Ended Generation LLM Pruning
2Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · 1ByteDance
Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training
Jul 6, 2026Jingwei Zuo, Cong Zeng, Ilyas Chahed +6Memorization in Language Models Neural Network Memorization
Technology Innovation Institute Abu Dhabi, UAE
How much of an LLM-generated clinical corpus is actually new? A production-scale measurement of content redundancy for provenance classification
Jun 28, 2026Ali H. Lazem, William J. TeahanData Provenance Clinical NLP
School of Computer Science and Engineering, Bangor University, Bangor, LL57 2DG, Gwynedd, United Kingdom. · University of Thi-Qar, Nasiriyah, 64001, Iraq.
Internal Data Repetition Destroys Language Models
Jun 23, 2026Jessica Chudnovsky, Joshua Kazdan, Noam Levi +6Memorization in Language Models Language Model Scaling Laws
1Stanford Computer Science. · 2Stanford Statistics. · 3Tel Aviv University. +1
Can Editing 1 Neuron Fix Repetition Loops in LLMs?
Jun 9, 2026Aristotelis Lazaridis, Aman Sharma, Dylan Bates +3LLM Self-Correction Mechanistic Interpretability
Edgerunner AI
Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
Jun 5, 2026Zhiwei Xu, Shihao Wu, Hanseul Cho +2Language Model Pretraining Language Model Scaling Laws
University of Michigan · KAIST AI
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
May 31, 2026Boqian Wu, Qiao Xiao, Patrik Okanovic +6Language Model Scaling Laws Efficient Language Model Training
University of Luxembourg · University of Twente · Eindhoven University of Technology +1
Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them
May 29, 2026Kevin Zhou, Lisa Alazraki, Kris Cao +1Language Model Pretraining Data Mixture Optimization
Imperial College London · Cohere
Scaling Laws for Mixture Pretraining Under Data Constraints
May 12, 2026Anastasiia Sedova, Skyler Seto, Natalie Schluter +1Language Model Pretraining Language Model Scaling Laws
Apple · ITU
InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition
May 4, 2026Fengze Liu, Weidong Zhou, Binbin Liu +7Language Model Pretraining Language Model Scaling Laws
1ByteDance · 2UC Santa Cruz
Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling
Apr 30, 2026Ansar Aynetdinov, Patrick Haller, Alan AkbikLanguage Model Pretraining Efficient Language Model Training
Humboldt-Universität zu Berlin

Oct 4, 2026, cs.CLJ/K move · Enter open · S save

What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch Pretraining

Yekun Chai, Haoyi Xiong

FloatAI · ETH Zurich · Independent Researcher

Language Model Pretraining Language Model Scaling Laws Data Repetition in Language Model Training

Open paper AskPDF arXiv

As pretraining increasingly repeats data, every run faces three questions: how many epochs to take, how that number should change with model size, and whether anything besides the epoch count matters. We answer them by pricing a repeated token against two references: one epoch on the same data, which gives its value, and fresh data at equal compute, which gives its cost. Against fresh data, the cost of repetition follows a single variable, the number of extra epochs divided by the unique tokens per parameter. Against the same data, a second epoch is worth nearly as much as a fresh one, and repeated tokens fall to half the value of fresh ones after a critical epoch count that grows with the training budget per parameter but hardly with model size. With unique data fixed, the predicted compute-optimal run grows model size and epochs together until loss stops improving, near the critical epoch count. The same variable accounts for the direction of size trends that appear to conflict: larger models tolerate fewer epochs when the corpus is fixed, from about 15 at 127M to 4 at 2B parameters, but not when unique data grow with the model. Counts alone do not determine loss: at identical counts, replaying shards consecutively raises loss by up to 0.46~bits per byte, concentrating repeats on fewer samples also raises it, lower-entropy sources degrade faster with repetition, and re-tokenizing repeats helps only under heavy repetition. These results offer an empirical guide to pretraining when unique data, rather than compute, are the binding constraint.

Momentum

Related topics

Latest papers 25