Model Collapse

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, with none the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 31

All topics

No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse
Oct 1, 2026Lewis MitchellLLM Fine-Tuning Model Collapse
Adelaide Data Science Centre School of Mathematical Sciences Adelaide University Adelaide SA 5005, Australia
Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training
Oct 1, 2026Hanna Malet, Gabriel TuriniciModel Collapse Diffusion Models
CEREMADE CNRS Université Paris Dauphine - PSL Paris 75116 FRANCE · Cornell University Ithaca, NY, USA
Source Identification Is Not Fitness Testing: Measuring the Limits of Synthetic-Data Attribution
Sep 30, 2026Joss ArmstrongModel Collapse Training Data Attribution
Ericsson, Athlone, Ireland
Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
Sep 16, 2026Matteo Marchi, João Pedro Silvestre, Bahman Gharesifard +1Synthetic Data Pretraining Model Collapse
Electrical and Computer Engineering Department, University of California at Los Angeles, Los Angeles, CA 90095 USA · Department of Mathematics and Statistics at Queens University, Kingston, ON, Canada
Break Step: Recursive Training Resonates with Replayed Sampling Noise
Sep 10, 2026Yangze Liu, Zhongyi HanSelf-Training for Language Models Model Collapse
Shandong University
Learning with Synthetic Data via SGD in High-Dimensional Linear Regression
Sep 10, 2026Jichu li, Difan ZouSynthetic Data Pretraining Model Collapse
Renmin University of China · The University of Hong Kong
Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuning
Jul 19, 2026Xiaonan Luo, Yue Huang, Kehan Guo +4Self-Training for Language Models Model Collapse
University of Notre Dame · Vanderbilt University · University of Pennsylvania
World-Model Collapse as a Phase Transition
Jun 30, 2026Xinyuan Song, Zekun CaiPhase Transitions Model Collapse
Emory University, Atlanta, GA, USA · The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan
When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMs
Jun 26, 2026Xinyuan Song, Zekun Cai, Liang ZhaoSelf-Training Model Collapse
Emory University, Atlanta, GA, USA · The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan
Model Collapse: On Recursion, Noise, and Uncharted Machine Visions
Jun 22, 2026Violaine Boutet de MonvelModel Collapse Generative Modeling
LIRA, IRCAV · Bibliotheca Hertziana – Max Planck Institute for Art History · MPRG Impett – Machine Visual Culture: Artificial Intelligence and the History of Seeing
Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterization
Jun 11, 2026Naïl B. Khelifa, Richard E. Turner, Ramji VenkataramananDiffusion Sampling Model Collapse
University of Cambridge
When Sample Selection Bias Precipitates Model Collapse
Jun 11, 2026Xinbao Qiao, Xianglong Du, Wei Liu +4Synthetic Data Pretraining Model Collapse
National University of Singapore (Chongqing) Research Institute · Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles, Chongqing, China · The Chinese University of Hong Kong +4
Entropy Minimization without Model Collapse: Mitigating Prediction Bias in Medical Imaging
Jun 1, 2026Tim Nielen, Sameer Ambekar, Johannes Kiechle +2Distribution Shift Robustness Test-Time Adaptation
School of Computation, Information and Technology, Technical University of Munich, Germany · Institute of Machine Learning in Biomedical Imaging, Helmholtz Munich, Germany · Munich Center for Machine Learning (MCML) +3
Benchmarking Recursive-Collapse Warning Claims Under Matched False-Positive Control
May 29, 2026David MullettBenchmark Design Model Collapse
Independent Researcher
Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables
May 28, 2026Masaaki Imaizumi, Masanori Koyama, Noboru Isobe +1Transformer Self-Attention
The University of Tokyo, Tokyo, Japan · RIKEN Advanced Intelligene Project, Tokyo, Japan · Kyoto University, Kyoto, Japan
When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loop
May 28, 2026Yang Zhang, Xiukun Wei, Xueru ZhangLLM Alignment Self-Training
Department of Computer Science and Engineering, The Ohio State University, Columbus, Ohio.
Model Collapse as Cultural Evolution
May 21, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuCultural Evolution Self-Training for Language Models
The University of Hong Kong · Hong Kong, China · Stellaris AI Limited
Self-Training Doesn't Flatten Language -- It Restructures It: Surface Markers Amplify While Deep Syntax Dies
May 20, 2026Ming LiuSelf-Training for Language Models Model Collapse
Amazon
When Does Model Collapse Occur in Structured Interactive Learning?
May 19, 2026Yuchen Wu, Kangjie Zhou, Weijie SuModel Collapse Generative Modeling
School of Operations Research and Information Engineering, Cornell University · Department of Statistics, Columbia University · Department of Statistics and Data Science, University of Pennsylvania
The Economics of Model Collapse: Equilibrium, Welfare, and Optimal Provenance Subsidies in Synthetic Data Markets
May 19, 2026Gustav Olaf Yunus Laitinen-Fredriksson Lundström-ImanovData Provenance Dataset Watermarking
Department of Economics Stockholm University Stockholm, Sweden
Silent Collapse in Recursive Learning Systems
May 14, 2026Zhipeng ZhangModel Collapse Adaptive Learning
China Mobile Research Institute, Beijing 100053, China · China Mobile GBA (Greater Bay Area) Innovation Institute, Guangzhou 510656, China
Curated Synthetic Data Doesn't Have to Collapse: A Theoretical Study of Generative Retraining with Pluralistic Preferences
May 8, 2026Ali Falahati, Mohammad Mohammadi Amiri, Kate Larson +1Synthetic Data Pretraining Model Collapse
University of Waterloo · 2Rensselaer Polytechnic Institute.
Human-AI Co-Evolution and Epistemic Collapse: A Dynamical Systems Perspective
May 7, 2026Xuening Wu, Yanlan Kang, Qianya Xu +5Model Collapse Human-AI Interaction
Fudan University, Shanghai, China · Pfizer China, Shanghai, China · University of California San Diego, CA, USA +4
On Semantic Loss Fine-Tuning Approach for Preventing Model Collapse in Causal Reasoning
May 6, 2026Pratik Deshmukh, Atirek GuptaLLM Fine-Tuning Model Collapse
Technical University of Vienna, Austria · HCLTech, Noida, India
Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communities
May 5, 2026Devon Jarvis, Richard Klein, Benjamin Rosman +2Model Collapse Responsible AI
School of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, South Africa · Machine Intelligence and Neural Discovery Institute, University of the Witwatersrand, Johannesburg, South Africa · Data Science and AI, Computer Science and Engineering, Chalmers University of Technology and University of Gothenburg
Language Generation with Replay: A Learning-Theoretic View of Model Collapse
Mar 12, 2026Giorgio Racca, Michal Valko, Amartya SanyalModel Collapse Data Contamination in Language Models
University of Copenhagen · Isara Labs
Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training
Feb 17, 2026Kevin Wang, Hongqian Niu, Didong LiModel Collapse Generative Modeling
Department of Biostatistics, University of North Carolina at Chapel Hill
Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
Oct 18, 2025Bingji Yi, Qiyuan Liu, Yuwei Cheng +1Synthetic Data Pretraining Model Collapse
Independent Researcher. Work done while visiting UChicago CS. · Work done while visiting UChicago CS. · Department of Statistics, University of Chicago. +1
Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity
Oct 1, 2025Jiayi Zhang, Simon Yu, Derek Chong +4LLM Prompting Model Collapse
Northeastern University · Stanford University · West Virginia University
Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework
Feb 25, 2025Hengzhi He, Shirong Xu, Guang ChengModel Collapse Generative Modeling
Department of Statistics and Data Science, University of California, Los Angeles · Wang Yanan Institute for Studies in Economics, Xiamen University

Oct 1, 2026, cs.CLJ/K move · Enter open · S save

No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse

Lewis Mitchell

Adelaide Data Science Centre School of Mathematical Sciences Adelaide University Adelaide SA 5005, Australia

LLM Fine-Tuning Model Collapse Data Repetition in Language Model Training

Open paper AskPDF arXiv

Iterative fine-tuning on synthetic data causes \emph{model collapse}: output diversity narrows as rare patterns are progressively lost, a signature most visible as phrase-level repetition. Existing mitigations either require model log-probabilities, an external oracle, or continued access to real human data. Here we develop a new approach grounded in mathematical information theory: the non-parametric Kontoyiannis entropy rate estimator

h_k

, computed entirely from raw text via match-length statistics, with no model of any kind. We show that this is in fact a \emph{superior} training-data filter on text-diversity metrics in a fully-synthetic, single-lineage fine-tuning setting. In a six-generation QLoRA collapse experiment on Llama-3.1-8B, logprob-based filtering (the most established model-access-requiring baseline) provides no significant text-diversity benefit on any metric (

p > 0.23

), whereas

h_k

-filtering yields

+42\%

unique trigrams,

+30\%

vocabulary, and

-19\%

repetition (all

p < 0.001

). We validate

h_k

as a cross-domain entropy proxy (

β= 0.924

R^2 = 0.746

) and collapse detector (

ρ= +0.454

p < 0.0001

) across 4~~domains, 2~~temperatures, 2~generator--scorer model pairs, and 1{,}520 generated documents. Our results demonstrate that information theoretic approaches to collapse mitigation are efficient, and suggest new approaches for maintaining multi-agent diversity.

Momentum

Related topics

Latest papers 31