No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse

Lewis Mitchell

cs.CLOct 1, 2026

No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse

Authors: Lewis Mitchell

Organizations: Adelaide Data Science Centre School of Mathematical Sciences Adelaide University Adelaide SA 5005, Australia

Abstract

Iterative fine-tuning on synthetic data causes \emph{model collapse}: output diversity narrows as rare patterns are progressively lost, a signature most visible as phrase-level repetition. Existing mitigations either require model log-probabilities, an external oracle, or continued access to real human data. Here we develop a new approach grounded in mathematical information theory: the non-parametric Kontoyiannis entropy rate estimator $h_k$ , computed entirely from raw text via match-length statistics, with no model of any kind. We show that this is in fact a \emph{superior} training-data filter on text-diversity metrics in a fully-synthetic, single-lineage fine-tuning setting. In a six-generation QLoRA collapse experiment on Llama-3.1-8B, logprob-based filtering (the most established model-access-requiring baseline) provides no significant text-diversity benefit on any metric ( $p > 0.23$ ), whereas $h_k$ -filtering yields $+42\%$ unique trigrams, $+30\%$ vocabulary, and $-19\%$ repetition (all $p < 0.001$ ). We validate $h_k$ as a cross-domain entropy proxy ( $β= 0.924$ , $R^2 = 0.746$ ) and collapse detector ( $ρ= +0.454$ , $p < 0.0001$ ) across 4~~domains, 2~~temperatures, 2~generator--scorer model pairs, and 1{,}520 generated documents. Our results demonstrate that information theoretic approaches to collapse mitigation are efficient, and suggest new approaches for maintaining multi-agent diversity.

Figures & tables

Figure 1 : Filtering on text-only entropy preserves diversity; logprob-based filtering does not. (a) Kontoyiannis entropy rate

\hat{H}_{K}

across six fine-tuning generations of Llama-3.1-8B-Instruct under three training-data conditions: unfiltered (rapid collapse),

H_{L}

-filtered (intermediate), and

\hat{H}_{K}

-filtered (slowest decline). Shaded bands:

\pm 1

SEM (

n=80

documents per generation). (b) Percentage change relative to the unfiltered baseline at generation 6 for three text-diversity metrics (Distinct-3, Vocabulary size, Rep-4). Error bars: bootstrap 95% CIs.

\hat{H}_{K}

-based filtering yields highly significant improvements on all three metrics (

{}^{***}p<0.001

), whereas

H_{L}

-based filtering is non-significant on all three.

Panel of Figure 2 — Figure 2 : $\hat{H}_{K}$ versus $H_{L}$ across domains and model pairs. Points are coloured by domain. The slope is domain-invariant in both panels: $\hat{H}_{K}$ scales with $H_{L}$ at a consistent rate regardless of domain, generating model, or scoring model.

Panel of Figure 3 — Figure 3 : $\hat{H}_{K}$ as a collapse detector under two regimes. Under rephrasing (a), $\hat{H}_{K}$ and $H_{L}$ co-decline but within-generation concordance is $\approx 0$ . Under fine-tuning (b), within-topic concordance is strongly positive ( $\rho=+0.454$ , $p<0.0001$ ), driven by phrase-level repetition that $\hat{H}_{K}$ directly measures.

Metric	Unfilt.	$H_{L}$ -filt.	$\hat{H}_{K}$ -filt.	$p$ ( $\hat{H}_{K}$ v. U)	$p$ ( $H_{L}$ v. U)	$p$ ( $\hat{H}_{K}$ v. $H_{L}$ )
$\hat{H}_{K}$ (bits/word)	0.79	1.16	1.43	${<}0.001$	$0.008$	${<}0.001$
Distinct-3 (%)	–	$+7$ %	+42 %	${<}0.001$	$0.23$	${<}0.001$
Vocabulary (%)	–	$+4$ %	+30 %	${<}0.001$	$0.31$	${<}0.001$
Rep-4 (%)	–	$-3$ %	-19 %	${<}0.001$	$0.27$	${<}0.001$

Table 1 : Generation-6 outcomes under three training-data conditions. Diversity metrics show % change relative to unfiltered.

p

-values from Welch

t

-tests.

n=80

documents per condition.

Figure 4 : (a)

H_{L}

(logprob entropy) trajectories across six fine-tuning generations for the three filtering conditions (

\pm 1

SEM bands). All collapse to low

H_{L}

by gen-1 and track nearly identically, illustrating why the

H_{L}

filter fails: post-collapse variance is too small to discriminate. (b) HellaSwag normalised accuracy (Llama-3.1-8B-Instruct) across six generations. All three conditions degrade from

0.729

(gen-0) to

\approx 0.652

–

0.657

(generation 6) with no significant difference between conditions, indicating that text-diversity preservation does not translate to preserved general reasoning ability.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5 : Three entropy metrics across six fine-tuning generations:

\hat{H}_{K}

(text-only),

H_{L}^{\text{Llama}}

(Llama-3.1-8B scorer), and

H_{L}^{\text{GPT-2}}

(independent GPT-2 scorer). All three metrics show the same directional ordering at generation 6. The

H_{L}^{\text{Llama}}

panel illustrates why logprob-based filtering fails: after generation 0, the three conditions are nearly indistinguishable, leaving the

H_{L}

-filter little variance to exploit.

Figure 6 : Six text-diversity metrics across generations under the three conditions.

\hat{H}_{K}

-filtering (green) maintains higher diversity on all within-document metrics (TTR, Distinct-

n

, vocabulary, Rep-4). Self-BLEU-4 (cross-document homogeneity) shows a non-monotonic pattern in early generations.

Figure 7 : Domain-level

\hat{H}_{K}

trajectories under the three conditions. The

\hat{H}_{K}

-filter advantage is consistent across all four domains.

Figure 8 : Generation-0

H_{L}

distribution. The base Llama-3.1-8B-Instruct model exhibits a wide, bimodal

H_{L}

distribution (mean

=3.36

, std

=2.17

), driven by topic-dependent familiarity. After one fine-tuning step the distribution collapses to a narrow regime (std

\approx 0.13

), eliminating the variance the

H_{L}

-filter relies on.

Explore similar work

Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuning
Jul 19, 2026Xiaonan Luo, Yue Huang, Kehan Guo +4Instruction Tuning Synthetic Training Data
University of Notre Dame · Vanderbilt University · University of Pennsylvania
Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Models
Oct 16, 2024Daniele Gambetta, Gizem Gezici, Fosca Giannotti +3Collapse Generative Artificial Intelligence
University of Pisa, Pisa, Italy · ISTI-CNR, Pisa, Italy · Scuola Normale Superiore, Pisa, Italy +1
Fine-Tuning Improves Information Conveyance in Language Models
May 29, 2026Yuwei Cheng, Weiyi Tian, Haifeng XuLarge Language Model Fine-Tuning Model Fine-Tuning
Department of Data Science University of Chicago Chicago, IL 60637 · Department of Computer Science University of Chicago Chicago, IL 60637

Jul 19, 2026, cs.CLJ/K move · Enter open · S save

Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuning

Xiaonan Luo, Yue Huang, Kehan Guo, Ping He+3

University of Notre Dame · Vanderbilt University · University of Pennsylvania

Instruction Tuning Synthetic Training Data Synthetic Data Collapse Strong Large Language Model Instruction

Open paper AskPDF arXiv

Model collapse is a central challenge in learning from synthetic data: as later-generation large language models (LLMs) are trained on an increasing proportion of model-generated data, performance can degrade due to narrowed coverage and accumulated bias. Existing work mainly studies how to bound this degradation. In iterative model evolution, however, the more meaningful objective is to ensure that each successive model improves over its predecessor, which requires diagnosing collapse at a granularity that is actionable for data curation. We study this problem in synthetic data self-improving for instruction tuning. We show that collapse in this setting is not simply uniform performance degradation, but can appear as a polarization of competence, where synthetic training reinforces already strong skills while further degrading weak ones. Motivated by this observation, we propose KITE (Knowledge-boundary Instruction Tuning via Exploration), a two-stage framework that combines failure-guided data generation with boundary-aware uncertainty curation. Experiments across several datasets and multiple open-source LLMs show that KITE yields more stable improvement than strong synthetic-data baselines.

Abstract

Explore similar work

Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuning

Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Models

Fine-Tuning Improves Information Conveyance in Language Models