Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing

Andrey Goncharov; Daniil Vyazhev; Simon Epanov; Alexey Zaytsev

cs.LGOct 14, 2026

Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing

Authors: Andrey Goncharov, Daniil Vyazhev, Simon Epanov, Alexey Zaytsev

Organizations: Applied AI Institute

Abstract

Distillation of reasoning chains from Large Language Models (LLMs) into their smaller siblings is a standard training approach. Frequently, researchers do not have full access to a teacher model and its internal representations, which limits the resulting performance. We propose a recursive offline distillation method combining two components: entropy-based sampling for selective distillation and answer mixing. Specifically, for three open models (3-4B), at each epoch, we sample reasoning chains weighted by entropy difference with an auxiliary medium model (70-72B) and, next, supplement thinking traces with single-token ground truth answers. Our approach consistently outperforms naive distillation with comparable data budgets ( $57.15 \pm 1.01$ vs 51.61 on average) and offers a much steeper learning curve. Additionally, we present a comparative analysis of methods to generate and normalize distillation traces.

Figures & tables

Figure 1: Recursive offline distillation scheme

Panel of Figure 2 — Figure 2: Distillation pipeline results across three models (average) on MMLU-Pro

Method	Answer Mix	LLaMA 3.2 3B	Qwen 2.5 3B	Phi4-mini	Mean
Direct reasoning trace	No	46.09 / 45.55	51.45 / 50.37	57.27 / 62.43	51.61 / 52.78
Corrected answer	No	47.80 / 47.26	50.87 / 49.42	58.81 / 61.26	52.49 / 52.65
Student entropy (deterministic)	Yes	48.84 / 49.09	55.86 / 55.49	58.02 / 63.30	54.24 / 55.96
Entropy gain (deterministic)	Yes	49.58 / 49.00	53.16 / 53.33	60.22 / 62.43	54.32 / 54.92
Student entropy (once, $k=1024$ )	No	41.35 / 40.11	39.44 / 39.07	50.17 / 52.87	43.65 / 44.01
Student entropy (once, $k=2048$ )	No	42.19 / 41.06	43.18 / 43.02	53.20 / 52.62	46.19 / 45.57

Table 1: Distillation pipeline last-epoch accuracy (4096-token / 2,048-token thinking budget) on MMLU-Pro.

Panel of Figure 3 — Figure 3: Comparison of synthetic traces across three models (average)

Sampler	Mix	Mean Accuracy
Direct reasoning trace (baseline)	No	51.61
Corrected answer (baseline)	No	52.49
Random	Yes	55.08	$\pm\,0.80$
Student entropy	Yes	55.87	$\pm\,1.40$
Discounted entropy gain	Yes	56.06	$\pm\,0.68$
Entropy gain	Yes	57.15	$\pm\,1.01$

Table 2: Distillation pipeline last-epoch accuracy for different sampling methods (4096-token thinking budget)

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Student	Cap	No mix	Mix
LLaMA 3.2 3B	2,048	1	2
Qwen 2.5 3B	2,048	1	0
Phi4-mini	2,048	3	8
LLaMA 3.2 3B	4,096	0	2
Qwen 2.5 3B	4,096	3	0
Phi4-mini	4,096	187	122

Appendix

Table 3: Invalid extracted answers. No mix denotes direct reasoning-trace distillation; mix denotes entropy-gain sampling with direct-answer examples in the data mix.

Panel of Figure 5 — Figure 5: Synthetic-trace distillation results for LLaMA 3.2 3B.

Panel of Figure 6 — Figure 6: Synthetic-trace distillation results for Qwen 2.5 3B.

Panel of Figure 7 — Figure 7: Synthetic-trace distillation results for Phi4-mini.

Budget (chars)	Direct traces	Corrected traces
3,072	3,603 (37.4%)	3,721 (38.7%)
8,192	1,879 (19.5%)	1,921 (20.0%)
24,000	738 (7.7%)	749 (7.8%)

Appendix

Table 4: Number (percentage) of training traces exceeding each truncation budget

Panel of Figure 8 — Figure 8: Training truncation policies at the nominal 24,000-character setting, evaluated with an 8,192-token reasoning cap. Dashed lines and epoch-zero points give the base model’s chain-of-thought accuracy.

Panel of Figure 9 — Figure 9: Best-checkpoint accuracy versus evaluation reasoning cap for each training truncation length.

Panel of Figure 10 — Figure 10: Last-checkpoint accuracy versus evaluation reasoning cap for each training truncation length.

Panel of Figure 11 — Figure 11: Distillation by sampling strategy for LLaMA 3.2 3B.

Panel of Figure 12 — Figure 12: Distillation by sampling strategy for Qwen 2.5 3B.

Panel of Figure 13 — Figure 13: Distillation by sampling strategy for Phi4-mini.

Panel of Figure 14 — Figure 14: Sampler comparisons and question coverage across all three models. In the accuracy panels, thin dashed curves show individual models, thick solid curves their equal-weight means, and the thick dashed curve the direct-trace baseline mean.

Model	Accuracy (%)	Mean $H$
LLaMA 70B	52.2	0.0126
Qwen 72B	54.2	0.0265
LLaMA 3B	28.9	0.1189
Qwen 3B	36.2	0.0287
Phi4-mini	37.8	0.1101

Appendix

Table 5: Single-token accuracy and mean normalized entropy on the full MMLU-Pro dataset

Models compared	$r$	$\rho$	$r$
	Entropy		Correctness
LLaMA 70B / Qwen 72B	0.240	0.533	0.554
Mean proxy / LLaMA 3B	0.369	0.435	0.322
Mean proxy / Qwen 3B	0.343	0.461	0.447
Mean proxy / Phi4-mini	0.502	0.582	0.441

Appendix

Table 6: Pearson

r

for entropy values and correctness, and Spearman

\rho

for entropy rankings. The first row compares the proxies; remaining rows compare their mean entropy and mean correctness scores with each student’s scores on the common test split.

Online Knowledge Distillation

See all 10 papers

Distribution Corrected Offline Data Distillation for Large Language Models
May 13, 2026Yumeng Zhang, Zhengbang Yang, Yevin Nikhel Goonatilake +1CoT Distillation Online Knowledge Distillation
George Mason University, Fairfax, VA, USA
Task-Specific Knowledge Distillation via Intermediate Probes
Feb 18, 2026Ryan Brown, Chris RussellLanguage Model Distillation Knowledge Distillation
Oxford Internet Institute, University of Oxford, Oxford, United Kingdom.
Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion
Feb 26, 2026Zhen Tan, Chengshuai Zhao, Song Wang +3Language Model Distillation Reasoning Distillation
Arizona State University · University of Virginia · University of North Carolina at Chapel Hill

May 13, 2026J/K move · Enter open · S save

Distribution Corrected Offline Data Distillation for Large Language Models

Yumeng Zhang, Zhengbang Yang, Yevin Nikhel Goonatilake, Zhuangdi Zhu

George Mason University, Fairfax, VA, USA

CoT Distillation Online Knowledge Distillation Efficient Language Model Reasoning Language Model Distillation Exposure Bias Teacher-Student Learning+1

Open paper AskPDF arXiv

Distilling reasoning traces from strong large language models into smaller ones is a promising route to improve intelligence in resource-constrained settings. Existing approaches face a fundamental trade-off: offline distillation from teacher-generated traces provides high-quality, sample-efficient supervision but suffers from distributional drift: during training, the student model conditions on teacher-generated prefixes, whereas during inference the student autoregresses on self-generated prefixes, leading to compounding errors over long reasoning trajectories. Meanwhile, on-policy or self-distillation methods better match the student's inference-time distribution, but require costly online sampling and often produce low-quality traces in early training. We propose a principled offline reasoning distillation framework that preserves the efficiency and supervision quality of offline teacher-generated data while correcting teacher-student distribution drift. It adaptively emphasizes teacher supervision that is better aligned with the student's on-policy distribution. Evaluations on mathematical reasoning benchmarks of GSM8K, MATH, MATH500, and harder held-out competition-style tasks, including AMC, AIME, and OlympiadBench, show that our method improves reasoning accuracy over prior offline distillation algorithms and yields more stable reasoning traces while preserving instruction-following capabilities. Our work shows that lightweight, distribution-correction-aware training can substantially strengthen offline reasoning distillation without online rollouts.

Abstract

Online Knowledge Distillation

Explore similar work

Distribution Corrected Offline Data Distillation for Large Language Models

Task-Specific Knowledge Distillation via Intermediate Probes

Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion