TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models

Yushi Guan; Jeanine Ohene-Agyei; Daniel Kwan; Jean Sebastien Dandurand; Yifei Zhang; Nandita Vijaykumar

cs.LGMar 25, 2026

TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models

Authors: Yushi Guan, Jeanine Ohene-Agyei, Daniel Kwan, Jean Sebastien Dandurand, Yifei Zhang, Nandita Vijaykumar

Organizations: Department of Computer Science, University of Toronto · Vector Institute, Toronto, Canada

Abstract

To embed domain-specific or specialized knowledge into pre-trained foundation models, fine-tuning using techniques such as parameter efficient fine-tuning (e.g. LoRA) is a common practice. However, as new LLM architectures and pre-trained models emerge, transferring this specialized knowledge to newer models becomes an important task. In many scenarios, the original specialized data may be unavailable due to privacy or commercial restrictions, necessitating distillation and transfer of this specialized knowledge from the fine-tuned base model to a different pre-trained model. We present TuneShift-KD, a novel approach that automatically distills specialized knowledge from a fine-tuned model to a target model using only a few examples representative of the specialized information. Our key insight is that specialized knowledge can be identified through perplexity differences between base and fine-tuned models: prompts where the fine-tuned model responds confidently (low perplexity), but the base model struggles (high perplexity), indicate queries corresponding to the specialized knowledge learned by the fine-tuned model. TuneShift-KD leverages this insight to create a synthetic training dataset to transfer the specialized knowledge. Using an iterative process, TuneShift-KD generates more prompts similar to those that generated responses with specialized knowledge. TuneShift-KD does not require training discriminators or access to training datasets. It is an automated approach that only requires the initial fine-tuned and base models and a few representative prompts. Our experiments demonstrate that models fine-tuned using TuneShift-KD achieve higher accuracy than prior approaches, enabling ease of deployment and more effective transfer of the specialized knowledge.

Figures & tables

Figure 1: TuneShift-KD Framework for Specialized Knowledge Transfer. This figure illustrates our approach to transferring specialized knowledge without requiring full access to original fine-tuning data. (a) The Synthetic Fine-tuning Dataset begins with just 5 seed examples from the original fine-tuning data and expands through our iterative process. (b) An instruction-tuned LLM generates similar prompts based on the prompt pattern ”Generate 20 more samples like these 5 [ the 5 sample data ]”. (c) Each synthetic prompt is fed to both the Source Base LLM and Source Fine-tuned LLM to obtain paired responses

\mathbf{y}^{(b)}

and

\mathbf{y}^{(f)}

. (d) Our key filtering mechanism selects prompts where the fine-tuned model shows high confidence (low perplexity

\mathrm{PPL}(\mathbf{y}^{(f)})

) while the base model shows low confidence (high perplexity

\mathrm{PPL}(\mathbf{y}^{(b)})

), indicating the prompt targets specialized knowledge. Qualified prompt-response pairs are added to the synthetic dataset, while others are discarded. (e) The curated synthetic dataset is used to perform knowledge distillation, transferring specialized capabilities to the Target LLM without requiring original training data.

		Trans‐LoRA			Ours
Source	Target	Source LoRA	Target (no LoRA)	Target LoRA	Source LoRA	Target (no LoRA)	Target LoRA
Llama2‐7B	Llama2‐13B	19.64%	28.86%	30.70% ( $\uparrow 1.84\%$ )	19.48%	27.89%	30.12% ( $\uparrow\textbf{2.23}\%$ )
Gemma‐2B	Gemma‐7B	14.94%	40.64%	44.58% ( $\uparrow 3.94\%$ )	15.20%	40.0%	44.80% ( $\uparrow\textbf{4.80}\%$ )

Table 1: GSM8K benchmark results

		Trans‐LoRA			Ours
Source	Target	Source LoRA	Target (no LoRA)	Target LoRA	Source LoRA	Target (no LoRA)	Target LoRA
Llama2‐7B	Llama2‐13B	27.2%	37.1%	39.7% ( $\uparrow 2.6\%$ )	28.2%	36.9%	40.2% ( $\uparrow\textbf{3.3}\%$ )
Gemma‐2B	Gemma‐7B	41.1%	37.9%	50.0% ( $\uparrow 12.1\%$ )	40.9%	37.8%	51.3% ( $\uparrow\textbf{13.5}\%$ )

Table 2: MBPP benchmark results (standard evaluation)

		Trans‐LoRA			Ours
Source	Target	Source LoRA	Target (no LoRA)	Target LoRA	Source LoRA	Target (no LoRA)	Target LoRA
Llama2‐7B	Llama2‐13B	43.32%	37.85%	43.41% ( $\uparrow 5.56\%$ )	42.02%	38.04%	44.92% ( $\uparrow\textbf{6.88}\%$ )
Gemma‐2B	Gemma‐7B	31.84%	37.75%	43.61% ( $\uparrow 5.86\%$ )	31.01%	38.15%	45.09% ( $\uparrow\textbf{6.94}\%$ )

Table 3: BBH benchmark results

Dataset	Source Base	Source LoRA	Target	Target (no LoRA)	Target LoRA	Our Acc Increase
GSM8K	Gemma-2B	Llama2-7B	Gemma-7B	40.0%	43.82%	$\uparrow$ 3.82%
MBPP	Llama2-7B	Gemma-2B	Gemma-7B	37.8%	51.2%	$\uparrow$ 13.4%

Table 4: Generic pre-trained model as base substitutes

Dataset	Source	Target	Target (no LoRA)	Low-High Filter (TuneShift-KD)	No Filter Acc	Low-Low Filter Acc	High-High Filter Acc
GSM8K	Gemma-2B	Gemma-7B	40.0%	44.8 %	36.7%	40.2 %	35.4%
GSM8K	Llama2-7B	Llama2-13B	27.9%	30.1 %	27.8%	28.4 %	26.9%
MBPP	Gemma-2B	Gemma-7B	37.8%	51.2 %	35.8%	39.5 %	34.4%
MBPP	Llama2-7B	Llama2-13B	36.9%	40.2 %	29.4%	37.4 %	30.2%

Table 5: Comparison of filtering rules: default filtering vs. alternative filtering rules

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

		Trans‐LoRA				Ours
Source	Target	Source LoRA	Target (no LoRA)	Target LoRA	Acc Increase	Source LoRA	Target (no LoRA)	Target LoRA	Acc Increase
Llama2‐7B	Llama2‐13B	45.89%	53.72%	55.09%	1.37%	47.19%	53.13%	53.62%	0.49%
Gemma‐2B	Gemma‐7B	42.34%	60.45%	61.23%	0.78%	41.39%	61.62%	62.38%	0.76%

Appendix

Table 6: MMLU benchmark results

		Trans‐LoRA				Ours
Source	Target	Source LoRA	Target (no LoRA)	Target LoRA	Acc Increase	Source LoRA	Target (no LoRA)	Target LoRA	Acc Increase
Llama2‐7B	Llama2‐13B	25.0%	31.7%	34.4%	2.7%	25.7%	31.6%	33.9%	2.3%
Gemma‐2B	Gemma‐7B	33.9%	32.1%	40.6%	8.5%	32.4%	40.7%	46.8%	6.1%

Appendix

Table 7: MBPP benchmark results (strict evaluation)

Dataset	Source	Target	Source LoRA	Target (no LoRA)	Target LoRA	Acc Increase
GSM8K	Qwen2.5-7B Instruct	Qwen2.5-14B Instruct	69.4%	43.9%	79.6%	35.7%
MBPP	Qwen2.5-7B Instruct	Qwen2.5-14B Instruct	52.8%	42.0%	60.1%	18.1%

Appendix

Table 8: TuneShift-KD results using Qwen models as source and target

Filter Setting	Source Model	Target Model	Target (no LoRA)	Target LoRA Acc.	Acc. Change (vs. default)
(a) Threshold = 1.3	Llama2-7B	Llama2-13B	27.89%	30.02%	–0.10%
(b) Fine-tuned $<1.2$ , Base $>1.6$	Llama2-7B	Llama2-13B	27.89%	30.10%	–0.02%
(c) Ratio-based: $1.5\times$ fine-tuned $\leq$ base	Llama2-7B	Llama2-13B	27.89%	30.25%	+0.13%

Appendix

Table 9: GSM8K performance under different perplexity filtering configurations. (a) Symmetric threshold filtering with both models using threshold 1.3. (b) Asymmetric threshold filtering with fine-tuned model perplexity

<1.2

and base model perplexity

>1.6

. (c) Ratio-based filtering where examples are retained if

1.5\times\text{fine-tuned perplexity}\leq\text{base perplexity}

Figure 2: t-SNE visualization of MBPP prompt embeddings generated by Llama2-13B versus GPT-4o. Embeddings computed using the MPNet encoder model demonstrate the superior diversity of GPT-4o generated prompts.

Dataset	Source LoRA	Target	Target (no LoRA)	Our Target LoRA	Acc Increase
Medical	Llama2-7B	Llama2-13B	33.8%	35.8%	2.0%
Medical	Gemma-2B	Gemma-7B	43.0%	45.1%	2.1%
Legal	Llama2-7B	Llama2-13B	25.2%	40.8%	15.6%
Legal	Gemma-2B	Gemma-7B	44.4%	48.4%	4.0%

Appendix

Table 10: TuneShift-KD results on specialized medical (MedMCQA) and legal (LexGLUE) domain knowledge transfer.

Dataset	Source LoRA	Target	Target LoRA (Original)	Target LoRA (New 1)	Target LoRA (New 2)	Average with Error Bar
GSM8K	Llama2-7B	Llama2-13B	30.1%	30.0%	29.9%	30.1% $\pm$ 0.058%
GSM8K	Gemma-2B	Gemma-7B	44.8%	44.6%	45.1%	44.8% $\pm$ 0.15%
MBPP	Llama2-7B	Llama2-13B	40.2%	40.3%	40.0%	40.2% $\pm$ 0.088%
MBPP	Gemma-2B	Gemma-7B	51.2%	50.9%	51.2%	51.1% $\pm$ 0.10%

Appendix

Table 11: Statistical significance analysis across random seeds on GSM8K and MBPP.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 512 papers

KDFP: A first-principles approach to knowledge distillation in large language models
Oct 7, 2026Ryan Swift, Konstantinos PsounisLanguage Model Distillation Knowledge Distillation
Thomas Lord Department of Computer Science University of Southern California Los Angeles, CA 90007, USA
Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails
Jun 22, 2026Xin Liu, Simin Ma, Shujian Liu +5Data-Free Knowledge Distillation Language Model Distillation
University of Michigan · 2Zoom Video Communications
Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities
Feb 17, 2026Shankar Padmanabhan, Mustafa Omer Gul, Tanya GoyalContinual Learning for LLMs Context Distillation
Department of Computer Science, Cornell University.

Oct 7, 2026J/K move · Enter open · S save

KDFP: A first-principles approach to knowledge distillation in large language models

Ryan Swift, Konstantinos Psounis

Thomas Lord Department of Computer Science University of Southern California Los Angeles, CA 90007, USA

Language Model Distillation Knowledge Distillation

Open paper AskPDF arXiv

Knowledge distillation is an established technique for improving the capabilities of small, efficient student models by training them with the representations of larger, more capable teacher models. Much of the recent work in the distillation of large language models (LLMs) has focused on distilling abilities learned during post-training, such as instruction following, chain-of-thought reasoning, and tool usage. This has left a large research gap in general knowledge distillation for LLMs, which is essential for developing efficient and private systems suitable for deployment on edge devices. We take a first-principles approach, evaluating previous lessons from prior works and conducting new explorations to develop a distillation methodology suitable for modern LLMs. We present KDFP, a novel methodology for white-box general knowledge distillation in LLMs. We demonstrate that KDFP outperforms existing methods by 1.6%

-

4.9% across 9 benchmarks while increasing training efficiency by up to 99.1% through ephemeral parameter reduction.

Abstract

Language Model Distillation

Explore similar work

KDFP: A first-principles approach to knowledge distillation in large language models

Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails

Updating Parametric Knowledge with Context Distillation Retains Post-Training Capabilities