To embed domain-specific or specialized knowledge into pre-trained foundation models, fine-tuning using techniques such as parameter efficient fine-tuning (e.g. LoRA) is a common practice. However, as new LLM architectures and pre-trained models emerge, transferring this specialized knowledge to newer models becomes an important task. In many scenarios, the original specialized data may be unavailable due to privacy or commercial restrictions, necessitating distillation and transfer of this specialized knowledge from the fine-tuned base model to a different pre-trained model. We present TuneShift-KD, a novel approach that automatically distills specialized knowledge from a fine-tuned model to a target model using only a few examples representative of the specialized information. Our key insight is that specialized knowledge can be identified through perplexity differences between base and fine-tuned models: prompts where the fine-tuned model responds confidently (low perplexity), but the base model struggles (high perplexity), indicate queries corresponding to the specialized knowledge learned by the fine-tuned model. TuneShift-KD leverages this insight to create a synthetic training dataset to transfer the specialized knowledge. Using an iterative process, TuneShift-KD generates more prompts similar to those that generated responses with specialized knowledge. TuneShift-KD does not require training discriminators or access to training datasets. It is an automated approach that only requires the initial fine-tuned and base models and a few representative prompts. Our experiments demonstrate that models fine-tuned using TuneShift-KD achieve higher accuracy than prior approaches, enabling ease of deployment and more effective transfer of the specialized knowledge.
Figures & tables
Figure 1: TuneShift-KD Framework for Specialized Knowledge Transfer. This figure illustrates our approach to transferring specialized knowledge without requiring full access to original fine-tuning data. (a) The Synthetic Fine-tuning Dataset begins with just 5 seed examples from the original fine-tuning data and expands through our iterative process. (b) An instruction-tuned LLM generates similar prompts based on the prompt pattern ”Generate 20 more samples like these 5 [ the 5 sample data ]”. (c) Each synthetic prompt is fed to both the Source Base LLM and Source Fine-tuned LLM to obtain paired responses y(b) and y(f) . (d) Our key filtering mechanism selects prompts where the fine-tuned model shows high confidence (low perplexity PPL(y(f)) ) while the base model shows low confidence (high perplexity PPL(y(b)) ), indicating the prompt targets specialized knowledge. Qualified prompt-response pairs are added to the synthetic dataset, while others are discarded. (e) The curated synthetic dataset is used to perform knowledge distillation, transferring specialized capabilities to the Target LLM without requiring original training data.
Table 8: TuneShift-KD results using Qwen models as source and target
Filter Setting
Source Model
Target Model
Target (no LoRA)
Target LoRA Acc.
Acc. Change (vs. default)
(a) Threshold = 1.3
Llama2-7B
Llama2-13B
27.89%
30.02%
–0.10%
(b) Fine-tuned <1.2 , Base >1.6
Llama2-7B
Llama2-13B
27.89%
30.10%
–0.02%
(c) Ratio-based: 1.5× fine-tuned ≤ base
Llama2-7B
Llama2-13B
27.89%
30.25%
+0.13%
Appendix
Table 9: GSM8K performance under different perplexity filtering configurations. (a) Symmetric threshold filtering with both models using threshold 1.3. (b) Asymmetric threshold filtering with fine-tuned model perplexity <1.2 and base model perplexity >1.6 . (c) Ratio-based filtering where examples are retained if 1.5×fine-tuned perplexity≤base perplexity .
Figure 2: t-SNE visualization of MBPP prompt embeddings generated by Llama2-13B versus GPT-4o. Embeddings computed using the MPNet encoder model demonstrate the superior diversity of GPT-4o generated prompts.
Dataset
Source LoRA
Target
Target (no LoRA)
Our Target LoRA
Acc Increase
Medical
Llama2-7B
Llama2-13B
33.8%
35.8%
2.0%
Medical
Gemma-2B
Gemma-7B
43.0%
45.1%
2.1%
Legal
Llama2-7B
Llama2-13B
25.2%
40.8%
15.6%
Legal
Gemma-2B
Gemma-7B
44.4%
48.4%
4.0%
Appendix
Table 10: TuneShift-KD results on specialized medical (MedMCQA) and legal (LexGLUE) domain knowledge transfer.
Dataset
Source LoRA
Target
Target LoRA (Original)
Target LoRA (New 1)
Target LoRA (New 2)
Average with Error Bar
GSM8K
Llama2-7B
Llama2-13B
30.1%
30.0%
29.9%
30.1% ± 0.058%
GSM8K
Gemma-2B
Gemma-7B
44.8%
44.6%
45.1%
44.8% ± 0.15%
MBPP
Llama2-7B
Llama2-13B
40.2%
40.3%
40.0%
40.2% ± 0.088%
MBPP
Gemma-2B
Gemma-7B
51.2%
50.9%
51.2%
51.1% ± 0.10%
Appendix
Table 11: Statistical significance analysis across random seeds on GSM8K and MBPP.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
Knowledge distillation is an established technique for improving the capabilities of small, efficient student models by training them with the representations of larger, more capable teacher models. Much of the recent work in the distillation of large language models (LLMs) has focused on distilling abilities learned during post-training, such as instruction following, chain-of-thought reasoning, and tool usage. This has left a large research gap in general knowledge distillation for LLMs, which is essential for developing efficient and private systems suitable for deployment on edge devices. We take a first-principles approach, evaluating previous lessons from prior works and conducting new explorations to develop a distillation methodology suitable for modern LLMs. We present KDFP, a novel methodology for white-box general knowledge distillation in LLMs. We demonstrate that KDFP outperforms existing methods by 1.6% − 4.9% across 9 benchmarks while increasing training efficiency by up to 99.1% through ephemeral parameter reduction.
Ryan Swift, Konstantinos Psounis
Thomas Lord Department of Computer Science University of Southern California Los Angeles, CA 90007, USA
Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model. While prior work has mainly examined task-specific or small-scale settings, the post-training stage for building general instruction-following models has received limited attention. In this paper, we conduct a systematic study of KD in post-training using the large-scale Tulu 3 dataset. We find that KD outperforms supervised fine-tuning (SFT) in low-data regimes, but its advantage diminishes as more training data is added. Distilling from a stronger instruction-tuned teacher restores substantial gains even with abundant data, indicating that KD remains effective when the teacher provides knowledge that the student cannot easily acquire from the training data alone. We further study domain-specific, low-resource scenarios and propose a two-stage KD strategy that leverages synthetic teacher-labeled data followed by refinement on human annotations. This method consistently improves student performance, providing practical guidance for building compact models in data-scarce environments.
Xin Liu, Simin Ma, Shujian Liu +5
University of Michigan · 2Zoom Video Communications
Post-training endows pretrained LLMs with a variety of desirable skills, including instruction-following, reasoning, and others. However, these post-trained LLMs only encode knowledge up to a cut-off date, necessitating continual adaptation. Unfortunately, existing solutions cannot simultaneously learn new knowledge from an adaptation document corpora and mitigate the forgetting of earlier learned capabilities. To address this, we introduce Distillation via Split Contexts (DiSC), a simple context-distillation based approach for continual knowledge adaptation. \methodname~derives student and teacher distributions by conditioning on distinct segments of the training example and minimizes the KL divergence between the shared tokens. This allows us to efficiently apply context-distillation without requiring explicit generation steps during training. We run experiments on four post-trained models and two adaptation domains. Compared to prior finetuning and distillation methods for continual adaptation, DiSC consistently reports the best trade-off between learning new knowledge and mitigating forgetting of previously learned skills like instruction-following, reasoning, and factual knowledge.
Shankar Padmanabhan, Mustafa Omer Gul, Tanya Goyal
Department of Computer Science, Cornell University.