Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization

Zihe Liu; Yulong Mao; Jinan Xu; Xinrui Peng; Kaiyu Huang

cs.CLApr 3, 2026

Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization

Authors: Zihe Liu, Yulong Mao, Jinan Xu, Xinrui Peng, Kaiyu Huang

Organizations: Key Laboratory of Big Data & Artificial Intelligence in Transportation · School of Computer Science and Technology, Beijing Jiaotong University

Abstract

Knowledge distillation is an effective technique for pre-trained language model compression. However, existing methods only focus on the knowledge distribution among layers, which may cause the loss of fine-grained information in the alignment process. To address this issue, we introduce the Multi-aspect Knowledge Distillation (MaKD) method, which mimics the self-attention and feed-forward modules in greater depth to capture rich language knowledge information at different aspects. Experimental results demonstrate that MaKD can achieve competitive performance compared with various strong baselines with the same storage parameter budget. In addition, our method also performs well in distilling auto-regressive architecture models.

Figures & tables

Figure 1: A comparison of logits-based (a), feature-based (b) and our multi-aspect (c) learning method.

Figure 2: Overview of multi-aspect knowledge distillation. We obtain student model with the same hidden dimensions as the teacher model by low-rank matrix decomposition. We introduce matrix distillation computed by intra-layer linear, which consists of MHA mappings (queries, keys and values) and FNN vectors (up projection and down projection). We choose multi-aspect hierarchical distillation to achieve a balance between performance and training speed.

Model	#Param	RTE	MRPC	STS-B	CoLA	SST-2	QNLI	QQP	MNLI-m/-mm	Avg
BERT ${}_{\text{BASE}}$	109M	63.4	90.4	87.4	59.6	92.5	90.7	88.2	83.3/83.6	82.1
BERT ${}_{\text{SMALL}}$	29.2M	58.3	83.1	82.0	29.5	89.9	82.2	83.4	77.6/77.2	73.6
DistilBERT 4	52.2M	55.7	82.6	82.9	31.4	90.2	82.7	83.8	78.9/78.1	74.0
TinyBERT 4	14.3M	67.1	86.8	86.6	34.9	90.7	84.9	85.2	81.6/81.5	77.6
MiniLM v2	22.7M	65.7	89.5	85.8	38.9	91.0	88.4	86.1	82.2/82.0	78.8
MaKD 64	24.1M	66.4	89.6	85.5	40.9	91.0	88.9	86.7	82.4/82.3	79.3

Table 1: Results of our student on development sets of GLUE. We report F1 scores for MRPC, Person correlation for STS-B, Matthew’s correlation for CoLA and accuracy for other datasets. The GLUE results of DistilBERT are taken from Sanh et al. (2020) . For TinyBERT and MiniLM v2, we fine-tune the lastest version of their public model for a fair comparison. The results of our fine-tuning experiments are an average of 5 runs for each task. The best results are in bold .

Model	SQuAD 1.1	SQuAD 2.0	Avg
BERT ${}_{\text{BASE}}$	76.0/83.0	69.5/72.6	75.2
DistilBERT 6	67.7/76.7	62.9/64.3	67.9
TinyBERT 6	68.2/77.4	65.8/69.6	70.2
MiniLM v2	73.9/81.4	68.2/71.8	73.8
MaKD ${}_{\textbf{256}}$	74.9/82.3	69.1/72.9	74.8

Table 2: Comparative Studies on SQuAD 1.1 and SQuAD 2.0. We report the exact match and F1 scores on the development set. The best results are in bold .

Model	SST-2	MNLI-m	CoLA	STS-B	MRPC	QNLI	SQuAD 2.0	Avg
MaKD 256	92.5	83.0	55.9	87.4	91.3	90.2	72.9	81.8
$\text{w/o}~\mathcal{L}_{\rm matrix}$	91.6	81.9	51.8	86.5	89.6	88.7	71.5	80.2
$\text{w/o}~\mathcal{L}_{\rm MHA}$	92.0	82.4	52.3	86.9	90.2	89.3	72.1	80.7
$\text{w/o}~\mathcal{L}_{\rm FNN}$	92.4	82.7	54.0	87.3	90.8	89.8	72.4	81.0
$\text{w/o}~\mathcal{L}_{\rm layer}$	92.2	82.3	53.3	86.6	89.9	89.5	72.0	80.8
$\text{w/o}~\mathcal{L}_{\rm model}$	90.9	82.0	49.2	86.7	89.4	89.1	71.2	79.7

Table 3: Ablation studies of different distillation objects in the MaKD learning. The variants are validated on the development set of GLUE benchmark and SQuAD 2.0.

$\rm\mathcal{L}_{matrix}$	$\rm\mathcal{L}_{layer}$	$\rm\mathcal{L}_{model}$	MRPC	STS-B	MNLI-m	Avg
$[1\text{,}~\frac{L}{2}]$	-	-	90.0	87.1	81.9	86.3
$[\frac{L}{2}\text{,}~L]$	-	-	87.2	85.5	81.5	84.7
$[1\text{,}~L]$	-	-	89.4	86.9	81.5	85.9
-	$[1\text{,}~\frac{L}{2}]$	-	87.5	85.5	81.5	84.8
-	$[\frac{L}{2}\text{,}~L]$	-	89.9	86.8	82.0	86.2
-	$[1\text{,}~L]$	-	89.7	86.6	81.7	86.0

Table 4: Experimental results of distillation objects acting on different layers. Here,

L

denotes the number of layers of the Transformer-based model (i.e.

L

=12, when model is BERT

{}_{\text{BASE}}

in this experiment).

Model	Role	#Param	Method	Dolly	SelfInst	Vicuna	Avg
GPT-2	Teacher	124M	SFT	22.5	9.05	13.99	15.18
	Student	77.4M	None	5.69	4.19	9.51	6.46
			SFT	15.47	6.31	12.24	11.34
			KD	16.25	6.18	11.82	11.41
			SeqKD	16.56	6.33	12.14	11.67
			MaKD	18.56	8.26	13.77	13.53

Table 5: Experimental results on auto-regressive models (i.e., GPT-2 and LLaMA-3). We report Rouge-L score for three instruction following tasks. KD Song et al. (2020) fine-tunes the student model on dataset using the teacher distribution as the supervision at each token step. SeqKD Zhou et al. (2023) fine-tunes the student model on the data generated by the teacher model.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 508 papers

From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers
Mar 11, 2026Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar +1Cross-Modal Knowledge Distillation Knowledge Distillation
Indian Institute of Technology Delhi, India · Indraprastha Institute of Information Technology Delhi, India
Knowledge Distillation for Large Language Models
Mar 14, 2026Alejandro Paredes La Torre, Barbara Flores, Diego RodriguezLanguage Model Distillation LLM Compression
Duke University
When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization
Jul 22, 2026Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela +4Selective Knowledge Distillation Text Summarization
BRAC University, Dhaka, Bangladesh

Mar 11, 2026J/K move · Enter open · S save

From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers

Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar, Tanmoy Chakraborty

Indian Institute of Technology Delhi, India · Indraprastha Institute of Information Technology Delhi, India

Cross-Modal Knowledge Distillation Knowledge Distillation Language Model Distillation VLM Distillation

Open paper AskPDF arXiv

Knowledge distillation (KD) methods are pivotal in compressing large pre-trained language models into smaller models, ensuring computational efficiency without significantly dropping performance. Traditional KD techniques assume homogeneity in modalities between the teacher (source) and the student (target) models. On the other hand, existing multimodal knowledge distillation methods require modality-specific pre-training of the teacher model, which is computationally infeasible in most cases. In this paper, we introduce ARMADA, an efficient cross-modal knowledge distillation framework designed to transfer knowledge from large vision-language models, including black-box models, to language-only models. Unlike existing KD techniques that rely on the internal structures of multimodal teachers or require computationally expensive pre-training, ARMADA leverages novel alignment techniques to distil knowledge without altering the teacher model, ensuring efficiency and scalability. We empirically validate ARMADA on twelve natural language understanding, eight complex generative reasoning and five instruction-tuning tasks, demonstrating consistent performance improvements in large models such as DeBERTa-v2-1.4B, OPT-1.3B, LLaMA-{3B, 7B, 8B}. ARMADA achieves up to 3.4% improvement on language understanding tasks and 2.6% boost in generative reasoning, all without requiring expensive multimodal pre-training or fine-tuning of the teacher model. Our findings challenge conventional knowledge distillation paradigms by demonstrating that even vision-language models, despite lacking direct textual understanding, can significantly enhance language models when distilled appropriately.

Abstract

Language Model Distillation

Explore similar work

From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers

Knowledge Distillation for Large Language Models

When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization