Knowledge distillation (KD) methods are pivotal in compressing large pre-trained language models into smaller models, ensuring computational efficiency without significantly dropping performance. Traditional KD techniques assume homogeneity in modalities between the teacher (source) and the student (target) models. On the other hand, existing multimodal knowledge distillation methods require modality-specific pre-training of the teacher model, which is computationally infeasible in most cases. In this paper, we introduce ARMADA, an efficient cross-modal knowledge distillation framework designed to transfer knowledge from large vision-language models, including black-box models, to language-only models. Unlike existing KD techniques that rely on the internal structures of multimodal teachers or require computationally expensive pre-training, ARMADA leverages novel alignment techniques to distil knowledge without altering the teacher model, ensuring efficiency and scalability. We empirically validate ARMADA on twelve natural language understanding, eight complex generative reasoning and five instruction-tuning tasks, demonstrating consistent performance improvements in large models such as DeBERTa-v2-1.4B, OPT-1.3B, LLaMA-{3B, 7B, 8B}. ARMADA achieves up to 3.4% improvement on language understanding tasks and 2.6% boost in generative reasoning, all without requiring expensive multimodal pre-training or fine-tuning of the teacher model. Our findings challenge conventional knowledge distillation paradigms by demonstrating that even vision-language models, despite lacking direct textual understanding, can significantly enhance language models when distilled appropriately.
Figures & tables
Figure 1: A schematic diagram of ARMADA . We highlight an example from the NLU task, where the text-to-image teacher model is used to distil knowledge to a language-only student model. Through manifold and output alignment loss objectives, the aligner model orchestrates the knowledge transfer from teacher to student modality.
Notation Type
Notation
Description
Dimension/Shape
Modalities
Mt
Teacher model modality
Ms
Student model modality
Architectures
Ft
Multi-modal teacher model
Fts
TS Aligner module
F~ts
Hidden non-linear projection mapping for TS Aligner
R128→Rh , h is hidden dimension of TS Aligner
OtsT
TS Aligner output layer for task T
Rh→Rc , h is hidden dimension of TS Aligner ; c is number of output classes for task T
Table 1: Glossary of all the mathematical notations in the paper and their descriptions.
Figure 2: Commutative diagram between the manifold, output and auxiliary output spaces. The bold lines denote the functions already defined between the spaces. The dashed lines indicate the functions we want to establish in the proof of Proposition 2.
Models
Distillation type
CoLA
MRPC
RTE
STS-B
CB
COPA
BoolQ
WiC
Mcc
Acc
Acc
Pear
Acc
Acc
Acc
Acc
BERT-6L
undistilled
42.8 ± 0.1
78.6 ± 0.0
64.1 ± 0.2
88.5 ± 0.4
75.0 ± 0.0
53.0 ± 0.0
71.3 ± 0.1
55.7 ± 0.2
ARMADA - Lcosine
46.1 ± 0.1
80.3 ± 0.0
67.5 ± 0.0
88.5 ± 0.1
76.8 ± 0.0
62.8 ± 0.1
73.1 ± 0.0
57.8 ± 0.1
ARMADA - Leuclid
47.6 ± 0.0
80.5 ± 0.0
67.4 ± 0.1
88.5 ± 0.2
78.5 ± 0.1
63.0 ± 0.0
73.1 ± 0.0
58.1 ± 0.1
ARMADA - Lelementwise
45.7 ± 0.2
79.5 ± 0.0
67.5 ± 0.1
88.6 ± 0.1
78.6 ± 0.0
63.0 ± 0.0
73.0 ± 0.1
57.9 ± 0.0
BERT-base
undistilled
60.8 ± 0.1
83.8 ± 0.1
71.0 ± 0.0
89.3 ± 0.0
83.7 ± 0.1
62.9 ± 0.0
75.1 ± 0.1
58.4 ± 0.1
Table 2: Results of different LMs (undistilled and distilled by ARMADA with Stable Diffusion teacher) on GLUE-small and SuperGLUE tasks - different training paradigms. Blue indicates the tasks where the student model distilled - ARMADA exhibits better performance than the undistilled fine-tuned variants. We report the average (and standard deviation) scores from three runs for each experiment.
Models
Distillation type
MNLI
QNLI
QQP
SST-2
Acc
Acc
Acc
Acc
BERT-6L
undistilled
77.7 ± 0.0
84.9 ± 0.1
89.0 ± 0.0
87.6 ± 0.1
ARMADA - Lcosine
80.2 ± 0.0
87.3 ± 0.0
88.6 ± 0.1
90.7 ± 0.0
ARMADA - Leuclid
80.3 ± 0.0
87.2 ± 0.1
88.9 ± 0.0
90.6 ± 0.0
ARMADA - Lelementwise
80.1 ± 0.0
87.3 ± 0.0
89.1 ± 0.1
90.7 ± 0.0
BERT-base
undistilled
79.3 ± 0.0
87.8 ± 0.1
83.1 ± 0.0
89.2 ± 0.1
Table 3: Results of BERT models (undistilled and distilled by ARMADA with Stable Diffusion teacher) on GLUE-large tasks. Blue indicates cases where ARMADA improves performance of the undistilled model.
Distillation type
Hellaswag
Winogrande
ARC-c
ARC-e
OpenBookQA
MultiArith
SingleEq
AQuA
Avg.
undistilled
65.2
70.5
65.5
80.9
75.6
96.0
83.5
14.8
69.0
ARMADA with Lcosine
67.8
69.5
64.7
81.1
76.8
97.0
83.1
15.9
69.5
ARMADA with Leuclid
67.0
70.0
65.7
80.8
76.5
96.5
83.3
15.6
69.4
ARMADA with Lelementwise
65.8
69.6
64.3
80.8
75.9
96.2
83.1
15.2
68.9
Table 4: Zero-shot accuracy on reasoning tasks for LLaMA-7B distilled with ARMADA with Stable Diffusion teacher model. Blue indicates the tasks where ARMADA improves performance over the undistilled LLaMA.
Models
Distillation type
CoLA
MRPC
RTE
STS-B
CB
COPA
BoolQ
WiC
Mcc
Acc
Acc
Pear
Acc
Acc
Acc
Acc
BERT-6L
undistilled
42.8 ± 0.1
78.6 ± 0.0
64.1 ± 0.2
88.5 ± 0.4
75.0 ± 0.0
53.0 ± 0.0
71.3 ± 0.1
55.7 ± 0.2
ARMADA - Lcosine
44.0 ± 0.0
80.1 ± 0.0
65.2 ± 0.1
87.9 ± 0.0
73.1 ± 0.1
49.9 ± 0.0
71.8 ± 0.0
56.6 ± 0.0
ARMADA - Leuclid
42.9 ± 0.0
79.1 ± 0.1
66.3 ± 0.0
88.3 ± 0.0
75.0 ± 0.0
62.9 ± 0.1
72.4 ± 0.0
58.2 ± 0.0
ARMADA - Lelementwise
46.9 ± 0.0
79.3 ± 0.0
67.0 ± 0.1
88.3 ± 0.0
78.4 ± 0.1
51.9 ± 0.0
73.3 ± 0.0
55.9 ± 0.0
BERT-base
undistilled
60.8 ± 0.1
83.8 ± 0.1
71.0 ± 0.0
89.3 ± 0.0
83.7 ± 0.1
62.9 ± 0.0
75.1 ± 0.1
58.4 ± 0.1
Table 5: Results of different LMs (undistilled and distilled by ARMADA ) on GLUE-small and SuperGLUE tasks with different training paradigms with Midjourney ( Borji, 2022 ) teacher model.
Teacher Model
CoLA
MRPC
RTE
STS-B
CB
COPA
BoolQ
WiC
Mcc
Acc
Acc
Pear
Acc
Acc
Acc
Acc
Stable Diffusion
7.8
66.6
53.4
3.7
55.4
62.0
60.9
53.0
Midjourney
1.8
66.5
52.3
1.9
50.0
56.0
56.6
50.6
Table 6: Results of TS Aligner on all GLUE-small and SuperGLUE tasks with different teacher models.
Methods
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
Avg.
KD ( Hinton et al., 2015 )
43.5
79.3
79.4
85.6
89.6
64.4
87.8
89.2
77.4
PKD ( Sun et al., 2019 )
43.9
79.4
78.9
85.9
89.6
64.3
87.9
89.2
77.4
TinyBERT ( Jiao et al., 2019 )
41.8
80.3
80.7
86.2
89.3
64.4
88.5
89.8
77.6
RCO ( Jin et al., 2019 )
43.0
79.1
79.3
86.1
89.3
64.3
88.0
89.2
77.3
TAKD ( Mirzadeh et al., 2020 )
43.2
79.2
79.2
86.0
89.4
65.0
88.0
89.3
77.4
DML ( Zhang et al., 2018 )
43.1
79.1
79.3
86.0
89.0
64.9
88.1
89.2
77.3
Table 7: Improvement for BERT-6L on GLUE tasks distilled with ARMADA and other contemporary unimodal distillation methods. For all the KD baselines except ARMADA , BERT-base ( 110M learnable parameters) is used as the teacher model. ARMADA uses Stable Diffusion as the teacher model ( 0.8M learnable parameters).
Methods
SST-2
QNLI
QQP
MNLI
Avg.
ViLBERT
90.3
89.6
88.4
82.4
87.7
VL-BERT
90.1
89.5
88.6
82.9
87.8
VisualBERT
90.3
88.9
88.4
82.4
87.5
Img-Voken
92.2
88.6
88.6
82.6
88.0
Visual Guidance
89.7
88.2
84.2
79.5
85.4
Vid-Voken w/ResNet
93.4
89.2
88.7
83.0
88.6
Table 8: Improvement for BERT-base model on GLUE-large tasks distilled by ARMADA with Stable Diffusion teacher model, compared against the improvements reported by multimodal KD methods.
Models
Distillation type
MM-IMDb
Meme
F1 Micro/Macro
Acc
BERT-12L
undistilled
64.9 48.1
55.4
ARMADA - Lcosine
65.7/ 49.3
60.2
ARMADA - Leuclid
66.2/ 48.9
60.4
ARMADA - Lelementwise
65.3/ 48.0
59.4
Table 9: Results of BERT-base model - and without cross-modal distillation on MM-IMDb genre and Hateful meme classification tasks.
Models
Loss type
CoLA
MRPC
RTE
STS-B
CB
COPA
BoolQ
WiC
Mcc
Acc
Acc
Pear
Acc
Acc
Acc
Acc
TS Aligner
-
0.7
66.4
50.2
1.3
48.9
41.0
45.4
48.9
BERT-base
cosine
59.0
83.4
70.4
88.6
87.5
64.0
74.7
57.1
euclid
60.1
84.1
67.5
88.2
85.7
63.0
73.9
55.2
elementwise
58.5
82.3
65.7
88.6
85.7
65.0
74.4
57.2
Table 10: Auxiliary output results for TS Aligner and BERT-base student model with different Lmanifold loss objectives on GLUE-small and SuperGLUE tasks.
Figure 3: Distribution of margin (referred as improvement ) between the undistilled and BERT-base distilled with ARMADA and Stable Diffusion teacher model for different values of α , β and γ . A. α highlights the importance of output alignment, B. β highlights the importance of manifold alignment between TS Aligner and the student, and C. γ highlights the importance of auxiliary output on the combined loss objective.
Figure 4: Performance improved in distilled BERT-base model over undistilled variant across all tasks for different Lmanifold functions.
Model
CoLA
MRPC
RTE
STS-B
CB
COPA
BoolQ
WiC
Mcc
Acc
Acc
Pear
Acc
Acc
Acc
Acc
TS Aligner
9.9
66.5
54.5
2.1
51.8
54/0
53.9
52.3
ARMADA with Lcosine
59.3
84.3
69.7
89.3
85.7
62.0
73.2
57.2
ARMADA with Leuclide
59.3
82.4
69.7
89.3
85.7
54.0
76.2
56.4
ARMADA with Lelementwise
59.3
82.4
69.7
89.3
83.9
55
76.7
55.6
Table 11: Results of TS Aligner and BERT-12L with ARMADA on all GLUE-small and SuperGLUE tasks with higher-capacity TS Aligner . We increase the number of hidden layers in TS Aligner and remove the projection mapping F~ts and auxiliary output head OauxtsT . The objective of this experiment is to corroborate that performance improvement by ARMADA is not entirely due to the capacity of the aligner model.
Teacher
Distillation Type
Dolly
SNI
SelfInst
UNI
Vicuna
-
undistilled
31.30
26.68
18.44
32.01
17.59
language-only
SeqKD
31.45
28.48
18.61
31.05
17.74
text-to-image
ARMADA with Lcosine
31.94
27.13
19.83
33.57
18.73
ARMADA with Leuclid
30.57
28.25
18.06
31.59
18.15
ARMADA with Lelementwise
31.38
26.68
18.28
32.27
18.16
text-to-video
ARMADA with Lcosine
33.02
26.98
18.49
32.07
18.44
Table 12: Results on instruction-tuning tasks with LLaMA-3.2-3B with different multimodal teacher models. We also highlight the RougeL score when distilled from language-only LLaMA-8B model with SeqKD ( Kim and Rush, 2016 ) . A p-value of 0.005 with one-sided t-test suggests that ARMADA provides better post-distillation performance than SeqKD. Interestingly, the teacher models used with ARMADA are significantly smaller (<2B) than the language-only teacher model (8B) used in SeqKD.
Figure 5: Distribution of improvement between the undistilled and ARMADA -distilled LLaMA-3B, for different values of β and γ . A two-way ANOVA using ordinary least square (OLS) regression model confirms the statistical significance (p-value 7e−5 and 5e−3 for β and γ , respectively) of both the hyperparameters.
Figure 6: Distribution of improvement between the undistilled and ARMADA -distilled LLaMA-3B, for different manifold alignment loss functions. A p-value of 0.52 with the ANOVA test suggests that we may not reject the null hypothesis of dissimilarity between the different loss functions. Therefore, the empirical evidence suggests that all the loss functions could be equally effective in the ARMADA framework.
Teacher
Distillation Type
Dolly
SNI
SelfInst
UNI
Vicuna
-
undistilled
35.34
28.19
18.01
32.78
17.85
text-to-image
ARMADA with Lcosine
33.67
28.77
19.36
32.59
18.89
ARMADA with Leuclid
33.71
28.11
19.14
34.53
18.72
ARMADA with Lelementwise
34.63
29.06
19.80
32.54
18.16
text-to-video
ARMADA with Lcosine
33.53
29.28
18.57
33.80
18.82
ARMADA with Leuclid
32.02
28.07
21.31
32.41
18.94
Table 13: Results on instruction-tuning tasks with LLaMA-3.1-8B with different multimodal teacher models.
Figure 7: t-SNE plots of embeddings obtained from different models on CoLA and COPA tasks.
Figure 8: Error analysis on CoLA and COPA tasks. Different output classes are highlighted with different sizes.
Methods
CoLA
COPA
Scen. 1
Scen. 2
Scen. 1
Scen. 2
Undistilled
0.31
0.40
0.31
0.59
ARMADA
0.52
0.48
0.57
0.38
Table 14: Silhouette scores (higher the better) for different scenarios with undistilled and ARMADA models.
Scenario
Text
Gold Label
Prediction
Dist. Before
Dist. After (% Change)
1
As you eat the most, you want the least.
0
0
0.097
0.001 (99%)
Who is she trying to make up to now?
1
1
0.575
0.007 (99%)
2
The more does Bill smoke, the more Susan hates him.
0
1
0.073
0.016 (78%)
Clearly, John probably will immediately learn French perfectly.
1
0
0.258
0.027 (89%)
Table 15: Cosine distance from the cluster centroid to the embeddings obtained by BERT-base model before and after distilled with ARMADA . Examples selected from CoLA classification task.
Loss type
CoLA
MRPC
RTE
STS-B
CB
COPA
BoolQ
WiC
Cosine
6.75 (0.00)
6.95 (0.00)
3.33 (0.00)
54.63 (0.00)
3.89 (0.00)
0.49 (0.62)
-0.29 (0.78)
12.91 (0.00)
Euclid
6.54 (0.00)
8.40 (0.00)
3.28 (0.00)
51.33 (0.00)
3.20 (0.00)
1.47 (0.14)
-3.20 (0.00)
16.29 (0.00)
Elementwise
6.42 (0.00)
8.03 (0.00)
3.16 (0.00)
51.51 (0.00)
4.48 (0.00)
5.85 (0.00)
-2.45 (0.01)
15.81 (0.00)
Table 16: T -statistic ( p -value) between student training loss distributions without and with TS Aligner training. A positive t -statistics (highlighted in bold) indicates that the expected student loss without a trained TS Aligner is higher than the student loss with a jointly trained TS Aligner model, indicating the importance of joint aligner-student training.
Figure 9: Categorization of examples where BERT-12L distilled with ARMADA predicts correctly and the undistiled variant fails.
Figure 10: Performance of distilled BERT-base under different training curricula. The performance improvement with a trained TS Aligner demonstrates the importance of TS Aligner training on cross-modal distillation.
Figure 11: Performance of TS Aligner and distilled BERT-base student under teacher input noises.
Figure 12: Sensitivity of TS Aligner and distilled BERT-base student models under input noises.
Figure 13: Improvement of ARMADA over undistilled LLaMA-3B for different teacher input noises. Higher noise indicates poorer quality of teacher representations.
Model
CoLA
MRPC
RTE
STS-B
CB
COPA
BoolQ
WiC
Mcc
Acc
Acc
Pear
Acc
Acc
Acc
Acc
TS Aligner
4.9
66.5
54.9
2.5
41.1
54.0
55.0
53.0
ARMADA with Lcosine
57.9
84.4
71.5
90.0
81.1
63.0
75.8
56.4
ARMADA with Leuclide
59.4
84.5
70.8
89.5
82.1
63.0
75.5
57.0
ARMADA with Lelementwise
60.4
84.8
70.0
89.8
83.9
63.0
76.7
57.4
Table 17: Results of TS Aligner and BERT-12L with ARMADA on all GLUE-small and SuperGLUE tasks with shuffled teacher inputs. We randomly shuffle the image embeddings obtained by the Stable Diffusion teacher model during training the student models.
Cross-modal knowledge distillation transfers useful representations or predictions between modalities, often enabling cheaper single-modality inference.
Turning a pretrained language model (LM) into a vision-language model (VLM) through multimodal fine-tuning often erodes its native language ability, a form of catastrophic forgetting that shows up even on text-only tasks. This loss is hard to undo with further fine-tuning, and existing remedies add adapters or alignment modules that increase architectural complexity and inference cost. We propose LinguDistill, an adapter-free knowledge distillation method that uses the original frozen LM as the teacher during multimodal post-training. To let a text-only teacher supervise vision-conditioned outputs, we introduce layer-wise KV-cache sharing, which exposes the teacher to the student's multimodal representations without changing either architecture. We then apply distillation selectively, on language-heavy data only, so the teacher restores linguistic ability while the student keeps its visual grounding on document and OCR tasks. LinguDistill recovers the language and knowledge performance lost during multimodal fine-tuning, matching the original VLM on average over text-only benchmarks (ARC, HellaSwag) and exceeding it on ScienceQA, while keeping vision-heavy performance close to standard fine-tuning. Since the teacher is dropped after training, the final model adds no parameters and no inference cost. More broadly, our results show that a model's own pre-adaptation backbone is a practical teacher for undoing forgetting, suggesting a simple recipe for keeping language ability intact as models are extended to new modalities.
Patrick Amadeus Irawan, Erland Hilman Fuadi, Shanu Kumar +2
Mohamed bin Zayed University of Artificial Intelligence
Knowledge distillation is an effective technique for pre-trained language model compression. However, existing methods only focus on the knowledge distribution among layers, which may cause the loss of fine-grained information in the alignment process. To address this issue, we introduce the Multi-aspect Knowledge Distillation (MaKD) method, which mimics the self-attention and feed-forward modules in greater depth to capture rich language knowledge information at different aspects. Experimental results demonstrate that MaKD can achieve competitive performance compared with various strong baselines with the same storage parameter budget. In addition, our method also performs well in distilling auto-regressive architecture models.
Zihe Liu, Yulong Mao, Jinan Xu +2
Key Laboratory of Big Data & Artificial Intelligence in Transportation · School of Computer Science and Technology, Beijing Jiaotong University
Knowledge distillation is crucial for deploying vision-language models on resource-constrained devices. However, existing methods typically impose uniform supervision across visual tokens or rely on static token selection, which confuses task-relevant cues with background noise and degrades cross-modal reasoning. To address this limitation, we propose Key Visual Evidence-guided Knowledge Distillation (KVE-KD), a framework that dynamically focuses feature distillation on task-relevant visual tokens identified by the teacher model. Specifically, KVE-KD appoints the final pre-generation textual token as a unified semantic anchor and identifies the target cross-modal fusion layer by analyzing changes in the anchor representation through iterative visual-token contribution removal. Within this layer, KVE-KD ranks visual tokens via the anchor-conditioned attention distribution and selects the most informative visual tokens as key visual evidence with normalized entropy. The key visual evidence subsequently guides focused visual feature distillation, making the student align closely with the teacher's task-relevant visual representations while suppressing irrelevant background information. Extensive experiments on six benchmarks demonstrate that KVE-KD outperforms state-of-the-art cross-modal distillation methods, with particularly pronounced gains on tasks requiring complex reasoning and fine-grained visual understanding. Importantly, these improvements are achieved without introducing any inference-time overhead. The source code is available at https://github.com/zhangjianbin07/KVE-KD.
Jianbin Zhang, Xin Sun, Shanwen Wang +2
Faculty of Data Science, City University of Macau, SAR Macao, China