Cross-modal knowledge distillation can incur negative transfer when source-modality supervision conflicts with the target modality's discriminative characteristics. We propose TALD, a two-stage framework for Transferability-Aware Logit Disentanglement in class-aligned logit space. Guided by task-dependent self-benefiting and cross-modal exclusivity criteria, a Residual Disentangler decomposes logits from a teacher and an independently trained target-modality reference student into modality-general and modality-specific components. A Sample-Adaptive Modulator regulates decomposition constraints and sample-specific fusion weights. The learned modality-general components are then adaptively fused to supervise the final student. Across five benchmarks spanning vision, audio, and text, TALD achieves the largest mean accuracy gain and highest positive-transfer frequency among the compared methods. Across ten transfer directions with five runs each, TALD improves over training without distillation in 84% of evaluations, compared with at most 70% for the baselines, while offering a favorable accuracy-efficiency trade-off. We plan to release the code.
Figures & tables
Figure 1: Class-structured negative transfer and the transferability-aware view (RAVDESS, V → A; single run). The visual and audio baselines differ in per-class accuracy by up to about 42 percentage points (pp), and the KD student falls below the audio baseline on several classes (e.g., C2, C4). Conventional KD uses the full teacher response, while TALD constructs its distillation target by adaptively fusing learned modality-general components from the teacher and reference student.
Figure 2: Overview of the TALD framework. In the first stage, the Disentangler decomposes teacher and reference student logits into modality-general and modality-specific components, while the Modulator predicts sample-specific margins and contribution-ratio bounds to regulate the decomposition. In the second stage, the modality-general components are fused using weights derived from the predicted margins and distilled to the target-modality student.
Figure 3: Architecture of the Residual Disentangler. Left: An MLP shared across the two modalities predicts the SPEC components, and the GEN components are obtained by residual subtraction. Right: A direction-specific CMP applies a class-conditioned affine transformation to a SPEC component to construct the cross-modal contribution used in the Stage-1 C2 constraint.
Figure 4: Architecture of the Sample-Adaptive Modulator. A backbone MLP encodes concatenated teacher and reference student logits, from which two independent heads predict margin δ^ and ratio bounds [ρ^min,ρ^max] , rescaled to predefined intervals via fixed range mapping.
Paradigm
Method
AV-MNIST
RAVDESS
VGGSound-50k
CrisisMMD-V2
Fakeddit
Δ Acc
PTF
Image
Audio
Visual
Audio
Visual
Audio
Image
Text
Image
Text
—
w/o KD
0.6322
0.4372
0.7472
0.6694
0.4080
0.5818
0.5524
0.5405
0.5729
0.5196
—
—
Response-based
KD
0.6328
0.4377
0.7035
0.6944
0.3855
0.5887
0.5501
0.5377
0.5745
0.4999
-0.56
42%
MLLD
0.6327
0.4380
0.7208
0.6799
0.4044
0.5898
0.5541
0.5127
0.5726
0.2610
-2.95
46%
Feature-based
FitNets
0.6201
0.4292
0.6889
0.6229
0.3742
0.5685
0.5458
0.5165
0.5492
0.5465
-1.99
10%
OFA
0.6334
0.4380
0.7292
0.6445
0.3861
0.5839
0.5493
0.5420
0.5727
0.5330
-0.49
50%
Table 1: Performance comparison on five multimodal classification benchmarks. Accuracy is averaged over five runs. Δ Acc is the mean gain (pp) over w/o KD across ten transfer directions. PTF is the percentage of the 50 evaluations (ten directions, five runs each) with strictly positive accuracy gains over the corresponding w/o KD baseline. The best and second-best results per column are bold , with the best additionally underlined .
Method
Stage
Params
FLOPs
Mem
Time
Δ Acc
(M) ↓
(T) ↓
(MiB) ↓
(s) ↓
(pp) ↑
MGDFR
S1
3.98
4871.12
1536.39
872.96
+1.00
S2
0.00
18260.23
863.03
814.39
S3
1.99
3589.71
2406.10
874.02
Total
5.96
26721.05
2406.10
2561.36
C 2 KD
S0
1.99
2435.64
1530.85
653.05
-5.87
Table 2: Efficiency comparison of four CMKD methods. Stage costs are averaged over five datasets and both directions. S0 denotes unimodal pretraining; the remaining stages follow each method’s pipeline. Total sums trainable parameters (Params), computation (FLOPs), and wall-clock time (Time), and takes the maximum of the stage-averaged peak GPU memory (Mem).
#
Framework
Sources
RAVDESS
Fakeddit
Avg
(a)
w/o KD
—
0.7083
0.5463
0.6273
(b)
Vanilla KD
T
0.6990
0.5372
0.6181
(c)
Vanilla KD
Sref
0.7260
0.5452
0.6356
(d)
Mean Fusion
T+Sref
0.7250
0.5439
0.6345
(e)
Single Stage
T+Sref
0.7365
0.5443
0.6404
(f)
Two Stage
T+Sref
0.7448
0.5483
0.6466
Table 3: Framework design ablation. T and Sref denote the cross-modal teacher and independently trained target-modality reference student. Each dataset score averages both directions; Avg averages the two datasets.
#
Logits
Fusion
RAVDESS
Fakeddit
Avg
(a)
ORG
Adaptive
0.7410
0.5443
0.6427
(b)
SPEC
Adaptive
0.7247
0.5428
0.6338
(c)
GEN
Mean
0.7358
0.5430
0.6394
(d)
GEN
Shuffled-S
0.7368
0.5431
0.6400
(e)
GEN
Shuffled-C
0.7375
0.5445
0.6410
(f)
GEN
Adaptive
0.7448
0.5483
0.6466
Table 4: Ablation of logit selection and adaptive fusion using both the teacher and reference student. ORG, GEN, and SPEC denote original, modality-general, and modality-specific logits. Mean and Adaptive use equal weights and the learned weight w , respectively. Shuffled-S permutes w across samples; Shuffled-C restricts the permutation to each class.
#
Disentangler
Modulator
RAVDESS
Fakeddit
Avg
Sharing
ResDis
Adapt
(a)
IND
✗
✗
0.7400
0.5481
0.6441
(b)
IND
✓
✓
0.7351
0.5460
0.6406
(c)
FS
✗
✗
0.7385
0.5492
0.6439
(d)
FS
✓
✓
0.7413
0.5480
0.6447
(e)
PS
✗
✗
0.7337
0.5474
0.6406
Table 5: Module design ablation. IND, FS, and PS denote independent, fully shared, and partially shared disentanglers; PS shares the MLP and uses separate CMPs. ResDis and Adapt denote residual disentanglement and sample-adaptive modulation.
#
Setting
Mapping
RAVDESS
Fakeddit
Avg
(a)
w/o CMP
ID
0.7441
0.5437
0.6439
(b)
Affine
CI
0.7396
0.5472
0.6434
(c)
Linear
CC
0.7439
0.5461
0.6450
(d)
Bias-only
CC
0.7425
0.5434
0.6430
(e)
Affine
CC
0.7448
0.5483
0.6466
Table 6: CMP design ablation. ID, CI, and CC denote identity, class-independent, and class-conditioned mappings, respectively. ID uses the other modality’s SPEC directly in C2. Linear omits the bias; Bias-only omits the SPEC-dependent term.
#
Lhyp
Rratio
Raux
RAVDESS
Fakeddit
Avg
LC1
LC2
(a)
✓
✗
✗
✗
0.7403
0.5448
0.6426
(b)
✗
✓
✗
✗
0.7399
0.5456
0.6428
(c)
✗
✗
✓
✓
0.7393
0.5452
0.6423
(d)
✓
✗
✓
✓
0.7410
0.5478
0.6444
(e)
✓
✓
✗
✗
0.7369
0.5453
0.6411
Table 7: Loss configuration ablation. LC1 and LC2 implement the self-benefiting and cross-modal exclusivity criteria; Rratio and Raux denote contribution and auxiliary regularization. Rows (c) and (d) denote noHyp and w/o C2, respectively. Rows (e)–(g) progressively add regularization to the joint criterion loss.
Figure 5: Hyperparameter sensitivity on RAVDESS: (a) distillation weight α ; (b) sensitivity coefficient β and mean fusion weight w ; (c) output ranges of ρmin and ρmax . Results are averaged over five runs; error bars in (a) and (b) indicate standard deviations.
Figure 6: Per-class heatmaps of the learned logit components. (a) Residual disentanglement (ours): GEN retains a clear class-discriminative diagonal, SPEC exhibits class-dependent corrections with near-zero mean, and the SPEC contribution −zspec/(∣zorg∣+∣zspec∣) is strongly negative at (true Sad , pred. Angry ). (b) The dual-head variant produces larger SPEC values, comparable in magnitude to GEN, with less distinct class structure in the displayed heatmaps.
Figure 7: Class-wise performance and projected logit distributions from a single RAVDESS V → A run. Left: per-class accuracy of the visual and audio baselines, the KD student, and TALD, where KD falls below the audio baseline on Sad and Calm while TALD stays above it on all classes. Right: in the displayed PCA projection, the KD student overlaps with the teacher’s source-modality cluster, while TALD lies closer to the target-modality cluster.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: PCA projections of model outputs on RAVDESS. (a) Logits from a same-modality model pair. (b) Logits from a cross-modal model pair. (c)–(f) Softmax probabilities obtained from the cross-modal logits at temperatures T=1 to 4 .
Figure 9: PCA projections of the original logits (ORG), modality-general components (GEN), and modality-specific components (SPEC) on RAVDESS, shown separately for the teacher (Visual) and reference student (Audio). In these projections, SPEC concentrates near the origin, while ORG and GEN largely overlap.
Dataset
Modality
Scale
Classes
Task
AV-MNIST
Image, Audio
70k
10
Handwritten digit recognition
RAVDESS (Speech)
Video, Audio
1,440
8
Emotion recognition
VGGSound-50k
Video, Audio
49k
141
Scene classification
CrisisMMD-V2
Image, Text
16k
8
Humanitarian classification
Fakeddit
Image, Text
50k
6
Fine-grained fake news detection
Appendix
Table 8: Overview of the five multimodal datasets utilized in our experiments, including modality composition, data scale, number of classes, and target task for each dataset.
Dataset
Unimodal Model 1
Unimodal Model 2
AV-MNIST
I: LeNet5
A: ThreeLayerCNN-2D
RAVDESS
V: ThreeLayerCNN-3D + MLP
A: ThreeLayerCNN-1D + MLP
VGGSound-50k
V: ThreeLayerCNN-3D + MLP
A: ThreeLayerCNN-1D + MLP
CrisisMMD-V2 / Fakeddit
I: MLP
T: MLP
Appendix
Table 9: Unimodal model configurations. I, V, A, and T denote image, video, audio, and text modalities, respectively.
#
Logits
Source
Fusion
RAVDESS
Fakeddit
Avg
(a)
GEN
Sref
—
0.7333
0.5436
0.6385
(b)
GEN
T
—
0.7021
0.5442
0.6232
(c)
SPEC
Sref
—
0.7222
0.5428
0.6325
(d)
SPEC
T
—
0.6962
0.5446
0.6204
(e)
GEN
Both
Mean
0.7358
0.5430
0.6394
(f)
SPEC
Both
Mean
0.7167
0.5447
0.6307
Appendix
Table 10: Complete distillation target ablation. ORG, GEN, and SPEC denote original, modality-general, and modality-specific logits. Both denotes T and Sref . Mean and Adaptive use equal weights and the learned weight w , respectively. Shuffled-S permutes w across samples; Shuffled-C restricts the permutation to each class. The best and second-best results per column are bold , with the best additionally underlined .
Dataset
Direction
Mean G
Within-class shuffle
Distillation
CMP
Original
SPEC=0
Bias=0
ΔG
∣ΔG∣
RAVDESS
V→A
V→A
4.106±3.220
2.688±2.022
1.412±1.206
0.142±0.103
1.807±0.241
A→V
4.548±3.046
2.492±1.685
2.047±1.421
0.074±0.147
1.929±0.309
A→V
A→V
4.298±3.186
2.639±1.984
1.651±1.295
0.062±0.211
1.856±0.324
V→A
4.307±2.970
2.555±1.846
1.747±1.119
0.077±0.169
1.883±0.192
Fakeddit
I→T
I→T
6.174±3.306
4.160±0.734
2.011±2.630
−0.104±0.137
10.002±1.258
Appendix
Table 11: Frozen CMP interventions, using saved original GPU values. All five numerical columns are in 10−3 nats and report mean ± sample standard deviation across five seeds after within-seed aggregation. Under Direction, Distillation denotes the teacher-to-student direction used to train the checkpoint, and CMP denotes the source-SPEC to recipient-GEN mapping evaluated within it. Each checkpoint contains both CMP directions. V/A and I/T denote visual/audio and image/text, respectively. SPEC=0 changes only the CMP input; Bias=0 changes only the CMP additive bias. No model is retrained.
Dataset
Direction
Robust sign flip rτ
Pair diagnostics ( τ=10−4 )
Distillation
CMP
τ=10−6
τ=10−5
τ=10−4
Pos.–near-zero qτ
Both non-near-zero cτ
RAVDESS
V→A
V→A
12.33±9.51
11.18±8.54
8.46±6.57
3.70±1.50
82.24±8.92
A→V
12.11±12.83
11.39±11.97
8.78±8.81
4.16±2.46
85.68±8.67
A→V
A→V
13.55±9.21
12.39±8.31
8.98±6.54
4.22±1.78
84.32±7.52
V→A
11.31±9.16
10.55±8.46
8.49±6.99
4.00±1.57
84.49±6.58
Fakeddit
I→T
I→T
29.27±6.20
29.20±6.19
28.54±6.12
1.03±0.06
98.43±0.25
Appendix
Table 12: Pairing diagnostics from full float64 recomputation of the cached logits and stored CMP parameters, using the same samples and donor permutations. Values are percentages, reported as mean ± sample standard deviation over five seeds. Robust sign-flip rates rτ are given at three tolerances. The diagnostic columns at τ=10−4 report positive–near-zero transitions ( qτ ) and the fraction of pairs with both values outside the near-zero band ( cτ ). Every rate uses all paired samples as its denominator.
#
Raux
VGGSound-50k
Fakeddit
Avg
Lcon
Lvar
M1(V)
M2(A)
M1(I)
M2(T)
(a)
✗
✗
0.4464
0.5946
0.5773
0.5158
0.5335
(b)
✓
✗
0.4493
0.5947
0.5759
0.5184
0.5346
(c)
✗
✓
0.4496
0.5940
0.5763
0.5145
0.5336
(d) Ours
✓
✓
0.4484
0.5939
0.5772
0.5194
0.5347
Appendix
Table 13: Ablation of auxiliary regularization Raux on VGGSound-50k and Fakeddit, averaged over five independent runs. Lcon denotes the supervised contrastive loss, and Lvar denotes the margin variance regularization.
Setting
Range
M1(V)
M2(A)
Avg
δ−−
[−0.3,−0.1]
0.7785
0.7076
0.7431
δ−
[−0.2,0.0]
0.7750
0.7007
0.7379
δ+
[0.0,0.2]
0.7604
0.7125
0.7365
δ++
[0.1,0.3]
0.7472
0.7070
0.7271
δ (Ours)
[−0.1,0.1]
0.7750
0.7146
0.7448
Appendix
Table 14: Ablation on the margin range [δmin,δmax] on RAVDESS, averaged over five independent runs. δ−− and δ− restrict the margin to non-positive ranges, whereas δ+ and δ++ restrict it to non-negative ranges. The proposed setting δ spans both negative and positive values.
Figure 10: Training dynamics of Lhyp and its components under different margin ranges on RAVDESS. Left column: A → V; right column: V → A. Each subplot shows the with-margin losses ( Lhyp , Lhyp,C1 , Lhyp,C2 ) alongside their margin-free counterparts ( Lhyp′ , Lhyp,C1′ , Lhyp,C2′ ).
Figure 11: Per-class heatmaps of the modality-general components and the fused distillation target on RAVDESS (V → A), with rows the true emotion and columns the logit dimension. Left and middle: the visual teacher and audio reference-student GEN logits zVgen and zAgen each keep a strong class-discriminative diagonal but differ in their off-diagonal patterns. Right: the adaptively fused target ztarget=wzVgen+(1−w)zAgen retains the diagonal while combining both modalities’ off-diagonal structure.
Figure 12: t-SNE visualization of disentangled logit components. T and RefS denote the teacher and reference student; ORG, GEN, and SPEC denote the original logits, modality-general components, and modality-specific components, respectively. All six groups are projected jointly. Red inset: T-SPEC and RefS-SPEC form separated compact clusters. Blue inset: ORG and GEN of the same modality are nearly co-located.
Method
D1
D2
D3
D4
D5
Avg
Single-Modal
KD
+0.06
−0.93
−0.78
−0.25
−0.90
−0.56
MLLD
+0.07
−0.80
+0.22
−1.30
−12.94
−2.95
FitNets
−1.00
−5.24
−2.35
−1.53
+0.16
−1.99
OFA
+0.10
−2.15
−0.99
−0.08
+0.66
−0.49
RKD
−2.53
+1.15
−0.48
−0.01
−17.53
−3.88
Appendix
Table 15: Accuracy gains (percentage points) over w/o KD, averaged over both transfer directions and five runs. Avg denotes the unweighted mean across D1–D5 and corresponds to Δ Acc in Table 1 . Blue and red denote positive and negative gains, respectively; shading intensity increases logarithmically with absolute gain.
Figure 28
Method
Stage
AV-MNIST
RAVDESS
VGGSound-50k
CrisisMMD-V2
Fakeddit
Params
FLOPs
Mem
Time
Params
FLOPs
Mem
Time
Params
FLOPs
Mem
Time
Params
FLOPs
Mem
Time
Params
FLOPs
Mem
Time
S0
0.2026
1500.89
3331.02
645.71
3.3301
134.39
2594.12
373.54
2.4632
10507.31
1578.35
1353.95
1.9717
8.66
75.38
440.56
1.9707
26.95
75.36
451.50
S1
0.2026
2007.40
3333.50
684.78
3.3301
189.35
2596.61
632.18
2.4632
15700.40
1583.22
2096.39
1.9717
12.46
84.19
457.07
1.9707
38.76
83.05
482.74
KD
Total
0.4053
3508.28
3333.50
1330.49
6.6603
323.74
2596.61
1005.73
4.9263
26207.71
1583.22
3450.34
3.9434
21.12
84.19
897.63
3.9414
65.71
83.05
934.23
S0
0.2026
1500.89
3331.02
645.71
3.3301
134.39
2594.12
373.54
2.4632
10507.31
1578.35
1353.95
1.9717
8.66
75.38
440.56
1.9707
26.95
75.36
451.50
S1
0.2026
2008.35
3333.50
713.96
3.3301
189.35
2596.61
623.16
2.4632
15711.32
1583.22
2005.57
1.9717
12.64
84.62
465.81
1.9707
39.19
84.51
107.18
Appendix
Table 16: Complete stage-wise and end-to-end cost records on five benchmarks, with stage entries averaged over both transfer directions. Params, FLOPs, Mem, and Time are reported in M, T, MiB, and seconds, respectively. Stage labels follow Table 2 . Within each dataset, Total sums Params, FLOPs, and Time and takes the maximum of the displayed stage Mem values.
Cross-modal knowledge distillation transfers useful representations or predictions between modalities, often enabling cheaper single-modality inference.
Cross-modal knowledge distillation (CMKD) studies how a (large) teacher model trained on one type of data (e.g., images) can guide a (smaller) student model building on another type of data (e.g., text/audio). Existing CMKD methods often require paired multi-modal data with aligned semantics, but obtaining such paired data are often costly and impractical. To mitigate this limitation, we develop a new CMKD framework for the more challenging setting where paired data are unavailable. In particular, we establish a cross-modal distributional relationship between teacher and student models, which reveals two fundamental quantities governing effective distillation: feature alignment and label alignment. These quantities characterize semantic discrepancy between modalities at the levels of representation and prediction distributions, respectively. Motivated by this insight, we propose a principled framework, with theoretical guarantees, that enables effective cross-modal knowledge distillation by aligning distributions rather than individual samples. Extensive experiments across a wide range of multimodal benchmarks show that our framework is highly effective in both unpaired and paired data settings, improving significantly over prior work.
Trong Khiem Tran, Anh Duc Chu, Quang Hung Pham +2
School of Electrical Engineering and Computer Science, Washington State University, Pullman, US · School of Information and Communications Technology, Hanoi University of Science and Technology, Hanoi, Vietnam
Cross-modal knowledge distillation transfers knowledge from a teacher modality to a student modality. Existing feature-level alignment methods typically assume that teacher and student features reside in structurally alignable representation spaces. However, this assumption does not hold when cross-modal features are structurally heterogeneous and lack clear unit-level correspondence, such as 2D spatial visual grids and 1D temporal audio sequences, thereby limiting the applicability of feature-level alignment. To address this challenge, we propose a cross-modal distillation framework that enables effective knowledge transfer across structurally heterogeneous feature spaces via a vector-quantized codebook. Specifically, teacher features are abstracted into a set of vector-form codes regardless of their original feature structure, and the selected codes serve as concept-level anchors for student learning. Code selection is guided by both task relevance and student compatibility, allowing the student to receive transferable teacher knowledge without requiring direct unit-level feature alignment. Experimental results across diverse cross-modal distillation scenarios demonstrate the effectiveness of the proposed framework on classification and semantic segmentation tasks.
Dae Ung Jo, Jongin Lim, YoungJoon Yoo +1
Kyungpook National University · AX/PI Center, Samsung Electronics · Chung-Ang University, SNUAILAB +1
Knowledge distillation (KD) methods are pivotal in compressing large pre-trained language models into smaller models, ensuring computational efficiency without significantly dropping performance. Traditional KD techniques assume homogeneity in modalities between the teacher (source) and the student (target) models. On the other hand, existing multimodal knowledge distillation methods require modality-specific pre-training of the teacher model, which is computationally infeasible in most cases. In this paper, we introduce ARMADA, an efficient cross-modal knowledge distillation framework designed to transfer knowledge from large vision-language models, including black-box models, to language-only models. Unlike existing KD techniques that rely on the internal structures of multimodal teachers or require computationally expensive pre-training, ARMADA leverages novel alignment techniques to distil knowledge without altering the teacher model, ensuring efficiency and scalability. We empirically validate ARMADA on twelve natural language understanding, eight complex generative reasoning and five instruction-tuning tasks, demonstrating consistent performance improvements in large models such as DeBERTa-v2-1.4B, OPT-1.3B, LLaMA-{3B, 7B, 8B}. ARMADA achieves up to 3.4% improvement on language understanding tasks and 2.6% boost in generative reasoning, all without requiring expensive multimodal pre-training or fine-tuning of the teacher model. Our findings challenge conventional knowledge distillation paradigms by demonstrating that even vision-language models, despite lacking direct textual understanding, can significantly enhance language models when distilled appropriately.
Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar +1
Indian Institute of Technology Delhi, India · Indraprastha Institute of Information Technology Delhi, India