Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Features

Dae Ung Jo; Jongin Lim; YoungJoon Yoo; Daeho Um

cs.CVSep 29, 2026

Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Features

Authors: Dae Ung Jo, Jongin Lim, YoungJoon Yoo, Daeho Um

Organizations: Kyungpook National University · AX/PI Center, Samsung Electronics · Chung-Ang University, SNUAILAB · University of Seoul

Abstract

Cross-modal knowledge distillation transfers knowledge from a teacher modality to a student modality. Existing feature-level alignment methods typically assume that teacher and student features reside in structurally alignable representation spaces. However, this assumption does not hold when cross-modal features are structurally heterogeneous and lack clear unit-level correspondence, such as 2D spatial visual grids and 1D temporal audio sequences, thereby limiting the applicability of feature-level alignment. To address this challenge, we propose a cross-modal distillation framework that enables effective knowledge transfer across structurally heterogeneous feature spaces via a vector-quantized codebook. Specifically, teacher features are abstracted into a set of vector-form codes regardless of their original feature structure, and the selected codes serve as concept-level anchors for student learning. Code selection is guided by both task relevance and student compatibility, allowing the student to receive transferable teacher knowledge without requiring direct unit-level feature alignment. Experimental results across diverse cross-modal distillation scenarios demonstrate the effectiveness of the proposed framework on classification and semantic segmentation tasks.

Figures & tables

Figure 1 : Overview of the proposed framework. We learn a teacher-side vector-quantized codebook (Sec. 3.3 ), select task-relevant and student-compatible codes (Sec. 3.4 ), and use the selected codes for student-side distillation (Sec. 3.5 ). The framework enables feature-level CMKD without requiring explicit unit-level correspondence between structurally different teacher and student representations.

General distillation baselines
Method	RAVDESS		VGG-Sound		CrisisMMD		MM-IMDB
Method	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A	T $\rightarrow$ I	I $\rightarrow$ T	T $\rightarrow$ I	I $\rightarrow$ T
w/o KD	0.9208	0.6826	0.5031	0.6058	0.5403	0.5500	0.6645	0.7294
KLD	0.9118	0.6972	0.5153	0.6176	0.5377	0.5578	0.6732	0.7310
MLLD	0.9292	0.6889	0.5135	0.6122	0.5357	0.5562	0.6699	0.7320
FitNets	0.9062	0.7146	0.4916	0.5962	0.5281	0.5507	0.6759	0.7296

Table 1: Classification results. A, V, T, and I denote audio, video, text, and image modalities, respectively, and each arrow indicates the teacher-to-student distillation direction. We report accuracy averaged over five independent runs, with the best result in each column shown in bold.

Method	Depth $\rightarrow$ RGB			RGB $\rightarrow$ Depth
Method	OA $\uparrow$	CA $\uparrow$	mIoU $\uparrow$	OA $\uparrow$	CA $\uparrow$	mIoU $\uparrow$
w/o KD	0.5582	0.2748	0.1850	0.5269	0.2100	0.1366
MLLD	0.5599	0.2792	0.1911	0.5250	0.2118	0.1399
CRD	0.5592	0.2677	0.1806	0.5261	0.2111	0.1385
OFA	0.5607	0.2881	0.1982	0.5242	0.2123	0.1409
MGDFR	0.5601	0.2737	0.1878	0.5213	0.2162	0.1442

Table 2 : Semantic segmentation results on NYU-Depth. Left : quantitative comparison for two distillation scenarios, where the first modality denotes the teacher and the second denotes the student. Right : class-wise recall comparison of RGB-only, depth-only, and our RGB

\rightarrow

Depth student models.

Figure 2 : Codebook analysis on RAVDESS. (a) Example frames from the dataset. (b,c) Spatial usage maps of top-ranked and bottom-ranked codes according to the proposed importance score.

Code	Pred	RAVDESS		VGG-Sound		CrisisMMD		MM-IMDB
Code	Match	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A	T $\rightarrow$ I	I $\rightarrow$ T	T $\rightarrow$ I	I $\rightarrow$ T
-	-	0.9208	0.6826	0.5031	0.6058	0.5403	0.5500	0.6645	0.7294
✓	-	0.9202	0.7333	0.5067	0.6098	0.5387	0.5534	0.6716	0.7363
-	✓	0.9285	0.7479	0.5294	0.6204	0.5367	0.5529	0.6770	0.7421
✓	✓	0.9431	0.7486	0.5299	0.6229	0.5433	0.5592	0.6768	0.7434

Table 3: Ablation study on Stage-3 loss components. “Code” denotes the student-side codebook-branch objective. “Pred-Match” denotes the codebook-induced prediction matching term.

Pred-Match Target	RAVDESS		VGG-Sound		CrisisMMD		MM-IMDB
Pred-Match Target	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A	T $\rightarrow$ I	I $\rightarrow$ T	T $\rightarrow$ I	I $\rightarrow$ T
Original branch $(\mathbf{p}^{(t)})$	0.8903	0.6201	0.5173	0.6171	0.5425	0.5539	0.6654	0.7351
Codebook branch $(\mathbf{q}^{(t)})$	0.9431	0.7486	0.5299	0.6229	0.5433	0.5592	0.6768	0.7434

Table 4: Ablation on the prediction matching target in Stage-3 Loss.

Selection Criteria	RAVDESS		VGG-Sound		CrisisMMD		MM-IMDB
Selection Criteria	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A	T $\rightarrow$ I	I $\rightarrow$ T	T $\rightarrow$ I	I $\rightarrow$ T
Random	0.9292	0.7396	0.5252	0.6211	0.5353	0.5536	0.6718	0.7353
Task relevance only	0.9396	0.7361	0.5276	0.6233	0.5435	0.5558	0.6727	0.7351
Student compatibility only	0.9375	0.7417	0.5275	0.6222	0.5437	0.5565	0.6722	0.7346
Combination (Ours)	0.9431	0.7486	0.5299	0.6229	0.5433	0.5592	0.6768	0.7434

Table 5: Ablation study on code selection criteria.

Figure 3 : Selected code ratio ablation. We vary the percentage of selected teacher codes used for student-side quantization from 25% to 100%. The dashed line denotes the w/o KD baseline.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Epochs	Learning rate	Weight decay	Batch size	#Code	Code Dim.	$\beta_{\mathrm{com}}$	$\lambda_{\mathrm{dist}}$
RAVDESS	100	$1\times 10^{-3}$	$1\times 10^{-4}$	64	512	128	0.25	$1\times 10^{-2}$
VGG-Sound	100	$1\times 10^{-3}$	$1\times 10^{-4}$	128	512	128	0.25	$1\times 10^{-2}$
CrisisMMD	100	$1\times 10^{-3}$	$1\times 10^{-4}$	64	256	128	0.25	$1\times 10^{-2}$
MM-IMDB	100	$1\times 10^{-3}$	$1\times 10^{-4}$	64	512	64	0.25	$1\times 10^{-2}$
NYU-Depth V2	100	$1\times 10^{-4}$	$1\times 10^{-4}$	6	1024	256	0.25	$1\times 10^{-2}$

Appendix

Table 6 : Implementation details for each dataset.

Class	# Samples
Drama	2,681
Comedy	1,337
Documentary	918
Horror	417
Western	173
Thriller	143

Appendix

Table 7: Number of samples per class in the single-label MM-IMDB dataset.

Method	RAVDESS		VGG-Sound
Method	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A
w/o KD	0.9208 $\pm$ 0.0164	0.6826 $\pm$ 0.0189	0.5031 $\pm$ 0.0035	0.6058 $\pm$ 0.0086
KLD	0.9118 $\pm$ 0.0156	0.6972 $\pm$ 0.0674	0.5153 $\pm$ 0.0067	0.6176 $\pm$ 0.0090
MLLD	0.9292 $\pm$ 0.0140	0.6889 $\pm$ 0.0461	0.5135 $\pm$ 0.0056	0.6122 $\pm$ 0.0047
FitNets	0.9062 $\pm$ 0.0188	0.7146 $\pm$ 0.0385	0.4916 $\pm$ 0.0105	0.5962 $\pm$ 0.0128
RKD	0.9236 $\pm$ 0.0132	0.7014 $\pm$ 0.0269	0.5066 $\pm$ 0.0068	0.6085 $\pm$ 0.0018

Appendix

Table 8: Classification results. A, V, T, and I denote audio, video, text, and image modalities, respectively, and each arrow indicates the teacher-to-student distillation direction. We report mean accuracy

\pm

standard deviation over five independent runs, with the best mean result in each column shown in bold.

Method	Depth $\rightarrow$ RGB			RGB $\rightarrow$ Depth
Method	OA $\uparrow$	CA $\uparrow$	mIoU $\uparrow$	OA $\uparrow$	CA $\uparrow$	mIoU $\uparrow$
w/o KD	0.5582 $\pm$ 0.0112	0.2748 $\pm$ 0.0218	0.1850 $\pm$ 0.0154	0.5269 $\pm$ 0.0070	0.2100 $\pm$ 0.0155	0.1366 $\pm$ 0.0110
MLLD	0.5599 $\pm$ 0.0122	0.2792 $\pm$ 0.0226	0.1911 $\pm$ 0.0169	0.5250 $\pm$ 0.0063	0.2118 $\pm$ 0.0150	0.1399 $\pm$ 0.0152
CRD	0.5592 $\pm$ 0.0101	0.2677 $\pm$ 0.0158	0.1806 $\pm$ 0.0108	0.5261 $\pm$ 0.0061	0.2111 $\pm$ 0.0125	0.1385 $\pm$ 0.0107
OFA	0.5607 $\pm$ 0.0119	0.2881 $\pm$ 0.0146	0.1982 $\pm$ 0.0107	0.5242 $\pm$ 0.0085	0.2123 $\pm$ 0.0125	0.1409 $\pm$ 0.0123
MGDFR	0.5601 $\pm$ 0.0123	0.2737 $\pm$ 0.0196	0.1878 $\pm$ 0.0165	0.5213 $\pm$ 0.0099	0.2162 $\pm$ 0.0097	0.1442 $\pm$ 0.0083

Appendix

Table 9: Semantic segmentation results on NYU-Depth. Quantitative comparison for two distillation scenarios, where the first modality denotes the teacher and the second denotes the student. Results are reported as mean

\pm

standard deviation over five independent runs.

Code	Pred-Match	RAVDESS		VGG-Sound
Code	Pred-Match	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A
-	-	0.9208 $\pm$ 0.0164	0.6826 $\pm$ 0.0189	0.5031 $\pm$ 0.0035	0.6058 $\pm$ 0.0086
✓	-	0.9202 $\pm$ 0.0260	0.7333 $\pm$ 0.0397	0.5067 $\pm$ 0.0029	0.6098 $\pm$ 0.0033
-	✓	0.9285 $\pm$ 0.0234	0.7479 $\pm$ 0.0175	0.5294 $\pm$ 0.0013	0.6204 $\pm$ 0.0056
✓	✓	0.9431 $\pm$ 0.0097	0.7486 $\pm$ 0.0231	0.5299 $\pm$ 0.0017	0.6229 $\pm$ 0.0059

Appendix

Table 10: Ablation study on Stage-3 loss components with standard deviations. “Code” denotes the student-side codebook-branch objective. “Pred-Match” denotes the codebook-induced prediction matching term. Results are reported as mean

\pm

standard deviation over five independent runs.

Pred-Match Target	RAVDESS		VGG-Sound
Pred-Match Target	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A
Original branch $(\mathbf{p}^{(t)})$	0.8903 $\pm$ 0.0140	0.6201 $\pm$ 0.0788	0.5173 $\pm$ 0.0024	0.6171 $\pm$ 0.0079
Codebook branch $(\mathbf{q}^{(t)})$	0.9431 $\pm$ 0.0097	0.7486 $\pm$ 0.0231	0.5299 $\pm$ 0.0017	0.6229 $\pm$ 0.0059

Appendix

Table 11: Ablation on the prediction matching target in Stage-3 loss with standard deviations. Results are reported as mean

\pm

standard deviation over five independent runs.

Selection Criteria	RAVDESS		VGG-Sound
Selection Criteria	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A
Random	0.9292 $\pm$ 0.0134	0.7396 $\pm$ 0.0214	0.5252 $\pm$ 0.0049	0.6211 $\pm$ 0.0051
Task relevance only	0.9396 $\pm$ 0.0063	0.7361 $\pm$ 0.0134	0.5276 $\pm$ 0.0055	0.6233 $\pm$ 0.0063
Student compatibility only	0.9375 $\pm$ 0.0078	0.7417 $\pm$ 0.0124	0.5275 $\pm$ 0.0077	0.6222 $\pm$ 0.0052
Combination (Ours)	0.9431 $\pm$ 0.0097	0.7486 $\pm$ 0.0231	0.5299 $\pm$ 0.0017	0.6229 $\pm$ 0.0059

Appendix

Table 12: Ablation study on code selection criteria with standard deviations. Results are reported as mean

\pm

standard deviation over five independent runs.

Method	RAVDESS		VGG-Sound
Method	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A
Runner-up	0.9271 $\pm$ 0.0080	0.7351 $\pm$ 0.0144	0.5219 $\pm$ 0.0046	0.6126 $\pm$ 0.0043
Ours	0.9420 $\pm$ 0.0071	0.7556 $\pm$ 0.0120	0.5273 $\pm$ 0.0041	0.6158 $\pm$ 0.0029
p-value	0.001	0.007	0.028	0.007

Appendix

Table 13: Controlled significance tests against the runner-up methods for the classification results. The data split and the teacher checkpoint are fixed, and results are reported as mean

\pm

standard deviation over ten runs. p-values are computed using a two-sided paired t-test. The runner-up is STXD for MM-IMDB T

\rightarrow

I and MGDFR for all other settings.

Method	RAVDESS		VGG-Sound		CrisisMMD		MM-IMDB
Method	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A	T $\rightarrow$ I	I $\rightarrow$ T	T $\rightarrow$ I	I $\rightarrow$ T
KLD	7.4	6.4	4.6	3.8	7.4	5.4	4.0	7.0
MLLD	4.8	7.2	4.4	5.8	8.6	5.0	5.8	6.4
FitNets	7.8	5.8	10.8	10.8	7.4	7.8	4.4	7.4
RKD	5.4	7.0	8.0	8.2	4.4	4.6	7.6	6.6
CRD	4.6	7.4	7.8	7.8	6.2	4.8	5.8	7.0

Appendix

Table 14: Average rank of classification methods across five runs with varying data splits and teacher checkpoints. Lower values indicate better performance, and the best average rank in each column is shown in bold.

Code	Pred-Match	RAVDESS		VGG-Sound
Code	Pred-Match	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A
-	✓	0.9274 $\pm$ 0.0110	0.7469 $\pm$ 0.0060	0.5246 $\pm$ 0.0031	0.6129 $\pm$ 0.0031
✓	✓	0.9420 $\pm$ 0.0071	0.7556 $\pm$ 0.0120	0.5273 $\pm$ 0.0041	0.6158 $\pm$ 0.0029
p-value		0.0031	0.0885	0.1215	0.0504

Appendix

Table 15: Controlled significance tests for the contribution of the “Code” term. The data split and the teacher checkpoint are fixed, and results are reported as mean

\pm

standard deviation over ten runs. p-values are computed using a two-sided paired t-test.

Selection Criteria	RAVDESS		VGG-Sound
Selection Criteria	A $\rightarrow$ V	V $\rightarrow$ A	A $\rightarrow$ V	V $\rightarrow$ A
Random	0.9316 $\pm$ 0.0100	0.7423 $\pm$ 0.0073	0.5242 $\pm$ 0.0029	0.6125 $\pm$ 0.0023
Combination (Ours)	0.9420 $\pm$ 0.0071	0.7556 $\pm$ 0.0120	0.5273 $\pm$ 0.0041	0.6158 $\pm$ 0.0029
p-value	0.0333	0.0205	0.0309	0.0287

Appendix

Table 16: Controlled significance tests for the code selection criterion. The data split and the teacher checkpoint are fixed, and results are reported as mean

\pm

standard deviation over ten runs. p-values are computed using a two-sided paired t-test.

Method	V $\rightarrow$ A			A $\rightarrow$ V
Method	Time	Mem	TFLOPs	Time	Mem	TFLOPs
w/o KD	82.69	25.40	–	168.44	77.25	–
KLD	172.95	76.29	0.62	176.04	80.34	0.62
MSE	171.78	76.35	0.62	165.68	80.40	0.62
CRD	325.45	88.22	0.63	324.03	94.71	0.63
FitNets	338.17	78.26	1.24	339.19	83.05	1.24

Appendix

Table 17: Training computational costs on VGG-Sound. Time and Mem denote the end-to-end wall-clock training time (s) and peak GPU memory usage (MB), respectively. TFLOPs are measured only for the method-specific scoring and alignment operations, excluding the teacher and student forward passes that are largely shared across methods.

\hookrightarrow

Stage-2 denotes the cost of the leave-one-code-out scoring in our method.

$B$	V $\rightarrow$ A				A $\rightarrow$ V
$B$	Stage-1	Stage-2	Stage-3	Acc.	Stage-1	Stage-2	Stage-3	Acc.
128	231.53	10.48	182.34	0.6202	224.55	10.56	179.26	0.5270
256	228.37	18.54	190.97	0.6213	225.65	18.65	193.02	0.5272
512	226.81	36.24	195.80	0.6229	225.70	34.87	191.72	0.5299
1024	226.90	79.93	191.29	0.6201	224.92	68.62	190.45	0.5283
2048	228.29	184.83	202.42	0.6205	221.61	131.78	199.46	0.5262

Appendix

Table 18: Stage-wise training time (s) and accuracy under different codebook sizes

B

on VGG-Sound. Accuracy is averaged over five independent runs. Our default setting is

B=512

Top-5 increased classes			Top-5 decreased classes
Class	Freq.	$\Delta$ Recall	Class	Freq.	$\Delta$ Recall
other_furniture	2.7%	+0.1807	person	0.3%	$-$ 0.1298
mirror	1.2%	+0.1519	counter	1.7%	$-$ 0.0478
television	0.7%	+0.1324	bookshelf	2.1%	$-$ 0.0398
desk	0.8%	+0.1315	clothes	0.8%	$-$ 0.0294
curtain	1.8%	+0.1311	wall	25.6%	$-$ 0.0274

Appendix

Table 19: Class-wise recall analysis on NYU-Depth V2 in the RGB

\rightarrow

Depth setting. Freq. denotes the percentage of pixels belonging to each class, and

\Delta

Recall denotes the recall difference between our method and the student trained without distillation.

Assignment	A $\rightarrow$ V	V $\rightarrow$ A
Hard	0.5299 $\pm$ 0.0017	0.6229 $\pm$ 0.0059
Soft	0.5254 $\pm$ 0.0024	0.6208 $\pm$ 0.0082

Appendix

Table 20: Comparison of hard and soft code assignment in Stage-3 on VGG-Sound. Results are reported as mean

\pm

standard deviation over five independent runs.

Panel of Figure 4 — Figure 4 : Qualitative analysis of learned codes. For each selected image-teacher code, the pair shows its associated image region and corresponding high-weight tokens from the distilled text student.

Explore similar work

Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm
Jun 9, 2026Trong Khiem Tran, Anh Duc Chu, Quang Hung Pham +2Cross-Modal Distillation Cross-Modal
School of Electrical Engineering and Computer Science, Washington State University, Pullman, US · School of Information and Communications Technology, Hanoi University of Science and Technology, Hanoi, Vietnam
Shared Semantic Codebook Distillation for Unpaired Cross-Modal Medical Classification
Jul 29, 2026Dillan Imans, Phuoc-Nguyen Bui, Duc-Tai Le +1Cross-Modal Distillation Cross-Modality
Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, South Korea · Convergence Research Institute, Sungkyunkwan University, Suwon, South Korea · Department of AI Systems Engineering, Sungkyunkwan University, Suwon, South Korea +1
UnDA: Unpaired Domain Alignment for Cross-Modal Knowledge Transfer in Medical Imaging
Jul 23, 2026Rafsan Jany, Shadab Tanjeed Ahmad, Ahsan Bulbul +3Cross-Modal Distillation Cross-Modal
Korea Institute of Oriental Medicine, Daejeon, South Korea · Islamic University of Technology, Gazipur, Bangladesh

Jun 9, 2026, cs.AIJ/K move · Enter open · S save

Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm

Trong Khiem Tran, Anh Duc Chu, Quang Hung Pham, Phi Le Nguyen+1

School of Electrical Engineering and Computer Science, Washington State University, Pullman, US · School of Information and Communications Technology, Hanoi University of Science and Technology, Hanoi, Vietnam

Cross-Modal Distillation Cross-Modal Value Alignment Theory

Open paper AskPDF arXiv

Cross-modal knowledge distillation (CMKD) studies how a (large) teacher model trained on one type of data (e.g., images) can guide a (smaller) student model building on another type of data (e.g., text/audio). Existing CMKD methods often require paired multi-modal data with aligned semantics, but obtaining such paired data are often costly and impractical. To mitigate this limitation, we develop a new CMKD framework for the more challenging setting where paired data are unavailable. In particular, we establish a cross-modal distributional relationship between teacher and student models, which reveals two fundamental quantities governing effective distillation: feature alignment and label alignment. These quantities characterize semantic discrepancy between modalities at the levels of representation and prediction distributions, respectively. Motivated by this insight, we propose a principled framework, with theoretical guarantees, that enables effective cross-modal knowledge distillation by aligning distributions rather than individual samples. Extensive experiments across a wide range of multimodal benchmarks show that our framework is highly effective in both unpaired and paired data settings, improving significantly over prior work.

Abstract

Explore similar work

Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm

Shared Semantic Codebook Distillation for Unpaired Cross-Modal Medical Classification

UnDA: Unpaired Domain Alignment for Cross-Modal Knowledge Transfer in Medical Imaging