Cross-Modal Knowledge Distillation

Cross-modal knowledge distillation studies how a model using one modality, such as images, audio, text, or medical signals, can teach a model that uses another. A common approach trains a rich multimodal teacher and transfers its knowledge through feature, logit, relational, or contrastive losses, so the student can work with fewer or cheaper inputs at deployment. Researchers also use attention and token-level guidance, on-policy distillation for reasoning, and distributional alignment when paired examples are unavailable; evaluations span recognition, retrieval, detection, segmentation, and multimodal reasoning. A central challenge is deciding what knowledge transfers reliably across different representations: noisy or mismatched teachers can cause negative transfer, while overly strong alignment may erase useful modality-specific information.