cs.CVNov 9, 2026

TMT: Adapting CLIP to Infrared Imagery via Text-Derived Modality Knowledge Transfer

Authors: Ruofan Wang, Puzuo Wang, Xin Chen, Zibo Yu, Jungang Yang, Yi Hou

Organizations: College of Electronic Science and Technology, National University of Defense Technology, Changsha, Hunan, China

Abstract

Vision-language models such as CLIP achieve strong performance across computer vision tasks, yet their image-text alignment is less reliable for infrared (IR) imagery because large-scale pre-training is dominated by visible-spectrum data. Due to the high cost of the infrared-text labeling, we investigate adapting CLIP to IR through only paired infrared-visible images, which requires considering modality-specific knowledge while preserving scene content and CLIP's image-text alignment. To achieve this, we propose TMT (Text-Derived Modality Knowledge Transfer), which adapts CLIP's image encoder through a text-defined infrared-to-visible direction and supervision from paired IR-visible images. First, we found that modality discrepancy can be described by corresponding text embeddings, thus using the direction of text embedding differences to guide the knowledge transfer between visible and infrared modalities without erasing infrared-specific information. To enhance the representation of semantics and details of infrared images, we introduce Generative-Perceptual Co-Supervision that combines semantic alignment from CLIP visible image encoder with diffusion noise-prediction feedbacks. On four source-disjoint IR classification benchmarks, TMT achieves the highest mean zero-shot accuracy among the compared methods (48.92%). It also achieves the best IR-text retrieval results among the compared methods without infrared-text supervision during adaptation and the highest open-vocabulary segmentation mIoU on both CART and SemanticRT among the CLIP-like methods.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. MonoIR-RS: Infrared Remote Sensing Vision-Language Learning with CLIP and VLM Adaptation

    Jul 7, 2026Jiaju Han, Ma Yaqi, Yahui Chai +8Remote Sensing Image UnderstandingVLM Adaptation

  2. T-CLIP: Enabling Thermal Perception for Contrastive Language-Image Pretraining

    May 30, 2026Tayeba Qazi, Ayush Maheshwari, Prerana Mukherjee +1Thermal ImagingImage Captioning