TMT: Adapting CLIP to Infrared Imagery via Text-Derived Modality Knowledge Transfer
Organizations: College of Electronic Science and Technology, National University of Defense Technology, Changsha, Hunan, China
Abstract
Vision-language models such as CLIP achieve strong performance across computer vision tasks, yet their image-text alignment is less reliable for infrared (IR) imagery because large-scale pre-training is dominated by visible-spectrum data. Due to the high cost of the infrared-text labeling, we investigate adapting CLIP to IR through only paired infrared-visible images, which requires considering modality-specific knowledge while preserving scene content and CLIP's image-text alignment. To achieve this, we propose TMT (Text-Derived Modality Knowledge Transfer), which adapts CLIP's image encoder through a text-defined infrared-to-visible direction and supervision from paired IR-visible images. First, we found that modality discrepancy can be described by corresponding text embeddings, thus using the direction of text embedding differences to guide the knowledge transfer between visible and infrared modalities without erasing infrared-specific information. To enhance the representation of semantics and details of infrared images, we introduce Generative-Perceptual Co-Supervision that combines semantic alignment from CLIP visible image encoder with diffusion noise-prediction feedbacks. On four source-disjoint IR classification benchmarks, TMT achieves the highest mean zero-shot accuracy among the compared methods (48.92%). It also achieves the best IR-text retrieval results among the compared methods without infrared-text supervision during adaptation and the highest open-vocabulary segmentation mIoU on both CART and SemanticRT among the CLIP-like methods.
Figures & tables
| Property | Visible | Infrared |
| Radiation source | Reflected scene illumination | Emitted thermal radiation (temperature, emissivity) |
| Contrast and color | Albedo- and shading-based; rich color | Thermal contrast; weak or absent color |
| Texture and noise | Strong high-frequency detail; near-uniform noise | Smoother texture; fixed-pattern noise |
| Shared structure | Object shape and spatial arrangement (preserved) | |
| Method | SMOD | JU-VNT-T | MFNet | PST900 | Avg. |
| LanguageBind † | 56.30 | 47.90 | 38.59 | 36.98 | 44.94 |
| un 2 CLIP | 57.87 | 32.63 | 35.66 | 34.19 | 40.09 |
| DIVA | 60.07 | 43.73 | 34.08 | 32.27 | 42.54 |
| ImageBind (official) | 25.73 | 4.17 | 6.07 | 27.04 | 15.75 |
| ImageBind (trained) | 63.00 | 36.28 | 36.28 | 38.50 | 43.52 |
| CLIP-FT † | 62.90 | 51.56 | 31.89 | 38.24 | 46.15 |
| Method | M3FD | FLIRv1 | FLIRv2 | LLVIP | All-8 avg. |
| LanguageBind † | 67.10 | 77.27 | 45.20 | 83.23 | 56.57 |
| LanguageBind ‡ | – | 82.90 | 48.00 | 87.20 | – |
| un 2 CLIP | 54.87 | 80.20 | 35.70 | 79.83 | 51.37 |
| DIVA | 61.77 | 71.77 | 30.33 | 85.20 | 52.40 |
| ImageBind (official) | 5.74 | 6.27 | 1.10 | 72.86 | 18.62 |
| ImageBind (trained) | 57.54 | 73.94 | 31.22 | 72.09 | 51.11 |
| Method | IR Text | Text IR | ||||||
| R@1 | R@5 | R@10 | R@20 | R@1 | R@5 | R@10 | R@20 | |
| Without infrared–text supervision | ||||||||
| OpenAI CLIP [ 21 ] | 5.00 | 11.72 | 16.72 | 25.47 | 10.47 | 20.31 | 30.31 | 43.44 |
| un 2 CLIP [ 15 ] | 3.59 | 10.00 | 15.00 | 22.03 | 10.31 | 21.41 | 29.22 | 39.38 |
| DIVA [ 29 ] | 4.84 | 11.41 | 16.09 | 25.00 | 9.06 | 19.22 | 29.84 | 42.81 |
| ImageBind (official) [ 6 ] | 1.09 | 4.06 | 6.25 | 9.53 | 1.41 | 4.37 | 7.66 | 11.25 |
| Method | CART | SemanticRT |
| OpenAI CLIP [ 21 ] | 16.06 | 15.16 |
| LanguageBind † [ 35 ] | 18.02 | 16.87 |
| un 2 CLIP [ 15 ] | 9.22 | 17.29 |
| DIVA [ 29 ] | 16.40 | 15.49 |
| ImageBind (official) [ 6 ] | 1.59 | 5.63 |
| ImageBind (trained) | 16.94 | 15.80 |
| Shift | R@1 | R@20 | Primary cls. | All cls. | Seg. mIoU | ||
| 7.73 | 34.45 | 42.48 | 52.33 | 15.61 | |||
| ✓ | ✓ | 8.83 | 37.81 | 48.23 | 59.78 | 17.69 | |
| ✓ | ✓ | 7.58 | 34.45 | 44.55 | 54.98 | 16.73 | |
| ✓ | ✓ | 7.88 | 34.14 | 46.18 | 55.79 | 16.27 | |
| ✓ | ✓ | ✓ | 8.83 | 38.28 | 48.92 | 60.40 | 17.81 |
| Variant | R@1 | R@20 | Primary cls. | All cls. | Seg. mIoU |
| ImageBind (trained) | 5.37 | 28.47 | 43.52 | 51.11 | 16.37 |
| Image-mean transfer | 9.14 | 35.31 | 46.40 | 56.38 | 16.24 |
| Random direction (3 seeds) | 8.90 | 36.85 | 46.06 | 55.53 | 16.27 |
| Text pair (modality names) | 8.52 | 37.26 | 46.57 | 56.51 | 16.12 |
| Reversed direction | 7.26 | 33.13 | 40.22 | 51.12 | 15.28 |
| TMT | 8.83 | 38.28 | 48.92 | 60.40 | 17.81 |
| Primary cls. | Retrieval R@20 | Seg. mIoU | |
| 0.0 | 46.18 | 34.14 | 16.27 |
| 0.1 | 48.49 | 37.97 | 17.96 |
| 0.2 | 48.15 | 36.80 | 17.92 |
| 0.3 | 48.92 | 38.28 | 17.81 |
| 0.4 | 47.61 | 37.42 | 17.76 |
| 0.5 | 45.41 | 34.45 | 17.23 |
| Dataset | Margin ratio | Fisher ratio | ||||
| CLIP | TMT | CLIP | TMT | |||
| M3FD | 0.102 | 0.116 | +13.52% | 0.132 | 0.175 | +32.86% |
| SMOD | 0.166 | 0.195 | +17.60% | 0.124 | 0.168 | +35.95% |
| FLIRv1 | 0.157 | 0.175 | +11.11% | 0.117 | 0.152 | +30.08% |
| FLIRv2 | 0.106 | 0.112 | +5.99% | 0.149 | 0.180 | +21.08% |
| LLVIP | 0.526 | 0.775 | +47.49% | 0.226 | 0.349 | +54.27% |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Probe target ( ) | OpenAI CLIP | diffusion branch | |
| Texture level | 0.654 | 0.677 | |
| Gain (signed brightness) | 0.263 | 0.067 |
VLM Adaptation
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
See all 564 papers