Vision-language models such as CLIP achieve strong performance across computer vision tasks, yet their image-text alignment is less reliable for infrared (IR) imagery because large-scale pre-training is dominated by visible-spectrum data. Due to the high cost of the infrared-text labeling, we investigate adapting CLIP to IR through only paired infrared-visible images, which requires considering modality-specific knowledge while preserving scene content and CLIP's image-text alignment. To achieve this, we propose TMT (Text-Derived Modality Knowledge Transfer), which adapts CLIP's image encoder through a text-defined infrared-to-visible direction and supervision from paired IR-visible images. First, we found that modality discrepancy can be described by corresponding text embeddings, thus using the direction of text embedding differences to guide the knowledge transfer between visible and infrared modalities without erasing infrared-specific information. To enhance the representation of semantics and details of infrared images, we introduce Generative-Perceptual Co-Supervision that combines semantic alignment from CLIP visible image encoder with diffusion noise-prediction feedbacks. On four source-disjoint IR classification benchmarks, TMT achieves the highest mean zero-shot accuracy among the compared methods (48.92%). It also achieves the best IR-text retrieval results among the compared methods without infrared-text supervision during adaptation and the highest open-vocabulary segmentation mIoU on both CART and SemanticRT among the CLIP-like methods.
Figures & tables
Figure 1: Illustration of the core idea. In order to guide the knowledge transfer between visible (rich) and infrared (lack) modalities and increase the performance of infrared–text alignment, a text-derived infrared-to-visible direction moves the infrared feature toward the visible-image region of CLIP’s embedding space, and paired visible images provide a semantic reference, and a frozen diffusion model provides noise-prediction feedback during adaptation.
Figure 2: Overview of TMT. A paired infrared–visible sample (purple zone) is processed by a trainable CLIP image encoder for the infrared image and a frozen CLIP image encoder for the visible counterpart. Generic visible–infrared modality descriptions are encoded by the frozen CLIP text encoder; their normalized difference produces the modality direction Δv , which is applied by Text-Derived Modality Transfer to the infrared feature and yields a pseudo-visible representation (red zone). Generative–Perceptual Co-Supervision (yellow and green zones) constrains this representation through two complementary paths: the frozen CLIP image encoder provides paired semantic alignment with the visible feature, while the frozen diffusion model uses the visible image and the pseudo-visible representation to compute a latent-space noise-prediction loss against a sampled standard-Gaussian target. Only the trainable infrared image encoder (blue zone) is updated during training. (best viewed in color)
Table 1: Modality-level properties underlying the description pair. The source and target descriptions verbalize the modality-dependent rows; the shared structural properties are stated in both descriptions as content to be preserved.
Method
SMOD
JU-VNT-T
MFNet
PST900
Avg.
LanguageBind †
56.30
47.90
38.59
36.98
44.94
un 2 CLIP
57.87
32.63
35.66
34.19
40.09
DIVA
60.07
43.73
34.08
32.27
42.54
ImageBind (official)
25.73
4.17
6.07
27.04
15.75
ImageBind (trained)
63.00
36.28
36.28
38.50
43.52
CLIP-FT †
62.90
51.56
31.89
38.24
46.15
Table 2: Zero-shot Top-1 accuracy (%) on four source-disjoint benchmarks, averaged over three shared prompt groups. ImageBind (trained) is the ImageBind-style paired CLIP control, distinct from the official checkpoint. † marks infrared–language-supervised references. Best values under our protocol are bold.
Method
M3FD
FLIRv1
FLIRv2
LLVIP
All-8 avg.
LanguageBind †
67.10
77.27
45.20
83.23
56.57
LanguageBind ‡
–
82.90
48.00
87.20
–
un 2 CLIP
54.87
80.20
35.70
79.83
51.37
DIVA
61.77
71.77
30.33
85.20
52.40
ImageBind (official)
5.74
6.27
1.10
72.86
18.62
ImageBind (trained)
57.54
73.94
31.22
72.09
51.11
Table 3: Supplementary Top-1 accuracy (%) on four benchmarks sharing source families with IR500K. ImageBind (trained) is the ImageBind-style paired CLIP control. ‡ gives originally reported LanguageBind values under a different protocol and is excluded from the eight-dataset average. Best values under our protocol are bold.
Method
IR → Text
Text → IR
R@1
R@5
R@10
R@20
R@1
R@5
R@10
R@20
Without infrared–text supervision
OpenAI CLIP [ 21 ]
5.00
11.72
16.72
25.47
10.47
20.31
30.31
43.44
un 2 CLIP [ 15 ]
3.59
10.00
15.00
22.03
10.31
21.41
29.22
39.38
DIVA [ 29 ]
4.84
11.41
16.09
25.00
9.06
19.22
29.84
42.81
ImageBind (official) [ 6 ]
1.09
4.06
6.25
9.53
1.41
4.37
7.66
11.25
Table 4: Infrared–text retrieval on IR-TD. Recall@K is reported in both retrieval directions. ImageBind (official) uses the released checkpoint; ImageBind (trained) is our paired CLIP alignment control. The upper block contains methods without infrared–text supervision during adaptation; the lower block contains supervised references. Methods marked with † use infrared–language supervision during pre-training or fine-tuning. Best results among the methods without infrared–text supervision are in bold .
Method
CART
SemanticRT
OpenAI CLIP [ 21 ]
16.06
15.16
LanguageBind † [ 35 ]
18.02
16.87
un 2 CLIP [ 15 ]
9.22
17.29
DIVA [ 29 ]
16.40
15.49
ImageBind (official) [ 6 ]
1.59
5.63
ImageBind (trained)
16.94
15.80
Table 5: Zero-shot open-vocabulary semantic segmentation mIoU (%) on infrared images. Both datasets are absent from TMT training. ImageBind (trained) denotes the paired CLIP alignment control. Methods marked with † use infrared–language supervision.
Shift
Lsem
Ldiff
R@1 ↑
R@20 ↑
Primary cls. ↑
All cls. ↑
Seg. mIoU ↑
×
×
×
7.73
34.45
42.48
52.33
15.61
✓
✓
×
8.83
37.81
48.23
59.78
17.69
✓
×
✓
7.58
34.45
44.55
54.98
16.73
×
✓
✓
7.88
34.14
46.18
55.79
16.27
✓
✓
✓
8.83
38.28
48.92
60.40
17.81
Table 6: Component ablation averaged across tasks. Each row switches the text-derived shift αΔv and the two supervision branches Lsem ( Lsemantic ) and Ldiff on (✓) or off ( × ); the first row is the unadapted OpenAI CLIP encoder without any further training. Retrieval averages both directions on IR-TD; primary classification averages the four source-disjoint datasets (SMOD, JU-VNT-T, MFNet, and PST900); all-dataset classification averages all eight datasets; segmentation averages CART and SemanticRT. All values are percentages.
Variant
R@1 ↑
R@20 ↑
Primary cls. ↑
All cls. ↑
Seg. mIoU ↑
ImageBind (trained)
5.37
28.47
43.52
51.11
16.37
Image-mean transfer
9.14
35.31
46.40
56.38
16.24
Random direction (3 seeds)
8.90
36.85
46.06
55.53
16.27
Text pair (modality names)
8.52
37.26
46.57
56.51
16.12
Reversed direction
7.26
33.13
40.22
51.12
15.28
TMT
8.83
38.28
48.92
60.40
17.81
Table 7: Transfer-direction controls. ImageBind (trained) is the ImageBind-style CLIP alignment control, not the official checkpoint. The variant without the shift is reported with the component ablation in Table 6 . Random direction averages three seeds; the modality-name pair uses “an image of visible” versus “an image of infrared” in place of the modality descriptions; the reversed direction negates Δv . Retrieval is bidirectional mean R@K; classification is Top-1 accuracy; segmentation is mean mIoU. All values are percentages. Primary classification averages SMOD, JU-VNT-T, MFNet, and PST900.
Figure 3: t-SNE visualization of CLIP ViT-L/14 embeddings for original infrared images, generated pseudo-visible images, and original visible images. Matched subsets are sampled from the source datasets for readability, and all images are encoded with the same CLIP backbone. Gray lines connect matched triplets, while colors indicate image source. The embeddings of generated pseudo-visible images appear closer to the visible-image distribution than those of the original infrared images; this is an indirect visualization of the transfer.
Figure 4: IR500K examples of the original infrared image, the pseudo-visible image, and the paired visible image from left to right. The pseudo-visible result preserves object identity and coarse structure, but it is not expected to match the visible image pixel by pixel.
α
Primary cls. ↑
Retrieval R@20 ↑
Seg. mIoU ↑
0.0
46.18
34.14
16.27
0.1
48.49
37.97
17.96
0.2
48.15
36.80
17.92
0.3
48.92
38.28
17.81
0.4
47.61
37.42
17.76
0.5
45.41
34.45
17.23
Table 8: Transfer-weight sensitivity. Primary classification averages four source-disjoint datasets; retrieval averages both R@20 directions; segmentation averages CART and SemanticRT mIoU. All values are percentages.
Figure 5: An IR500K/KAIST example: infrared input, pseudo-visible outputs of the diffusion model at six transfer weights, and the paired visible image.
Dataset
Margin ratio ↑
Fisher ratio ↑
CLIP
TMT
Δ
CLIP
TMT
Δ
M3FD
0.102
0.116
+13.52%
0.132
0.175
+32.86%
SMOD
0.166
0.195
+17.60%
0.124
0.168
+35.95%
FLIRv1
0.157
0.175
+11.11%
0.117
0.152
+30.08%
FLIRv2
0.106
0.112
+5.99%
0.149
0.180
+21.08%
LLVIP
0.526
0.775
+47.49%
0.226
0.349
+54.27%
Table 9: Feature separability of infrared object embeddings. Both metrics are computed on the high-dimensional L2-normalized CLIP features before t-SNE projection. Δ reports the relative change of TMT over OpenAI CLIP, computed from unrounded values.
Figure 6: t-SNE visualization of infrared object features. The upper block shows M3FD, FLIRv1, FLIRv2, and LLVIP; the lower block shows JU-VNT-T, MFNet, PST900, and SMOD. Separability values in Table 9 are computed in the original embedding space.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Training curves of the two diffusion supervision strategies.
Figure A.2: Gradient distributions in the trainable CLIP image encoder.
Figure A.3: Training curves of Strategy B under different learning rates.
Probe target ( R2 )
OpenAI CLIP
diffusion branch
Δ
Texture level
0.654
0.677
+0.023
Gain (signed brightness)
0.263
0.067
−0.196
Appendix
Table A.1: Linear-probe R2 for reading the perturbation target from frozen CLIP embeddings. Higher texture R2 with lower gain R2 indicates detail-specific enhancement rather than a generic sensitivity gain.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
Infrared remote-sensing imagery captures intensity structure, object-background contrast, and illumination-invariant cues often invisible in RGB imagery. Yet, most remote-sensing vision-language resources and models focus on visible-band semantics, leaving infrared vision-language understanding underexplored. We introduce MonoIR-RS, a large-scale infrared remote-sensing vision-language dataset and benchmark that couples IR-aware data construction with CLIP-style contrastive adaptation and VLM instruction tuning. Built from the same source pool and split as FusionRS, MonoIR-RS retains the infrared image as the model-facing modality, yielding 600,000 synthesized infrared images and 59,032 retained IR-aware caption records. The model experiments use this retained language-supervision subset, whose captions rewrite supervision around grayscale structure and infrared-style contrast instead of RGB appearance. We show that the synthesized infrared imagery is markedly closer to real thermal imagery than a grayscale conversion on the AVIID benchmark. We fine-tune five CLIP backbones and six VLM backbones, and calibrate them against zero-shot behavior: IR-aware adaptation lifts CLIP mean recall by up to 12.8 points and drives VLM captioning IR-cue coverage to 100% while reducing residual RGB-color leakage to near zero. By isolating the infrared modality from RGB-IR dual-modal learning, MonoIR-RS offers a controlled, reproducible testbed for aligning infrared remote-sensing evidence with language.
Jiaju Han, Ma Yaqi, Yahui Chai +8
China University of Petroleum-Beijing at Karamay, Karamay, Xinjiang, China · Guizhou University, Guiyang, China · Shenzhen Research Institute of Big Data, Shenzhen, China +1
Thermal imaging offers a powerful alternative to visible-spectrum vision under challenging conditions such as low illumination and adverse weather, yet foundational vision-language models like CLIP fail to align thermal images with textual descriptions due to a fundamental thermal perception gap. We identify three major challenges: the lack of captioned thermal datasets, the inability of standard LLMs to reason about thermal phenomena, and a key representational challenge in thermal imaging where global scene context and object-level heat signatures conflict when learned together in a single embedding space. To address these, we introduce IR-Cap, the first physics-aware thermal captioning pipeline and dataset providing complementary global and fine-grained thermal descriptions across three public benchmarks, and T-CLIP, a decoupled dual-LoRA framework that independently adapts CLIP for scene-level and object-level thermal understanding. T-CLIP achieves consistent improvements over all baselines across three thermal benchmarks in cross-modal retrieval, and we provide an exploratory demonstration of its applicability to text-conditioned thermal image generation.
Vision-language models such as CLIP enable zero-shot classification from arbitrary class names, but under test-time corruptions, their visual encoder projects distorted features onto incorrect textual anchors with high confidence: a phenomenon we term modality bias. Standard entropy minimization reinforces these overconfident errors, which creates a confirmation-bias loop and collapses the predicted label distribution. We propose an online test-time adaptation (TTA) method that breaks this loop at three levels. At the representation level, a frozen self-supervised backbone (e.g., DINO v2) is combined with CLIP through a cross-covariance logit ensemble. CLIP text prototypes are projected into the feature space of this text-free backbone, so the self-supervised branch classifies native image features and is less affected by the text bias of CLIP. At the optimization level, a gated test-time information-maximization objective pairs per-sample sharpening with batch-level diversity, while an adaptive gate excludes unreliable samples from the gradient. At the prediction level, an online logit adjustment corrects the residual class-distribution skew. CDIM outperforms the compared CLIP-based online TTA baselines on ImageNet-C, CIFAR10-C, and CIFAR100-C, and remains competitive in continual and mixed-domain TTA. Extensive analyses and ablation studies verify the contribution of each of its components. Our code is available at https://github.com/B1ueC1oud/CDIM