Cross-Domain Few-Shot Learning (CDFSL) adapts models trained with large-scale general data (source domain) to downstream target domains with only scarce training data, where the research on vision-language models (e.g., CLIP) is still in the early stages. Typical downstream domains, such as medical diagnosis, require fine-grained visual cues for interpretable recognition, but we find that current fine-tuned CLIP models can hardly focus on these cues, albeit they can roughly focus on important regions in source domains. Although current works have demonstrated CLIP's shortcomings in capturing local subtle patterns, in this paper, we find that the domain gap and scarce training data further exacerbate such shortcomings, much more than that of holistic patterns, which we call the local misalignment problem in CLIP-based CDFSL. To address this problem, due to the lack of supervision in aligning local visual features and text semantics, we turn to self-supervision information. Inspired by the translation task, we propose the CC-CDFSL method with cycle consistency, which translates local visual features into text features and then translates them back into visual features (and vice versa), and constrains the original features close to the translated back features. To reduce the noise imported by richer information in the visual modality, we further propose a Semantic Anchor mechanism, which first augments visual features to provide a larger corpus for the text-to-image mapping, and then shrinks the image features to filter out irrelevant image-to-text mapping. Extensive experiments on various benchmarks, backbones, and fine-tuning methods show we can (1) effectively improve the local vision-language alignment, (2) enhance the interpretability of learned patterns and model decisions by visualizing patches, and (3) achieve state-of-the-art performance.
Figures & tables
Figure 1: Fine-grained visual cues (marked as red boxes) are crucial in specialized fields like interpretable medical diagnosis (a). However, in target domains, the fine-tuned CLIP cannot focus on these subtle patterns (b), but in source domains (c), CLIP can still roughly capture all important regions for recognition. Therefore, we hypothesize that the domain gap and scarce training data exacerbate CLIP’s shortcomings in capturing subtle patterns, much more than that of holistic patterns, which we aim to address.
Figure 2: (a) To validate our hypothesis, we measure the alignment score between global / local features and text features, and we find the alignments of both global and local features are harmed under the CDFSL task, but the local features show a larger decline in the alignment score, verifying our hypothesis. (b) Our proposed method effectively improves the local alignment in target domains.
Figure 3: Overview of our framework, consisting of three key components: (a) the Text-to-Image-to-Text (T-I-T) cycle-consistency module, (b) the Semantic Anchor (SA) module, and (c) the Image-to-Text-to-Image (I-T-I) cycle-consistency module. The process begins with the SA module augmenting raw images to create a larger corpus, followed by extracting local image features from these images and transforming them via MLP to align with the text feature space. The T-I-T cycle then uses text features to select semantically relevant patches and maps them back to reconstruct text features, enhancing local feature alignment. Subsequently, the SA module shrinks the feature set to select class-relevant anchor patches, which are used in the I-T-I cycle to map these anchor visual features through text features to augmented image features. Our model improves local alignment and interpretability in cross-domain few-shot learning.
Figure 4: Base-to-new generalization on 11 datasets.
T-I-T
I-T-I
SA
ISIC
ChestX
EuroSAT
Crop.
Ave.
50.68
24.44
92.63
96.20
65.98
✓
51.13
25.15
93.79
96.37
66.61
✓
✓
54.30
25.35
94.33
96.95
67.73
✓
✓
53.81
25.14
93.83
97.01
67.45
✓
✓
✓
54.72
25.47
94.35
97.08
67.90
Table 2: Ablation study of the 5-way 5-shot task.
Retrieval
ISIC
ChestX
EuroSAT
Crop.
Ave.
Cross-view
53.77
25.13
93.75
96.84
67.37
Intra-image
53.59
25.01
93.63
96.77
67.25
All-images
53.66
25.02
93.46
96.64
67.19
Table 3: Impact of different retrieval strategies.
Figure 5: Ablation study on hybrid coefficient in Eq. 16 .
Figure 6: Ablation study on hyperparameter k in Eq. 10 .
Figure 7: Semantic Anchor (SA) selection across four domains. Red boxes indicate the top- k=10 patch proposals retained by the SA shrinking phase (Eq. 11 ).
Figure 8: Our method (the third row) outperforms the baseline (the second row) in highlighting key details, offering a clearer view of important features in data-scarce scenarios.
Figure 9: The T-I-T pathway indicates that the model can focus on disease-relevant regions and capture fine-grained semantic relationships between different diseases, providing valuable interpretability for medical analysis even when the reconstructed text does not fully match the initial label.
Figure 10: The I-T-I cycle pathway demonstrates how the model extracts and retains essential semantics in the text domain, which are then used to re-focus on meaningful regions within the image’s augmented feature space.
ISIC
ChestX
EuroSAT
CropDiseases
CLIP-LoRA
49.83
23.79
93.0
95.85
+ Ours
50.68
24.51
93.64
96.24
Table 4: Prototype-classification accuracy of the vision encoder.
Figure 11: Samples from CropDiseases.
Figure 12: Samples from EuroSAT.
Figure 13: Samples from ISIC2018.
Figure 14: Samples from ChestX.
Backbone
Method
ChestX
ISIC
EuroSAT
Crop.
Ave.
ViT-B/32
Baseline
21.23
36.32
79.79
81.13
54.62
+ Ours
21.36
38.26
82.88
83.98
56.62
ViT-L/14
Baseline
22.90
38.45
85.83
89.83
59.25
+ Ours
23.19
38.72
90.11
90.77
60.70
Table 5: Comparison with ViT-based methods in 1-shot.
Dataset
Global Feature
Local Features
Far Domain
miniImageNet
0.7056
0.3821
✗
ChestX
0.6768
0.2391
✓
ISIC2018
0.7261
0.2360
✓
EuroSAT
0.6768
0.3845
✓
CropDiseases
0.7012
0.3811
✓
Table 6: Alignment Scores of Global and Local Features.
Method
ChestX
ISIC
EuroSAT
Crop.
Ave.
Baseline
0.2391
0.2360
0.3845
0.3811
0.3102
+ Ours
0.3293
0.2661
0.3796
0.3877
0.3407
Table 7: Comparison of Local Feature Alignment Scores.
Table 8: Base-to-new generalization on 11 datasets. Ours (CC-CDFSL) consistently improves new-class accuracy and harmonic mean (HM) over MaPLe, validating the cross-task generality of cycle-consistent patch-level alignment.
Figure 15: t-SNE visualization of feature distributions on different datasets. The first row shows results from CLIP-LoRA. The second row shows results from our proposed method. Different colors denote different classes.
Figure 16: Row 2 and Row 3 display the attention maps of the baseline and CC-CDFSL, respectively, showing that the latter localizes critical regions more precisely.
Figure 17: The heatmap for CLIP-LoRA (the second row) and our CC-CDFSL (the third row) in four target domains.
Figure 18: Illustration of the Text-to-Image-patch-to-Text (T-I-T) cycle consistency pathway. Each specific class text is first used to locate the most semantically relevant local patch within the image. The selected image patch is then mapped back to the text space in an attempt to reconstruct the original class label. And the patches in the figure are resized to a resolution of 64×64 for better presentation.
Figure 19: Additional visualizations of the Image-to-Text-to-Image (I-T-I) cycle consistency pathway. Each pathway shows a selected patch within the initial image, the most similar text derived from this patch, and a semantically re-focused patch within its corresponding image in the augmented image space.
Method
Backbone
Mark
Source
Target
CropDiseases
EuroSAT
ISIC
ChestX
Ave.
MEM-FS
ViT/DINO
TIP-23
93.74
86.49
47.38
26.67
63.57
StyleAdv
ViT/DINO
CVPR-23
✓
-
94.85
88.57
47.73
26.97
64.53
FLoR
ViT/DINO
CVPR-24
✓
-
95.28
90.41
49.52
27.28
65.48
DAMIM
ViT/DINO
AAAI-25
✓
-
95.52
89.50
50.76
27.28
65.77
AttnTemp
ViT/DINO
NeurIPS-24
✓
-
95.53
90.13
53.09
27.72
66.62
CD-CLS
ViT/DINO
NeurIPS-24
✓
✓
96.27
91.53
54.69
27.66
67.54
Table 9: Comparison with state-of-the-art CDFSL works by the 5-way 5-shot classification.
Method
Backbone
Mark
Source
Target
CropDiseases
EuroSAT
ISIC
ChestX
Ave.
MEM-FS
ViT/DINO
TIP-23
81.11
68.11
32.97
22.76
51.24
StyleAdv
ViT/DINO
CVPR-23
✓
-
81.22
72.15
33.05
22.92
52.34
FLoR
ViT/DINO
CVPR-24
✓
-
81.81
72.39
34.20
22.78
52.80
DAMIM
ViT/DINO
AAAI-25
✓
-
82.34
72.87
34.66
22.97
53.21
AttnTemp
ViT/DINO
NeurIPS-24
✓
-
84.02
74.35
34.92
23.19
54.12
CD-CLS
ViT/DINO
NeurIPS-24
✓
✓
84.53
74.97
35.56
23.39
54.62
Table 10: Comparison with state-of-the-art CDFSL works by the 5-way 1-shot classification.
Method
Backbone
ISIC
ChestX
EuroSAT
CropDiseases
Average
CLIP-LoRA
RN50/CLIP
32.01
21.76
57.79
65.24
44.20
+ OURS
RN50/CLIP
35.21
22.75
59.23
72.85
47.51
SigLIP2-LoRA
ViT/SigLip2
26.48
20.53
63.05
81.84
47.98
+ OURS
ViT/SigLip2
29.53
22.00
68.12
83.39
50.76
PE-Core-LoRA
ViT/PE-Core
38.05
22.45
82.16
89.01
57.92
+ OURS
ViT/PE-Core
40.72
22.67
83.92
90.48
59.45
Table 11: Accuracy on different backbones in 1-shot.
Method
Feature type
ISIC
ChestX
EuroSAT
CropDiseases
Average
SigLIP2-LoRA
global
0.1511
0.1380
0.1226
0.1691
0.14520
SigLIP2-LoRA
local
0.0330
0.0310
0.0126
0.0383
0.02872
+ OURS
local
0.0421
0.0363
0.0181
0.0486
0.03627
PE-Core-LoRA
global
0.2795
0.2493
0.2459
0.2810
0.26392
PE-Core-LoRA
local
0.0191
0.0096
0.0264
0.0088
0.01597
+ OURS
local
0.0238
0.0111
0.0286
0.0123
0.01895
Table 12: Feature alignments of other baselines.
ChestX
ISIC2018
EuroSAT
CropDiseases
λ1
3
3
1.5
1
λ2
0.5
2
0.2
1.5
Table 13: Hyperparameters λ1 and λ2 for each dataset.
Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.
School of Computer Science and Technology, Huazhong University of Science and Technology · National Key Laboratory for Multimedia Information Processing, Peking University