Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment

Yaze Zhao; Yixiong Zou; Yuhua Li; Ruixuan Li

cs.CVMar 18, 2026

Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment

Authors: Yaze Zhao, Yixiong Zou, Yuhua Li, Ruixuan Li

Organizations: School of Computer Science and Technology, Huazhong University of Science and Technology

Abstract

Cross-Domain Few-Shot Learning (CDFSL) adapts models trained with large-scale general data (source domain) to downstream target domains with only scarce training data, where the research on vision-language models (e.g., CLIP) is still in the early stages. Typical downstream domains, such as medical diagnosis, require fine-grained visual cues for interpretable recognition, but we find that current fine-tuned CLIP models can hardly focus on these cues, albeit they can roughly focus on important regions in source domains. Although current works have demonstrated CLIP's shortcomings in capturing local subtle patterns, in this paper, we find that the domain gap and scarce training data further exacerbate such shortcomings, much more than that of holistic patterns, which we call the local misalignment problem in CLIP-based CDFSL. To address this problem, due to the lack of supervision in aligning local visual features and text semantics, we turn to self-supervision information. Inspired by the translation task, we propose the CC-CDFSL method with cycle consistency, which translates local visual features into text features and then translates them back into visual features (and vice versa), and constrains the original features close to the translated back features. To reduce the noise imported by richer information in the visual modality, we further propose a Semantic Anchor mechanism, which first augments visual features to provide a larger corpus for the text-to-image mapping, and then shrinks the image features to filter out irrelevant image-to-text mapping. Extensive experiments on various benchmarks, backbones, and fine-tuning methods show we can (1) effectively improve the local vision-language alignment, (2) enhance the interpretability of learned patterns and model decisions by visualizing patches, and (3) achieve state-of-the-art performance.

Figures & tables

Figure 1: Fine-grained visual cues (marked as red boxes) are crucial in specialized fields like interpretable medical diagnosis (a). However, in target domains, the fine-tuned CLIP cannot focus on these subtle patterns (b), but in source domains (c), CLIP can still roughly capture all important regions for recognition. Therefore, we hypothesize that the domain gap and scarce training data exacerbate CLIP’s shortcomings in capturing subtle patterns, much more than that of holistic patterns, which we aim to address.

Figure 2: (a) To validate our hypothesis, we measure the alignment score between global / local features and text features, and we find the alignments of both global and local features are harmed under the CDFSL task, but the local features show a larger decline in the alignment score, verifying our hypothesis. (b) Our proposed method effectively improves the local alignment in target domains.

Figure 3: Overview of our framework, consisting of three key components: (a) the Text-to-Image-to-Text (T-I-T) cycle-consistency module, (b) the Semantic Anchor (SA) module, and (c) the Image-to-Text-to-Image (I-T-I) cycle-consistency module. The process begins with the SA module augmenting raw images to create a larger corpus, followed by extracting local image features from these images and transforming them via MLP to align with the text feature space. The T-I-T cycle then uses text features to select semantically relevant patches and maps them back to reconstruct text features, enhancing local feature alignment. Subsequently, the SA module shrinks the feature set to select class-relevant anchor patches, which are used in the I-T-I cycle to map these anchor visual features through text features to augmented image features. Our model improves local alignment and interpretability in cross-domain few-shot learning.

Figure 4: Base-to-new generalization on 11 datasets.

T-I-T	I-T-I	SA	ISIC	ChestX	EuroSAT	Crop.	Ave.
			50.68	24.44	92.63	96.20	65.98
✓			51.13	25.15	93.79	96.37	66.61
✓		✓	54.30	25.35	94.33	96.95	67.73
	✓	✓	53.81	25.14	93.83	97.01	67.45
✓	✓	✓	54.72	25.47	94.35	97.08	67.90

Table 2: Ablation study of the 5-way 5-shot task.

Retrieval	ISIC	ChestX	EuroSAT	Crop.	Ave.
Cross-view	53.77	25.13	93.75	96.84	67.37
Intra-image	53.59	25.01	93.63	96.77	67.25
All-images	53.66	25.02	93.46	96.64	67.19

Table 3: Impact of different retrieval strategies.

Figure 5: Ablation study on hybrid coefficient in Eq. 16 .

Figure 6: Ablation study on hyperparameter $k$ in Eq. 10 .

Figure 7: Semantic Anchor (SA) selection across four domains. Red boxes indicate the top- $k\!=\!10$ patch proposals retained by the SA shrinking phase (Eq. 11 ).

Figure 8: Our method (the third row) outperforms the baseline (the second row) in highlighting key details, offering a clearer view of important features in data-scarce scenarios.

Figure 9: The T-I-T pathway indicates that the model can focus on disease-relevant regions and capture fine-grained semantic relationships between different diseases, providing valuable interpretability for medical analysis even when the reconstructed text does not fully match the initial label.

Figure 10: The I-T-I cycle pathway demonstrates how the model extracts and retains essential semantics in the text domain, which are then used to re-focus on meaningful regions within the image’s augmented feature space.

	ISIC	ChestX	EuroSAT	CropDiseases
CLIP-LoRA	49.83	23.79	93.0	95.85
+ Ours	50.68	24.51	93.64	96.24

Table 4: Prototype-classification accuracy of the vision encoder.

Backbone	Method	ChestX	ISIC	EuroSAT	Crop.	Ave.
ViT-B/32	Baseline	21.23	36.32	79.79	81.13	54.62
ViT-B/32	+ Ours	21.36	38.26	82.88	83.98	56.62
ViT-L/14	Baseline	22.90	38.45	85.83	89.83	59.25
ViT-L/14	+ Ours	23.19	38.72	90.11	90.77	60.70

Table 5: Comparison with ViT-based methods in 1-shot.

Dataset	Global Feature	Local Features	Far Domain
miniImageNet	0.7056	0.3821	✗
ChestX	0.6768	0.2391	✓
ISIC2018	0.7261	0.2360	✓
EuroSAT	0.6768	0.3845	✓
CropDiseases	0.7012	0.3811	✓

Table 6: Alignment Scores of Global and Local Features.

Method	ChestX	ISIC	EuroSAT	Crop.	Ave.
Baseline	0.2391	0.2360	0.3845	0.3811	0.3102
+ Ours	0.3293	0.2661	0.3796	0.3877	0.3407

Table 7: Comparison of Local Feature Alignment Scores.

Table 8: Base-to-new generalization on 11 datasets. Ours (CC-CDFSL) consistently improves new-class accuracy and harmonic mean (HM) over MaPLe, validating the cross-task generality of cycle-consistent patch-level alignment.

Panel of Figure 15 — Figure 15: t-SNE visualization of feature distributions on different datasets. The first row shows results from CLIP-LoRA. The second row shows results from our proposed method. Different colors denote different classes.

Figure 16: Row 2 and Row 3 display the attention maps of the baseline and CC-CDFSL, respectively, showing that the latter localizes critical regions more precisely.

Figure 17: The heatmap for CLIP-LoRA (the second row) and our CC-CDFSL (the third row) in four target domains.

Figure 18: Illustration of the Text-to-Image-patch-to-Text (T-I-T) cycle consistency pathway. Each specific class text is first used to locate the most semantically relevant local patch within the image. The selected image patch is then mapped back to the text space in an attempt to reconstruct the original class label. And the patches in the figure are resized to a resolution of 64×64 for better presentation.

Figure 19: Additional visualizations of the Image-to-Text-to-Image (I-T-I) cycle consistency pathway. Each pathway shows a selected patch within the initial image, the most similar text derived from this patch, and a semantically re-focused patch within its corresponding image in the augmented image space.

Method	Backbone	Mark	Source	Target	CropDiseases	EuroSAT	ISIC	ChestX	Ave.
MEM-FS	ViT/DINO	TIP-23			93.74	86.49	47.38	26.67	63.57
StyleAdv	ViT/DINO	CVPR-23	✓	-	94.85	88.57	47.73	26.97	64.53
FLoR	ViT/DINO	CVPR-24	✓	-	95.28	90.41	49.52	27.28	65.48
DAMIM	ViT/DINO	AAAI-25	✓	-	95.52	89.50	50.76	27.28	65.77
AttnTemp	ViT/DINO	NeurIPS-24	✓	-	95.53	90.13	53.09	27.72	66.62
CD-CLS	ViT/DINO	NeurIPS-24	✓	✓	96.27	91.53	54.69	27.66	67.54

Table 9: Comparison with state-of-the-art CDFSL works by the 5-way 5-shot classification.

Method	Backbone	Mark	Source	Target	CropDiseases	EuroSAT	ISIC	ChestX	Ave.
MEM-FS	ViT/DINO	TIP-23			81.11	68.11	32.97	22.76	51.24
StyleAdv	ViT/DINO	CVPR-23	✓	-	81.22	72.15	33.05	22.92	52.34
FLoR	ViT/DINO	CVPR-24	✓	-	81.81	72.39	34.20	22.78	52.80
DAMIM	ViT/DINO	AAAI-25	✓	-	82.34	72.87	34.66	22.97	53.21
AttnTemp	ViT/DINO	NeurIPS-24	✓	-	84.02	74.35	34.92	23.19	54.12
CD-CLS	ViT/DINO	NeurIPS-24	✓	✓	84.53	74.97	35.56	23.39	54.62

Table 10: Comparison with state-of-the-art CDFSL works by the 5-way 1-shot classification.

Method	Backbone	ISIC	ChestX	EuroSAT	CropDiseases	Average
CLIP-LoRA	RN50/CLIP	32.01	21.76	57.79	65.24	44.20
+ OURS	RN50/CLIP	35.21	22.75	59.23	72.85	47.51
SigLIP2-LoRA	ViT/SigLip2	26.48	20.53	63.05	81.84	47.98
+ OURS	ViT/SigLip2	29.53	22.00	68.12	83.39	50.76
PE-Core-LoRA	ViT/PE-Core	38.05	22.45	82.16	89.01	57.92
+ OURS	ViT/PE-Core	40.72	22.67	83.92	90.48	59.45

Table 11: Accuracy on different backbones in 1-shot.

Method	Feature type	ISIC	ChestX	EuroSAT	CropDiseases	Average
SigLIP2-LoRA	global	0.1511	0.1380	0.1226	0.1691	0.14520
SigLIP2-LoRA	local	0.0330	0.0310	0.0126	0.0383	0.02872
+ OURS	local	0.0421	0.0363	0.0181	0.0486	0.03627
PE-Core-LoRA	global	0.2795	0.2493	0.2459	0.2810	0.26392
PE-Core-LoRA	local	0.0191	0.0096	0.0264	0.0088	0.01597
+ OURS	local	0.0238	0.0111	0.0286	0.0123	0.01895

Table 12: Feature alignments of other baselines.

	ChestX	ISIC2018	EuroSAT	CropDiseases
$\lambda_{1}$	3	3	1.5	1
$\lambda_{2}$	0.5	2	0.2	1.5

Table 13: Hyperparameters

\lambda_{1}

and

\lambda_{2}

for each dataset.

Explore similar work

May 12, 2026

Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning

Cross-Domain Few-Shot Learning (CDFSL) aims to adapt large-scale pretrained models to specialized target domains with limited samples, yet the few-shot fine-tuning of vision-language models like CLIP remains underexplored. By establishing multiple fine-tuning baselines of CLIP for CDFSL, we find adapter-based methods (e.g., LoRA) consistently outperform prompt-based ones (e.g., MaPLe), contrary to in-domain scenarios. To make those effective in-domain methods competitive again in CDFSL, we analyze this phenomenon and discover LoRA's superiority stems from rectifying the collapsed attention of visual CLS token, enhancing modality alignment and class separation by focusing on text-related visual regions. Further, we find textual EOS token exhibit much better attention to visual samples, and CLIP's standard contrastive loss weakly constrains modality alignment. Based on these insights, we propose Semantic Probe, a plug-and-play attention rectification framework for both adapter- and prompt-based methods. Extensive experiments on four CDFSL benchmarks validate our rationale, achieving state-of-the-art performance and benefiting both fine-tuning paradigms. Codes will be released.

Yaze Zhao, Yicong Liu, Yixiong Zou +2

Source-Free Domain Adaptation Few-Shot Domain Adaptation VLM Adaptation Few-Shot Learning Vision-Language Alignment Parameter-Efficient Fine-Tuning

Ask PDF

May 28, 2026

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning

Vision-Language Models (VLMs) such as CLIP demonstrate strong zero-shot generalization, but their performance significantly degrades in cross-domain scenarios with scarce target-domain training data (Cross-Domain Few-Shot Learning, CDFSL). In this paper, we focus on the target-domain few-shot finetuning in the CLIP-based CDFSL task. Prevailing finetuning paradigms uniformly align all image patch tokens with their corresponding textual embeddings. However, we find a counterintuitive phenomenon: actively pushing away certain low-similarity image tokens, termed "tail tokens", from their textual embeddings consistently improves target-domain performance. We delve into this phenomenon and provide a novel interpretation: under great domain shifts and scarce training data, the model can hardly extract semantic information from visual inputs; therefore, the common belief of alignment is valid only for tokens already containing sufficient semantic information; for tail tokens, forcing the alignment would lead to excessive overfitting to the scarce training, while breaking the alignment is more useful. Motivated by this, we propose Adaptive Tail-Head Alignment (ATHA), a novel fine-tuning strategy for CLIP that transforms the conventional uniform alignment paradigm to an adaptive alignment paradigm, with both alignment strengthening and weakening. Extensive experiments on four challenging CDFSL benchmarks validate our state-of-the-art performance. Our code is available at https://github.com/shuaiyi308/ATHA.

Shuai Yi, Yixiong Zou, Yuhua Li +1

Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.

Source-Free Domain Adaptation Few-Shot Domain Adaptation VLM Adaptation Few-Shot Learning Vision-Language Alignment

Ask PDF

Mar 7, 2026

Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learning

Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) focuses on fine-tuning with limited training data from target domains (e.g., medical or satellite images), where Vision-Language Models (VLMs) such as CLIP and SigLIP have shown promising results. Current works in traditional visual models suggest that improving visual discriminability enhances performance. However, in VLM-based SF-CDFSL tasks, we find that strengthening visual-modal discriminability actually suppresses VLMs' performance. In this paper, we aim to delve into this phenomenon for an interpretation and a solution. By both theoretical and experimental proofs, our study reveals that fine-tuning with the typical cross-entropy loss (

\mathcal{L}_{\mathrm{vlm}}

) inherently includes a visual learning part and a cross-modal learning part, where the cross-modal part is crucial for rectifying the heavily disrupted modality misalignment in SF-CDFSL. However, we find that the visual learning essentially acts as a shortcut that encourages the model to reduce

\mathcal{L}_{\mathrm{vlm}}

without considering the cross-modal part, therefore hindering the cross-modal alignment and harming the performance. Based on this interpretation, we further propose an approach to address this problem: first, we perturb the visual learning to guide the model to focus on the cross-modal alignment. Then, we use the visual-text semantic relationships to gradually align the visual and textual modalities during the fine-tuning. Extensive experiments on various settings, backbones (CLIP, SigLip, PE-Core), and tasks (4 CDFSL datasets and 11 FSL datasets) show that we consistently set new state-of-the-art results. Code is available at https://github.com/zhenyuZ-HUST/CVPR26-Mind-the-Discriminability-Trap.

Zhenyu Zhang, Yixiong Zou, Yuhua Li +2

School of Computer Science and Technology, Huazhong University of Science and Technology · National Key Laboratory for Multimedia Information Processing, Peking University

Few-Shot Learning VLM Adaptation Few-Shot Domain Adaptation Vision-Language Alignment Source-Free Domain Adaptation

Ask PDF