Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learning

Zhenyu Zhang; Yixiong Zou; Yuhua Li; Ruixuan Li; Guangyao Chen

cs.CVMar 7, 2026

Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learning

Authors: Zhenyu Zhang, Yixiong Zou, Yuhua Li, Ruixuan Li, Guangyao Chen

Organizations: School of Computer Science and Technology, Huazhong University of Science and Technology · National Key Laboratory for Multimedia Information Processing, Peking University

Abstract

Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) focuses on fine-tuning with limited training data from target domains (e.g., medical or satellite images), where Vision-Language Models (VLMs) such as CLIP and SigLIP have shown promising results. Current works in traditional visual models suggest that improving visual discriminability enhances performance. However, in VLM-based SF-CDFSL tasks, we find that strengthening visual-modal discriminability actually suppresses VLMs' performance. In this paper, we aim to delve into this phenomenon for an interpretation and a solution. By both theoretical and experimental proofs, our study reveals that fine-tuning with the typical cross-entropy loss ( $\mathcal{L}_{\mathrm{vlm}}$ ) inherently includes a visual learning part and a cross-modal learning part, where the cross-modal part is crucial for rectifying the heavily disrupted modality misalignment in SF-CDFSL. However, we find that the visual learning essentially acts as a shortcut that encourages the model to reduce $\mathcal{L}_{\mathrm{vlm}}$ without considering the cross-modal part, therefore hindering the cross-modal alignment and harming the performance. Based on this interpretation, we further propose an approach to address this problem: first, we perturb the visual learning to guide the model to focus on the cross-modal alignment. Then, we use the visual-text semantic relationships to gradually align the visual and textual modalities during the fine-tuning. Extensive experiments on various settings, backbones (CLIP, SigLip, PE-Core), and tasks (4 CDFSL datasets and 11 FSL datasets) show that we consistently set new state-of-the-art results. Code is available at https://github.com/zhenyuZ-HUST/CVPR26-Mind-the-Discriminability-Trap.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Unlike traditional visual models, we find that enhancing visual learning consistently decreases the performance of the VLM-based CDFSL, although the visual-modal accuracy increases. Appropriately inhibiting visual learning improves cross-modal performance. (b) We also find that this phenomenon widely exists in cross-domain scenarios, where the best-performing zero-shot CLIP model does not necessarily extract the best visual features. (c) In this paper, we explore the phenomenon for an interpretation, finding that fine-tuning involves two learning directions: the visual and the cross-modal ones. We interpret visual learning as a shortcut that reduces the VLM’s classification loss but disrupts cross-modal learning. (d) Based on this, we propose a method to suppress and guide visual learning, enhancing the model’s cross-modal learning.

Panel of Figure 2 — Figure 2: (a) When sample $i$ and sample $k$ belong to different classes, $\Delta\cos(\theta_{ik})$ in 5-way 1-shot fine-tuning is always less than 0. (b) At any stage of fine-tuning (each epoch), visual learning can effectively reduce the loss value ( $\mathcal{L}_{\mathrm{vlm}}$ ).

Figure 3: Recording the model at each epoch during training and further training these models using visual learning, the target loss value $\mathcal{L}_{\mathrm{vlm}}$ can be effectively reduced even further.

Figure 4: We employ three strategies to restrict visual learning during the VLM fine-tuning process and measure the distance of the resulting models from the optimal cross-modal model. We find that visual learning increases the distance from the optimal model, which hinders visual-text alignment.

Panel of Figure 5 — Figure 5: (a) Gap shift operation [ 38 ] . (b) The model’s initial state is marked by the vertical dashed line. In the cross-domain dataset EuroSAT, CLIP’s modalities are misaligned, meaning that through gap shift, lower loss and higher accuracy can be achieved. (c) Fine-tuning does not effectively re-align the modalities. (d) When visual learning is enhanced ( $\mathcal{L}_{\mathrm{v}}$ ), the misalignment becomes more pronounced. (e) When visual learning is suppressed ( $\mathcal{L}_{\mathrm{ad}}$ ), the misalignment is mitigated.

Panel of Figure 6 — Figure 6: (a)(b) The Attention maps of the three models. From left to right: the original image, strategy 2 result (trained with $\mathcal{L}_{\mathrm{vlm}}$ ), strategy 3 result (trained with $\mathcal{L}_{\mathrm{vlm}}+\mathcal{L}_{\mathrm{v}}$ ), and strategy 1 result (trained with $\mathcal{L}_{\mathrm{vlm}}+\mathcal{L}_{\mathrm{ad}}$ ). $s_{i,j}$ represents the cosine similarity between the image features and the text features. A higher similarity indicates better alignment.

Figure 7: The overall architecture of the model, with the pink region highlighting our proposed method. (a) Firstly, we propose suppressing the visual learning to resist its shortcut during the finetuning, which constrains the model to focus more on the cross-modal alignment. (b) Then, we propose using semantic relationships to gradually align the internal relationships of the visual modality with those of the text modality, thereby enhancing cross-modal alignment.

Figure 8: Average few-shot learning result of 11 datasets.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: (a) When sample $i$ and sample $k$ belong to different classes, $\Delta\cos(\theta_{ik})$ in 5-way 1-shot fine-tuning is always less than 0, which means that the visual features of samples $i$ and $k$ from different classes will become increasingly dissimilar. (b) Hyperparameter study.

Figure 10: Results for different initial epochs settings.

Panel of Figure 11 — Figure 11: The attention maps of the three models are shown. From left to right: the original image, the Baseline result (trained with $\mathcal{L}_{\mathrm{vlm}}$ ), the result of the model with enhanced visual learning (trained with $\mathcal{L}_{\mathrm{vlm}}+\mathcal{L}_{\mathrm{v}}$ ), and the result of our method (trained with $\mathcal{L}_{\mathrm{vlm}}+\mathcal{L}_{\mathrm{ad}}$ ). Here, $s_{i,j}$ represents the cosine similarity between the image features and the text features. A higher similarity indicates better alignment. It is evident that our model has the most appropriate attention scope and achieves the highest similarity between sample pairs.

Explore similar work

Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning
May 12, 2026Yaze Zhao, Yicong Liu, Yixiong Zou +2Source-Free Domain Adaptation Few-Shot Domain Adaptation
Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learning
Mar 5, 2026Zhenyu Zhang, Guangyao Chen, Yixiong Zou +2Few-Shot Learning Few-Shot Domain Adaptation
School of Computer Science and Technology, Huazhong University of Science and Technology · National Key Laboratory for Multimedia Information Processing, Peking University
Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learning
May 25, 2026Shuai Yi, Yixiong Zou, Yuhua Li +1Source-Free Domain Adaptation Few-Shot Domain Adaptation
Huazhong University of Science and Technology

May 12, 2026J/K move · Enter open · S save

Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning

Yaze Zhao, Yicong Liu, Yixiong Zou, Yuhua Li+1

Source-Free Domain Adaptation Few-Shot Domain Adaptation VLM Adaptation Few-Shot Learning Vision-Language Alignment Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Cross-Domain Few-Shot Learning (CDFSL) aims to adapt large-scale pretrained models to specialized target domains with limited samples, yet the few-shot fine-tuning of vision-language models like CLIP remains underexplored. By establishing multiple fine-tuning baselines of CLIP for CDFSL, we find adapter-based methods (e.g., LoRA) consistently outperform prompt-based ones (e.g., MaPLe), contrary to in-domain scenarios. To make those effective in-domain methods competitive again in CDFSL, we analyze this phenomenon and discover LoRA's superiority stems from rectifying the collapsed attention of visual CLS token, enhancing modality alignment and class separation by focusing on text-related visual regions. Further, we find textual EOS token exhibit much better attention to visual samples, and CLIP's standard contrastive loss weakly constrains modality alignment. Based on these insights, we propose Semantic Probe, a plug-and-play attention rectification framework for both adapter- and prompt-based methods. Extensive experiments on four CDFSL benchmarks validate our rationale, achieving state-of-the-art performance and benefiting both fine-tuning paradigms. Codes will be released.

Abstract

Explore similar work

Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning

Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learning

Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learning