Organizations: School of Computer Science and Technology, Huazhong University of Science and Technology · National Key Laboratory for Multimedia Information Processing, Peking University
Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) focuses on fine-tuning with limited training data from target domains (e.g., medical or satellite images), where Vision-Language Models (VLMs) such as CLIP and SigLIP have shown promising results. Current works in traditional visual models suggest that improving visual discriminability enhances performance. However, in VLM-based SF-CDFSL tasks, we find that strengthening visual-modal discriminability actually suppresses VLMs' performance. In this paper, we aim to delve into this phenomenon for an interpretation and a solution. By both theoretical and experimental proofs, our study reveals that fine-tuning with the typical cross-entropy loss (Lvlm) inherently includes a visual learning part and a cross-modal learning part, where the cross-modal part is crucial for rectifying the heavily disrupted modality misalignment in SF-CDFSL. However, we find that the visual learning essentially acts as a shortcut that encourages the model to reduce Lvlm without considering the cross-modal part, therefore hindering the cross-modal alignment and harming the performance. Based on this interpretation, we further propose an approach to address this problem: first, we perturb the visual learning to guide the model to focus on the cross-modal alignment. Then, we use the visual-text semantic relationships to gradually align the visual and textual modalities during the fine-tuning. Extensive experiments on various settings, backbones (CLIP, SigLip, PE-Core), and tasks (4 CDFSL datasets and 11 FSL datasets) show that we consistently set new state-of-the-art results. Code is available at https://github.com/zhenyuZ-HUST/CVPR26-Mind-the-Discriminability-Trap.
Figures & tables
Figure 1: (a) Unlike traditional visual models, we find that enhancing visual learning consistently decreases the performance of the VLM-based CDFSL, although the visual-modal accuracy increases. Appropriately inhibiting visual learning improves cross-modal performance. (b) We also find that this phenomenon widely exists in cross-domain scenarios, where the best-performing zero-shot CLIP model does not necessarily extract the best visual features. (c) In this paper, we explore the phenomenon for an interpretation, finding that fine-tuning involves two learning directions: the visual and the cross-modal ones. We interpret visual learning as a shortcut that reduces the VLM’s classification loss but disrupts cross-modal learning. (d) Based on this, we propose a method to suppress and guide visual learning, enhancing the model’s cross-modal learning.
Figure 2: (a) When sample i and sample k belong to different classes, Δcos(θik) in 5-way 1-shot fine-tuning is always less than 0. (b) At any stage of fine-tuning (each epoch), visual learning can effectively reduce the loss value ( Lvlm ).
Figure 3: Recording the model at each epoch during training and further training these models using visual learning, the target loss value Lvlm can be effectively reduced even further.
Figure 4: We employ three strategies to restrict visual learning during the VLM fine-tuning process and measure the distance of the resulting models from the optimal cross-modal model. We find that visual learning increases the distance from the optimal model, which hinders visual-text alignment.
Figure 5: (a) Gap shift operation [ 38 ] . (b) The model’s initial state is marked by the vertical dashed line. In the cross-domain dataset EuroSAT, CLIP’s modalities are misaligned, meaning that through gap shift, lower loss and higher accuracy can be achieved. (c) Fine-tuning does not effectively re-align the modalities. (d) When visual learning is enhanced ( Lv ), the misalignment becomes more pronounced. (e) When visual learning is suppressed ( Lad ), the misalignment is mitigated.
Figure 6: (a)(b) The Attention maps of the three models. From left to right: the original image, strategy 2 result (trained with Lvlm ), strategy 3 result (trained with Lvlm+Lv ), and strategy 1 result (trained with Lvlm+Lad ). si,j represents the cosine similarity between the image features and the text features. A higher similarity indicates better alignment.
Figure 7: The overall architecture of the model, with the pink region highlighting our proposed method. (a) Firstly, we propose suppressing the visual learning to resist its shortcut during the finetuning, which constrains the model to focus more on the cross-modal alignment. (b) Then, we propose using semantic relationships to gradually align the internal relationships of the visual modality with those of the text modality, thereby enhancing cross-modal alignment.
Figure 8: Average few-shot learning result of 11 datasets.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: (a) When sample i and sample k belong to different classes, Δcos(θik) in 5-way 1-shot fine-tuning is always less than 0, which means that the visual features of samples i and k from different classes will become increasingly dissimilar. (b) Hyperparameter study.
Figure 10: Results for different initial epochs settings.
Figure 11: The attention maps of the three models are shown. From left to right: the original image, the Baseline result (trained with Lvlm ), the result of the model with enhanced visual learning (trained with Lvlm+Lv ), and the result of our method (trained with Lvlm+Lad ). Here, si,j represents the cosine similarity between the image features and the text features. A higher similarity indicates better alignment. It is evident that our model has the most appropriate attention scope and achieves the highest similarity between sample pairs.
Cross-Domain Few-Shot Learning (CDFSL) aims to adapt large-scale pretrained models to specialized target domains with limited samples, yet the few-shot fine-tuning of vision-language models like CLIP remains underexplored. By establishing multiple fine-tuning baselines of CLIP for CDFSL, we find adapter-based methods (e.g., LoRA) consistently outperform prompt-based ones (e.g., MaPLe), contrary to in-domain scenarios. To make those effective in-domain methods competitive again in CDFSL, we analyze this phenomenon and discover LoRA's superiority stems from rectifying the collapsed attention of visual CLS token, enhancing modality alignment and class separation by focusing on text-related visual regions. Further, we find textual EOS token exhibit much better attention to visual samples, and CLIP's standard contrastive loss weakly constrains modality alignment. Based on these insights, we propose Semantic Probe, a plug-and-play attention rectification framework for both adapter- and prompt-based methods. Extensive experiments on four CDFSL benchmarks validate our rationale, achieving state-of-the-art performance and benefiting both fine-tuning paradigms. Codes will be released.
Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) focuses on fine-tuning with limited training data from target domains (e.g., medical or satellite images), where CLIP has recently shown promising results due to its generalizability to downstream tasks. Current works indicate CLIP's text encoder is more suitable for cross-domain tasks, however, we find that removing certain middle layers of the text encoder can effectively improve performance in SF-CDFSL, which we call the Lost Layers. In this paper, we delve into this phenomenon for a deeper understanding. We discover that instead of being harmful for the SF-CDFSL task, the information in these layers is actually beneficial, but visual gaps prevent this useful information from being fully utilized, making these layers seem redundant. Based on this understanding, unlike current works that simply remove these layers, we propose a method to teachs the model to re-utilize information in these lost layers at both the layer and encoder levels, guiding the re-learning of the visual branch under domain shifts. Our approach effectively addresses the issue of underutilized information in the text encoder. Extensive experiments across various settings, backbones (CLIP, SigLip, PE-Core), and tasks (4 CDFSL datasets and 10 Meta-dataset datasets) demonstrate the effectiveness of our method. Code is available at https://github.com/zhenyuZ-HUST/CVPR26-VtT.
Zhenyu Zhang, Guangyao Chen, Yixiong Zou +2
School of Computer Science and Technology, Huazhong University of Science and Technology · National Key Laboratory for Multimedia Information Processing, Peking University
Vision-language models (VLMs) like CLIP have shown impressive generalization capabilities, yet their potential for Cross-Domain Few-Shot Learning (CDFSL) remains underexplored, where the model needs to transfer source-domain information to target domains with scarce training data. While the attention sink phenomenon has been observed in VLMs for certain tasks, its role in CDFSL scenarios has not been studied. In this paper, we uncover a critical issue overlooked by prior works: standard target-domain few-shot fine-tuning in CDFSL significantly exacerbates the attention sink problem, leading to poor discriminability across classes. To understand this phenomenon, through extensive experiments, we interpret it as the model's shortcut learning for domain adaptation: to overcome the huge domain gap between the source and target domains, the model shows a high tendency to push tokens that are initially closer to target-domain classes (i.e., simple tokens) to be even closer to these classes, exacerbating the attention sink and wasting the capability of learning other discriminative but initially further tokens (i.e., hard tokens). To address this, we propose a novel approach to dynamically re-weight tokens according to their relevance with target-domain classes during the target-domain finetuning, which explicitly suppresses the model's reliance on these simple tokens and enhances the learning of hard tokens, reducing sink tokens and enhancing discriminability. Extensive experiments on four benchmark datasets validate the rationale of our method, demonstrating new state-of-the-art performance. Our codes are available at https://github.com/shuaiyi308/TIR.