Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learning

Zhenyu Zhang; Guangyao Chen; Yixiong Zou; Yuhua Li; Ruixuan Li

cs.AIMar 5, 2026

Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learning

Authors: Zhenyu Zhang, Guangyao Chen, Yixiong Zou, Yuhua Li, Ruixuan Li

Organizations: School of Computer Science and Technology, Huazhong University of Science and Technology · National Key Laboratory for Multimedia Information Processing, Peking University

Abstract

Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) focuses on fine-tuning with limited training data from target domains (e.g., medical or satellite images), where CLIP has recently shown promising results due to its generalizability to downstream tasks. Current works indicate CLIP's text encoder is more suitable for cross-domain tasks, however, we find that removing certain middle layers of the text encoder can effectively improve performance in SF-CDFSL, which we call the Lost Layers. In this paper, we delve into this phenomenon for a deeper understanding. We discover that instead of being harmful for the SF-CDFSL task, the information in these layers is actually beneficial, but visual gaps prevent this useful information from being fully utilized, making these layers seem redundant. Based on this understanding, unlike current works that simply remove these layers, we propose a method to teachs the model to re-utilize information in these lost layers at both the layer and encoder levels, guiding the re-learning of the visual branch under domain shifts. Our approach effectively addresses the issue of underutilized information in the text encoder. Extensive experiments across various settings, backbones (CLIP, SigLip, PE-Core), and tasks (4 CDFSL datasets and 10 Meta-dataset datasets) demonstrate the effectiveness of our method. Code is available at https://github.com/zhenyuZ-HUST/CVPR26-VtT.

Figures & tables

Panel of Figure 1 — Figure 1: (a) CLIP has two branches: a visual encoder and a text encoder. However, we find that removing certain layers of the text encoder can significantly enhance its performance in SF-CDFSL tasks. (b) Performance of 5-way 1-shot fine-tuned CLIP after removing the i-th layer (x-axis) of the text encoder. The horizontal dashed line represents the performance achieved using the full text encoder. Masking certain layers results in better performance. (c) After applying our method, the optimal performance is achieved using the full text encoder (dashed line), indicating that the lost layer no longer exists.

Panel of Figure 2 — Figure 2: In SF-CDFSL tasks, (a) the lost layer is commonly present in various CLIP structures (Improvement: increased performance achieved by masking a specific layer of the text encoder), and (b) different fine-tuning methods do not effectively address this issue. Please refer to the Appendix for more detailed results. (c) Two strategies for leveraging the lost layer: Remove - eliminating the layer; Emphasize - enhancing the output of the layer using a residual approach in the final output. (d) In the source domain (ImageNet), the performance using the full text encoder (blue dashed line) is consistently optimal, thus no lost layer exists. However, after a change in the visual domain (ImageNet-R), masking the 7th layer of the text encoder significantly improves performance, indicating the reappearance of the lost layer.

Figure 3: (a) The overall architecture of the VtT model. First, the V-T Fusion module integrates visual and textual features at the layer level (yellow lines). Then, the TIA module absorbs information from the text encoder at the encoder level (pink lines). Finally, the DGSO module optimizes the model using the outputs from the previous modules and gradient information (orange lines). (b) The V-T Fusion module interleaves the outputs of the visual and text encoders from deep to shallow layers and integrates them using the SSM network. (c) The DGSO module removes gradients that conflict with the main task (classification) before optimizing the model. It also determines when to stop using the VtT model based on the extent of gradient conflicts.

Figure 4: The 5-way 5-shot results on 10 Meta-datasts [ 50 ] , see Appendix for the 5-way 1-shot and detailed results.

Panel of Figure 5 — Figure 5: The attention maps. From left to right: the original image, the baseline result, baseline + remove (see Figure 2(c) ) result, and the result of ours. Black boxes highlight areas of incorrect attention, while white boxes highlight missing attention. $Sim$ represents the cosine similarity between the image features and the text features. A higher similarity indicates better alignment.

Panel of Figure 6 — Figure 6: Hyperparameter experiments for $\beta$ and $\lambda$ . The vertical axis represents the average performance on 5-way 1-shot tasks across four datasets.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: (a) The attention distribution across different layers of the vision encoder. (b) The red line quantitatively shows the attention weight percentage of the top 10 tokens with the highest attention weights in each layer, while the blue line shows the similarity of features extracted by different layers of the vision encoder for visual samples from different domains but of the same category. A higher similarity indicates more domain-independent features. (c) Visualization of the attention weight distribution across different layers of the text encoder. (d) The red line quantitatively shows the attention weight percentage of the category name token in each layer of the text encoder, while the blue line shows the similarity of features extracted by different layers of the text encoder for textual descriptions with different domain information but containing the same category name. A higher similarity indicates more domain-independent features. (e) Domain distance between visual features and text features extracted by the CLIP model when transferred from the ImageNet dataset (source domain) to downstream cross-domain datasets. (f) Cross-domain transfer experiments of visual and textual features from different domains.

Panel of Figure 8 — Figure 8: (a) In the source domain (ImageNet), performance on sampled sub-tasks containing 100 different categories shows that optimal performance is almost always achieved using the complete text encoder, suggesting that the category composition is not the cause of the shortcut layer. (b) Cosine similarity between the visual features and the corresponding textual features extracted by the model before and after applying our method.

Panel of Figure 9 — Figure 9: t-SNE [ 38 ] visualization results for the ImageNet-R [ 20 ] (5 classes: glodfish, hammerhead shark, tiger shark, stingray and hen). Different colors denote samples from different domain, and different shapes indicate samples of different classes.

Panel of Figure 10 — Figure 10: The attention maps of the three models. From left to right: the original image, the baseline result, baseline + remove (see Figure 2(c) in the main text) result, and the result of ours. Black boxes highlight areas of incorrect attention, while white boxes highlight areas of missing attention. $Sim$ represents the cosine similarity between the image features and the text features. A higher similarity indicates better alignment.

Explore similar work

Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learning
Mar 7, 2026Zhenyu Zhang, Yixiong Zou, Yuhua Li +2Few-Shot Learning VLM Adaptation
School of Computer Science and Technology, Huazhong University of Science and Technology · National Key Laboratory for Multimedia Information Processing, Peking University
Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning
May 12, 2026Yaze Zhao, Yicong Liu, Yixiong Zou +2Source-Free Domain Adaptation Few-Shot Domain Adaptation
Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment
Mar 18, 2026Yaze Zhao, Yixiong Zou, Yuhua Li +1Few-Shot Learning Few-Shot Domain Adaptation
School of Computer Science and Technology, Huazhong University of Science and Technology

Mar 7, 2026J/K move · Enter open · S save

Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learning

Zhenyu Zhang, Yixiong Zou, Yuhua Li, Ruixuan Li+1

School of Computer Science and Technology, Huazhong University of Science and Technology · National Key Laboratory for Multimedia Information Processing, Peking University

Few-Shot Learning VLM Adaptation Few-Shot Domain Adaptation Vision-Language Alignment Source-Free Domain Adaptation

Open paper AskPDF arXiv

Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) focuses on fine-tuning with limited training data from target domains (e.g., medical or satellite images), where Vision-Language Models (VLMs) such as CLIP and SigLIP have shown promising results. Current works in traditional visual models suggest that improving visual discriminability enhances performance. However, in VLM-based SF-CDFSL tasks, we find that strengthening visual-modal discriminability actually suppresses VLMs' performance. In this paper, we aim to delve into this phenomenon for an interpretation and a solution. By both theoretical and experimental proofs, our study reveals that fine-tuning with the typical cross-entropy loss (

\mathcal{L}_{\mathrm{vlm}}

) inherently includes a visual learning part and a cross-modal learning part, where the cross-modal part is crucial for rectifying the heavily disrupted modality misalignment in SF-CDFSL. However, we find that the visual learning essentially acts as a shortcut that encourages the model to reduce

\mathcal{L}_{\mathrm{vlm}}

without considering the cross-modal part, therefore hindering the cross-modal alignment and harming the performance. Based on this interpretation, we further propose an approach to address this problem: first, we perturb the visual learning to guide the model to focus on the cross-modal alignment. Then, we use the visual-text semantic relationships to gradually align the visual and textual modalities during the fine-tuning. Extensive experiments on various settings, backbones (CLIP, SigLip, PE-Core), and tasks (4 CDFSL datasets and 11 FSL datasets) show that we consistently set new state-of-the-art results. Code is available at https://github.com/zhenyuZ-HUST/CVPR26-Mind-the-Discriminability-Trap.

Abstract

Explore similar work

Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learning

Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning

Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment