Language-Aware Information Maximization for Transductive Few-Shot CLIP
Authors: Ghassen Baklouti, Maxime Zanella, Ismail Ben Ayed
Organizations: École de Technologie Supérieure (ÉTS), Montréal, Canada · Université Catholique de Louvain (UCLouvain), Louvain-La-Neuve, Belgium · Université de Mons (UMons), Mons, Belgium
Transductive few-shot learning has triggered an abundant literature focusing on vision-only models, but is still at a nascent stage within the recent context of foundational vision-language models (VLMs). Only a few recent methods addressed the problem, pointing to the potential of tranduction in VLMs and to the need for VLM-tailored methods. Building on this momentum, we leverage information-theoretic concepts and recent progress in parameter-efficient fine-tuning (PEFT), developing a highly competitive transductive few-shot CLIP method. Specifically, we introduce a novel Language-aware Information MaximizatiOn (LIMO) loss integrating three complementary terms: (i) the mutual information between the vision inputs and the textual class descriptions; (ii) a Kullback-Leibler (KL) divergence penalizing deviation of the network's probabilistic outputs from the text-driven zero-shot predictions; and (iii) a standard cross-entropy loss based on the labeled shots. Furthermore, we challenge the commonly followed fine-tuning practices in the context of transductive few-shot learning, and explore PEFT strategies, completely overlooked in this context. Surprisingly, we observe substantial boosts in performances, which points to the potential of adapting a subset of the model's parameters in the transductive few-shot setting. We report comprehensive evaluations, which show that LIMO outperforms the very recent transductive few-shot CLIP methods by a large margin and yields significant gains over the best-performing inductive methods. Our code is publicly available at:https://github.com/ghassenbaklouti/LIMO
Figures & tables
Figure 1: The reported performance is the average accuracy over the 11 datasets studied in this paper. LIMO outperforms a recent transductive VLM method (TransCLIP ( Zanella et al., 2024 ) ) and a related vision-only method (TIM ( Boudiaf et al., 2020b ) ) by large margins. It also demonstrates a significant gain when compared to a related inductive method (CLIP-LoRA ( Zanella and Ben Ayed, 2024a ) ).
ImageNet
Sun
FGVC
EuroSAT
Cars
Food
Pets
Flowers
Caltech
DTD
UCF
Average
Zero-Shot
66.7
62.6
24.7
47.5
65.3
86.1
89.1
71.4
92.9
43.6
66.7
65.1
2-shot
Vision-only
TF ( Dhillon et al., 2019 )
✓
40.5
51.6
25.3
63.1
45.1
58.8
54.8
83.2
87.0
47.3
59.4
56.0
BD-CSPN ( Liu et al., 2020 )
✓
46.1
56.1
26.7
64.7
50.7
67.5
64.6
89.6
89.6
48.9
64.0
60.8
LaplacianShot ( Ziko et al., 2020 )
✓
45.8
55.9
27.1
68.2
51.1
68.2
66.0
89.7
89.6
48.9
65.1
61.4
PT-MAP ( Hu et al., 2021b )
✓
50.7
63.1
28.6
71.7
57.5
77.5
75.7
73.9
59.1
53.8
68.7
61.9
TIM ( Boudiaf et al., 2020b )
✓
47.9
60.7
28.1
75.8
55.7
78.7
70.6
91.4
86.6
52.3
66.4
64.9
Table 1: Comparison of state-of-the-art methods in few-shot learning : Top-1 classification accuracy on 11 datasets. Bolded values indicate highest accuracy. ✓ denotes transductive methods, ✗ denotes non-transductive methods.
Figure 2: Statistical comparison of LIMO, TransCLIP ( Zanella et al., 2024 ) , and CLIP-LoRA ( Zanella and Ben Ayed, 2024a ) under the 2-, 4- and 16-shot settings. Error bars represent the standard deviation averaged across four datasets (FGVC, EuroSAT, Caltech, and DTD) using 10 seeds.
2-shot
4-shot
16-shot
ViT-B/32
TIM ( Boudiaf et al., 2020b )
✓
58.7
65.5
73.3
CoOp ( Zhou et al., 2022b )
✗
65.5
69.5
75.7
CLIP-LoRA ( Zanella and Ben Ayed, 2024a )
✗
70.7
72.9
78.9
TransCLIP ( Zanella et al., 2024 )
✓
70
71.5
74.3
LIMO (ours)
✓
73.2
75.7
81.1
ViT-L/14
TIM
✓
73.2
78
83.4
Table 2: Average Top-1 accuracy over 11 datasets for the ViT-B/32 and ViT-L/14 visual backbones . Bolded values indicate highest accuracy. ✓ denotes transductive methods, ✗ denotes non-transductive methods.
Method
Loss
Sun
FGVC
EuroSAT
Cars
Food
Pets
Flowers
Caltech
DTD
UCF
Average
LoRA ( Hu et al., 2021a )
C(θv,θt)
72.78
37.69
84.76
77.41
82.96
90.39
94.19
95.39
64.19
80.77
78.05
C(θv,θt) - I(θv,θt)
73.69
39.72
91.83
78.11
85.89
93.98
96.78
95.73
68.3
83.87
80.79
C(θv,θt) + T(θv,θt)
73.09
36.51
56.65
77.82
84.77
90.14
91.77
95.11
54.87
79.68
74.04
C(θv,θt) - I(θv,θt) + T(θv,θt)
73.94
39.78
92.69
78.75
86.67
94.3
96.08
95.91
67.93
84.01
81.01
CoOp ( Zhou et al., 2022b )
C(θv,θt)
71.22
33.67
76.2
72.03
84.69
89.91
92.11
94.92
59.1
77.45
75.13
C(θv,θt) - I(θv,θt)
71.11
31.69
76.98
70.87
86.05
92.61
93.12
94.97
62.96
78.87
75.92
Table 3: Ablation study on the contribution of each term of our proposed loss function LIMO under different fine-tuning strategies: LoRA (Low-Rank Adaptation), LVP (fine-tuning only the Last Visual Projector of CLIP), and CoOp (learning only the textual prompts). We report 4-shot results as top-1 accuracy and the average top-1 accuracy across 10 datasets (excluding ImageNet).
0.1
1
10
λent
77.27
77.29
77.49
λcond
73.5
77.49
71.81
λtext
77.49
70.22
64.12
Table 4: Ablations on hyper-parameters: Average top-1 accuracy across 7 datasets {DTD, EuroSAT, Food, UCF, FGVC, Caltech and Flowers} using CLIP ViT-B/16 under a 2-shot setting. Each hyperparameter is studied individually, while keeping other parameters fixed as in the main configuration.
ImageNet
Sun
FGVC
EuroSAT
Cars
Food
Pets
Flowers
Caltech
DTD
UCF
Average
2-shot
TransCLIP ( Zanella et al., 2024 )
70.3
70.9
30.0
77.1
71.7
87.0
91.7
90.6
93.5
55.1
78.5
74.2
TransCLIP-LoRA
71.18
72.44
27.83
90.69
73.33
86.35
92.79
85.22
94.52
58.04
76.28
75.33
LIMO (Ours)
71.08
72.31
33.8
91.68
74.47
86.28
93.78
92.43
94.94
63.32
81.21
77.75
4-shot
TransCLIP
70.3
71.9
34.0
79.4
74.0
86.4
91.6
93.6
94.0
61.1
79.1
75.9
TransCLIP-LoRA
71.99
73.8
33.3
90.9
77.3
86.86
93.79
88.47
95.36
60.56
77.38
77.25
LIMO (Ours)
71.94
74.08
40.3
93.1
78.61
86.59
94.26
95.99
95.81
67.57
83.97
80.2
Table 5: Comparing LIMO with a TransCLIP variant that fine-tunes CLIP’s text and vision encoders using the TransCLIP objective and LoRA : Top-1 classification accuracy is reported over 11 datasets. The highest accuracy values are highlighted in bold , and the second highest are underlined .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
ImageNet
Sun
FGVC
EuroSAT
Cars
Food
Pets
Flowers
Caltech
DTD
UCF
Average
2-shot
Vision-only
TF ( Dhillon et al., 2019 )
✓
34.7
49.5
19.3
56.5
37.4
48.7
47.4
75.1
83.9
44.5
57.7
50.4
BD-CSPN ( Liu et al., 2020 )
✓
39.2
53.1
20.7
57.2
42.1
55.5
55.2
82.4
86.8
45.6
61.6
54.5
LaplacianShot ( Ziko et al., 2020 )
✓
39.1
53.9
20.4
58.3
42.4
57.7
57.3
82.5
86.7
45.9
62.6
55.2
PT-MAP ( Hu et al., 2021b )
✓
42.6
60.1
22.3
63.7
46.0
63.9
64.0
69.5
55.6
50.4
66.8
55.0
TIM ( Boudiaf et al., 2020b )
✓
41.1
59.0
21.1
68.9
44.1
66.2
60.1
86.5
81.5
48.6
68.1
58.7
Vision-Language
CoOp ( Zhou et al., 2022b )
✗
61.3
63.8
18.6
62.4
62.3
73.3
85.8
82.0
91.2
49.6
70.4
65.5
Appendix
Table 6: Detailed results of state-of-the-art methods in the few-shot setting for the 11 datasets with the ViT-B/32 as visual backbone : Top-1 classification accuracy averaged over 3 random seeds is reported. Bolded values indicate highest accuracy. ✓denotes transductive methods, ✗denotes non-transductive methods.
ImageNet
Sun
FGVC
EuroSAT
Cars
Food
Pets
Flowers
Caltech
DTD
UCF
Average
2-shot
Vision-only
TF ( Dhillon et al., 2019 )
✓
50.1
56.6
33.5
71.7
58.3
71.6
65.7
93.0
90.5
49.8
69.4
64.6
BD-CSPN ( Liu et al., 2020 )
✓
57.0
61.2
35.6
72.6
65.1
79.9
77.2
95.7
92.8
52.3
74.7
69.5
LaplacianShot ( Ziko et al., 2020 )
✓
56.5
61.3
35.9
76.8
65.4
80.3
77.4
96.2
93.3
52.4
74.8
70.0
PT-MAP ( Hu et al., 2021b )
✓
61.3
68.0
37.0
78.4
68.4
87.3
86.7
77.9
61.1
56.5
75.2
68.9
TIM ( Boudiaf et al., 2020b )
✓
59.7
67.6
35.4
82.2
69.3
87.4
85.5
95.1
91.4
53.2
78.6
73.2
Vision-Language
CoOp ( Zhou et al., 2022b )
✗
73.1
69.3
38.8
72.7
81.2
89.4
93.5
93.5
95.0
59.2
80.0
76.9
Appendix
Table 7: Detailed results of state-of-the-art methods in the few-shot setting for the 11 datasets with the ViT-L/14 as visual backbone : Top-1 classification accuracy averaged over 3 random seeds is reported. Bolded values indicate highest accuracy. ✓denotes transductive methods, ✗denotes non-transductive methods.
Current Large Vision Language Models (LVLMs) excel at many zero-shot tasks like image captioning, visual question answering and OCR. However, these same models suffer from poor performance at image classification tasks, underperforming against CLIP-based methods. Notably, this gap is surprising because many LVLMs use CLIP-pretrained vision encoders. Yet LVLMs are not inherently limited by CLIP's architecture with independent vision and text encoders. In CLIP, this separation biases classification toward class-name matching rather than joint visual-text reasoning. In this paper we show that, despite their poor raw performance, LVLMs can improve visual feature class separability at inference using prompt conditioning, and LVLMs' internal representations, especially attention heads, can outperform the model itself at zero-shot and few-shot classification. We introduce Head Ensemble Classifiers (HEC) to bridge the performance gap between CLIP-based and LVLM-based classification methods. Inspired by Gaussian Discriminant Analysis, HEC ranks the most discriminative vision and text heads and combines them into a training-free classifier. We show that HEC achieves state-of-the-art performance in few-shot and zero-shot classification across 12 datasets.
Adhemar de Senneville, Xavier Bou, Jérémy Anger +2
Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · École Polytechnique, AMIAD, France · Institut Universitaire de France
Training-free few-shot adaptation of vision-language models typically constructs a classifier from CLIP's final projected image representation, leaving intermediate representations unused. Combining them is not straightforward. Class scores from different representations differ in spread by orders of magnitude for the same query. Direct averaging allocates influence by numerical scale rather than predictive information. We propose Layer-Ensemble Gaussian Discriminant Analysis (LE-GDA), a method that fits regularized shared-covariance discriminants independently to intermediate and final representations extracted in one frozen encoder pass. LE-GDA standardizes each discriminant's class scores per query, selects a small representation multiset on labeled validation data, and combines it with the zero-shot classifier without gradient-based optimization. Across 11 datasets, four CLIP backbones, five shot counts, and three independently sampled K-shot training sets, LE-GDA achieves the highest mean accuracy of all evaluated methods on every backbone, raising the 16-shot mean from 78.01% to 79.54% over a matched GDA-CLIP (final-feature) baseline and improving that baseline for every backbone-shot combination. At one shot, LE-GDA retains a 1.20-point mean gain over the baseline even though covariance estimation is unavailable and per-query standardization is neutral. A control with richer per-class structure inside a single representation does not reproduce the gain, indicating that the benefit comes from combining representations. These results show that intermediate CLIP representations carry information beyond the final embedding and can improve training-free few-shot adaptation. Code and evaluation resources are available at https://github.com/pavodi-NM/le-gda.
Vision-Language Models (VLMs) such as CLIP demonstrate strong zero-shot generalization, but their performance significantly degrades in cross-domain scenarios with scarce target-domain training data (Cross-Domain Few-Shot Learning, CDFSL). In this paper, we focus on the target-domain few-shot finetuning in the CLIP-based CDFSL task. Prevailing finetuning paradigms uniformly align all image patch tokens with their corresponding textual embeddings. However, we find a counterintuitive phenomenon: actively pushing away certain low-similarity image tokens, termed "tail tokens", from their textual embeddings consistently improves target-domain performance. We delve into this phenomenon and provide a novel interpretation: under great domain shifts and scarce training data, the model can hardly extract semantic information from visual inputs; therefore, the common belief of alignment is valid only for tokens already containing sufficient semantic information; for tail tokens, forcing the alignment would lead to excessive overfitting to the scarce training, while breaking the alignment is more useful. Motivated by this, we propose Adaptive Tail-Head Alignment (ATHA), a novel fine-tuning strategy for CLIP that transforms the conventional uniform alignment paradigm to an adaptive alignment paradigm, with both alignment strengthening and weakening. Extensive experiments on four challenging CDFSL benchmarks validate our state-of-the-art performance. Our code is available at https://github.com/shuaiyi308/ATHA.
Shuai Yi, Yixiong Zou, Yuhua Li +1
Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.