Language-Aware Information Maximization for Transductive Few-Shot CLIP
Organizations: École de Technologie Supérieure (ÉTS), Montréal, Canada · Université Catholique de Louvain (UCLouvain), Louvain-La-Neuve, Belgium · Université de Mons (UMons), Mons, Belgium
Abstract
Transductive few-shot learning has triggered an abundant literature focusing on vision-only models, but is still at a nascent stage within the recent context of foundational vision-language models (VLMs). Only a few recent methods addressed the problem, pointing to the potential of tranduction in VLMs and to the need for VLM-tailored methods. Building on this momentum, we leverage information-theoretic concepts and recent progress in parameter-efficient fine-tuning (PEFT), developing a highly competitive transductive few-shot CLIP method. Specifically, we introduce a novel Language-aware Information MaximizatiOn (LIMO) loss integrating three complementary terms: (i) the mutual information between the vision inputs and the textual class descriptions; (ii) a Kullback-Leibler (KL) divergence penalizing deviation of the network's probabilistic outputs from the text-driven zero-shot predictions; and (iii) a standard cross-entropy loss based on the labeled shots. Furthermore, we challenge the commonly followed fine-tuning practices in the context of transductive few-shot learning, and explore PEFT strategies, completely overlooked in this context. Surprisingly, we observe substantial boosts in performances, which points to the potential of adapting a subset of the model's parameters in the transductive few-shot setting. We report comprehensive evaluations, which show that LIMO outperforms the very recent transductive few-shot CLIP methods by a large margin and yields significant gains over the best-performing inductive methods. Our code is publicly available at:https://github.com/ghassenbaklouti/LIMO
Figures & tables
| ImageNet | Sun | FGVC | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average | ||||
| Zero-Shot | 66.7 | 62.6 | 24.7 | 47.5 | 65.3 | 86.1 | 89.1 | 71.4 | 92.9 | 43.6 | 66.7 | 65.1 | |||
| 2-shot | Vision-only | TF ( Dhillon et al., 2019 ) | ✓ | 40.5 | 51.6 | 25.3 | 63.1 | 45.1 | 58.8 | 54.8 | 83.2 | 87.0 | 47.3 | 59.4 | 56.0 |
| BD-CSPN ( Liu et al., 2020 ) | ✓ | 46.1 | 56.1 | 26.7 | 64.7 | 50.7 | 67.5 | 64.6 | 89.6 | 89.6 | 48.9 | 64.0 | 60.8 | ||
| LaplacianShot ( Ziko et al., 2020 ) | ✓ | 45.8 | 55.9 | 27.1 | 68.2 | 51.1 | 68.2 | 66.0 | 89.7 | 89.6 | 48.9 | 65.1 | 61.4 | ||
| PT-MAP ( Hu et al., 2021b ) | ✓ | 50.7 | 63.1 | 28.6 | 71.7 | 57.5 | 77.5 | 75.7 | 73.9 | 59.1 | 53.8 | 68.7 | 61.9 | ||
| TIM ( Boudiaf et al., 2020b ) | ✓ | 47.9 | 60.7 | 28.1 | 75.8 | 55.7 | 78.7 | 70.6 | 91.4 | 86.6 | 52.3 | 66.4 | 64.9 | ||
| 2-shot | 4-shot | 16-shot | |||
|---|---|---|---|---|---|
| ViT-B/32 | TIM ( Boudiaf et al., 2020b ) | ✓ | 58.7 | 65.5 | 73.3 |
| CoOp ( Zhou et al., 2022b ) | ✗ | 65.5 | 69.5 | 75.7 | |
| CLIP-LoRA ( Zanella and Ben Ayed, 2024a ) | ✗ | 70.7 | 72.9 | 78.9 | |
| TransCLIP ( Zanella et al., 2024 ) | ✓ | 70 | 71.5 | 74.3 | |
| LIMO (ours) | ✓ | 73.2 | 75.7 | 81.1 | |
| ViT-L/14 | TIM | ✓ | 73.2 | 78 | 83.4 |
| Method | Loss | Sun | FGVC | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LoRA ( Hu et al., 2021a ) | 72.78 | 37.69 | 84.76 | 77.41 | 82.96 | 90.39 | 94.19 | 95.39 | 64.19 | 80.77 | 78.05 | |
| - | 73.69 | 39.72 | 91.83 | 78.11 | 85.89 | 93.98 | 96.78 | 95.73 | 68.3 | 83.87 | 80.79 | |
| + | 73.09 | 36.51 | 56.65 | 77.82 | 84.77 | 90.14 | 91.77 | 95.11 | 54.87 | 79.68 | 74.04 | |
| - + | 73.94 | 39.78 | 92.69 | 78.75 | 86.67 | 94.3 | 96.08 | 95.91 | 67.93 | 84.01 | 81.01 | |
| CoOp ( Zhou et al., 2022b ) | 71.22 | 33.67 | 76.2 | 72.03 | 84.69 | 89.91 | 92.11 | 94.92 | 59.1 | 77.45 | 75.13 | |
| - | 71.11 | 31.69 | 76.98 | 70.87 | 86.05 | 92.61 | 93.12 | 94.97 | 62.96 | 78.87 | 75.92 |
| 0.1 | 1 | 10 | |
|---|---|---|---|
| 77.27 | 77.29 | 77.49 | |
| 73.5 | 77.49 | 71.81 | |
| 77.49 | 70.22 | 64.12 |
| ImageNet | Sun | FGVC | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2-shot | TransCLIP ( Zanella et al., 2024 ) | 70.3 | 70.9 | 30.0 | 77.1 | 71.7 | 87.0 | 91.7 | 90.6 | 93.5 | 55.1 | 78.5 | 74.2 |
| TransCLIP-LoRA | 71.18 | 72.44 | 27.83 | 90.69 | 73.33 | 86.35 | 92.79 | 85.22 | 94.52 | 58.04 | 76.28 | 75.33 | |
| LIMO (Ours) | 71.08 | 72.31 | 33.8 | 91.68 | 74.47 | 86.28 | 93.78 | 92.43 | 94.94 | 63.32 | 81.21 | 77.75 | |
| 4-shot | TransCLIP | 70.3 | 71.9 | 34.0 | 79.4 | 74.0 | 86.4 | 91.6 | 93.6 | 94.0 | 61.1 | 79.1 | 75.9 |
| TransCLIP-LoRA | 71.99 | 73.8 | 33.3 | 90.9 | 77.3 | 86.86 | 93.79 | 88.47 | 95.36 | 60.56 | 77.38 | 77.25 | |
| LIMO (Ours) | 71.94 | 74.08 | 40.3 | 93.1 | 78.61 | 86.59 | 94.26 | 95.99 | 95.81 | 67.57 | 83.97 | 80.2 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| ImageNet | Sun | FGVC | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2-shot | Vision-only | TF ( Dhillon et al., 2019 ) | ✓ | 34.7 | 49.5 | 19.3 | 56.5 | 37.4 | 48.7 | 47.4 | 75.1 | 83.9 | 44.5 | 57.7 | 50.4 |
| BD-CSPN ( Liu et al., 2020 ) | ✓ | 39.2 | 53.1 | 20.7 | 57.2 | 42.1 | 55.5 | 55.2 | 82.4 | 86.8 | 45.6 | 61.6 | 54.5 | ||
| LaplacianShot ( Ziko et al., 2020 ) | ✓ | 39.1 | 53.9 | 20.4 | 58.3 | 42.4 | 57.7 | 57.3 | 82.5 | 86.7 | 45.9 | 62.6 | 55.2 | ||
| PT-MAP ( Hu et al., 2021b ) | ✓ | 42.6 | 60.1 | 22.3 | 63.7 | 46.0 | 63.9 | 64.0 | 69.5 | 55.6 | 50.4 | 66.8 | 55.0 | ||
| TIM ( Boudiaf et al., 2020b ) | ✓ | 41.1 | 59.0 | 21.1 | 68.9 | 44.1 | 66.2 | 60.1 | 86.5 | 81.5 | 48.6 | 68.1 | 58.7 | ||
| Vision-Language | CoOp ( Zhou et al., 2022b ) | ✗ | 61.3 | 63.8 | 18.6 | 62.4 | 62.3 | 73.3 | 85.8 | 82.0 | 91.2 | 49.6 | 70.4 | 65.5 |
| ImageNet | Sun | FGVC | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2-shot | Vision-only | TF ( Dhillon et al., 2019 ) | ✓ | 50.1 | 56.6 | 33.5 | 71.7 | 58.3 | 71.6 | 65.7 | 93.0 | 90.5 | 49.8 | 69.4 | 64.6 |
| BD-CSPN ( Liu et al., 2020 ) | ✓ | 57.0 | 61.2 | 35.6 | 72.6 | 65.1 | 79.9 | 77.2 | 95.7 | 92.8 | 52.3 | 74.7 | 69.5 | ||
| LaplacianShot ( Ziko et al., 2020 ) | ✓ | 56.5 | 61.3 | 35.9 | 76.8 | 65.4 | 80.3 | 77.4 | 96.2 | 93.3 | 52.4 | 74.8 | 70.0 | ||
| PT-MAP ( Hu et al., 2021b ) | ✓ | 61.3 | 68.0 | 37.0 | 78.4 | 68.4 | 87.3 | 86.7 | 77.9 | 61.1 | 56.5 | 75.2 | 68.9 | ||
| TIM ( Boudiaf et al., 2020b ) | ✓ | 59.7 | 67.6 | 35.4 | 82.2 | 69.3 | 87.4 | 85.5 | 95.1 | 91.4 | 53.2 | 78.6 | 73.2 | ||
| Vision-Language | CoOp ( Zhou et al., 2022b ) | ✗ | 73.1 | 69.3 | 38.8 | 72.7 | 81.2 | 89.4 | 93.5 | 93.5 | 95.0 | 59.2 | 80.0 | 76.9 |