Recent adapter-based CLIP tuning (e.g., Tip-Adapter) is a strong few-shot learner, achieving efficiency by caching support features for fast prototype matching. However, these methods rely on global uni-modal feature vectors, overlooking fine-grained patch relations and their structural alignment with class text. To bridge this gap without incurring inference costs, we introduce a novel asymmetric training-only framework. Instead of altering the lightweight adapter, we construct a high-capacity auxiliary Heterogeneous Graph Teacher that operates solely during training. This teacher (i) integrates multi-scale visual patches and text prompts into a unified graph, (ii) performs deep cross-modal reasoning via a Modality-aware Graph Transformer (MGT), and (iii) applies discriminative node filtering to extract high-fidelity class features. Crucially, we employ a cache-aware dual-objective strategy to supervise this relational knowledge directly into the Tip-Adapter's key-value cache, effectively upgrading the prototypes while the graph teacher is discarded at test time. Thus, inference remains identical to Tip-Adapter with zero extra latency or memory. Across standard 1-16-shot benchmarks, our method consistently establishes a new state-of-the-art. Ablations confirm that the auxiliary graph supervision, text-guided reasoning, and node filtering are the essential ingredients for robust few-shot adaptation. Code is available at https://github.com/MR-Sherif/TOGA.git.
Figures & tables
Figure 1: The core trade-off of VLM adaptation and our proposed solution. (a) Lightweight adapters (e.g., Tip-Adapter [ 43 ] ) are fast but limited, as they only reason over the single, global image feature. (b) Heavyweight adapters (e.g., GraphAdapter [ 22 ] ) are more powerful, reasoning over visual patches, but their complex GNN adds a permanent, significant cost at inference time. (c) Our proposed method , TOGA (the best of both), uses a powerful auxiliary teacher. This teacher, our modality-aware graph transformer (MGT), performs deep bi-modal reasoning on a unified graph of patches and text. It supervises its knowledge into the lightweight “student” adapter, which is the only component used at inference, achieving high performance with the efficiency of (a).
Figure 2: Our Training-Only Graph Adapter ( TOGA ) is an asymmetric supervision pipeline. At train time (red dotted region and → denote training only), we use a three-branch ensemble: (1) a frozen LZS branch, (2) a lightweight student (Cache Model) LCache , and (3) our powerful, auxiliary graph teacher LGraph . The teacher enriches multi-scale patches and text embeddings via unimodal Transformers, then constructs a modality-aware graph transformer to perform cross-modal reasoning. A dual-loss objective regularizes the teacher’s knowledge into the student’s adapter A . At test time , the entire teacher branch is discarded, resulting in zero additional inference cost.
Figure 3
Method
ImageNet
-R
-A
-V2
-Sketch
Avg
CLIP [ 27 ]
66.7
74.0
47.8
60.8
46.2
59.1
CoOp [ 47 ]
71.5
75.2
49.7
64.2
48.0
61.1
CoCoOp [ 46 ]
71.0
76.2
50.6
64.1
48.8
62.1
MaPLe [ 19 ]
70.7
77.0
50.9
64.1
49.2
62.4
ProGrad [ 48 ]
72.2
74.6
49.4
64.7
47.6
61.7
KgCoOp [ 37 ]
71.2
76.7
50.7
64.1
49.0
62.3
Table 2: Out-of-distribution (OOD) robustness accuracy (%) on ImageNet and its shifted variants; Avg: mean across five datasets.
Loss Objective
1 Shot
2 Shot
4 Shot
8 Shot
16 Shot
Euro
LCE
65.1
73.1
77.9
83.2
88.1
LCE+LCEGraph
67.1
74.5
79.3
82.2
87.5
LCE+LFocalGraph
67.4
74.9
80.3
84.1
89.4
Cal
LCE
94.0
94.3
95.4
95.5
95.8
LCE+LCEGraph
94.0
94.6
95.4
95.5
96.1
LCE+LFocalGraph
94.3
95.0
95.7
95.8
96.3
Table 3: Ablation of the dual-objective co-training strategy. LCE denotes the cross-entropy loss on the final logits. LCEGraph and LFocalGraph are the auxiliary losses applied to the teacher. Dataset abbreviation follows the same as table 1 .
Pooling Strategy
1 Shot
2 Shot
4 Shot
8 Shot
16 Shot
EuroSAT
∑All (Global)
63.4
72.2
79.5
83.4
88.7
∑Top-N=25
65.3
73.3
79.9
83.7
88.9
∑Top-N=50
67.4
74.9
80.3
84.1
89.4
∑Top-N=75
64.9
73.0
79.6
83.4
88.8
Caltech
∑All (Global)
94.1
94.6
95.5
95.6
96.2
∑Top-N=25
94.2
94.8
95.6
95.8
96.1
Table 4: Global pooling ( ∑All ) vs Top- N filtering ( ∑Top-N ). The Top- N strategy explicitly filters out non-discriminative patches.
Figure 4: Qualitative visualization of Discriminative Node Filtering. Our Top- N filtering learns to retain high-scoring, discriminative foreground patches (Green) while suppressing non-informative background nodes (Blue). This resolves feature dilution inherent in global pooling, yielding a cleaner signal by focusing on key object parts (e.g., ant’s head, flower pistil, cat’s eyes). Samples from Caltech-101; more visualizations in supplementary.
Patch Strategy
1 Shot
2 Shot
4 Shot
8 Shot
16 Shot
Euro
Grid 2×2
65.0
72.9
80.0
83.7
88.8
Grid 3×3
61.7
72.5
78.8
83.6
88.8
Grid 4×4
66.6
74.5
77.9
83.7
88.9
MultiScale
67.4
74.9
80.3
84.1
89.4
Cal
Grid 2×2
93.7
94.5
95.5
95.6
96.2
Grid 3×3
93.8
94.6
95.6
95.6
96.0
Table 5: Ablation of patch grid granularity. Comparison of fixed-scale grids vs MultiScale strategy (18 patches).
Configuration
1 Shot
2 Shot
4 Shot
8 Shot
16 Shot
Euro
w/o MGT
61.9
71.5
77.2
80.3
85.7
w/ MGT
67.4
74.9
80.3
84.1
89.4
Cal
w/o MGT
94.0
94.6
95.4
95.2
95.8
w/ MGT
94.3
95.0
95.7
95.8
96.3
Food
w/o MGT
86.0
86.3
86.5
86.4
87.2
w/ MGT
86.2
86.5
86.7
86.8
87.5
Table 6: Impact of Modality-aware Graph Transformer (MGT).
T
M
F
P
1-shot
2-shot
4-shot
8-shot
16-shot
EuroSAT
✓
✓
✓
✓
67.4
74.9
80.3
84.1
89.4
✓
✓
✓
✗
64.1
71.3
76.8
81.0
86.7
✓
✓
✗
✓
63.4
72.2
79.5
83.4
86.5
✓
✗
✓
✓
61.9
71.5
77.2
80.3
85.7
✗
✓
✓
✓
66.2
72.6
78.9
81.8
87.4
Aircraft
✓
✓
✓
✓
31.0
34.8
38.3
44.2
48.4
Table 7: Internal teacher ablation. T: unimodal Transformer, M: MGT, F: TopK filtering, P: patch–text edges. Cross-modal graph reasoning is the primary contributor ( M + P ), while T and F offer smaller complementary gains.
Figure S1 : Performance trajectories of different adapter-based architectures. The curves show the few-shot accuracy of the model on several additional datasets, complementing the main results presented in the paper.
Shots
Method
Venue
INet
SUN
Air
Euro
Cars
Food
Pets
Flow
Cal
DTD
UCF
Avg
0
CLIP [ 27 ]
ICML’22
66.7
62.6
24.7
47.5
65.3
86.1
89.1
71.4
92.9
43.6
66.7
65.1
1
CoOp [ 47 ]
IJCV’22
68.0
67.3
26.2
50.9
67.1
82.6
90.3
72.7
93.2
50.1
70.7
67.2
CoOp [ 47 ]
IJCV’22
65.7
67.0
20.8
56.4
67.5
84.3
90.2
78.3
92.5
50.1
71.2
67.6
CoCoOp [ 46 ]
CVPR’22
69.4
68.7
28.1
55.4
67.6
84.9
91.9
73.4
94.1
52.6
70.4
68.8
PLOT++ [ 3 ]
ICLR’23
66.4
66.7
28.6
65.4
68.8
86.1
91.8
80.4
94.3
54.5
74.3
70.7
KgCoOp [ 37 ]
CVPR’23
68.9
68.4
26.8
61.9
66.7
86.4
92.1
74.7
94.2
52.7
72.8
69.6
Table S1 : Comparison of few-shot classification accuracy (%) on 11 benchmark datasets. We evaluate our method against several SOTA prompt-based approaches. The best performance in each shot-group is marked in bold . Our results are highlighted in gray. Dataset abbreviations: INet (ImageNet), SUN (SUN397), Air (Aircraft), Euro (EuroSAT), Cars (Stanford Cars), Food (Food101), Pets (OxfordPets), Flow (Flowers102), Cal (Caltech101), DTD (Describable Textures), UCF (UCF101).
Figure S2 : t-SNE visualization of logits on the Caltech101 dataset ( 16 -shot). The plots depict the separation of 8 object categories, ranging from distinct objects like stop sign to complex shapes like windsor chair. (a) Baseline CLIP provides a strong initial separation ( 0.618 ). (b) Tip-Adapter-F introduces minor variance, slightly reducing the score to 0.592 . (c) Ours further refines the manifold, tightening the clusters for difficult classes (e.g., crocodile and brontosaurus) and achieving the highest distinctiveness with a Silhouette score of 0.693 . This highlights the benefit of our discriminative node filtering in suppressing background noise.
Figure S3 : This visualization highlights the model’s ability to distinguish between fine-grained culinary categories. (a) Baseline CLIP and (b) Tip-Adapter-F show comparable performance ( ≈0.58 ), with noticeable overlap between visually similar classes (e.g., macarons and cup cakes). (c) Ours (ViT-B/16) significantly enhances class purity, leading to superior clustering of features and raising the Silhouette score to 0.732 . This confirms that our MGT-guided distillation effectively transfers local ingredient-level details into the student cache.
Figure S4 : t-SNE visualization of logits on the EuroSAT dataset ( 16 -shot). We visualize the embedding space for 8 randomly selected categories. (a) Baseline CLIP fails to form distinct clusters for texture-heavy classes, resulting in a negative Silhouette score of −0.043 . (b) Tip-Adapter-F improves separability ( 0.211 ) but retains ambiguity between vegetation classes. (c) Ours demonstrates that the asymmetric graph supervision effectively disentangles complex spectral signatures, achieving a Silhouette score of 0.522 and showing clear separation between semantically adjacent classes like Pasture Land and Herbaceous Vegetation Land.
Figure S5 : Extended Qualitative Visualization of Discriminative Node Filtering. This figure provides additional samples demonstrating the efficacy of our Top-N filtering across various Caltech-101 classes. The learned filter successfully identifies and retains high-scoring, discriminative foreground patches (Green) covering key object features (e.g., the segmented body of the scorpion, the silhouette of the scissors, and the spike pattern of the stegosaurus). Concurrently, it suppresses non-informative background regions (Blue). This mechanism ensures that the GNN Teacher’s final visual feature is robust against feature dilution, yielding a high-fidelity relational signal for cache supervision.
Shots
Backbone
Method
Datasets
SUN
Air
Euro
Cars
Food
Pets
Flowers
Calt
DTD
UCF
1
ResNet-101
Base
60.9
19.9
38.7
64.2
80.6
87.1
64.8
91.0
43.4
63.4
Ours
63.5
22.7
62.1
66.1
81.0
87.8
81.1
91.9
50.5
68.0
ViT-B/32
Base
63.8
20.8
49.9
61.1
80.3
87.2
67.8
91.5
48.2
65.9
Ours
66.2
24.3
60.8
62.9
81.0
88.5
84.5
92.6
52.5
70.2
ViT-B/16
Base
65.0
25.9
53.4
67.1
85.0
89.2
69.7
93.6
47.1
70.9
Table S2 : Comparison of few-shot classification accuracy (%). The table compares the baseline Tip-Adapter-F (denoted as Base ) against the Proposed method (denoted as Ours ) across various CLIP backbones. Distinct blocks represent different shot settings ( K={1,2,4,8} ). Best results are highlighted in bold .
Figure S6 : Accuracy vs. Efficiency Trade-off (8-Shot Average). We compare the total inference cost (GFLOPs) against average few-shot accuracy across 10 benchmark datasets. Our method demonstrates strong scaling efficiency : while the transition from ViT-B/16 to ViT-L/14 incurs higher computational cost (due to the larger backbone resolution and depth), it yields a massive +5.1% performance gain , reaching a state-of-the-art average accuracy of 85.9%. This highlights that our GNN-based adaptation module effectively leverages the capacity of stronger backbones without hitting a saturation point.
Backbone
Res.
Dim
Backbone
GNN (Ours)
Cache
GFLOPs
Param
GFLOPs
Param
GFLOPs
Param
RN101
224
512
19.6
56.3
4.5
13.5
0.02
8.2
ViT-B/32
224
512
8.7
87.9
4.5
13.5
0.02
8.2
ViT-B/16
224
512
33.7
86.2
4.5
13.5
0.02
8.2
ViT-L/14
224
768
155.7
304.0
10.2
30.4
0.03
12.3
Table S3 : Efficiency Analysis. We report GFLOPs (billions) and Parameters (millions) for the Backbone, our GNN module, and the Cache Adapter. Note that our GNN introduces marginal overhead compared to the backbone.
Figure S7 : Efficiency Profile: Inference vs. Training. Left: Breakdown of inference computational cost (GFLOPs). Our method (GNN + Cache) introduces only marginal overhead compared to the pre-trained backbone, scaling efficiently to larger models (e.g., only +6.1% overhead for ViT-L/14). Right: Analysis of trainable parameters. By keeping the massive backbone frozen (grey), our approach requires optimizing only a lightweight set of parameters ( ∼ 21M–43M) for the GNN and Adapter, significantly reducing training memory and time costs compared to full fine-tuning.
Figure S8 : Hyperparameter Sensitivity Analysis. We visualize the impact of key hyperparameters on test accuracy, highlighting the necessity of our validation-based tuning protocol.
Hyperparameter
Symbol
Search Range
Scale
Learning Rate
η
[10−5,5×10−2]
Log-Uniform
Contrastive Weight
λcon
[0.1,1.0]
Uniform
Graph Logit Scale
γ
[0.1,5.0]
Uniform
Focal Loss Gamma
γfocal
[0.5,3.0]
Uniform
Tip-Adapter Alpha
α
[1.0,10.0]
Uniform
Table S4 : Hyperparameter Optimization Space. Optimization parameters are tuned on the validation set to adapt to domain granularity and shot availability.
Figure S9 : Quantitative comparison between the Baseline (Tip-Adapter) and Ours (Graph Refined) models on four test examples from the Caltech-101 dataset. Each panel displays the ground truth (GT) label, the input image, and the top-5 classifications with their associated probabilities. Green bars denote the correct class, while red bars indicate an incorrect top-1 prediction. The results demonstrate that the graph-based method corrects the baseline’s misclassifications (e.g., in the first and third panels) and significantly improves confidence in correct predictions (e.g., in the second panel).
Nodes
1-Shot
2-Shot
4-Shot
8-Shot
16-Shot
Caltech101
5
94.0
94.7
95.7
95.7
96.0
9
94.0
94.7
95.6
95.7
96.0
14
94.1
94.7
95.6
95.7
96.1
18
94.3
95.0
95.7
95.8
96.3
21
94.1
94.8
95.6
95.7
96.2
Table S5 : Few-shot classification accuracy (in %) for the model across varying numbers of Nodes (rows) and Shots per class (columns) on the Caltech and EuroSAT datasets.
Configuration
1 Shot
2 Shot
4 Shot
8 Shot
16 Shot
EuroSAT
w/o Text
64.6
70.5
75.5
79.8
84.2
w/ Text
67.4
74.9
80.3
84.1
89.4
Caltech101
w/o Text
93.6
93.6
95.1
95.2
95.7
w/ Text
94.3
95.0
95.7
95.8
96.3
Table S6 : Impact of Text Nodes in MGT.
MGT
Trans
1 Shot
2 Shot
4 Shot
8 Shot
16 Shot
EuroSAT
✗
✓
62.9
72.1
77.4
80.8
87.5
✓
✗
67.4
74.9
80.3
84.1
89.4
Aircraft
✗
✓
27.4
31.6
34.9
41.5
45.7
✓
✗
31.0
34.8
38.3
44.2
48.4
Table S7 : Comparison of MGT vs. Transformer architectures across different datasets and shot configurations.
In many CLIP adaptation methods, a blending ratio hyperparameter controls the trade-off between general pretrained CLIP knowledge and the limited, dataset-specific supervision from the few-shot cases. Most few-shot CLIP adaptation techniques report results by ablation of the blending ratio on the test set or require additional validation sets to select the blending ratio per dataset, and thus are not strictly few-shot. We present a simple, validation-free method for learning the blending ratio in CLIP adaptation. Hold-One-Shot-Out (HOSO) presents a novel approach for CLIP-Adapter-style methods to compete in the newly established validation-free setting. CLIP-Adapter with HOSO (HOSO-Adapter) learns the blending ratio using a one-shot, hold-out set, while the adapter trains on the remaining few-shot support examples. Under the validation-free few-shot protocol, HOSO-Adapter outperforms the CLIP-Adapter baseline by more than 4 percentage points on average across 11 standard few-shot datasets. Interestingly, in the 8- and 16-shot settings, HOSO-Adapter outperforms CLIP-Adapter even with the optimal blending ratio selected on the test set. Ablation studies validate the use of a one-shot hold-out mechanism, decoupled training, and improvements over the naively learnt blending ratio baseline. Code is released here: https://github.com/chris-vorster/HOSO-Adapter
Chris Vorster, Mayug Maniparambil, Noel E. O'Connor +2
Cross-Domain Few-Shot Learning (CDFSL) aims to adapt large-scale pretrained models to specialized target domains with limited samples, yet the few-shot fine-tuning of vision-language models like CLIP remains underexplored. By establishing multiple fine-tuning baselines of CLIP for CDFSL, we find adapter-based methods (e.g., LoRA) consistently outperform prompt-based ones (e.g., MaPLe), contrary to in-domain scenarios. To make those effective in-domain methods competitive again in CDFSL, we analyze this phenomenon and discover LoRA's superiority stems from rectifying the collapsed attention of visual CLS token, enhancing modality alignment and class separation by focusing on text-related visual regions. Further, we find textual EOS token exhibit much better attention to visual samples, and CLIP's standard contrastive loss weakly constrains modality alignment. Based on these insights, we propose Semantic Probe, a plug-and-play attention rectification framework for both adapter- and prompt-based methods. Extensive experiments on four CDFSL benchmarks validate our rationale, achieving state-of-the-art performance and benefiting both fine-tuning paradigms. Codes will be released.
We propose GP-Adapter, a training-free framework that augments CLIP (Contrastive Language-Image Pre-training) with Gaussian Process (GP) uncertainty modeling for few-shot classification and out-of-distribution (OOD) detection. While CLIP achieves strong zero-shot recognition, it yields deterministic similarity scores and offers limited uncertainty information, which is critical under distribution shift and data scarcity. GP-Adapter constructs modality-specific, class-wise one-class GPs on top of frozen CLIP embeddings using an RBF kernel for image features and a linear kernel for text prompts and fuses their predictive statistics to produce a variance-aware confidence score for OOD detection. The method requires no fine-tuning of the CLIP backbone and relies only on a small K-shot cache and lightweight hyperparameter selection, with memory cost scaling as O(CK2) for C classes and K shots. Experiments on ImageNet and multiple OOD benchmarks show that GP-Adapter provides competitive few-shot performance and consistently improves OOD detection when combined with prompt-learning baselines, highlighting the complementarity between GP-based uncertainty modeling and prompt learning. Overall, our results suggest that integrating probabilistic inference with large pre-trained vision-language models can improve reliability in low-data and distribution-shifted settings. Code is available at https://github.com/tms-byte/GP-Adapter