Few-shot adaptation of vision-language models must balance two competing goals: learning fine-grained visual distinctions from very limited supervision and preserving the efficiency that makes frozen CLIP-like models attractive for deployment. Existing lightweight adaptation methods, including prompt tuning, LoRA, and cache-based adapters, typically supervise global image embeddings and therefore underuse token-level structure. Conversely, structure-aware methods exploit local evidence but often introduce additional inference-time computation. We propose SD-LoRA, a structural distillation framework that uses token-level graph reasoning only during training and transfers its effects into a lightweight LoRA-adapted CLIP model. During training, a heterogeneous graph teacher models patch-patch, global-local, and visual-text relations over high-resolution ViT tokens. Rather than retaining this graph at test time, SD-LoRA transfers its structural bias into the shared CLIP-LoRA backbone through implicit distillation and Prototype Predictive Alignment, which aligns graph-conditioned representations with support-derived class prototypes. At inference, the graph teacher is discarded; prediction requires only a single CLIP-LoRA forward pass and a cache-based classifier. Across 11 few-shot benchmarks, SD-LoRA improves over a strong CLIP-LoRA baseline by 2.4/1.8/3.0/2.4/2.0 percentage points in the 1/2/4/8/16-shot settings, respectively, without additional graph computation at test time. Results on base-to-novel generalization, cross-dataset transfer, out-of-distribution robustness, and calibration further show that training-time structural supervision improves both accuracy and transfer reliability. Code is available at https://github.com/MR-Sherif/SD-LoRA.git.
Figures & tables
Figure 1: Structural distillation sharpens token-level evidence. We visualize patch-level attention maps for frozen CLIP and SD-LoRA using cosine similarity between the final [CLS] token and patch tokens. Frozen CLIP attends strongly to background foliage and misclassifies the image as clematis , while SD-LoRA shifts focus toward discriminative flower regions and correctly predicts balloon flower . The graph teacher is used only during training; the visualization is produced from the deployed LoRA backbone without test-time graph computation.
Figure 2: SD-LoRA framework. It trains a deployable CLIP-LoRA + cache student sθ,η using an auxiliary structural teacher gθ,ψ present only during adaptation. Teacher and student share the LoRA backbone, but the teacher reasons over high-resolution ViT tokens and class-text embeddings with a relation-aware graph transformer. Structural supervision from the graph branch plus Prototype Predictive Alignment updates the shared LoRA parameters θ . At inference, the teacher parameters ψ and graph computation are discarded, and only the lightweight student sθ,η makes predictions.
Figure 3: Few-shot accuracy trends across 11 benchmarks. Each panel shows top-1 accuracy for K∈{1,2,4,8,16} shots on one dataset; the first panel reports the macro-average over all 11 benchmarks. SD-LoRA is compared to recent CLIP adaptation methods. Positive values mean structural distillation improves the deployed LoRA + cache model, with especially strong gains in low-shot and fine-grained scenarios.
Shots
TIP-F
TaskRes
CLIP-Adapt
CCA
PLOT++
MaPLe
PromptSRC
CLIP-LoRA
CLIP-AST
SD-LoRA
1
70.9 ± 0.5
70.8 ± 0.7
65.7 ± 0.6
66.3 ± 0.5
70.7 ± 0.6
66.4 ± 0.5
72.1 ± 0.7
72.5 ± 0.5
72.3 ± 0.2
74.9 ± 0.1
2
73.7 ± 0.4
72.9 ± 0.4
66.8 ± 0.6
68.9 ± 0.4
74.0 ± 0.6
70.5 ± 0.3
75.2 ± 0.2
75.4 ± 0.6
75.8 ± 0.3
77.2 ± 0.3
4
75.6 ± 0.1
70.3 ± 0.2
74.7 ± 0.4
72.2 ± 0.3
76.9 ± 0.1
75.8 ± 0.4
78.3 ± 0.4
77.4 ± 0.1
78.9 ± 0.2
80.4 ± 0.1
8
78.3 ± 0.3
78.4 ± 0.3
71.4 ± 0.2
75.0 ± 0.3
79.6 ± 0.2
76.4 ± 0.3
80.6 ± 0.3
80.3 ± 0.2
81.7 ± 0.1
82.7 ± 0.1
16
80.7 ± 0.2
80.8 ± 0.2
75.5 ± 0.1
77.6 ± 0.2
82.1 ± 0.1
81.8 ± 0.1
82.7 ± 0.2
83.0 ± 0.1
84.6 ± 0.1
85.0 ± 0.2
Table 1: Macro-average few-shot accuracy across 11 benchmarks. We report average top-1 accuracy (%) over all datasets for top-10 representative adapter-, prompt-, and LoRA-based CLIP adaptation methods. Full per-dataset results are provided in Appendix B . The best result is shown in bold , and the strongest previous baseline in each shot setting is underlined.
Method
Avg.(11)
Cars
EuroSAT
Flowers
UCF
DTD
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
CLIP
69.34
74.22
71.70
63.37
74.89
68.65
56.48
64.05
60.03
72.08
77.80
74.83
70.53
77.50
73.85
53.24
59.90
56.37
CoOp
82.69
63.22
71.66
78.12
60.40
68.13
92.19
54.74
68.69
97.60
59.67
74.06
84.69
56.05
67.46
79.44
41.18
54.24
CoCoOp
80.47
71.69
75.83
70.49
73.59
72.01
87.49
60.04
71.21
94.87
71.75
81.71
82.33
73.45
77.64
77.01
56.00
64.85
ProGrad
82.48
70.75
76.16
77.68
68.63
72.88
90.11
60.89
72.67
95.54
71.87
82.03
84.33
74.94
79.35
77.35
52.35
62.45
KgCoOp
80.73
73.60
77.00
71.76
75.04
73.36
85.64
64.34
73.48
95.00
74.73
83.65
82.89
76.67
79.65
77.55
54.99
64.35
Table 2: Base-to-novel generalization under the MaPLe protocol. We report accuracy (%) on base classes, novel classes, and their harmonic mean (HM). Avg.(11) is the unweighted mean over 11 benchmarks. The Δ row shows the improvement of SD-LoRA over the strongest baseline for each metric. Best results are in bold .
Dataset
CLIP
CoOp
CoCoOp
MaPLe
ProGrad
KgCoOp
GraphAdapter
TransCLIP
CLIP-AST
Ours
ImageNet
66.73
71.51
71.02
70.72
72.24
71.20
73.68
70.30
73.87
74.20
ImageNet-R
73.96
75.21
76.18
76.98
74.58
76.70
77.20
75.00
—
77.32
ImageNet-A
47.77
49.71
50.63
50.90
49.39
50.69
49.23
49.50
—
49.63
ImageNet-V2
60.83
64.20
64.07
64.07
64.73
64.10
65.57
62.30
66.37
65.78
ImageNet-Sketch
46.15
47.99
48.75
49.15
47.61
48.97
48.57
49.07
48.37
49.38
Avg.
59.09
61.72
62.13
62.23
61.71
62.33
62.85
61.23
—
63.26
Table 3: Out-of-distribution (OOD) generalization. Models are adapted on ImageNet and evaluated on ImageNet and four OOD variants. Top-1 accuracy (%) is reported, and “Avg.” denotes the unweighted mean over all five datasets. Best results are shown in bold .
Variant
Components
Aircraft
Caltech101
EuroSAT
Cache
Graph
1-Shot
4-shot
16-shot
1-shot
4-shot
16-shot
1-shot
4-shot
16-shot
LoRA + Cache
✓
✗
32.2
43.5
59.2
92.7
93.0
95.1
74.6
88.3
91.8
LoRA + Graph
✗
✓
33.1
44.7
60.7
93.2
94.8
96.5
75.3
90.2
93.1
SD-LoRA
✓
✓
34.8
46.2
61.4
94.4
96.2
97.2
76.8
91.3
94.0
Table 4: Main component ablation. LoRA is enabled in all variants. Cache is a deployed inference component, while the graph is a training-only teacher and is removed after adaptation. SD-LoRA combines both components and achieves the best performance across all datasets and shot settings.
Figure 4: Effect of LoRA placement. We compare fixed LoRA placement and adaptive layer selection under 1/4/16-shot settings. SD-LoRA outperforms both fixed CLIP-LoRA and adaptive CLIP-AST, indicating that adaptive placement complements SD-LoRA: layer selection optimizes where LoRA is inserted, while structural distillation improves how its parameters are trained.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Shots
Type
Method
Venue
INet
SUN
Air
Euro
Cars
Food
Pets
Flow
Cal
DTD
UCF
Avg
0
CLIP
ICML’22
66.7
71.4
62.6
24.7
47.5
65.3
86.1
89.1
92.9
43.6
66.7
65.1
1
Adapter
TIP-Adapter-F
ECCV’22
69.4
67.2
28.8
67.8
67.1
85.8
90.6
83.8
94.0
51.6
73.4
70.9
TaskRes
CVPR’23
69.6
68.1
31.3
65.4
68.8
84.6
90.2
81.7
93.6
53.8
71.7
70.8
GraphAdapter
NeurIPS’23
61.5
61.9
20.9
63.3
59.7
75.4
84.4
80.0
88.9
51.8
64.9
64.8
CLIP-Adapter
IJCV’24
67.9
71.3
25.2
65.4
49.3
65.7
86.1
89.0
92.0
44.2
66.9
65.7
CLAP
CVPR’24
58.5
61.1
20.6
59.2
56.3
73.0
83.6
79.9
88.4
47.5
62.5
62.8
Appendix
Table 6: Full few-shot comparison on 11 benchmarks. We report top-1 accuracy (%) for 1-, 2-, and 4-shot classification against representative adapter- and prompt-based CLIP adaptation methods. The best result in each shot group is shown in bold . SD-LoRA is reported in the final row of each shot group. Dataset abbreviations: INet: ImageNet; SUN: SUN397; Air: FGVC-Aircraft; Euro: EuroSAT; Cars: StanfordCars; Food: Food101; Pets: OxfordPets; Flow: Flowers102; Cal: Caltech101; DTD: Describable Textures; UCF: UCF101.
Shots
Type
Method
Venue
INet
SUN
Air
Euro
Cars
Food
Pets
Flow
Cal
DTD
UCF
Avg
8
Adapter
TIP-Adapter-F
ECCV’22
71.7
73.5
39.5
81.3
78.3
86.9
91.8
94.3
95.2
66.7
82.0
78.3
TaskRes
CVPR’23
72.3
74.6
40.3
77.5
79.6
86.4
92.0
96.0
95.3
66.7
81.6
78.4
GraphAdapter
NeurIPS’23
64.2
68.9
31.4
80.2
70.5
77.7
87.6
94.1
92.4
64.5
75.7
73.4
CLIP-Adapter
IJCV’24
62.7
67.5
26.2
77.9
67.9
78.0
87.6
91.7
91.4
61.0
73.3
71.4
CLAP
CVPR’24
62.9
68.6
28.9
76.7
70.3
77.4
87.7
92.1
91.4
63.2
73.3
72.1
CCA
ICCV’25
64.9
71.0
35.0
83.1
74.3
79.2
89.0
94.4
92.5
65.0
77.5
75.0
Appendix
Table 7: Full 8- and 16-shot comparison on 11 benchmarks. We report top-1 accuracy (%) against representative adapter- and prompt-based CLIP adaptation methods. The best result in each shot group is shown in bold . SD-LoRA is reported in the final row of each shot group. Dataset abbreviations follow Table 6 .
Method
Aircraft
Caltech
Food
Imagenet
Pets
SUN
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
CLIP
27.19
36.29
31.09
96.84
94.00
95.40
90.10
91.22
90.66
72.43
68.14
70.22
91.17
97.26
94.12
69.36
75.35
72.23
CoCoOp
33.41
23.71
27.74
97.96
93.81
95.84
90.70
91.29
90.99
75.98
70.43
73.10
95.20
97.69
96.43
79.74
76.86
78.27
MaPLe
37.44
35.61
36.50
97.74
94.36
96.02
90.71
92.05
91.38
76.66
70.54
73.47
95.43
97.76
96.58
80.82
78.70
79.75
PromptSRC
42.73
37.87
40.15
98.10
94.03
96.02
90.67
91.53
91.10
77.60
70.73
74.01
95.33
97.30
96.30
82.67
78.47
80.52
CLIP-AST
48.98
38.21
42.93
98.71
94.00
96.30
90.57
91.11
90.84
78.44
70.22
74.10
96.23
97.37
96.80
83.05
78.12
80.51
Appendix
Table 8: Additional base-to-novel generalization results under the MaPLe protocol. Due to space constraints in the main paper, we report the remaining per-dataset results here. We show accuracy (%) on base classes, novel classes, and their harmonic mean (HM).
Shots
Method
SUN
Air
Euro
Cars
Food
Pets
Flow
Cal
DTD
UCF
Avg
1
CoOp (4)
64.8
18.7
49.3
60.5
74.4
87.7
68.0
91.5
47.1
66.5
62.85
CoOp (16)
63.3
15.7
52.9
59.6
75.9
87.6
71.5
91.7
47.2
66.3
63.17
CoCoOp
65.6
15.5
46.7
60.4
79.5
88.2
68.6
92.2
45.7
67.3
62.97
TIP-Adapter-F
65.4
22.2
59.7
61.1
80.4
87.5
81.1
92.4
50.9
66.5
66.72
CLIP-Adapter
63.2
19.8
46.9
60.7
80.6
87.6
66.7
91.6
44.1
62.9
62.41
PLOT++
64.5
22.0
67.3
61.4
79.2
89.4
73.7
93.0
50.6
69.7
67.08
Appendix
Table 9: Few-shot results with CLIP ViT-B/32. Top-1 accuracy (%) for K∈{1,2,4,8,16} shots on 10 datasets (excluding ImageNet); Avg is their unweighted mean. TOGA Mohammad et al. (2026) results for 1/2/4/8 shots are reproduced from supplementary Table S2. Best results are in bold.
Method
Source
Target datasets
Avg.
ImageNet
Caltech
DTD
EuroSAT
Aircraft
Food
Flowers
Pets
Cars
SUN
UCF
MaPLe
72.30
93.53
46.49
48.06
24.74
86.20
72.23
90.49
65.57
67.01
68.69
66.30
PromptSRC
72.80
93.60
46.87
45.50
23.90
86.15
70.25
90.25
65.70
67.10
68.75
65.81
PromptKD
—
93.61
55.08
63.74
26.24
88.84
75.33
91.59
73.93
68.57
76.39
71.33
Ours
74.20
95.17
61.55
69.05
31.76
86.90
85.25
92.48
76.23
72.98
78.43
74.98
Δ vs. PromptKD
—
+1.56
+6.47
+5.31
+5.52
-1.94
+9.92
+0.89
+2.30
+4.41
+2.04
+3.65
Appendix
Table 10: Cross-dataset generalization from ImageNet. Models are adapted on ImageNet and evaluated directly on target datasets without target-domain training. The ImageNet column reports source-domain accuracy, while Avg. denotes the unweighted mean over the ten target datasets. Δ reports the absolute difference between SD-LoRA and PromptKD. Best results are shown in bold .
Metric
Method
Caltech
DTD
EuroSAT
Aircraft
Food
Flowers
Pets
Cars
SUN
UCF
Avg.
ECE ↓
PromptKD
5.01
4.96
17.41
3.04
2.00
5.49
5.47
7.94
1.49
2.93
5.57
ECE ↓
Ours
1.99
3.54
6.83
2.32
1.56
3.40
0.83
3.64
1.28
1.13
2.65
NLL ↓
PromptKD
0.24
1.86
0.97
2.46
0.39
1.48
0.25
0.79
1.12
0.91
1.05
NLL ↓
Ours
0.16
1.18
0.28
1.70
0.28
0.32
0.23
0.76
0.78
0.51
0.62
Brier ↓
PromptKD
0.11
0.62
0.46
0.81
0.16
0.33
0.11
0.37
0.44
0.34
0.38
Brier ↓
Ours
0.07
0.34
0.13
0.44
0.09
0.09
0.10
0.35
0.31
0.12
0.20
Appendix
Table 11: Calibration under cross-dataset generalization. After ImageNet-source adaptation, models are evaluated on ten target datasets. We report expected calibration error (ECE), negative log-likelihood (NLL), and Brier score. Lower values indicate better calibration. Best results are shown in bold .
Variant
Enabled edge families
Aircraft
Caltech101
Spatial
Global–local
CLS–text
Patch–text
1-shot
4-shot
16-shot
1-shot
4-shot
16-shot
Full graph
✓
✓
✓
✓
34.80
46.20
61.40
94.40
96.20
97.20
w/o spatial edges
✗
✓
✓
✓
33.21
45.47
60.44
92.75
95.40
96.28
w/o global–local edges
✓
✗
✓
✓
32.36
43.98
58.12
91.92
93.92
93.84
w/o semantic text–vision edges
✓
✓
✗
✗
31.57
43.87
57.32
91.02
93.80
93.28
CLS-semantic only
✓
✓
✓
✗
33.54
45.80
60.68
93.22
95.33
96.44
Appendix
Table 13: Edge-type ablation of the graph teacher. Accuracy (%) is reported on Aircraft and Caltech101 under 1-, 4-, and 16-shot settings. The edge columns indicate whether each relation family is enabled: spatial patch–patch, global–local CLS–patch, semantic CLS–text , and semantic patch–text interactions . The full graph performs best across all settings. Removing semantic visual–text edges gives the largest degradation, followed by global–local edges, while spatial patch–patch edges provide a smaller but consistent gain.
Variant
Enabled components
Aircraft
Caltech101
EuroSAT
PPA
Graph
1-shot
4-Shot
16-Shot
1-shot
4-Shot
16-Shot
1-shot
4-Shot
16-Shot
HR only
✗
✗
32.2
43.5
59.2
92.7
93.0
95.1
74.6
88.3
91.8
HR + PPA
✓
✗
31.7
43.4
59.5
92.5
93.0
95.3
73.9
86.4
92.1
HR + Graph
✗
✓
33.6
45.1
60.2
94.2
96.1
97.1
76.1
89.9
93.2
Full model
✓
✓
34.8
46.2
61.4
94.4
96.2
97.2
76.8
91.3
94.0
Appendix
Table 14: Effect of PPA and the graph teacher. LoRA and cache-based inference are enabled in all variants. We compare high-resolution adaptation alone (HR only), adding Prototype Predictive Alignment (HR + PPA), adding the training-only graph teacher (HR + Graph), and the full model with both graph and PPA. Accuracy (%) is reported under 1-, 4-, and 16-shot settings on Aircraft, Caltech101, and EuroSAT. The graph teacher provides the dominant gain, while PPA is most effective when combined with graph-conditioned representations.
Pooling Strategy
1-shot
2-shot
4-shot
8-shot
16-shot
EuroSAT
∑All (Global)
75.9
80.1
89.5
92.2
93.6
∑Top-M=25
77.0
81.2
90.0
92.6
93.9
∑Top-M=50
76.8
84.9
91.3
92.7
94.0
∑Top-M=75
76.7
81.0
88.8
92.4
93.2
Caltech
∑All (Global)
94.1
94.8
94.9
96.0
96.8
∑Top-M=25
94.2
95.0
95.0
96.2
96.9
Appendix
Table 15: Effect of token filtering in the graph-teacher readout. We compare global pooling over all graph-refined tokens with saliency-based Top- M filtering. Top- M retains only the highest-scoring patch tokens before aggregation, reducing the influence of background or weakly informative tokens. A moderate selection size, Top- M=50 , provides the best overall trade-off across datasets and shot settings.
Resolution ( H′×W′ )
1 Shot
2 Shot
4 Shot
8 Shot
16 Shot
Caltech
224×224 (Standard)
94.1
95.6
95.4
96.8
96.9
336×336
94.4
95.8
96.2
96.3
97.2
512×512
91.5
92.6
93.7
94.8
95.9
Euro
224×224 (Standard)
76.5
86.6
91.5
92.4
93.9
336×336
76.8
84.9
91.3
92.7
94.0
512×512
75.1
82.7
90.2
91.5
92.4
Appendix
Table 16: Effect of input resolution. We compare the standard CLIP input resolution, 224×224 , with higher resolutions used for training-time graph reasoning. The 336×336 setting generally provides the best trade-off between finer token granularity and pretrained feature stability, while 512×512 introduces a stronger resolution shift and degrades performance.
Components
Aircraft
Caltech101
EuroSAT
Trans
RGT
Filtering
1-shot
4-shot
16-shot
1-shot
4-shot
16-shot
1-shot
4-shot
16-shot
✗
✓
✓
34.1
45.1
58.8
93.2
94.8
95.4
74.2
89.8
91.1
✓
✗
✓
33.5
45.3
57.6
92.8
93.7
95.5
73.1
88.6
89.8
✓
✓
✗
33.4
44.2
57.4
92.4
94.2
95.7
73.7
89.1
91.6
✗
✓
✗
32.7
44.9
56.5
91.5
93.3
94.5
72.8
88.2
90.8
✓
✗
✗
33.2
44.0
55.3
91.5
92.5
93.3
72.1
87.5
90.0
Appendix
Table 18: Graph teacher architecture ablation. We evaluate the contributions of the visual refinement Transformer (Trans), the relation-aware graph Transformer (RGT), and discriminative node filtering. Accuracy (%) is reported on Aircraft, Caltech101, and EuroSAT under 1-, 4-, and 16-shot settings. The full architecture performs best across all datasets and shot settings, showing that token refinement, relation-aware message passing, and foreground-focused filtering provide complementary structural supervision.
Loss Components
EuroSAT
Caltech101
Food101
LStudent
LTeacher
LPPA
1-shot
4-shot
8-shot
1-shot
4-shot
8-shot
1-shot
4-shot
8-shot
✓
✗
✗
74.6
88.3
91.8
92.7
93.0
95.1
85.2
86.3
86.9
✓
✓
✗
76.2
89.7
93.5
94.2
94.9
96.8
85.7
86.7
87.2
✓
✗
✓
76.4
91.0
93.8
94.3
95.5
97.0
86.0
86.9
87.6
✓
✓
✓
76.8
91.3
94.0
94.4
96.2
97.2
86.2
87.1
87.8
Appendix
Table 19: Loss component ablation. We evaluate the contributions of the student loss LStudent , the graph-teacher loss LTeacher , and Prototype Predictive Alignment LPPA . Accuracy (%) is reported on EuroSAT, Caltech101, and Food101 under 1-, 4-, and 8-shot settings. The full objective performs best across all settings, showing that graph-based structural supervision and prototype-level alignment provide complementary training signals.
Figure 5: Effect of cross-modal interaction frequency. We ablate where semantic text–visual interaction is applied across graph layers on Aircraft under 1/4/16-shot settings. Each marker reports the peak accuracy, while the text label indicates the number of epochs required to reach 95% of the peak. Applying interaction in all layers consistently achieves the best peak accuracy across shot regimes and also converges efficiently in the 16-shot setting, reaching 95% of peak in 13 epochs compared with 18 epochs for last only and 22 epochs for none . This suggests that frequent cross-modal interaction improves both final adaptation quality and optimization efficiency.
Figure 6: Computational and deployment analysis of our method. A shows the one-time wall-clock overhead associated with the additional high-resolution training and re-encoding passes. B reports the computational overhead relative to the CLIP-LoRA training system, showing that the added graph branch increases computation by only 0.61 GFLOPs ( +1.31% ) while keeping parameter growth modest ( +8.0% ). C summarizes practical deployment-time metrics on ViT-B/16, including training time, memory footprint, and inference latency. D shows that the method remains well behaved across backbone choices: ViT-B/32 retains competitive average few-shot accuracy, while the higher cost of ViT-L/14 is primarily attributable to the larger backbone rather than to additional method-specific steps.
Component / Setting
Params (M)
Trainable (M)
GFLOPs
Role
Standard adaptation reference
CLIP-LoRA, 224 × 224
152.08
1.23
17.50
Standard train/infer path
SD-LoRA training path
Frozen CLIP backbone, 336 × 336
150.85
0.00
46.73
HR token extraction
+ LoRA
1.23
1.23
≈ 0.00
Shared deployed update
Cache head
0.05
0.05
≈ 0.00
Deployed classifier
Appendix
Table 20: Computational analysis. GFLOPs are reported per image using the same profiler. SD-LoRA separates training-time structural supervision from deployment. The 336 × 336 pass is used only during adaptation to expose a denser ViT token grid for fine-grained graph supervision. The graph branch itself adds 0.61 GFLOPs on top of this high-resolution pass, corresponding to 1.31% overhead relative to the 336 × 336 backbone pass. At inference, the graph reasoning module, saliency scorer, PPA head, and high-resolution teacher path are removed, reducing deployment to the 224 × 224 CLIP-LoRA+cache student.
Hyperparameter
Symbol
Search Range
Scale
Optimizer & Backbone
Learning Rate
η
[10−5,5×10−2]
Log-Uniform
Weight Decay
wd
[10−6,10−3]
Log-Uniform
LoRA Rank
r
[2,4,8,16]
Integer
LoRA Alpha
αLoRA
{8,16,32}
Discrete
Training Objectives
Appendix
Table 21: Hyperparameter Optimization Space. The search selects one global configuration on validation data; selected values are reported in Table 22 and are fixed across datasets and shots.
Training-free few-shot adaptation of vision-language models typically constructs a classifier from CLIP's final projected image representation, leaving intermediate representations unused. Combining them is not straightforward. Class scores from different representations differ in spread by orders of magnitude for the same query. Direct averaging allocates influence by numerical scale rather than predictive information. We propose Layer-Ensemble Gaussian Discriminant Analysis (LE-GDA), a method that fits regularized shared-covariance discriminants independently to intermediate and final representations extracted in one frozen encoder pass. LE-GDA standardizes each discriminant's class scores per query, selects a small representation multiset on labeled validation data, and combines it with the zero-shot classifier without gradient-based optimization. Across 11 datasets, four CLIP backbones, five shot counts, and three independently sampled K-shot training sets, LE-GDA achieves the highest mean accuracy of all evaluated methods on every backbone, raising the 16-shot mean from 78.01% to 79.54% over a matched GDA-CLIP (final-feature) baseline and improving that baseline for every backbone-shot combination. At one shot, LE-GDA retains a 1.20-point mean gain over the baseline even though covariance estimation is unavailable and per-query standardization is neutral. A control with richer per-class structure inside a single representation does not reproduce the gain, indicating that the benefit comes from combining representations. These results show that intermediate CLIP representations carry information beyond the final embedding and can improve training-free few-shot adaptation. Code and evaluation resources are available at https://github.com/pavodi-NM/le-gda.
Recent adapter-based CLIP tuning (e.g., Tip-Adapter) is a strong few-shot learner, achieving efficiency by caching support features for fast prototype matching. However, these methods rely on global uni-modal feature vectors, overlooking fine-grained patch relations and their structural alignment with class text. To bridge this gap without incurring inference costs, we introduce a novel asymmetric training-only framework. Instead of altering the lightweight adapter, we construct a high-capacity auxiliary Heterogeneous Graph Teacher that operates solely during training. This teacher (i) integrates multi-scale visual patches and text prompts into a unified graph, (ii) performs deep cross-modal reasoning via a Modality-aware Graph Transformer (MGT), and (iii) applies discriminative node filtering to extract high-fidelity class features. Crucially, we employ a cache-aware dual-objective strategy to supervise this relational knowledge directly into the Tip-Adapter's key-value cache, effectively upgrading the prototypes while the graph teacher is discarded at test time. Thus, inference remains identical to Tip-Adapter with zero extra latency or memory. Across standard 1-16-shot benchmarks, our method consistently establishes a new state-of-the-art. Ablations confirm that the auxiliary graph supervision, text-guided reasoning, and node filtering are the essential ingredients for robust few-shot adaptation. Code is available at https://github.com/MR-Sherif/TOGA.git.
Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan +2
Edge Hill University, Ormskirk, Lancashire, L39 4QP
The limited supervision available for the few-shot adaptation of vision-language models makes gradient estimation particularly noisy. We find that the gradient magnitudes are severely miscalibrated, while the signs of a small top-magnitude subset remain informative. This motivates SignSparseSGD and DynSignSparseSGD, two state-free optimizers that use gradient magnitudes only to select where to update and fixed-magnitude signs to decide how to update. The static variant fixes the support after the first minibatch. The dynamic variant refreshes it at every step. Across 11 datasets, three CLIP backbones, and five shot levels, the proposed methods are competitive with or outperform parameter-efficient and optimization-based baselines on average, while exhibiting markedly lower sensitivity to long training schedules. They introduce no task-specific modules, maintain no first- or second-moment optimizer states, and use one shared configuration across datasets, backbones, and shot levels. Our code is included in the supplementary materials and will be released upon publication.
Nairouz Mrabah, Ismail Ben Ayed, Mahdi S. Hosseini +1
Department of Systems Engineering, ETS Montreal, Canada · Department of Computer Science and Software Engineering, Concordia, Canada