SD-LoRA: Training-Time Structural Distillation into LoRA for Few-Shot Vision-Language Adaptation

Mohammed Rahman Sherif Khan Mohammad; Ardhendu Behera; Sandip Pradhan; Swagat Kumar; Yonghuai Liu

cs.CVNov 4, 2026

SD-LoRA: Training-Time Structural Distillation into LoRA for Few-Shot Vision-Language Adaptation

Authors: Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan, Swagat Kumar, Yonghuai Liu

Organizations: Edge Hill University, UK

Abstract

Few-shot adaptation of vision-language models must balance two competing goals: learning fine-grained visual distinctions from very limited supervision and preserving the efficiency that makes frozen CLIP-like models attractive for deployment. Existing lightweight adaptation methods, including prompt tuning, LoRA, and cache-based adapters, typically supervise global image embeddings and therefore underuse token-level structure. Conversely, structure-aware methods exploit local evidence but often introduce additional inference-time computation. We propose SD-LoRA, a structural distillation framework that uses token-level graph reasoning only during training and transfers its effects into a lightweight LoRA-adapted CLIP model. During training, a heterogeneous graph teacher models patch-patch, global-local, and visual-text relations over high-resolution ViT tokens. Rather than retaining this graph at test time, SD-LoRA transfers its structural bias into the shared CLIP-LoRA backbone through implicit distillation and Prototype Predictive Alignment, which aligns graph-conditioned representations with support-derived class prototypes. At inference, the graph teacher is discarded; prediction requires only a single CLIP-LoRA forward pass and a cache-based classifier. Across 11 few-shot benchmarks, SD-LoRA improves over a strong CLIP-LoRA baseline by 2.4/1.8/3.0/2.4/2.0 percentage points in the 1/2/4/8/16-shot settings, respectively, without additional graph computation at test time. Results on base-to-novel generalization, cross-dataset transfer, out-of-distribution robustness, and calibration further show that training-time structural supervision improves both accuracy and transfer reliability. Code is available at https://github.com/MR-Sherif/SD-LoRA.git.

Figures & tables

Figure 1: Structural distillation sharpens token-level evidence. We visualize patch-level attention maps for frozen CLIP and SD-LoRA using cosine similarity between the final [CLS] token and patch tokens. Frozen CLIP attends strongly to background foliage and misclassifies the image as clematis , while SD-LoRA shifts focus toward discriminative flower regions and correctly predicts balloon flower . The graph teacher is used only during training; the visualization is produced from the deployed LoRA backbone without test-time graph computation.

Figure 2: SD-LoRA framework. It trains a deployable CLIP-LoRA $+$ cache student $s_{\theta,\eta}$ using an auxiliary structural teacher $g_{\theta,\psi}$ present only during adaptation. Teacher and student share the LoRA backbone, but the teacher reasons over high-resolution ViT tokens and class-text embeddings with a relation-aware graph transformer. Structural supervision from the graph branch plus Prototype Predictive Alignment updates the shared LoRA parameters $\theta$ . At inference, the teacher parameters $\psi$ and graph computation are discarded, and only the lightweight student $s_{\theta,\eta}$ makes predictions.

Figure 3: Few-shot accuracy trends across 11 benchmarks. Each panel shows top-1 accuracy for

K\in\{1,2,4,8,16\}

shots on one dataset; the first panel reports the macro-average over all 11 benchmarks. SD-LoRA is compared to recent CLIP adaptation methods. Positive values mean structural distillation improves the deployed LoRA

+

cache model, with especially strong gains in low-shot and fine-grained scenarios.

Shots	TIP-F	TaskRes	CLIP-Adapt	CCA	PLOT++	MaPLe	PromptSRC	CLIP-LoRA	CLIP-AST	SD-LoRA
1	70.9 $\pm$ 0.5	70.8 $\pm$ 0.7	65.7 $\pm$ 0.6	66.3 $\pm$ 0.5	70.7 $\pm$ 0.6	66.4 $\pm$ 0.5	72.1 $\pm$ 0.7	72.5 $\pm$ 0.5	72.3 $\pm$ 0.2	74.9 $\pm$ 0.1
2	73.7 $\pm$ 0.4	72.9 $\pm$ 0.4	66.8 $\pm$ 0.6	68.9 $\pm$ 0.4	74.0 $\pm$ 0.6	70.5 $\pm$ 0.3	75.2 $\pm$ 0.2	75.4 $\pm$ 0.6	75.8 $\pm$ 0.3	77.2 $\pm$ 0.3
4	75.6 $\pm$ 0.1	70.3 $\pm$ 0.2	74.7 $\pm$ 0.4	72.2 $\pm$ 0.3	76.9 $\pm$ 0.1	75.8 $\pm$ 0.4	78.3 $\pm$ 0.4	77.4 $\pm$ 0.1	78.9 $\pm$ 0.2	80.4 $\pm$ 0.1
8	78.3 $\pm$ 0.3	78.4 $\pm$ 0.3	71.4 $\pm$ 0.2	75.0 $\pm$ 0.3	79.6 $\pm$ 0.2	76.4 $\pm$ 0.3	80.6 $\pm$ 0.3	80.3 $\pm$ 0.2	81.7 $\pm$ 0.1	82.7 $\pm$ 0.1
16	80.7 $\pm$ 0.2	80.8 $\pm$ 0.2	75.5 $\pm$ 0.1	77.6 $\pm$ 0.2	82.1 $\pm$ 0.1	81.8 $\pm$ 0.1	82.7 $\pm$ 0.2	83.0 $\pm$ 0.1	84.6 $\pm$ 0.1	85.0 $\pm$ 0.2

Table 1: Macro-average few-shot accuracy across 11 benchmarks. We report average top-1 accuracy (%) over all datasets for top-10 representative adapter-, prompt-, and LoRA-based CLIP adaptation methods. Full per-dataset results are provided in Appendix B . The best result is shown in bold , and the strongest previous baseline in each shot setting is underlined.

Method	Avg.(11)			Cars			EuroSAT			Flowers			UCF			DTD
Method	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
CLIP	69.34	74.22	71.70	63.37	74.89	68.65	56.48	64.05	60.03	72.08	77.80	74.83	70.53	77.50	73.85	53.24	59.90	56.37
CoOp	82.69	63.22	71.66	78.12	60.40	68.13	92.19	54.74	68.69	97.60	59.67	74.06	84.69	56.05	67.46	79.44	41.18	54.24
CoCoOp	80.47	71.69	75.83	70.49	73.59	72.01	87.49	60.04	71.21	94.87	71.75	81.71	82.33	73.45	77.64	77.01	56.00	64.85
ProGrad	82.48	70.75	76.16	77.68	68.63	72.88	90.11	60.89	72.67	95.54	71.87	82.03	84.33	74.94	79.35	77.35	52.35	62.45
KgCoOp	80.73	73.60	77.00	71.76	75.04	73.36	85.64	64.34	73.48	95.00	74.73	83.65	82.89	76.67	79.65	77.55	54.99	64.35

Table 2: Base-to-novel generalization under the MaPLe protocol. We report accuracy (%) on base classes, novel classes, and their harmonic mean (HM). Avg.(11) is the unweighted mean over 11 benchmarks. The

\Delta

row shows the improvement of SD-LoRA over the strongest baseline for each metric. Best results are in bold .

Dataset	CLIP	CoOp	CoCoOp	MaPLe	ProGrad	KgCoOp	GraphAdapter	TransCLIP	CLIP-AST	Ours
ImageNet	66.73	71.51	71.02	70.72	72.24	71.20	73.68	70.30	73.87	74.20
ImageNet-R	73.96	75.21	76.18	76.98	74.58	76.70	77.20	75.00	—	77.32
ImageNet-A	47.77	49.71	50.63	50.90	49.39	50.69	49.23	49.50	—	49.63
ImageNet-V2	60.83	64.20	64.07	64.07	64.73	64.10	65.57	62.30	66.37	65.78
ImageNet-Sketch	46.15	47.99	48.75	49.15	47.61	48.97	48.57	49.07	48.37	49.38
Avg.	59.09	61.72	62.13	62.23	61.71	62.33	62.85	61.23	—	63.26

Table 3: Out-of-distribution (OOD) generalization. Models are adapted on ImageNet and evaluated on ImageNet and four OOD variants. Top-1 accuracy (%) is reported, and “Avg.” denotes the unweighted mean over all five datasets. Best results are shown in bold .

Variant	Components		Aircraft			Caltech101			EuroSAT
Variant	Cache	Graph	1-Shot	4-shot	16-shot	1-shot	4-shot	16-shot	1-shot	4-shot	16-shot
LoRA + Cache	✓	✗	32.2	43.5	59.2	92.7	93.0	95.1	74.6	88.3	91.8
LoRA + Graph	✗	✓	33.1	44.7	60.7	93.2	94.8	96.5	75.3	90.2	93.1
SD-LoRA	✓	✓	34.8	46.2	61.4	94.4	96.2	97.2	76.8	91.3	94.0

Table 4: Main component ablation. LoRA is enabled in all variants. Cache is a deployed inference component, while the graph is a training-only teacher and is removed after adaptation. SD-LoRA combines both components and achieves the best performance across all datasets and shot settings.

Figure 4: Effect of LoRA placement. We compare fixed LoRA placement and adaptive layer selection under 1/4/16-shot settings. SD-LoRA outperforms both fixed CLIP-LoRA and adaptive CLIP-AST, indicating that adaptive placement complements SD-LoRA: layer selection optimizes where LoRA is inserted, while structural distillation improves how its parameters are trained.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Shots	Type	Method	Venue	INet	SUN	Air	Euro	Cars	Food	Pets	Flow	Cal	DTD	UCF	Avg
0		CLIP	ICML’22	66.7	71.4	62.6	24.7	47.5	65.3	86.1	89.1	92.9	43.6	66.7	65.1
1	Adapter	TIP-Adapter-F	ECCV’22	69.4	67.2	28.8	67.8	67.1	85.8	90.6	83.8	94.0	51.6	73.4	70.9
		TaskRes	CVPR’23	69.6	68.1	31.3	65.4	68.8	84.6	90.2	81.7	93.6	53.8	71.7	70.8
		GraphAdapter	NeurIPS’23	61.5	61.9	20.9	63.3	59.7	75.4	84.4	80.0	88.9	51.8	64.9	64.8
		CLIP-Adapter	IJCV’24	67.9	71.3	25.2	65.4	49.3	65.7	86.1	89.0	92.0	44.2	66.9	65.7
		CLAP	CVPR’24	58.5	61.1	20.6	59.2	56.3	73.0	83.6	79.9	88.4	47.5	62.5	62.8

Appendix

Table 6: Full few-shot comparison on 11 benchmarks. We report top-1 accuracy (%) for 1-, 2-, and 4-shot classification against representative adapter- and prompt-based CLIP adaptation methods. The best result in each shot group is shown in bold . SD-LoRA is reported in the final row of each shot group. Dataset abbreviations: INet: ImageNet; SUN: SUN397; Air: FGVC-Aircraft; Euro: EuroSAT; Cars: StanfordCars; Food: Food101; Pets: OxfordPets; Flow: Flowers102; Cal: Caltech101; DTD: Describable Textures; UCF: UCF101.

Shots	Type	Method	Venue	INet	SUN	Air	Euro	Cars	Food	Pets	Flow	Cal	DTD	UCF	Avg
8	Adapter	TIP-Adapter-F	ECCV’22	71.7	73.5	39.5	81.3	78.3	86.9	91.8	94.3	95.2	66.7	82.0	78.3
		TaskRes	CVPR’23	72.3	74.6	40.3	77.5	79.6	86.4	92.0	96.0	95.3	66.7	81.6	78.4
		GraphAdapter	NeurIPS’23	64.2	68.9	31.4	80.2	70.5	77.7	87.6	94.1	92.4	64.5	75.7	73.4
		CLIP-Adapter	IJCV’24	62.7	67.5	26.2	77.9	67.9	78.0	87.6	91.7	91.4	61.0	73.3	71.4
		CLAP	CVPR’24	62.9	68.6	28.9	76.7	70.3	77.4	87.7	92.1	91.4	63.2	73.3	72.1
		CCA	ICCV’25	64.9	71.0	35.0	83.1	74.3	79.2	89.0	94.4	92.5	65.0	77.5	75.0

Appendix

Table 7: Full 8- and 16-shot comparison on 11 benchmarks. We report top-1 accuracy (%) against representative adapter- and prompt-based CLIP adaptation methods. The best result in each shot group is shown in bold . SD-LoRA is reported in the final row of each shot group. Dataset abbreviations follow Table 6 .

Method	Aircraft			Caltech			Food			Imagenet			Pets			SUN
Method	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
CLIP	27.19	36.29	31.09	96.84	94.00	95.40	90.10	91.22	90.66	72.43	68.14	70.22	91.17	97.26	94.12	69.36	75.35	72.23
CoCoOp	33.41	23.71	27.74	97.96	93.81	95.84	90.70	91.29	90.99	75.98	70.43	73.10	95.20	97.69	96.43	79.74	76.86	78.27
MaPLe	37.44	35.61	36.50	97.74	94.36	96.02	90.71	92.05	91.38	76.66	70.54	73.47	95.43	97.76	96.58	80.82	78.70	79.75
PromptSRC	42.73	37.87	40.15	98.10	94.03	96.02	90.67	91.53	91.10	77.60	70.73	74.01	95.33	97.30	96.30	82.67	78.47	80.52
CLIP-AST	48.98	38.21	42.93	98.71	94.00	96.30	90.57	91.11	90.84	78.44	70.22	74.10	96.23	97.37	96.80	83.05	78.12	80.51

Appendix

Table 8: Additional base-to-novel generalization results under the MaPLe protocol. Due to space constraints in the main paper, we report the remaining per-dataset results here. We show accuracy (%) on base classes, novel classes, and their harmonic mean (HM).

Shots	Method	SUN	Air	Euro	Cars	Food	Pets	Flow	Cal	DTD	UCF	Avg
1	CoOp (4)	64.8	18.7	49.3	60.5	74.4	87.7	68.0	91.5	47.1	66.5	62.85
	CoOp (16)	63.3	15.7	52.9	59.6	75.9	87.6	71.5	91.7	47.2	66.3	63.17
	CoCoOp	65.6	15.5	46.7	60.4	79.5	88.2	68.6	92.2	45.7	67.3	62.97
	TIP-Adapter-F	65.4	22.2	59.7	61.1	80.4	87.5	81.1	92.4	50.9	66.5	66.72
	CLIP-Adapter	63.2	19.8	46.9	60.7	80.6	87.6	66.7	91.6	44.1	62.9	62.41
	PLOT++	64.5	22.0	67.3	61.4	79.2	89.4	73.7	93.0	50.6	69.7	67.08

Appendix

Table 9: Few-shot results with CLIP ViT-B/32. Top-1 accuracy (%) for

K\in\{1,2,4,8,16\}

shots on 10 datasets (excluding ImageNet); Avg is their unweighted mean. TOGA Mohammad et al. (2026) results for 1/2/4/8 shots are reproduced from supplementary Table S2. Best results are in bold.

Method	Source	Target datasets										Avg.
Method	ImageNet	Caltech	DTD	EuroSAT	Aircraft	Food	Flowers	Pets	Cars	SUN	UCF	Avg.
MaPLe	72.30	93.53	46.49	48.06	24.74	86.20	72.23	90.49	65.57	67.01	68.69	66.30
PromptSRC	72.80	93.60	46.87	45.50	23.90	86.15	70.25	90.25	65.70	67.10	68.75	65.81
PromptKD	—	93.61	55.08	63.74	26.24	88.84	75.33	91.59	73.93	68.57	76.39	71.33
Ours	74.20	95.17	61.55	69.05	31.76	86.90	85.25	92.48	76.23	72.98	78.43	74.98
$\Delta$ vs. PromptKD	—	+1.56	+6.47	+5.31	+5.52	-1.94	+9.92	+0.89	+2.30	+4.41	+2.04	+3.65

Appendix

Table 10: Cross-dataset generalization from ImageNet. Models are adapted on ImageNet and evaluated directly on target datasets without target-domain training. The ImageNet column reports source-domain accuracy, while Avg. denotes the unweighted mean over the ten target datasets.

\Delta

reports the absolute difference between SD-LoRA and PromptKD. Best results are shown in bold .

Metric	Method	Caltech	DTD	EuroSAT	Aircraft	Food	Flowers	Pets	Cars	SUN	UCF	Avg.
ECE $\downarrow$	PromptKD	5.01	4.96	17.41	3.04	2.00	5.49	5.47	7.94	1.49	2.93	5.57
ECE $\downarrow$	Ours	1.99	3.54	6.83	2.32	1.56	3.40	0.83	3.64	1.28	1.13	2.65
NLL $\downarrow$	PromptKD	0.24	1.86	0.97	2.46	0.39	1.48	0.25	0.79	1.12	0.91	1.05
NLL $\downarrow$	Ours	0.16	1.18	0.28	1.70	0.28	0.32	0.23	0.76	0.78	0.51	0.62
Brier $\downarrow$	PromptKD	0.11	0.62	0.46	0.81	0.16	0.33	0.11	0.37	0.44	0.34	0.38
Brier $\downarrow$	Ours	0.07	0.34	0.13	0.44	0.09	0.09	0.10	0.35	0.31	0.12	0.20

Appendix

Table 11: Calibration under cross-dataset generalization. After ImageNet-source adaptation, models are evaluated on ten target datasets. We report expected calibration error (ECE), negative log-likelihood (NLL), and Brier score. Lower values indicate better calibration. Best results are shown in bold .

Variant	Enabled edge families				Aircraft			Caltech101
Variant	Spatial	Global–local	CLS–text	Patch–text	1-shot	4-shot	16-shot	1-shot	4-shot	16-shot
Full graph	✓	✓	✓	✓	34.80	46.20	61.40	94.40	96.20	97.20
w/o spatial edges	✗	✓	✓	✓	33.21	45.47	60.44	92.75	95.40	96.28
w/o global–local edges	✓	✗	✓	✓	32.36	43.98	58.12	91.92	93.92	93.84
w/o semantic text–vision edges	✓	✓	✗	✗	31.57	43.87	57.32	91.02	93.80	93.28
CLS-semantic only	✓	✓	✓	✗	33.54	45.80	60.68	93.22	95.33	96.44

Appendix

Table 13: Edge-type ablation of the graph teacher. Accuracy (%) is reported on Aircraft and Caltech101 under 1-, 4-, and 16-shot settings. The edge columns indicate whether each relation family is enabled: spatial patch–patch, global–local CLS–patch, semantic CLS–text , and semantic patch–text interactions . The full graph performs best across all settings. Removing semantic visual–text edges gives the largest degradation, followed by global–local edges, while spatial patch–patch edges provide a smaller but consistent gain.

Variant	Enabled components		Aircraft			Caltech101			EuroSAT
Variant	PPA	Graph	1-shot	4-Shot	16-Shot	1-shot	4-Shot	16-Shot	1-shot	4-Shot	16-Shot
HR only	✗	✗	32.2	43.5	59.2	92.7	93.0	95.1	74.6	88.3	91.8
HR + PPA	✓	✗	31.7	43.4	59.5	92.5	93.0	95.3	73.9	86.4	92.1
HR + Graph	✗	✓	33.6	45.1	60.2	94.2	96.1	97.1	76.1	89.9	93.2
Full model	✓	✓	34.8	46.2	61.4	94.4	96.2	97.2	76.8	91.3	94.0

Appendix

Table 14: Effect of PPA and the graph teacher. LoRA and cache-based inference are enabled in all variants. We compare high-resolution adaptation alone (HR only), adding Prototype Predictive Alignment (HR + PPA), adding the training-only graph teacher (HR + Graph), and the full model with both graph and PPA. Accuracy (%) is reported under 1-, 4-, and 16-shot settings on Aircraft, Caltech101, and EuroSAT. The graph teacher provides the dominant gain, while PPA is most effective when combined with graph-conditioned representations.

	Pooling Strategy	1-shot	2-shot	4-shot	8-shot	16-shot
EuroSAT	$\sum_{\text{All}}$ (Global)	75.9	80.1	89.5	92.2	93.6
	$\sum_{\text{Top-}M=25}$	77.0	81.2	90.0	92.6	93.9
	$\sum_{\text{Top-}M=50}$	76.8	84.9	91.3	92.7	94.0
	$\sum_{\text{Top-}M=75}$	76.7	81.0	88.8	92.4	93.2
Caltech	$\sum_{\text{All}}$ (Global)	94.1	94.8	94.9	96.0	96.8
	$\sum_{\text{Top-}M=25}$	94.2	95.0	95.0	96.2	96.9

Appendix

Table 15: Effect of token filtering in the graph-teacher readout. We compare global pooling over all graph-refined tokens with saliency-based Top-

M

filtering. Top-

M

retains only the highest-scoring patch tokens before aggregation, reducing the influence of background or weakly informative tokens. A moderate selection size, Top-

M{=}50

, provides the best overall trade-off across datasets and shot settings.

	Resolution ( $H^{\prime}\times W^{\prime}$ )	1 Shot	2 Shot	4 Shot	8 Shot	16 Shot
Caltech	$224\times 224$ (Standard)	94.1	95.6	95.4	96.8	96.9
	$336\times 336$	94.4	95.8	96.2	96.3	97.2
	$512\times 512$	91.5	92.6	93.7	94.8	95.9
Euro	$224\times 224$ (Standard)	76.5	86.6	91.5	92.4	93.9
	$336\times 336$	76.8	84.9	91.3	92.7	94.0
	$512\times 512$	75.1	82.7	90.2	91.5	92.4

Appendix

Table 16: Effect of input resolution. We compare the standard CLIP input resolution,

224\times 224

, with higher resolutions used for training-time graph reasoning. The

336\times 336

setting generally provides the best trade-off between finer token granularity and pretrained feature stability, while

512\times 512

introduces a stronger resolution shift and degrades performance.

Components			Aircraft			Caltech101			EuroSAT
Trans	RGT	Filtering	1-shot	4-shot	16-shot	1-shot	4-shot	16-shot	1-shot	4-shot	16-shot
✗	✓	✓	34.1	45.1	58.8	93.2	94.8	95.4	74.2	89.8	91.1
✓	✗	✓	33.5	45.3	57.6	92.8	93.7	95.5	73.1	88.6	89.8
✓	✓	✗	33.4	44.2	57.4	92.4	94.2	95.7	73.7	89.1	91.6
✗	✓	✗	32.7	44.9	56.5	91.5	93.3	94.5	72.8	88.2	90.8
✓	✗	✗	33.2	44.0	55.3	91.5	92.5	93.3	72.1	87.5	90.0

Appendix

Table 18: Graph teacher architecture ablation. We evaluate the contributions of the visual refinement Transformer (Trans), the relation-aware graph Transformer (RGT), and discriminative node filtering. Accuracy (%) is reported on Aircraft, Caltech101, and EuroSAT under 1-, 4-, and 16-shot settings. The full architecture performs best across all datasets and shot settings, showing that token refinement, relation-aware message passing, and foreground-focused filtering provide complementary structural supervision.

Loss Components			EuroSAT			Caltech101			Food101
$\mathcal{L}_{\text{Student}}$	$\mathcal{L}_{\text{Teacher}}$	$\mathcal{L}_{\text{PPA}}$	1-shot	4-shot	8-shot	1-shot	4-shot	8-shot	1-shot	4-shot	8-shot
✓	✗	✗	74.6	88.3	91.8	92.7	93.0	95.1	85.2	86.3	86.9
✓	✓	✗	76.2	89.7	93.5	94.2	94.9	96.8	85.7	86.7	87.2
✓	✗	✓	76.4	91.0	93.8	94.3	95.5	97.0	86.0	86.9	87.6
✓	✓	✓	76.8	91.3	94.0	94.4	96.2	97.2	86.2	87.1	87.8

Appendix

Table 19: Loss component ablation. We evaluate the contributions of the student loss

\mathcal{L}_{\mathrm{Student}}

, the graph-teacher loss

\mathcal{L}_{\mathrm{Teacher}}

, and Prototype Predictive Alignment

\mathcal{L}_{\mathrm{PPA}}

. Accuracy (%) is reported on EuroSAT, Caltech101, and Food101 under 1-, 4-, and 8-shot settings. The full objective performs best across all settings, showing that graph-based structural supervision and prototype-level alignment provide complementary training signals.

Figure 5: Effect of cross-modal interaction frequency. We ablate where semantic text–visual interaction is applied across graph layers on Aircraft under 1/4/16-shot settings. Each marker reports the peak accuracy, while the text label indicates the number of epochs required to reach 95% of the peak. Applying interaction in all layers consistently achieves the best peak accuracy across shot regimes and also converges efficiently in the 16-shot setting, reaching 95% of peak in 13 epochs compared with 18 epochs for last only and 22 epochs for none . This suggests that frequent cross-modal interaction improves both final adaptation quality and optimization efficiency.

Figure 6: Computational and deployment analysis of our method. A shows the one-time wall-clock overhead associated with the additional high-resolution training and re-encoding passes. B reports the computational overhead relative to the CLIP-LoRA training system, showing that the added graph branch increases computation by only

0.61

GFLOPs (

+1.31\%

) while keeping parameter growth modest (

+8.0\%

). C summarizes practical deployment-time metrics on ViT-B/16, including training time, memory footprint, and inference latency. D shows that the method remains well behaved across backbone choices: ViT-B/32 retains competitive average few-shot accuracy, while the higher cost of ViT-L/14 is primarily attributable to the larger backbone rather than to additional method-specific steps.

Component / Setting	Params (M)	Trainable (M)	GFLOPs	Role
Standard adaptation reference
CLIP-LoRA, 224 $\times$ 224	152.08	1.23	17.50	Standard train/infer path
SD-LoRA training path
Frozen CLIP backbone, 336 $\times$ 336	150.85	0.00	46.73	HR token extraction
+ LoRA	1.23	1.23	$\approx$ 0.00	Shared deployed update
Cache head	0.05	0.05	$\approx$ 0.00	Deployed classifier

Appendix

Table 20: Computational analysis. GFLOPs are reported per image using the same profiler. SD-LoRA separates training-time structural supervision from deployment. The 336

\times

336 pass is used only during adaptation to expose a denser ViT token grid for fine-grained graph supervision. The graph branch itself adds 0.61 GFLOPs on top of this high-resolution pass, corresponding to 1.31% overhead relative to the 336

\times

336 backbone pass. At inference, the graph reasoning module, saliency scorer, PPA head, and high-resolution teacher path are removed, reducing deployment to the 224

\times

224 CLIP-LoRA+cache student.

Hyperparameter	Symbol	Search Range	Scale
Optimizer & Backbone
Learning Rate	$\eta$	$[10^{-5},5\times 10^{-2}]$	Log-Uniform
Weight Decay	$wd$	$[10^{-6},10^{-3}]$	Log-Uniform
LoRA Rank	$r$	$[2,4,8,16]$	Integer
LoRA Alpha	$\alpha_{\text{LoRA}}$	$\{8,16,32\}$	Discrete
Training Objectives

Appendix

Table 21: Hyperparameter Optimization Space. The search selects one global configuration on validation data; selected values are reported in Table 22 and are fixed across datasets and shots.

Few-Shot VLM Adaptation

See all 9 papers

LE-GDA: Layer Ensemble Gaussian Discriminant Analysis for Training-Free Few-Shot CLIP Adaptation
Oct 26, 2026Pavodi Maniamfu, Vogel Kiketa, Zezina Kuzulu Kuetu +4VLM Adaptation Few-Shot Image Classification
Temple University Japan Campus · Universit´e de Kinshasa · Universit´e Protestante au Congo +2
Training-Only Heterogeneous Image-Patch-Text Graph Supervision for Advancing Few-Shot Learning Adapters
Mar 18, 2026Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan +2Few-Shot Image Classification VLM Adaptation
Edge Hill University, Ormskirk, Lancashire, L39 4QP
On the Few-Shot Adaptation of Vision-Language Models: A Few Gradient Signs Are All You Need
Nov 5, 2026Nairouz Mrabah, Ismail Ben Ayed, Mahdi S. Hosseini +1Few-Shot VLM Adaptation VLM Adaptation
Department of Systems Engineering, ETS Montreal, Canada · Department of Computer Science and Software Engineering, Concordia, Canada

Oct 26, 2026J/K move · Enter open · S save

LE-GDA: Layer Ensemble Gaussian Discriminant Analysis for Training-Free Few-Shot CLIP Adaptation

Pavodi Maniamfu, Vogel Kiketa, Zezina Kuzulu Kuetu, Jakin Mukendi Kabongo+3

Temple University Japan Campus · Universit´e de Kinshasa · Universit´e Protestante au Congo+2

VLM Adaptation Few-Shot Image Classification Few-Shot Learning

Open paper AskPDF arXiv

Training-free few-shot adaptation of vision-language models typically constructs a classifier from CLIP's final projected image representation, leaving intermediate representations unused. Combining them is not straightforward. Class scores from different representations differ in spread by orders of magnitude for the same query. Direct averaging allocates influence by numerical scale rather than predictive information. We propose Layer-Ensemble Gaussian Discriminant Analysis (LE-GDA), a method that fits regularized shared-covariance discriminants independently to intermediate and final representations extracted in one frozen encoder pass. LE-GDA standardizes each discriminant's class scores per query, selects a small representation multiset on labeled validation data, and combines it with the zero-shot classifier without gradient-based optimization. Across 11 datasets, four CLIP backbones, five shot counts, and three independently sampled K-shot training sets, LE-GDA achieves the highest mean accuracy of all evaluated methods on every backbone, raising the 16-shot mean from 78.01% to 79.54% over a matched GDA-CLIP (final-feature) baseline and improving that baseline for every backbone-shot combination. At one shot, LE-GDA retains a 1.20-point mean gain over the baseline even though covariance estimation is unavailable and per-query standardization is neutral. A control with richer per-class structure inside a single representation does not reproduce the gain, indicating that the benefit comes from combining representations. These results show that intermediate CLIP representations carry information beyond the final embedding and can improve training-free few-shot adaptation. Code and evaluation resources are available at https://github.com/pavodi-NM/le-gda.

Abstract

Few-Shot VLM Adaptation

Explore similar work

LE-GDA: Layer Ensemble Gaussian Discriminant Analysis for Training-Free Few-Shot CLIP Adaptation

Training-Only Heterogeneous Image-Patch-Text Graph Supervision for Advancing Few-Shot Learning Adapters

On the Few-Shot Adaptation of Vision-Language Models: A Few Gradient Signs Are All You Need