LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation

Zhipeng Ye; Feng Jiang; Qiufeng Wang; Hao Li

cs.CVOct 15, 2026

LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation

Authors: Zhipeng Ye, Feng Jiang, Qiufeng Wang, Hao Li

Organizations: Taizhou Institute of Science and Technology, Nanjing University of Science and Technology, Taizhou 225300, Jiangsu, China · Department of Intelligence Science, Xi’an Jiaotong-Liverpool University, Suzhou 215123, Jiangsu, China · School of Computer Science and Technology, University of Arizona, Tucson 85705, AZ, USA

Abstract

Few-shot adaptation often relies on an encoder's final global representation, leaving complementary information in intermediate layers underused. Combining these features is challenging because their dimensions can differ and their usefulness varies across tasks. We propose LAMP, Layerwise Adaptive Mixing of Predictors, a training-free framework that combines global and regional information from multiple encoder layers. A single forward pass per view supplies all descriptors. Global and support-calibrated regional kernel predictors independently fit the same label residuals. Labeled validation data select their mixture in class-score space, preserving native feature dimensions without learned projections. LAMP-F further adapts text-anchored class prototypes, using a cached response to update the fixed-kernel residual predictions exactly. Experiments cover eleven datasets and ViT and ResNet backbones. LAMP achieves state-of-the-art mean accuracy among the compared published training-free ResNet-50 methods at 4, 8, and 16 shots, reaching 78.73% at 16 shots. Averaged across five support budgets, LAMP improves over a paired global-feature-only baseline by 1.14 and 1.59 percentage points on ViT-B/16 and ResNet-50; LAMP-F raises these gains to 1.96 and 2.54 points. LAMP-F reaches 16-shot accuracies of 83.57% and 79.10%, respectively. Controlled experiments identify multi-layer integration as the main source of improvement and examine pooling, calibration, validation labels, and computational cost. These findings show that combining layer-specific predictions can exploit complementary features from frozen encoders to improve few-shot classification.

Figures & tables

Figure 1: Training-free LAMP with ViT-B/16 for one query $q$ . (a) The frozen encoder provides global descriptors, query regional matrices $\mathbf{Q}_{\ell}(q)$ , and support tensors $\mathbf{S}_{\ell}$ ; branches $\ell=1,2,3$ correspond to blocks 4, 8, and 12. Row labels $1,2,3,\ldots,C$ denote class indices. (b) Independent predictors fit the shared support-label residual $\mathbf{R}_{\mathbf{T}}$ . The effective kernel $\mathcal{K}_{b}$ is $K_{g}$ for the global branch and $\widehat{K}_{\ell}$ for a calibrated regional branch. (c) Their corrections $\boldsymbol{\Delta}_{b}(q)$ are weighted and added to the CLIP prior. Here $N=VCK$ , $V=10$ , $J=4$ , $D_{\ell}=768$ , and $N_{p}=196$ . Prediction uses the selected parameters; Section 3.5 describes LAMP-F. Asterisks mark the displayed support subset and illustrative prediction.

Method	Venue/year	1	2	4	8	16
CoOp (4) [ 2 ]	IJCV, 2022	67.2	70.5	73.0	76.4	79.0
CoOp (16) [ 2 ]	IJCV, 2022	67.6	70.8	74.0	76.8	80.0
CoCoOp [ 12 ]	CVPR, 2022	68.8	70.4	71.7	73.8	75.4
Tip-Adapter-F [ 5 ]	ECCV, 2022	70.9	73.7	75.6	78.3	80.7
CLIP-Adapter [ 18 ]	IJCV, 2024	65.7	66.8	67.7	71.5	75.5
PLOT++ [ 16 ]	ICLR, 2023	70.7	74.0	76.9	79.6	82.1

Table 1: ViT-B/16 comparison: mean top-1 accuracy (%) over eleven datasets. Published scores are taken from CLIP-LoRA [ 19 ] and CAPEL [ 17 ] ; results for LAMP and LAMP-F are averaged over three seeds. Boldface and underlining indicate the highest and second-highest means in each column. The numbers in CoOp (4)/(16) denote context length. Venue/year identifies each method’s publication; PLOT++ is the enhanced PLOT implementation used in CLIP-LoRA. PR denotes Pattern Recognition, and CVPRW denotes CVPR Workshops. Evaluation protocols are described in Section 4.1 .

Method	Venue/year	1	2	4	8	16
GDA [ 21 ]	ICLR, 2024	62.19	66.19	69.77	73.30	76.04
Tip-Adapter [ 5 ]	ECCV, 2022	62.83	64.84	66.50	68.64	70.02
Tip-Adapter with RFF [ 5 , 6 ]	CVPR, 2025	62.77	64.58	66.29	68.15	69.60
APE [ 20 ]	ICCV, 2023	64.43	66.48	68.66	70.75	72.81
APE with RFF [ 20 , 6 ]	CVPR, 2025	64.16	66.11	68.76	70.37	72.02
APE+SSP [ 7 ]	ACM MM, 2024	66.03	68.04	69.90	71.63	73.72

Table 2: RN50 comparison: mean top-1 accuracy (%) over eleven datasets. Published scores are taken from Table 2 of ProKeR [ 6 ] , including variants using random Fourier features (RFF), and Table 3 of SSP [ 7 ] . Results for LAMP and LAMP-F are averaged over three seeds. The published methods and LAMP are training-free; LAMP-F optimizes class prototypes. Boldface and underlining indicate the highest and second-highest values in each column across all rows. APE+SSP uses the CuPL prompts specified in the source publication. Venue/year identifies the publication reporting each method or RFF variant. Evaluation protocols are described in Section 4.1 .

Dataset	Tip-Adapter-F	PLOT++	CLIP-LoRA	CAPEL	LAMP	LAMP-F
	[ 5 ]	[ 16 ]	[ 19 ]	[ 17 ]
ImageNet [ 25 ]	73.4	72.6	73.6	73.73	74.10	74.39
SUN397 [ 26 ]	76.0	76.0	76.1	77.11	76.35	76.36
FGVC Aircraft [ 27 ]	44.6	46.7	54.7	48.09	49.83	50.58
EuroSAT [ 28 ]	85.9	92.0	92.1	88.73	93.16	93.68
Stanford Cars [ 29 ]	82.3	84.6	86.3	84.40	84.66	84.85

Table 3: ViT-B/16 16-shot top-1 accuracy (%) for representative published methods. Boldface and underlining indicate the highest and second-highest scores in each row. We retain the numerical precision and mean accuracies reported in the source publications. Evaluation protocols are described in Section 4.1 .

ViT-B/16
$K$	Global control	LAMP	LAMP-F	$\Delta$ LAMP	$\Delta$ LAMP-F
1	$71.17\pm 0.47$	$72.18\pm 0.46$	$73.28\pm 0.40$	$1.00\pm 0.06$	$2.11\pm 0.20$
2	$73.32\pm 0.49$	$74.27\pm 0.43$	$75.62\pm 0.45$	$0.96\pm 0.10$	$2.30\pm 0.10$
4	$76.26\pm 0.12$	$77.70\pm 0.23$	$78.80\pm 0.20$	$1.45\pm 0.26$	$2.54\pm 0.09$
8	$79.34\pm 0.09$	$80.65\pm 0.11$	$80.94\pm 0.16$	$1.30\pm 0.09$	$1.59\pm 0.19$
16	$82.30\pm 0.09$	$83.30\pm 0.15$	$83.57\pm 0.13$	$0.99\pm 0.08$	$1.27\pm 0.08$

Table 4: Benchmark accuracy on eleven datasets, including ImageNet. Accuracy is averaged equally across datasets for each support-sampling seed; entries report the mean

\pm

sample SD across seeds 1–3. The final row also averages the five shot counts. Gains are paired differences from our paired reproduction of the ProKeR global predictor, in percentage points.

	LAMP minus global		LAMP-F minus LAMP
Dataset	ViT-B/16	RN50	ViT-B/16	RN50
ImageNet	$+0.168\pm 0.137$	$+0.127\pm 0.030$	$+0.575\pm 0.066$	$+0.835\pm 0.035$
Caltech101	$+0.419\pm 0.328$	$+0.852\pm 0.226$	$+0.203\pm 0.361$	$+1.041\pm 0.366$
Pets	$+0.191\pm 0.330$	$+0.145\pm 0.186$	$+0.372\pm 0.576$	$+0.073\pm 0.126$
Cars	$+0.858\pm 0.402$	$+1.729\pm 1.296$	$+1.206\pm 0.914$	$+1.190\pm 0.549$
Flowers102	$+1.164\pm 0.394$	$+2.328\pm 0.131$	$+0.392\pm 0.680$	$+0.758\pm 0.587$

Table 5: Paired accuracy gains at 4 shots on all eleven benchmark datasets, in percentage points. LAMP minus global measures the gain from training-free multi-layer integration; LAMP-F minus LAMP measures the additional gain from prototype adaptation. Entries report the mean

\pm

sample SD across three support-sampling seeds. For the mean row, dataset gains are first averaged within each seed.

Comparison or representation	ViT-B/16	RN50
Native-layer versus global information (gain)
REGION $-$ MULTIBAND	+1.286	+1.635
Depth and pooling (accuracy)
REGION: four descriptors per depth	82.993	77.060
CENTROID: mean of regional descriptors	82.991	77.067
WHOLE: one descriptor per depth	83.021	76.988

Table 6: Representation controls at 4-shot with 113 mixture candidates per family. Top: mean gains in percentage points over ten non-ImageNet datasets. Bottom: mean accuracy (%) over Pets, Food101, DTD, and EuroSAT, averaged equally across datasets and three support seeds. MULTIBAND uses the final CLIP global descriptor; FINAL uses the deepest native feature map.

Figure 2: Effect of the validation-label budget at four shots on DTD and EuroSAT. Points show mean test accuracy over ten new support draws, and error bars indicate one sample SD. The full validation split contains 24 images per class on DTD and 540 on EuroSAT. Methods and support draws use the same nested validation subsets; the complete parameter search is repeated for each setting.

Models	ResNet-50	ResNet-101	ViT-B/32	ViT-B/16
Zero-shot CLIP	60.33	62.53	63.80	68.73
Tip-Adapter	61.43	64.08	65.18	70.25
APE	62.60	65.61	66.31	71.37
IDEA	62.58	65.51	65.93	71.07
ProKeR	64.45	67.39	68.12	73.25
LAMP (Ours)	65.73	68.52	69.12	74.10

Table 7: Training-free ImageNet 16-shot accuracy (%) across four CLIP backbones. External scores come from ProKeR Table 6 [ 6 ] and IDEA Table 2 [ 22 ] . LAMP reports means over seeds 1–3. Bold marks the highest accuracy among the listed methods in each column; gains over ProKeR are in percentage points.

Shot	Global-only	LAMP	Gain	95% interval
1	65.750	66.946	+1.196	[+0.873, +1.523]
2	68.491	70.030	+1.539	[+1.240, +1.832]
4	72.333	74.146	+1.814	[+1.536, +2.104]
8	75.873	78.206	+2.333	[+1.989, +2.685]
16	79.790	81.927	+2.137	[+1.893, +2.384]

Table 8: Training-free RN101 accuracy averaged equally over the ten non-ImageNet datasets. LAMP and the global-feature-only baseline use the same support banks. Accuracy is in percent; gains and pointwise 95% paired-bootstrap intervals are in percentage points.

Backbone	Target	Global-only	LAMP	Gain [95% interval]
ViT-B/16	ImageNet-V2	65.843	66.297	+0.453 [+0.110, +0.837]
ViT-B/16	ImageNet-Sketch	49.258	49.910	+0.652 [+0.363, +0.914]
RN50	ImageNet-V2	56.220	57.077	+0.857 [+0.537, +1.177]
RN50	ImageNet-Sketch	35.771	36.631	+0.860 [+0.689, +1.059]

Table 9: Transfer of frozen 16-shot models: mean top-1 accuracy (%) over three seeds and LAMP’s paired gain over the global-feature-only baseline. Gains are in percentage points, with pointwise 95% confidence intervals from 5,000 paired bootstrap draws.

Figure 3: Regional parameter profiles at 4-shot, averaged equally over eleven datasets and three support seeds. Each point is the validation gain over the global baseline after reselecting the other regional parameters. Left: mixture strength

\rho

in Eq. ( 10 ), with an inset showing the small-

\rho

values. Right: bandwidth ratio

r=\beta_{f}/\beta_{g}

for the regional kernel in Eq. ( 6 ).

Figure 4: Effective weights of the global and regional corrections at 4 shots, averaged over seeds 1–3. The four coefficients in Eq. ( 10 ) are $(1-\rho,\rho w_{1},\rho w_{2},\rho w_{3})$ ; their weighted corrections are added to the shared CLIP prior. Both panels use the same 0–1 scale.

Dataset	Backbone	Method	Latency	Throughput	GPU peak
DTD	ViT-B/16	Global-only	12.52	128.71	1.82
		LAMP	42.38	124.15	2.23
		LAMP-F	42.70	127.02	2.29
	RN50	Global-only	12.67	138.47	1.84
		LAMP	23.40	137.79	2.02
		LAMP-F	23.64	137.43	2.11

Table 10: Deployment cost at 16 shots, measured on one RTX 4090 with seed 1. Latency is the median time from image decoding to classification (ms) at batch size one. Throughput (images/s) and peak allocated GPU memory (GiB) are measured at batch size 128, separately for each method. Timing includes decoding, preprocessing, encoding, and classification; disk retrieval is excluded.

Few-Shot Learning

Few-shot learning develops methods that recognize or adapt to new tasks from only a handful of labeled examples.

See all 213 papers

LE-GDA: Layer Ensemble Gaussian Discriminant Analysis for Training-Free Few-Shot CLIP Adaptation
Oct 26, 2026Pavodi Maniamfu, Vogel Kiketa, Zezina Kuzulu Kuetu +4VLM Adaptation Few-Shot Image Classification
Temple University Japan Campus · Universit´e de Kinshasa · Universit´e Protestante au Congo +2
PRiSM: Prototype Regularization for Few-Shot VLMs
Jul 20, 2026Ghassen Baklouti, Omprakash Chakraborty, Jose Dolz +1VLM Adaptation Prototype Learning
ÉTS Montreal
Prompt-Anchored Residual Adaptation for Biomedical Vision-Language Models
Sep 27, 2026Jingxuan Kang, Qianying Yue, Che Liu +1VLM Adaptation Few-Shot Learning
Imperial College London · The Chinese University of Hong Kong

Oct 26, 2026J/K move · Enter open · S save

LE-GDA: Layer Ensemble Gaussian Discriminant Analysis for Training-Free Few-Shot CLIP Adaptation

Pavodi Maniamfu, Vogel Kiketa, Zezina Kuzulu Kuetu, Jakin Mukendi Kabongo+3

Temple University Japan Campus · Universit´e de Kinshasa · Universit´e Protestante au Congo+2

VLM Adaptation Few-Shot Image Classification Few-Shot Learning

Open paper AskPDF arXiv

Training-free few-shot adaptation of vision-language models typically constructs a classifier from CLIP's final projected image representation, leaving intermediate representations unused. Combining them is not straightforward. Class scores from different representations differ in spread by orders of magnitude for the same query. Direct averaging allocates influence by numerical scale rather than predictive information. We propose Layer-Ensemble Gaussian Discriminant Analysis (LE-GDA), a method that fits regularized shared-covariance discriminants independently to intermediate and final representations extracted in one frozen encoder pass. LE-GDA standardizes each discriminant's class scores per query, selects a small representation multiset on labeled validation data, and combines it with the zero-shot classifier without gradient-based optimization. Across 11 datasets, four CLIP backbones, five shot counts, and three independently sampled K-shot training sets, LE-GDA achieves the highest mean accuracy of all evaluated methods on every backbone, raising the 16-shot mean from 78.01% to 79.54% over a matched GDA-CLIP (final-feature) baseline and improving that baseline for every backbone-shot combination. At one shot, LE-GDA retains a 1.20-point mean gain over the baseline even though covariance estimation is unavailable and per-query standardization is neutral. A control with richer per-class structure inside a single representation does not reproduce the gain, indicating that the benefit comes from combining representations. These results show that intermediate CLIP representations carry information beyond the final embedding and can improve training-free few-shot adaptation. Code and evaluation resources are available at https://github.com/pavodi-NM/le-gda.

Abstract

Few-Shot Learning

Explore similar work

LE-GDA: Layer Ensemble Gaussian Discriminant Analysis for Training-Free Few-Shot CLIP Adaptation

PRiSM: Prototype Regularization for Few-Shot VLMs

Prompt-Anchored Residual Adaptation for Biomedical Vision-Language Models