Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols

Xu Luo; Ji Zhang; Lianli Gao; Heng Tao Shen; Jingkuan Song

cs.LGFeb 28, 2026

Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols

Authors: Xu Luo, Ji Zhang, Lianli Gao, Heng Tao Shen, Jingkuan Song

Organizations: University of Electronic Science and Technology of China, Chengdu, China. · Southwest Jiaotong University, Chengdu, China. · Tongji University, Shanghai, China.

Abstract

Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation algorithms.However, there lacks a unified, rigorous evaluation protocol that is both challenging and realistic for real-world usage. In this work, we establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets, and propose the Hyperparameter Ensemble (HPE) protocol to overcome the "validation set illusion" in data-scarce regimes. Our empirical findings demonstrate that the choice of pre-trained model is the dominant factor for performance, while many sophisticated transfer methods offer negligible practical advantages over a simple full-parameter fine-tuning baseline. To explain this surprising effectiveness, we provide an in-depth mechanistic analysis showing that full fine-tuning succeeds via distributed micro-adjustments and more flexible reshaping of high-level semantic presentations without suffering from overfitting. Additionally, we quantify the performance collapse of multimodal models in specialized domains as a result of linguistic rarity using adjusted Zipf frequency scores. By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research. We make the FEWTRANS benchmark publicly available at https://github.com/Frankluox/FewTrans.

Figures & tables

Fig. 1: Average accuracy and 95% confidence intervals of single-task few-shot transfer evaluation of a pretrained DINOv2-s model on EuroSAT [ 30 ] .

Task ID	0	1	2	3	4	5	6	7	8	9
Epoch	15	15	15	40	15	40	30	20	20	30
Backbone lr	5e-05	5e-06	5e-06	1e-05	2e-06	5e-06	1e-05	2e-05	2e-05	1e-05
Head lr	0.05	0.01	0.2	0.02	0.01	0.05	0.01	0.05	0.05	0.2

TABLE I: Optimal hyperparameters vary from task to task. The pretrained model is DINOv2-small, and all tasks are 1-shot sampled from EuroSAT.

Fig. 2: The heatmaps showing how the few-shot transfer performance of a single 1-shot task sampled from EuroSAT changes with hyperparameters. We fix the number of epochs to $50$ in the left plot, and fix the head lr to $0.01$ in the second plot. The black rectangles highlight the optimal hyperparameter areas.

Fig. 3: Cross-validation cannot find good hyperparameters when the number of shots is small, regardless of the domain shift between pretraining and downstream dataset. We use a subset class of ImageNet as the training set, and use the remaining part as the downstream dataset for the left plot.

	CIFAR-100	UCF	Plant Disease	Aircraft	DTD	EuroSAT
Epoch	30	10	50	30	20	30
Backbone lr	1e-05	1e-05	2e-05	1e-06	1e-05	1e-05
Head lr	0.0005	0.01	0.005	0.005	0.001	0.01

TABLE II: Average optimal hyperparameters of few-shot transfer vary from dataset to dataset. The pretrained model is DINOv2-small, and all tasks are 1-shot.

configuration	(1,1)	(1,2)	(1,3)	(2,1)	(2,2)	(2,3)	(3,1)	(3,2)	(3,3)	Avg	lr ensemble	lr+epoch ensemble
EuroSAT	67.52	67.23	68.00	70.75	71.01	61.87	45.44	45.55	42.48	59.98	70.21	70.72
Aircraft	61.92	61.55	61.44	61.49	61.79	61.23	61.44	61.31	60.45	61.40	63.39	63.28

TABLE III: Average 1-shot transfer performance of pretrained DINOv2-small over 50 tasks: hyperparameter ensemble vs. individual hyperparameter configurations. See appendix for details.

Fig. 4: Robustness and fairness analysis of the Hyperparameter Ensemble (HPE) protocol. Left: Positive Pearson correlation ( $r=0.38$ ) between hyperparameter sensitivity and the HPE penalty, confirming that the protocol naturally penalizes volatile methods. Right: Stability of HPE Top-1 accuracy across various grid spacings (3x, 5x, 10x), demonstrating that the protocol effectively buffers against the specific choice of hyperparameter boundaries across all 10 datasets.

Models	Dataset	ImageNet-S	DTD	CIFAR-100	Flowers	UCF	EuroSAT	Quick Draw	Fungi	Plant Disease	Aircraft	Average
ResNet-50	IN-1M	63.6 $\pm$ 1.5	69.3 $\pm$ 1.1	74.3 $\pm$ 1.1	84.1 $\pm$ 1.1	76.7 $\pm$ 1.2	84.1 $\pm$ 1.0	64.8 $\pm$ 1.3	47.6 $\pm$ 1.4	72.5 $\pm$ 1.4	51.7 $\pm$ 1.4	68.9 $\pm$ 1.3
MAE-base	IN-1M	72.3 $\pm$ 1.5	74.0 $\pm$ 1.1	86.3 $\pm$ 0.8	92.1 $\pm$ 0.6	88.2 $\pm$ 0.8	88.1 $\pm$ 0.9	72.6 $\pm$ 1.2	62.1 $\pm$ 1.3	85.9 $\pm$ 0.8	52.3 $\pm$ 1.3	77.4 $\pm$ 1.1
Swin-B	IN-1M	71.5 $\pm$ 1.5	74.7 $\pm$ 1.1	87.0 $\pm$ 0.8	90.9 $\pm$ 0.8	87.8 $\pm$ 0.8	87.6 $\pm$ 0.9	74.2 $\pm$ 1.1	60.7 $\pm$ 1.3	86.1 $\pm$ 0.8	55.4 $\pm$ 1.4	77.6 $\pm$ 1.1
EsViT-SwinB	IN-1M	69.7 $\pm$ 1.5	75.3 $\pm$ 1.0	84.7 $\pm$ 0.9	93.6 $\pm$ 0.6	88.0 $\pm$ 0.8	87.8 $\pm$ 0.9	70.8 $\pm$ 1.2	62.9 $\pm$ 1.3	86.2 $\pm$ 0.8	57.7 $\pm$ 1.5	77.7 $\pm$ 1.1
ConvNext-B	IN-1M	74.6 $\pm$ 1.5	73.9 $\pm$ 1.1	88.3 $\pm$ 0.7	90.8 $\pm$ 0.8	89.1 $\pm$ 0.8	86.4 $\pm$ 1.0	75.1 $\pm$ 1.1	60.8 $\pm$ 1.3	84.2 $\pm$ 0.9	55.7 $\pm$ 1.4	77.9 $\pm$ 1.1
IBOT-ViT	IN-1M	69.7 $\pm$ 1.5	74.9 $\pm$ 1.1	89.6 $\pm$ 0.7	93.6 $\pm$ 0.6	89.1 $\pm$ 0.8	89.1 $\pm$ 0.8	69.9 $\pm$ 1.2	60.8 $\pm$ 1.3	86.3 $\pm$ 0.8	56.7 $\pm$ 1.5	78.0 $\pm$ 1.1

TABLE IV: Sub-benchmark of FewTrans that compares the few-shot transferability of different pretrained models. We use all-parameter finetune as the transfer algorithm for all models. We temporarily do not evaluate pretrained models that use larger architectures.

	ImageNet-S	DTD	CIFAR-100	Flowers	UCF	EuroSAT	Quick Draw	Fungi	Plant Disease	Aircraft	Average
Linear	72.1 $\pm$ 1.5	76.7 $\pm$ 1.1	83.7 $\pm$ 0.8	95.5 $\pm$ 0.4	91.6 $\pm$ 0.7	81.5 $\pm$ 1.0	70.8 $\pm$ 1.1	56.8 $\pm$ 1.4	75.3 $\pm$ 1.1	68.0 $\pm$ 1.3	77.2 $\pm$ 1.1
Finetune	73.1 $\pm$ 1.5	79.9 $\pm$ 1.0	88.0 $\pm$ 0.8	95.9 $\pm$ 0.5	93.0 $\pm$ 0.6	87.6 $\pm$ 0.9	78.9 $\pm$ 1.0	58.9 $\pm$ 1.4	83.7 $\pm$ 0.9	70.7 $\pm$ 1.3	81.0 $\pm$ 1.0
LoRA	73.8 $\pm$ 1.5	80.7 $\pm$ 1.0	88.7 $\pm$ 0.7	96.1 $\pm$ 0.4	93.3 $\pm$ 0.6	87.7 $\pm$ 0.9	78.0 $\pm$ 1.1	59.4 $\pm$ 1.4	83.3 $\pm$ 0.9	71.0 $\pm$ 1.3	81.2 $\pm$ 1.0
BitFit	73.6 $\pm$ 1.5	79.7 $\pm$ 1.0	89.3 $\pm$ 0.7	96.8 $\pm$ 0.4	93.3 $\pm$ 0.6	86.5 $\pm$ 0.9	77.3 $\pm$ 1.1	61.3 $\pm$ 1.3	83.5 $\pm$ 0.9	71.0 $\pm$ 1.2	81.2 $\pm$ 1.0
SSF	74.2 $\pm$ 1.5	80.3 $\pm$ 1.0	89.0 $\pm$ 0.7	96.7 $\pm$ 0.4	93.2 $\pm$ 0.6	87.4 $\pm$ 0.9	77.3 $\pm$ 1.1	60.8 $\pm$ 1.3	84.4 $\pm$ 0.9	70.7 $\pm$ 1.3	81.4 $\pm$ 1.0
Adapter	74.1 $\pm$ 1.5	80.5 $\pm$ 1.0	89.8 $\pm$ 0.7	96.9 $\pm$ 0.4	93.6 $\pm$ 0.5	86.5 $\pm$ 0.9	77.3 $\pm$ 1.0	61.2 $\pm$ 1.3	83.2 $\pm$ 0.9	70.9 $\pm$ 1.2	81.4 $\pm$ 1.0

TABLE V: Sub-benchmark of FewTrans that compares different transfer algorithms for pure vision pretrained models. The visual encoder of CLIP-base is chosen as the pretrained model.

	ImageNet-S	DTD	CIFAR-100	Flowers	UCF	EuroSAT	Quick Draw	Fungi	Plant Disease	Aircraft	Average
Zero-shot	72.6 $\pm$ 1.5	73.0 $\pm$ 1.0	92.9 $\pm$ 0.4	86.3 $\pm$ 0.9	90.5 $\pm$ 0.6	64.4 $\pm$ 1.2	57.4 $\pm$ 1.2	38.7 $\pm$ 1.5	46.0 $\pm$ 1.4	69.2 $\pm$ 1.2	69.1 $\pm$ 1.2
CoOp	79.3 $\pm$ 1.3	83.8 $\pm$ 0.8	93.8 $\pm$ 0.4	97.8 $\pm$ 0.2	95.1 $\pm$ 0.4	84.3 $\pm$ 0.8	73.8 $\pm$ 0.9	51.9 $\pm$ 1.5	70.9 $\pm$ 1.2	70.0 $\pm$ 1.4	80.1 $\pm$ 1.0
ProGrad	79.4 $\pm$ 1.3	82.3 $\pm$ 0.8	93.9 $\pm$ 0.4	96.2 $\pm$ 0.3	94.7 $\pm$ 0.4	84.1 $\pm$ 0.8	72.5 $\pm$ 0.9	53.8 $\pm$ 1.4	71.6 $\pm$ 1.1	73.2 $\pm$ 1.2	80.2 $\pm$ 0.9
VPT	78.8 $\pm$ 1.3	81.3 $\pm$ 0.8	94.5 $\pm$ 0.3	95.5 $\pm$ 0.4	94.5 $\pm$ 0.4	88.3 $\pm$ 0.7	75.1 $\pm$ 0.9	47.4 $\pm$ 1.5	72.9 $\pm$ 1.1	76.5 $\pm$ 1.1	80.5 $\pm$ 0.9
MaPLe	79.2 $\pm$ 1.3	82.5 $\pm$ 0.8	94.6 $\pm$ 0.3	96.5 $\pm$ 0.4	95.1 $\pm$ 0.4	88.8 $\pm$ 0.7	76.3 $\pm$ 0.9	48.9 $\pm$ 1.5	74.6 $\pm$ 1.1	74.5 $\pm$ 1.1	81.1 $\pm$ 0.9
KgCoOp	79.9 $\pm$ 1.2	84.1 $\pm$ 0.7	94.1 $\pm$ 0.4	97.5 $\pm$ 0.2	95.3 $\pm$ 0.4	84.7 $\pm$ 0.8	74.1 $\pm$ 0.9	55.2 $\pm$ 1.5	72.9 $\pm$ 1.1	73.9 $\pm$ 1.2	81.2 $\pm$ 0.9

TABLE VI: Sub-benchmark of FewTrans that compares different transfer algorithms for base evaluation of multi-modal pretrained models. CLIP-base is chosen as the pretrained model.

	ImageNet-S	DTD	CIFAR-100	Flowers	UCF	EuroSAT	Quick Draw	Fungi	Plant Disease	Aircraft	Average
CoCoOp	67.5 $\pm$ 1.4	66.7 $\pm$ 1.1	86.8 $\pm$ 0.5	77.5 $\pm$ 1.1	87.9 $\pm$ 0.7	67.5 $\pm$ 1.3	61.4 $\pm$ 1.1	21.8 $\pm$ 1.2	59.4 $\pm$ 1.4	47.9 $\pm$ 1.7	64.4 $\pm$ 1.2
CoOp	64.3 $\pm$ 1.5	71.5 $\pm$ 1.0	86.5 $\pm$ 0.6	85.0 $\pm$ 0.8	86.1 $\pm$ 0.7	71.5 $\pm$ 1.2	60.9 $\pm$ 1.2	31.5 $\pm$ 1.4	65.0 $\pm$ 1.2	46.2 $\pm$ 1.9	66.8 $\pm$ 1.2
ProGrad	65.0 $\pm$ 1.5	71.7 $\pm$ 1.0	86.7 $\pm$ 0.5	85.0 $\pm$ 0.8	86.3 $\pm$ 0.7	72.0 $\pm$ 1.2	61.1 $\pm$ 1.2	32.5 $\pm$ 1.4	65.7 $\pm$ 1.2	49.7 $\pm$ 1.8	67.6 $\pm$ 1.2
VPT	71.7 $\pm$ 1.3	67.7 $\pm$ 1.0	87.5 $\pm$ 0.6	84.5 $\pm$ 0.8	86.4 $\pm$ 0.7	68.1 $\pm$ 1.4	56.7 $\pm$ 1.2	37.0 $\pm$ 1.3	56.9 $\pm$ 1.4	61.5 $\pm$ 1.3	67.8 $\pm$ 1.1
MaPLe	70.4 $\pm$ 1.3	62.1 $\pm$ 1.2	88.3 $\pm$ 0.5	82.4 $\pm$ 0.8	87.3 $\pm$ 0.6	77.1 $\pm$ 1.3	60.8 $\pm$ 1.1	34.3 $\pm$ 1.3	62.2 $\pm$ 1.3	56.2 $\pm$ 1.4	68.1 $\pm$ 1.1
KgCoOp	68.9 $\pm$ 1.4	72.7 $\pm$ 0.9	87.0 $\pm$ 0.5	86.6 $\pm$ 0.7	87.8 $\pm$ 0.7	70.6 $\pm$ 1.2	60.6 $\pm$ 1.1	33.9 $\pm$ 1.4	66.7 $\pm$ 1.2	51.9 $\pm$ 1.8	68.7 $\pm$ 1.2

TABLE VII: Sub-benchmark of FewTrans that compares different transfer algorithms for base-to-novel evaluation of multi-modal pretrained models. CLIP-base is chosen as the pretrained model.

Dataset	Full-FT (%)	LoRA (%)	Diff (%)	$p$ -value	Cohen’s $d$
Aircraft	63.39	63.81	-0.42	0.4859	-0.10
CIFAR100	82.21	84.45	-2.24	0.0003	-0.55
Fungi	55.73	55.84	-0.11	0.8646	-0.02
ILSVRC	85.49	87.17	-1.68	0.0004	-0.54
Quick Draw	72.45	73.23	-0.78	0.1794	-0.19
Textures	68.69	69.76	-1.07	0.0562	-0.28

TABLE VIII: Paired t-test results and Cohen’s

d

effect sizes between Full-FT and LoRA (

N=600

tasks per dataset).

Method	Total Time (600 Tasks)	Peak GPU Memory
Full Fine-tuning	13.3 hours	$\approx$ 12,100 MB
LoRA	5.5 hours	$\approx$ 7,100 MB

TABLE IX: Computational requirements for evaluating 600 tasks on a single GPU.

Fig. 5: Analysis of the adaptation mechanism for full fine-tuning and LoRA. (Left) Layer-wise parameter update scales ( $L_{2}$ norm of $\Delta W$ across 12 Transformer blocks, showing that Full fine-tuning relies on distributed micro-adjustments ( $0.01\sim 0.07$ ) to avoid overfitting. (Right) Feature distribution shift measured by Centered Kernel Alignment (CKA) similarity at deep layers, where Full fine-tuning demonstrates more effective reshaping of high-level semantic representations compared to the constrained adaptation of LoRA.

Fig. 6: Quantitative analysis of text domain shift on CLIP few-shot adaptation. The scatter plot illustrates a strong negative Spearman rank correlation (

\rho=-0.881,p=0.0039

) between text semantic rarity (adjusted Zipf score) and few-shot adaptation gain. Datasets with rarer linguistic terms (e.g., Fungi, plantD) exhibit significant zero-shot performance collapses, necessitating indispensable correction via full fine-tuning to realign visual features.

Explore similar work

Are We Really Doing Few-Shot Learning? A Critical Examination of Pre-Training Assumptions
Sep 9, 2026Alejandro Galan-Cuenca, Marcelo Saval-Calvo, Antonio Javier GallegoDomain Adaptation Few-Shot Learning
University Institute for Computer Research, University of Alicante, Alicante, Spain
Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation
Jul 20, 2026Claudia Grosser, Maike Heuer, Denis Krompass +1Language Model Pretraining LLM Evaluation
Technical University Munich · Siemens AG
The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data
Mar 17, 2026Christina Baek, Ricardo Pio Monti, David Schwab +31Language Model Pretraining Domain-Adaptive Pretraining
DatologyAI

Sep 9, 2026J/K move · Enter open · S save

Are We Really Doing Few-Shot Learning? A Critical Examination of Pre-Training Assumptions

Alejandro Galan-Cuenca, Marcelo Saval-Calvo, Antonio Javier Gallego

University Institute for Computer Research, University of Alicante, Alicante, Spain

Domain Adaptation Few-Shot Learning Few-Shot Image Classification

Open paper AskPDF arXiv

Few-shot learning is commonly evaluated under protocols that pre-train a model on a large auxiliary set whose classes are disjoint from the target episodes yet drawn from the same visual domain. This paper examines whether such protocols truly reflect low-data learning. We systematically compare no pre-training, class-disjoint in-domain pre-training, supervised out-of-domain pre-training, and label-free out-of-domain pre-training across eight datasets, three few-shot architectures, and multiple way-shot settings. Our results show that class disjointness alone is insufficient to remove the influence of target-domain data. In-domain pre-training improves over no pre-training by 33.41 percentage points on average, whereas supervised out-of-domain pre-training yields 23.75 percentage points, revealing a 9.66-point optimistic bias associated with domain overlap. Although out-of-domain pre-training is more realistic in applications where target-domain data are scarce, its effectiveness depends strongly on the compatibility between source and target domains. We further show that labeled source data are not strictly required, with an augmentation-based label-free strategy reaching an average gain of 27.71 percentage points and closely matching supervised out-of-domain pre-training at 27.97 percentage points. Finally, we introduce a descriptor-based source-selection strategy that estimates source-domain suitability before pre-training, reaching a median gap of only 1.37 percentage points to oracle selection. These findings highlight the need to move beyond in-domain pre-training as the default few-shot evaluation protocol, since it can overestimate performance in realistic scenarios where target-domain data are scarce.

Abstract

Explore similar work

Are We Really Doing Few-Shot Learning? A Critical Examination of Pre-Training Assumptions

Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation

The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data