Organizations: Department of Mathematics and Applied Mathematics, School of Mathematics and Information Science, Guangxi University, Nanning, Guangxi, 530004, China · Department of Mathematics, School of Science, Shanghai University, Shanghai, 200444, China · School of Economics, Guangxi University, Nanning, Guangxi, 530004, China
The feature-extraction quality of pre-trained models sets the ceiling of downstream transfer learning, yet few-shot model selection remains difficult: supervised evaluation on scarce labels is unstable and yields fluctuating rankings. We propose Entropy-Guided Model Evaluation (EGiME), a semi-supervised criterion that assesses a candidate model from the clustering structure of its feature space without any classifier training. EGiME performs K-means clustering over all labeled and unlabeled samples, then combines two entropy terms: an inter-cluster entropy measuring global separability, and an intra-cluster entropy measuring the compactness of labeled anchors, fused through an inverse-frequency weight that emphasizes small clusters. On five few-shot benchmarks and nine pre-trained models, EGiME achieves a correlation score of 0.966 with the full-pool ground-truth accuracy obtained by averaging the accuracies of K-means, KNN, and SVM. At labeling ratios of 4%--20%, it produces markedly more stable model rankings than cross-validated supervised baselines, establishing it as a practical alternative to conventional supervised evaluation for pre-trained model selection under data scarcity.
Figures & tables
Figure 1: The feature-extracted sample distributions derived by AlexNet (left) and ViT-B/16 (right) from the Mini-ImageNet dataset and visualized by t -SNE. Different colors represent different categories.
Figure 2: Overall framework of the proposed EGiME methodology. The pipeline consists of three main stages: Feature embedding, structure discovery, and entropy-based evaluation.
Algorithm 1 Fold-Averaged EGiME
Input: Raw image dataset D , model set M={M1,…,Mq} , cluster number K=s , labeled sample indices and corresponding labels, number of stratified folds F
2: Extract the full-pool feature set Df(r) and partition the labeled samples into F stratified folds;
3: for each fold j=1,…,F do
4: Compute the class-mean centroids of the current labeled fold, and run K-means once from these centroids ( ninit=1 ) on the complete pool Df(r) ;
Table 1: EGiME: Fold-Averaged Entropy-Guided Model Evaluation
Model
Architecture
Feature Dim.
AlexNet
CNN
4096
VGG11
CNN
4096
VGG13
CNN
4096
GoogLeNet
CNN
1024
ResNet18
CNN
512
ResNet34
CNN
512
Table 2: Specifications of the Pre-trained Models Evaluated in the Experiments
Figure 3: EGiME as a low-label proxy for the three-predictor ground-truth accuracy. The overall regression score of R2=0.966 indicates that EGiME preserves the accuracy trend and can serve as a practical pre-screening alternative to fully supervised model selection.
Dataset
AlexNet
VGG11
VGG13
GoogLeNet
ResNet18
ResNet34
ViT-S
ViT-B
ViT-L
Mini-val
7
9
8
6
5
3
4
1
2
STL-10
7
8
9
6
5
4
3
2
1
CIFAR-10
7
8
9
4
5
3
6
1
2
EuroSAT
7
8
9
4
3
1
5
6
2
CIFAR-100
7
8
9
6
5
4
2
3
1
Table 3: Ground-Truth Rankings of Pre-trained Models on Five Benchmark Datasets, Obtained by Averaging the Accuracies of K-means, KNN, and SVM
Dataset
Method
Labels Per Class ( nshot )
4
6
8
10
12
14
16
18
20
Mini-val
CV(KNN)
3.01
2.88
2.72
2.43
2.25
2.33
2.23
1.92
1.81
CV(SVM)
3.38
3.74
4.56
4.70
4.82
4.59
4.70
4.64
4.53
LEEP
12.59
11.91
12.04
12.22
11.82
11.80
11.83
11.54
11.28
LogME
3.87
3.96
4.14
4.12
4.29
4.26
4.30
4.22
4.24
PACTran
8.23
8.78
9.24
9.31
9.42
9.52
9.62
9.71
9.74
Table 4: Average Ranking Mistakes on Five Datasets Under Different Labeling Proportions
Figure 4: Average ranking mistakes of the six methods as a function of the number of labeled samples per class, drawn as waterfall surfaces; the vertical axis is logarithmic, since LEEP and PACTran exceed the remaining methods by up to an order of magnitude. Within each panel the methods run from front (EGiME) to back (LEEP).
Figure 6: Score distributions across different labeled sample sizes (continued). (D) EuroSAT, (E) CIFAR-100.
Dataset
Strategy
Labels Per Class ( nshot )
4
6
8
10
12
14
16
18
20
Mini-val
K-means
2.87
2.62
2.09
2.00
1.92
1.83
1.70
1.54
1.38
Equal Partition
35.97
35.97
35.97
35.97
35.97
35.97
35.97
35.97
35.97
STL-10
K-means
2.14
1.71
1.60
1.37
1.30
1.29
1.28
1.11
1.18
Equal Partition
13.63
14.95
15.41
15.16
17.85
13.06
16.41
14.06
17.85
CIFAR-10
K-means
3.66
3.48
3.39
3.11
2.83
2.28
2.75
2.40
2.15
Table 5: Ablation Study (A): Ranking Errors of K-means vs. Equal Partition
Dataset
Average ΔRE
ErA=Er(inter)
ErB=1/(1+Er(intra))
Mini-val
+1.00
+2.00
STL-10
+1.33
+1.33
CIFAR-10
+3.89
+3.00
EuroSAT
+1.78
+10.11
CIFAR-100
+0.11
+4.11
Table 6: The contribution of ErA and ErB . Increase in Ranking Error ( ΔRE=REvariant−REFull ) after Removing Each Entropy Component. Smaller ΔRE indicates the removed component is less critical.
Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation algorithms.However, there lacks a unified, rigorous evaluation protocol that is both challenging and realistic for real-world usage. In this work, we establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets, and propose the Hyperparameter Ensemble (HPE) protocol to overcome the "validation set illusion" in data-scarce regimes. Our empirical findings demonstrate that the choice of pre-trained model is the dominant factor for performance, while many sophisticated transfer methods offer negligible practical advantages over a simple full-parameter fine-tuning baseline. To explain this surprising effectiveness, we provide an in-depth mechanistic analysis showing that full fine-tuning succeeds via distributed micro-adjustments and more flexible reshaping of high-level semantic presentations without suffering from overfitting. Additionally, we quantify the performance collapse of multimodal models in specialized domains as a result of linguistic rarity using adjusted Zipf frequency scores. By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research. We make the FEWTRANS benchmark publicly available at https://github.com/Frankluox/FewTrans.
Xu Luo, Ji Zhang, Lianli Gao +2
University of Electronic Science and Technology of China, Chengdu, China. · Southwest Jiaotong University, Chengdu, China. · Tongji University, Shanghai, China.
Few-shot learning is commonly evaluated under protocols that pre-train a model on a large auxiliary set whose classes are disjoint from the target episodes yet drawn from the same visual domain. This paper examines whether such protocols truly reflect low-data learning. We systematically compare no pre-training, class-disjoint in-domain pre-training, supervised out-of-domain pre-training, and label-free out-of-domain pre-training across eight datasets, three few-shot architectures, and multiple way-shot settings. Our results show that class disjointness alone is insufficient to remove the influence of target-domain data. In-domain pre-training improves over no pre-training by 33.41 percentage points on average, whereas supervised out-of-domain pre-training yields 23.75 percentage points, revealing a 9.66-point optimistic bias associated with domain overlap. Although out-of-domain pre-training is more realistic in applications where target-domain data are scarce, its effectiveness depends strongly on the compatibility between source and target domains. We further show that labeled source data are not strictly required, with an augmentation-based label-free strategy reaching an average gain of 27.71 percentage points and closely matching supervised out-of-domain pre-training at 27.97 percentage points. Finally, we introduce a descriptor-based source-selection strategy that estimates source-domain suitability before pre-training, reaching a median gap of only 1.37 percentage points to oracle selection. These findings highlight the need to move beyond in-domain pre-training as the default few-shot evaluation protocol, since it can overestimate performance in realistic scenarios where target-domain data are scarce.
Alejandro Galan-Cuenca, Marcelo Saval-Calvo, Antonio Javier Gallego
University Institute for Computer Research, University of Alicante, Alicante, Spain
Machine learning models -- including prominent zero-shot models -- are often trained on datasets whose labels are only a small proportion of a larger label space. Such spaces are commonly equipped with a metric that relates the labels via distances between them. We propose a simple approach to exploit this information to adapt the trained model to reliably predict new classes -- or, in the case of zero-shot prediction, to improve its performance -- without any additional training. Our technique is a drop-in replacement of the standard prediction rule, swapping argmax with the Fréchet mean. We provide a comprehensive theoretical analysis for this approach, studying (i) learning-theoretic results trading off label space diameter, sample complexity, and model dimension, (ii) characterizations of the full range of scenarios in which it is possible to predict any unobserved class, and (iii) an optimal active learning-like next class selection procedure to obtain optimal training classes for when it is not possible to predict the entire range of unobserved classes. Empirically, using easily-available external metrics, our proposed approach, Loki, gains up to 29.7% relative improvement over SimCLR on ImageNet and scales to hundreds of thousands of classes. When no such metric is available, Loki can use self-derived metrics from class embeddings and obtains a 10.5% improvement on pretrained zero-shot models such as CLIP.