Enabling Validation for Robust Few-Shot Recognition

Hanxin Wang; Tian Liu; Shu Kong

cs.CVJun 5, 2025

Enabling Validation for Robust Few-Shot Recognition

Authors: Hanxin Wang, Tian Liu, Shu Kong

Organizations: University of Macau · Texas A&M University · Institute of Collaborative Innovation

Abstract

A key challenge in Few-Shot Recognition (FSR) is the scarcity of labeled task-specific data for training. Contemporary methods mitigate this challenge by exploiting pretrained Vision-Language Models (VLMs) for their learned transferable features. Some recent ones augment the scarce data by retrieving relevant yet out-of-distribution (OOD) examples from external data sources. However, they largely overlook an equally fundamental challenge: the absence of validation data for model selection and hyperparameter tuning. As a result, the adapted VLMs overfit the few-shot training data and generalize poorly to OOD test data. We show that retrieved data provide a natural source of validation, but their direct use creates a fundamental paradox. Since the retrieved samples are inherently OOD w.r.t the few-shot ID training data, adapting a VLM to the ID data constantly degrades its performance on the retrieved data. Consequently, conventional validation relying on the accuracy of retrieved data favors no adaptation and prevents improved generalization. To resolve this dilemma, we formulate validation as a multi-objective optimization (MOO) problem between performance gains on the few-shot ID data and the performance preservation on the retrieved data. We introduce a simple yet effective validation strategy by choosing an operating point on the Pareto front of the MOO problem. This strategy enables hyperparameter tuning, which had been unrealistically done by using test data, thereby mitigating overfitting and improving generalization. We integrate it into a stage-wise finetuning pipeline, termed Validation-Enabled Stage-wise Tuning (VEST). Extensive experiments on five established benchmarks show that VEST consistently outperforms existing approaches on both ID and OOD test data.

Figures & tables

Figure 1: Summary of benchmarking results of VLM adaptation methods w.r.t accuracies on the ImageNet ID and OOD test data. We use circle size to mark the number of learned parameters of each method. All the compared methods adapt a pretrained CLIP ViT-B/16 VLM [ 43 ] with the 16-shot ImageNet ID data. Refer to § 4 for dataset details, Fig. 6 for the compared methods, and Table 2 for detailed results. Our method VEST (§ 3.4 ) significantly outperforms existing methods in both ID and OOD accuracies, with a moderate number of learned parameters. Notably, unlike VEST, CLAP, FFT, and SWAT, other methods artificially use the ImageNet ID test set for tuning.

Figure 2: From the ImageNet-OOD benchmark, we display two classes with images from the ID training set (marked by black border), the OOD test sets (marked by blue border ), and the retrieved images from the external data source LAION-400M [ 51 ] (marked by orange border ). The retrieved examples are not only OOD compared with the ID images, but also noisy due to linguistic ambiguity. For example, lemon also means a pale yellow color, hence an image of a lemon/yellow umbrella, on the top right, is retrieved.

Figure 3: Left: Solely relying on the accuracy on training data or the retrieved data (serving as the validation set) selects the checkpoints at the last epoch or the first epoch. Right: These checkpoints yield notably lower accuracy on the ID and, especially, the OOD test data than what an oracle would select, e.g., checkpoints at epoch 3-7 marked by the red circle. It is also worth noting that checkpoints selected by the oracle would achieve generally higher accuracy on both ID and OOD test data than most others.

Figure 4: Left: Maximizing the gF1 score selects the epoch-3 checkpoint. Right: We compare this checkpoint with others selected by different methods: the arithmetic mean of accuracies on the ID training data and retrieved data, the accuracy on the retrieved data only, the accuracy on the ID training data only, the rank aggregation method Borda count [ 1 ] , F-

\beta

score, and no selection (i.e., using the last checkpoint). Clearly, gF1’s selection performs robustly well and generalizes to both ID and OOD test data.

harmonic mean	ID test	OOD test
harmonic mean	IN	avg.	IN-V2	IN-S	IN-A	IN-R
w/o min-sub	72.24	60.24	65.21	48.36	50.95	76.43
w/ min-sub	72.58	60.94	65.54	49.39	50.72	78.12

Table 1: Min-subtraction helps focus on the change regions in the Pareto front (Fig. 4 -left) and thus derives a more robust metric for validation.

Panel of Figure 5 — Figure 5: Layer selection via the proposed validation strategy. (A) With a specific $k$ , we partially finetune (PFT) the top- $k$ layers of the pretrained visual encoder and plot the accuracies of checkpoints on the ID training data and retrieved data. (B) For each checkpoint, we calculate its gF1 score. These scores are not comparable across $k$ due to their inconsistent start and end points in (A). (C) We address this issue by first selecting the checkpoint based on gF1 for each PFT of top- $k$ layers, and then plotting the accuracies on the ID training data and retrieved data of all the selected checkpoints. This plot allows fair comparisons of the gF1 scores of these selected checkpoints. (D) We recompute their gF1 scores and determine to PFT the top-4 layers. Right: We list the detailed results on ID and OOD test sets of finetuning different top- $k$ layers, showing that the selected checkpoint, or finetuning the top-4 layers, indeed performs robustly well, remarkably outperforming linear probing and full finetuning.

Figure 6: Diagrams of prevailing VLM adaptation methods. We categorize them into four groups. (a) Linear Probing (LP) learns a linear classifier on top of a pretrained visual encoder [ 43 ] . Finetuning (FT) learns to update either (b) the visual encoder via a cross entropy loss [ 32 ] or (c) the entire VLM via a contrastive loss [ 11 ] . Prompt Tuning (PT) learns either (d) visual prompts [ 20 ] or (e) text prompts [ 70 ] in the input space with a frozen VLM. Adapter Learning (AL) freezes the VLM and learns lightweight modules, such as (f) low-rank matrices [ 17 , 66 ] or (g) some layers of parameters [ 67 ] .

				16-shot						8-shot		4-shot
				ID	OOD					ID	OOD	ID	OOD
	method	venue & year	validation	IN	avg.	IN-V2	IN-S	IN-A	IN-R	IN	avg.	IN	avg.
Zero-Shot	Prompting	ICML’21	–	66.75	57.15	60.91	46.16	47.53	73.99	66.75	57.15	66.75	57.15
Zero-Shot	CuPL	ICCV’23	–	69.64	60.04	63.37	49.03	50.68	77.07	69.64	60.04	69.64	60.04
Prompt Tuning	CoOp	IJCV’22	ID test acc.	71.69	59.24	64.34	47.20	50.20	75.21	70.80	59.68	70.29	59.74
	CoCoOp ∗	CVPR’22	ID test acc.	71.02	59.91	64.07	48.75	50.63	76.18	–	–	–	–

Table 2: Benchmarking results w.r.t top-1 accuracy on the ID test set ImageNet (IN) and the OOD test sets ImageNet-V2 (IN-V2), ImageNet-Sketch (IN-S), ImageNet-A (IN-A), and ImageNet-Rendition (IN-R). Zero-shot methods serve as references. For convenient comparison, for each method, we report averaged accuracy (marked in blue ) across the four OOD test sets. We run all the methods with their original validation protocols over the CLIP ViT-B/16 model [ 43 ] , under 16-, 8-, and 4-shot settings. Numbers for marked methods are copied from their papers. Particularly, for methods that originally exploit a validation set of ID data for tuning, we use the accuracy on the ID test set to tune them, reporting their over-optimistic results and marking them as “ ID test acc. ”. Both SWAT and VEST adopt RA, using the same retrieved data during training. Enabled by our validation strategy, VEST resoundingly outperforms SWAT on the ID and OOD test data.

Stage-1		Stage-2	accuracy		wall-clock time ( $h$ )
tuning	RA	tuning	ID test	OOD test	wall-clock time ( $h$ )
F	✕	✕	72.58	60.94	0.5
F	✓	✕	72.89 +0.31	63.28 +2.34	6.4
F	✓	F	74.49 +1.91	63.58 +2.64	6.4 + 0.5
F	✓	P	74.74 +2.16	64.09 +3.15	6.4 + 0.4
P	✕	✕	73.35	61.80	0.4

Table 3: Ablation study. We compare different combinations of stage-1 and stage-2 finetuning strategies, with or without RA, and with either full finetuning (“F”) or partial finetuning (“P”). All models use our gF1 for checkpoint selection, with wall-clock time reported for 10 epochs of finetuning. Incorporating RA in stage-1 yields notable gains in OOD accuracy, while subsequent stage-2 further improves both ID and OOD performance. VEST, highlighted in the last row , achieves the best ID and OOD accuracy with decent training efficiency.

	ckpt	ID test set	OOD test set
w/o val	50	72.32	59.33
rand-25	4	72.75	60.82
rand-50	3	72.58	60.94
rand-100	3	72.58	60.94
high-quality	4	72.75	60.82
low-quality	3	72.58	60.94

Table 4: Robustness of gF1 to validation data quality and quantity. We construct validation sets of varying quality (top vs. bottom 50 retrieved images by similarities between retrieved image caption and target class names), and quantity (random 25, 50, and 100 images per class). Using these sets for FFT checkpoint selection demonstrates that gF1 is highly robust to retrieved data variations used for validation.

Table 11

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

dataset	# classes	# test data
ImageNet	1,000	50,000
ImageNet-V2	1,000	10,000
ImageNet-S	1,000	50,000
ImageNet-A	200	7,500
ImageNet-R	200	30,000
Living17	17	3,400

Appendix

Table 7: Statistics of datasets for evaluating OOD generalization. We list the number of classes and images in the test sets for all the datasets we used. Following the few-shot OOD evaluation protocol, we randomly sample

\{4,8,16\}

images from the ID training set to adapt the VLM (referred to as our training set ) and evaluate the model’s performance on both ID and OOD test sets. Notably, we do not exploit a validation set. Instead, we propose a novel validation strategy by exploiting retrieved open data.

Figure 9: Validation for tuning the learning rate. We apply the proposed validation strategy to tune the learning rate of the backbone and classifier for stage-2 finetuning of VEST. For each candidate learning rate combination, we use gF1 to select a checkpoint and report its ID test accuracy (z-axis) and OOD test accuracy (surface color). We observe strong performance on both ID and OOD test data across a broad range of backbone learning rates from 1e-6 to 1e-8 when the classifier learning rate is fixed at 1e-2. Importantly, gF1 selects a learning rate combination within the high-performing region.

		Validation-free			Validation by our gF1
	method	ID test	OOD test	avg.	ID test	OOD test	avg.
PT	VPT shallow [ 20 ]	72.60	57.46	65.03	72.43	60.74	66.59 +1.56
AL	LoRA [ 17 ]	71.53	56.55	64.04	72.84	60.66	66.75 +2.71
LP	CMLP [ 30 ]	73.16	59.32	66.24	72.49	60.66	66.58 +0.34
FT	FLYP [ 11 ]	73.89	58.15	66.02	73.63	59.48	66.56 +0.54

Appendix

Table 8: Performance comparison between validation-free and validation by our gF1 across four representative finetuning methods. We apply gF1 for checkpoint selection and report the ID and OOD test accuracy, along with their average. Clearly, gF1 consistently improves the performance of different finetuning methods.

Figure 10: Checkpoint selection on DINOv2. We evaluate different validation strategies for checkpoint selection after fully finetuning a DINOv2 ViT-B/14 for 150 epochs. The right table presents the selected checkpoints (ref. ckpt) and their detailed performance on ID and OOD test data. The results demonstrate that our gF1 score generalizes effectively to vision foundation models, consistently selecting a checkpoint that achieves robust performance on both ID and OOD test data.

finetuned layers	gF1 ( $\times 10^{-2}$ )	overall average	ID test	OOD test
finetuned layers	gF1 ( $\times 10^{-2}$ )	overall average	IN	avg.	IN-V2	IN-S	IN-A	IN-R
linear probing	0.00	68.29	76.72	59.86	68.82	52.45	51.20	66.97
top-1	0.43	68.89	77.15	60.62	69.44	52.85	52.76	67.44
top-2	0.33	68.87	77.66	60.07	69.58	52.35	53.07	65.29
top-3	0.00	68.70	77.80	59.60	69.86	51.72	53.32	63.51
top-4	0.10	68.62	77.82	59.42	69.97	51.48	53.49	62.73

Appendix

Table 9: Layer selection on DINOv2. We perform partial finetuning (PFT) on the top-

k

layers of the vision model. We report top-1 accuracy on ID and OOD test data, along with the overall average, computed as the mean of ID accuracy and average OOD accuracy. A clear trade-off exists between ID and OOD test accuracy across different values of

k

. Our proposed validation method selects PFT top-1 layers, which provides a good balance between ID and OOD performance and remarkably outperforms linear probing and full finetuning.

			ID test	OOD test
	methods	validation	IN	IN-V2	IN-S	IN-A	IN-R	avg.
Prompt Tuning	CoOp IJCV’22	ID test acc.	71.74 $\pm$ 0.16	64.57 $\pm$ 0.22	47.54 $\pm$ 0.31	50.40 $\pm$ 0.25	75.58 $\pm$ 0.36	59.52 $\pm$ 0.28
	MaPLe CVPR’23	ID test acc.	71.05 $\pm$ 0.17	64.27 $\pm$ 0.33	48.74 $\pm$ 0.45	50.72 $\pm$ 0.20	77.22 $\pm$ 0.34	60.24 $\pm$ 0.29
	VPT shallow ECCV’22	ID test acc.	72.61 $\pm$ 0.20	65.51 $\pm$ 0.30	49.08 $\pm$ 0.21	49.72 $\pm$ 0.47	78.51 $\pm$ 0.13	60.70 $\pm$ 0.23
	VPT deep ECCV’22	ID test acc.	72.63 $\pm$ 0.14	65.14 $\pm$ 0.23	47.67 $\pm$ 0.23	47.28 $\pm$ 0.50	76.59 $\pm$ 0.16	59.17 $\pm$ 0.26
Adapter Learning	Tip-Adapter ECCV’22	ID test acc.	70.53 $\pm$ 0.03	63.26 $\pm$ 0.09	48.68 $\pm$ 0.06	50.90 $\pm$ 0.09	77.80 $\pm$ 0.07	60.16 $\pm$ 0.04

Appendix

Table 10: Averaged accuracy with standard deviations. For each method, we run three times using different few-shot training data and report the average accuracy and standard deviation on ImageNet (as ID test set) and its variants (as OOD test sets). Our proposed methods demonstrate superior accuracy and stability.

Table 19

method	validation	ID	OOD
method	validation	Sketch	avg.	Real	Quickdraw	Painting	Infograph	Clipart
zero-shot	–	62.83	55.80	83.23	13.84	65.12	45.61	70.19
FFT	val-free	70.17	58.83	83.70	17.52	67.67	51.02	74.23
FFT	gF1	70.34	59.34	84.37	18.06	68.50	50.85	74.93
SWAT	val-free	69.95	59.48	83.17	18.71	68.30	53.06	74.16
PFT ours	gF1	70.41	59.50	84.54	18.09	68.93	51.26	74.70

Appendix

Table 13: gF1 enables VEST to achieve strong performance on DomainNet dataset. Using gF1 for checkpoint selection improves performance over the final checkpoint of few-shot finetuning (FFT), a default validation-free configuration. Furthermore, gF1 enables layer selection for partial finetuning (PFT), leading to further gains. With the proposed validation strategy, our VEST significantly outperforms SWAT on both ID and OOD test data.

Figure 11: Comparing ID training data, OOD test data (marked by blue border ), and retrieved data from LAION-400M (marked by orange border ). To illustrate the distributional characteristics, we present examples of the training and testing images by randomly selecting two classes from each dataset. The comparison reveals significant distribution shifts between ID and OOD data. Notably, although the retrieved data enhances sample diversity, it also introduces noisy images due to linguistic ambiguity in the retrieval process [ 32 ] .

Explore similar work

Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective
Dec 11, 2025Tian Liu, Anwesha Basu, James Caverlee +1VLM Adaptation Few-Shot Learning
Texas A&M University · University of Macau · Institute of Collaborative Innovation
Training-Free Spectral Transductive Refinement for Cross-Domain Few-Shot Classification
Sep 20, 2026Fahim Rahman, S. M. Tanjeeb Meheran Rohan, Md. Taimum Ibne Sayed +2Few-Shot Learning Spectral Methods
Department of Computer Science and Engineering Islamic University of Technology Gazipur, Bangladesh
PRiSM: Prototype Regularization for Few-Shot VLMs
Jul 20, 2026Ghassen Baklouti, Omprakash Chakraborty, Jose Dolz +1VLM Adaptation Prototype Learning
ÉTS Montreal

Dec 11, 2025J/K move · Enter open · S save

Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective

Tian Liu, Anwesha Basu, James Caverlee, Shu Kong

Texas A&M University · University of Macau · Institute of Collaborative Innovation

VLM Adaptation Few-Shot Learning Pseudo-Labeling Semi-Supervised Learning

Open paper AskPDF arXiv

Semi-supervised few-shot learning (SSFSL) resembles real-world applications such as auto-annotation, as it aims to learn a model from a few labeled and abundant unlabeled task-specific examples to annotate the unlabeled ones. Despite the availability of powerful open-source Vision-Language Models (VLMs) and open-world data, existing SSFSL literature largely neglects these resources. In contrast, the related area few-shot learning (FSL) has already exploited them to boost performance. Arguably, to solve real-world auto-annotation, SSFSL should leverage such open resources. To bridge this gap, we explore established SSL methods to finetune a VLM. Unexpectedly, they significantly underperform FSL baselines that do not use unlabeled data. Our in-depth analysis reveals the root cause of failure: VLMs produce flat distributions of softmax probabilities, resulting in zero utilization of unlabeled data and weak supervision signals. To address this challenge, we propose an embarrassingly simple solution that uses temperatures to sharpen the softmax output, which not only increases the confidence scores of pseudo-labels to improve the utilization of unlabeled data, but also strengthens training supervision for effective finetuning. Furthermore, we exploit task-relevant open data, e.g., those retrieved from VLMs' publicly available pretraining set. To mitigate the imbalance and domain gaps in retrieved data, we employ a stage-wise training strategy. Building on the successful finetuning of VLMs and the exploitation of open data, we present a simple yet effective SSFSL method, Stage-Wise Finetuning with Temperatures (SWIFT). Across five benchmarks, SWIFT outperforms recent FSL and SSL methods by

\sim

5 accuracy points. SWIFT even rivals supervised learning, which finetunes a VLM assuming unlabeled data having ground-truth labels!

Abstract

Explore similar work

Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective

Training-Free Spectral Transductive Refinement for Cross-Domain Few-Shot Classification

PRiSM: Prototype Regularization for Few-Shot VLMs