Hold-One-Shot-Out (HOSO) for Validation-Free Few-Shot CLIP Adapters

Chris Vorster; Mayug Maniparambil; Noel E. O'Connor; Noel Murphy; Derek Molloy

cs.CVMar 4, 2026

Hold-One-Shot-Out (HOSO) for Validation-Free Few-Shot CLIP Adapters

Authors: Chris Vorster, Mayug Maniparambil, Noel E. O'Connor, Noel Murphy, Derek Molloy

Organizations: ML-Labs, Dublin City University, Dublin, Ireland

Abstract

In many CLIP adaptation methods, a blending ratio hyperparameter controls the trade-off between general pretrained CLIP knowledge and the limited, dataset-specific supervision from the few-shot cases. Most few-shot CLIP adaptation techniques report results by ablation of the blending ratio on the test set or require additional validation sets to select the blending ratio per dataset, and thus are not strictly few-shot. We present a simple, validation-free method for learning the blending ratio in CLIP adaptation. Hold-One-Shot-Out (HOSO) presents a novel approach for CLIP-Adapter-style methods to compete in the newly established validation-free setting. CLIP-Adapter with HOSO (HOSO-Adapter) learns the blending ratio using a one-shot, hold-out set, while the adapter trains on the remaining few-shot support examples. Under the validation-free few-shot protocol, HOSO-Adapter outperforms the CLIP-Adapter baseline by more than 4 percentage points on average across 11 standard few-shot datasets. Interestingly, in the 8- and 16-shot settings, HOSO-Adapter outperforms CLIP-Adapter even with the optimal blending ratio selected on the test set. Ablation studies validate the use of a one-shot hold-out mechanism, decoupled training, and improvements over the naively learnt blending ratio baseline. Code is released here: https://github.com/chris-vorster/HOSO-Adapter

Figures & tables

Figure 1: CLIP 1-shot versus full test-set accuracy is strongly correlated across multiple runs. Inspired by this, we use a single hold-out shot to learn the blending ratio for CLIP adapters.

Figure 2: The optimal blending ratio reflects a dataset-specific trade-off between the CLIP prior and task adaptation. Fine-grained datasets like Stanford Cars benefit from a higher

\alpha

to learn new features, while the general-domain ImageNet performs better with a lower

\alpha

that preserves the strong prior. This variability makes any fixed ratio suboptimal, motivating our adaptive, validation-free approach.

Method	ImageNet	Caltech101	OxfordPets	StanfordCars	Flowers102	Food101	FGVCAAircraft	SUN397	DTD	EuroSAT	UCF101	Average
CLIP-Adapter ${}_{\text{ IJCV'23}}$ [ 10 ] †	63.59	92.49	87.84	74.01	93.90	78.25	32.10	69.55	65.96	84.43	76.76	74.44
Zero-Shot ${}_{\text{ ICML'21}}$ [ 23 ]	60.35	83.81	82.86	55.69	65.94	74.85	17.16	56.80	42.32	37.53	57.47	57.71
CLIP-Adapter ${}_{\text{ IJCV'23}}$ [ 10 ]	59.02	92.28	84.92	73.49	94.56	73.96	34.19	68.14	65.70	83.24	77.30	73.35
TIP-Adapter ${}_{\text{ ECCV'22}}$ [ 30 ]	57.81	88.44	81.09	58.83	78.41	72.96	21.96	64.00	54.79	67.90	64.52	64.61
SVL-Adapter ${}_{\text{ BMVC'22}}$ [ 21 ] ∗	–	90.00	89.50	54.00	87.00	78.00	29.70	65.50	63.00	97.50	70.50	–
SVL-Adapter ${}_{\text{ BMVC'22}}$ [ 21 ] §	51.40	82.90	71.10	35.00	81.40	38.80	26.40	59.60	53.80	75.10	63.70	58.11

Table 1: Transfer learning performance (16-shot, RN50).

Method	Caltech101	DTD	EuroSAT	FGVCAircraft	Food101	ImageNet	Flowers102	OxfordPets	StanfordCars	SUN397	UCF101	Average
CLIP-Adapter (best $\alpha$ ) †	95.90	71.70	85.80	45.80	89.30	71.50	97.40	92.70	82.10	75.60	84.00	81.07
CLIP-Adapter ( $\alpha=0.2$ ) ∗	94.90	59.70	70.50	34.10	89.10	71.50	93.10	92.60	73.90	74.20	80.40	75.82
HOSO-Adapter (ours)	95.40	70.67	85.30	43.23	88.97	70.93	97.23	92.27	81.50	74.67	83.43	80.33

Table 2: ViT-B/16 16-shot performance comparison. CLIP-Adapter results are shown with a fixed residual ratio (

\alpha=0.2

) and with the best-performing ratio selected per dataset (best

\alpha

). Results are averaged over three runs.

Method	Average
HOSO-Adapter	76.43
HOSO-Adapter w/o decoupled training	73.02
HOSO-Adapter w/o removing 1-shot from few-shot train set	73.35
HOSO-Adapter with 2-shots for blending ratio	76.04
HOSO-Adapter with 8-shots for blending ratio	73.68

Table 3: Ablation study on RN50 with 16-shot. We report average accuracy across 10 datasets. Per dataset table provided in Suppl. Section D.

Panel of Figure 3 — Figure 3: HOSO-Adapter performance across few-shot settings (K=2, 4, 8, 16) on the ResNet-50 backbone with averaged results over 11 datasets. Our method achieves state-of-the-art performance at K=8 and K=16, surpassing even the CLIP-Adapter Oracle baseline. The Oracle baseline, which uses a blending ratio grid-searched on the test set for each dataset, is included for reference and is not directly comparable in the validation-free setting. CLIP-Adapter’s results are the validation-free values from [ 24 ] .

Figure 4: Three example datasets that show the consistent trend: hold-one-shot-out reduces the blending ratio (green) compared to the naively learnt blending ratio that enables overfitting on the limited few-shot cases (red). See Suppl. Section E for full table.

Figure 5: Three example datasets that show the consistent trend: hold-one-shot-out (green) overfits less compared to the naively learnt blending ratio (red). See Suppl. Section F for full table.

Figure S1 : A single hold-out shot is optimal. An ablation study of the hold-out cache size in the 16-shot setting using ViT-B/16. The average accuracy across ten datasets is plotted against the cache size, expressed as a ratio of the total shots. Performance peaks at a cache size of one (ratio $\approx 0.06$ ), validating the core design choice. Larger cache sizes lead to performance declines, indicating a trade-off between allocating samples to learn the blending ratio and retaining samples for adapter training.

Method	Caltech101	DTD	EuroSAT	FGVC	Food101	ImageNet	Flowers	Pets	Cars	SUN397	UCF101	Average
HOSO-Adapter (2-shot)	89.10	47.83	53.27	19.13	79.63	59.63	73.90	84.10	58.00	62.10	66.60	63.03
HOSO-Adapter (4-shot)	90.17	56.40	35.00	23.47	79.80	60.70	85.50	87.73	62.43	65.03	70.20	65.13
HOSO-Adapter (8-shot)	91.40	61.47	77.83	27.80	80.63	61.75	91.90	87.73	67.13	67.67	74.27	71.78
HOSO-Adapter (16-shot)	93.03	66.77	83.27	34.60	80.93	62.93	95.07	89.47	73.80	69.83	78.03	75.25

Table S1 : Main results with RN50 backbone, average over three runs. Performance of HOSO-Adapter across different few-shot settings.

Method	Caltech101	DTD	EuroSAT	FGVC	Food101	Flowers	Pets	Cars	SUN397	UCF101	Average
Baseline (remove cached samples)	93.0	66.6	83.0	34.8	81.1	95.1	89.5	73.7	69.6	77.9	76.43
Keep cache in training set	91.2	63.9	84.2	34.4	73.5	94.2	79.5	70.8	65.5	76.3	73.35
Cache size of two per class	92.6	66.5	82.1	33.9	80.8	94.6	89.0	73.6	69.5	77.8	76.04
Cache size of eight per class	91.7	62.6	80.9	29.3	80.7	92.5	88.0	69.1	67.3	74.7	73.68

Table S2 : Ablation on design choices for HOSO-Adapter (RN50, 16-shot). The baseline updates the ratio every epoch using a cache of size one per class, removing cached samples from the training set.

Shots	Caltech101	DTD	EuroSAT	FGVCAircraft	Food101	Flowers102	OxfordPets	StanfordCars	SUN397	UCF101	Average
2 shots	88.5	47.7	50.7	19.1	78.8	73.6	84.7	57.5	62.0	65.7	62.83
4 shots	89.4	56.4	32.6	22.9	79.4	85.1	87.1	61.7	65.2	69.5	64.93
8 shots	91.2	62.1	77.6	28.2	80.2	91.4	86.9	66.7	67.5	74.0	72.58
16 shots	92.5	66.3	81.6	34.9	80.7	94.6	89.2	73.3	69.4	78.0	76.05

Table S3 : Performance of HOSO-Adapter with data augmentation. Results use an RN50 backbone.

Method	Caltech101	DTD	EuroSAT	FGVCAircraft	Food101	ImageNet	Flowers102	OxfordPets	StanfordCars	SUN397	UCF101	Average
CLIP-Adapter ( $\alpha=0.2$ )	94.90	59.70	70.50	34.10	89.10	71.50	93.10	92.60	73.90	74.20	80.40	75.82
CLIP-Adapter (best $\alpha$ ) †	95.90	71.70	85.80	45.80	89.30	71.50	97.40	92.70	82.10	75.60	84.00	81.07
HOSO-Adapter (ours)	95.40	70.67	85.30	43.23	88.97	70.93	97.23	92.27	81.50	74.67	83.43	80.33

Table S4 : ViT-B/16 16-shot performance comparison. CLIP-Adapter results are shown with a fixed blending ratio (

\alpha=0.2

) and with the best-performing ratio selected per dataset. HOSO-Adapter results are averaged over three runs.

Figure S2 : Per-class accuracy as a function of class type for different blending ratios (

\alpha

) on the EuroSAT dataset. The top plot shows the results for 1-shot learning, while the bottom plot shows 16-shot learning. Each coloured line represents a fixed blending ratio. The analysis highlights that the optimal ratio varies significantly between classes, particularly in the 1-shot scenario.

Figure S3 : Per-class accuracy as a function of the blending ratio (

\alpha

) on the EuroSAT dataset. The top plot corresponds to 1-shot and the bottom to 16-shot settings. Each coloured line represents a specific class. The plots show that performance generally improves from the 0-shot baseline (

\alpha=0.0

) to an optimal intermediate ratio, with the 16-shot setting yielding a broader high-performance plateau.

Figure S4 : Most datasets exhibit a trend: hold-one-shot-out reduces the blending ratio (green) compared with a learnt blending ratio that leads to overfitting on the limited few-shot cases (red).

Figure S5 : All datasets exhibit a consistent trend: hold-one-shot-out (green) exhibits less overfitting than the naively learnt blending ratio (red).

Dataset	Classes	Splits (Tr/V/Te)	Task Category	Prompt Template
ImageNet	1000	1.28M / – / 50k	objects	"a photo of a {}."
Caltech101	100	4128 / 1649 / 2465	objects	"a photo of a {}."
OxfordPets	37	2944 / 736 / 3669	pets	"a photo of a {}, a type of pet."
StanfordCars	196	6509 / 1635 / 8041	cars	"a photo of a {}."
Flowers102	102	4093 / 1633 / 2463	flowers	"a photo of a {}, a type of flower."
Food101	101	50.5k / 20.2k / 30.3k	food	"a photo of {}, a type of food."

Table S5 : Details of the 11 datasets used for few-shot evaluation.

Dataset	$\alpha=0.2$	$\alpha=0.4$	$\alpha=0.6$	$\alpha=0.8$
Caltech101	94.9	95.7	95.7	95.3
DTD	59.7	69.7	71.5	71.6
EuroSAT	70.5	80.3	84.1	85.8
FGVCAircraft	34.1	40.9	44.1	45.8
Flowers102	93.1	97.0	97.3	96.8
OxfordPets	92.6	92.7	92.2	90.7

Table S6 : CLIP-Adapter ViT-B/16 blending search (16-shot). Classification accuracy for different blending weights (

\alpha

). A low

\alpha

favours CLIP features, while a high

\alpha

favours the adapter.

Dataset	$\alpha=0.2$	$\alpha=0.4$	$\alpha=0.6$	$\alpha=0.8$
Caltech101	91.4	92.7	92.6	91.8
DTD	59.2	66.1	67.0	66.2
EuroSAT	65.4	81.4	83.3	84.0
FGVCAircraft	24.5	32.1	35.0	35.6
Food101	81.3	79.9	77.9	75.7
Flowers102	89.7	94.0	94.7	94.8

Table S7 : CLIP-Adapter RN50 blending ratio search (16-shot).

Dataset	Fixed $\alpha=0.6$	Random $\alpha\in[0,1]$	HOSO-Adapter
Caltech101	95.7	95.4	96.1
DTD	71.5	67.0	72.5
EuroSAT	84.1	79.6	86.6
FGVCAircraft	44.1	37.3	46.4
Flowers102	97.3	94.0	97.1
OxfordPets	92.2	88.3	90.4

Table S8 : Comparison of different blending ratio strategies for CLIP-Adapter (ViT-B/16, 16-shot). HOSO-Adapter uses a dynamically learned blending ratio.

Dataset	Feature Blend	Logit Blend
Caltech101	95.9	94.3
DTD	71.7	70.7
EuroSAT	85.1	85.5
FGVC Aircraft	45.4	46.4
Flowers102	97.4	96.9
OxfordPets	91.6	90.5

Table S9 : CLIP-Adapter ViT-B/16 feature vs. logit blending (16-shot; top-1 accuracy, %). Both blending methods use a blending weight of

\alpha=0.7

Dataset	0.001	0.01	0.1	0.4
DTD	65.5	66.5	66.7	66.6
EuroSAT	83.9	83.6	84.0	83.0
UCF101	76.2	77.4	78.1	77.9
Average	75.20	75.83	76.27	75.83

Table S10 : Ablation on the learning rate for the blending ratio optimiser. Experiments use RN50 with 16-shots.

Method	2-shot	4-shot	8-shot	16-shot	Average
CLIP-Adapter	59.63	60.67	61.67	63.07	61.26
HOSO-Adapter	59.63	60.70	61.75	62.93	61.25

Table S11 : ImageNet accuracy with RN50 backbone. Average of three runs.

Method	Caltech101	DTD	EuroSAT	FGVC	Food101	ImageNet	Flowers	Pets	Cars	SUN397	UCF101	Average
HOSO-Adapter (2-shot)	94.20	50.40	59.23	27.73	88.20	67.53	82.07	90.00	67.73	67.03	73.27	69.76
HOSO-Adapter (4-shot)	94.67	59.57	45.10	32.57	87.97	68.67	89.43	91.63	70.90	70.13	77.10	71.61
HOSO-Adapter (8-shot)	94.83	65.40	79.87	37.47	88.90	69.83	95.17	91.57	75.37	72.60	80.73	77.43
HOSO-Adapter (16-shot)	95.40	70.67	85.30	43.23	88.97	70.93	97.23	92.27	81.50	74.67	83.43	80.33

Table S12 : Main results with ViT-B/16 backbone, averaged over three runs. Performance of HOSO-Adapter across different few-shot settings.

Explore similar work

Training-Only Heterogeneous Image-Patch-Text Graph Supervision for Advancing Few-Shot Learning Adapters
Mar 18, 2026Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan +2Few-Shot Image Classification VLM Adaptation
Edge Hill University, Ormskirk, Lancashire, L39 4QP
Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning
May 12, 2026Yaze Zhao, Yicong Liu, Yixiong Zou +2Source-Free Domain Adaptation Few-Shot Domain Adaptation
The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models
Aug 23, 2026Liangzhi Li, Bowen Wang, Yiming Qian +3VLM Adaptation
Department of Computer Science, Qufu Normal University, China, and with Climind · Institute of Scientific and Industrial Research (SANKEN), The University of Osaka, Japan · Institute of High Performance Computing (IHPC), A*STAR, Singapore +3

Mar 18, 2026J/K move · Enter open · S save

Training-Only Heterogeneous Image-Patch-Text Graph Supervision for Advancing Few-Shot Learning Adapters

Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan, Swagat Kumar+1

Edge Hill University, Ormskirk, Lancashire, L39 4QP

Few-Shot Image Classification VLM Adaptation Cross-Modal Knowledge Distillation

Open paper AskPDF arXiv

Recent adapter-based CLIP tuning (e.g., Tip-Adapter) is a strong few-shot learner, achieving efficiency by caching support features for fast prototype matching. However, these methods rely on global uni-modal feature vectors, overlooking fine-grained patch relations and their structural alignment with class text. To bridge this gap without incurring inference costs, we introduce a novel asymmetric training-only framework. Instead of altering the lightweight adapter, we construct a high-capacity auxiliary Heterogeneous Graph Teacher that operates solely during training. This teacher (i) integrates multi-scale visual patches and text prompts into a unified graph, (ii) performs deep cross-modal reasoning via a Modality-aware Graph Transformer (MGT), and (iii) applies discriminative node filtering to extract high-fidelity class features. Crucially, we employ a cache-aware dual-objective strategy to supervise this relational knowledge directly into the Tip-Adapter's key-value cache, effectively upgrading the prototypes while the graph teacher is discarded at test time. Thus, inference remains identical to Tip-Adapter with zero extra latency or memory. Across standard 1-16-shot benchmarks, our method consistently establishes a new state-of-the-art. Ablations confirm that the auxiliary graph supervision, text-guided reasoning, and node filtering are the essential ingredients for robust few-shot adaptation. Code is available at https://github.com/MR-Sherif/TOGA.git.

Abstract

Explore similar work

Training-Only Heterogeneous Image-Patch-Text Graph Supervision for Advancing Few-Shot Learning Adapters

Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning

The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models