Few-shot Classification as Multi-instance Verification: Effective Backbone-agnostic Transfer across Domains

Xin Xu; Eibe Frank; Geoffrey Holmes

cs.CVJul 1, 2025

Few-shot Classification as Multi-instance Verification: Effective Backbone-agnostic Transfer across Domains

Authors: Xin Xu, Eibe Frank, Geoffrey Holmes

Organizations: Department of Computer Science University of Waikato Hamilton, New Zealand

Abstract

We investigate cross-domain few-shot learning under the constraint that fine-tuning of backbones (i.e., feature extractors) is impossible or infeasible---a scenario that is increasingly common in practical use cases---and show how representing few-shot classification as a series of multiple instance verification (MIV) tasks enables competitive classification performance using the static embeddings produced by frozen, "black-box" backbones. More specifically, based on this novel representation, we introduce a new approach to few-shot domain adaptation named the "MIV-head": a classification head that is backbone-agnostic and computationally efficient. We show that the core components of the MIV-head, when trained on few-shot data from a target domain, collectively yield strong performance on test data from that domain. Importantly, they do so without fine-tuning the backbone, requiring only the "meta-testing" phase. In experiments under various settings on an extension of the Meta-Dataset benchmark for cross-domain few-shot image classification, using representative off-the-shelf convolutional neural network and vision transformer backbones pretrained on ImageNet1K, we demonstrate that the MIV-head achieves highly competitive accuracy when compared to state-of-the-art "adapter" (or "partial fine-tuning") methods applied to the same backbones, while incurring substantially lower adaptation cost. Moreover, compared to our approach, well-known "classification head" approaches achieve a substantially lower level of accuracy. We share our code at: https://github.com/xxweka/MIV-head.

Figures & tables

Figure 1: Few-shot classification (FSC) represented as a series of multi-instance verification (MIV) tasks. The upper panel illustrates a standard MIV task, where a target bag and a query are paired for binary classification. The lower panel shows that FSC can be represented as a series of MIV tasks: the support set forms a set of target bags, with each class as a bag. All the bags (or classes) are paired with a query, forming a series of MIV tasks, to perform multi-class classification.

Panel of Figure 3 — Figure 3: Embedding visualizations of a support set (circles) from the Aircraft dataset, obtained with t-SNE. All embeddings are colored according to their class labels, as specified by the legend. The left and right panels of Figure 3(a) are visualizations of embeddings from the last and second-to-last block of a frozen, off-the-shelf ResNet-50 backbone, respectively, and also show the prototypes (squares) and the query (star) after processing by the MIV-head. The third panel (i.e., Figure 3(b) ) shows the embeddings of the support set, the prototypes and the query from the same backbone when applying trained TSA adapters.

Regarding …	Previous literature	Novelty of our work
Backbone	Fine-tuning backbone to	NEW: No modification of backbone model
models	achieve SOTA accuracy	weights or architecture, while being competitive
	1) Classification problem	1) “Verification with noise” (MIV) problem
Representations	2) Support samples are	2) Same-class support samples are treated as bag
and solution	treated as “noise-free”	of “noisy” instances with unknown relevance
design	& with equal relevance	3) NEW: Introduction of competitions to

Table 1: Summary of the innovations in this paper, compared to existing practices from the prior literature.

Figure 4: Illustration of the end-to-end data flow through the MIV-head (described in Section 3.1 ), including the shapes of inputs and outputs for each core component of the MIV-head. This data-flow diagram assumes a hypothetical backbone that, given an input image, produces an output in the shape of $(7\times 7\times 2048)$ from the last block, and $(14\times 14\times 1024)$ from the second-to-last block. $L$ denotes the number of classes (i.e., “number of ways”) in the support set, and $S_{l}\,(l=1,\ldots,L)$ denotes the number of support samples (i.e., “number of shots”) within the $l^{th}$ class.

Figure 5: Component 1 of the MIV-head: “pooling by attention”, described in Section 3.2 . This component derives candidates from patch-level embeddings extracted from the backbone, and transforms them into an image-level feature map. The first and second “pooling by attention” denote Equations ( 1 ) and ( 2 ), respectively.

Figure 6: Algorithm 1 lists the PyTorch-style pseudo-code that describes Component 1 of the MIV-head, where the algorithm processes an output retrieved from a single block of the backbone. Note that it uses a utility function (named “attn_pool”) that implements Equations ( 1 ) and ( 2 )—the definition of this function (in pseudo-code) is also included in Algorithm 1 .

Figure 7: Component 2 of the MIV-head: “Cross Attention Pooling” (CAP), described in Section 3.3 . This component creates prototypes, one for each class, based on the feature vectors of a query image and all images of the same class in the support set. Q , K , V represent the “query”, “key”, and “value” elements of a multi-head attention mechanism. “ Linear ” stands for linear projections from channel dimension to individual heads. The red, green, white fill-colors of Linear denote $W_{j}^{K},W_{j}^{V},\kappa$ , respectively, described by Equations ( 4 ) and ( 6 ). The cross-attention, co-excitation, and “in-attention skip-connection” mechanisms are highlighted by arrowed lines in blue, yellow and purple colors (specified by the legend). We also highlight that the “in-attention skip-connection” mechanism is a new element created for the MIV-head, non-existent in the original CAP ( Xu et al., 2025 ) . $\odot$ , $\oplus$ , “MatMul”, “Concat”, “Sigmoid” denote element-wise multiplication, element-wise addition, matrix multiplication, concatenation (of multi-heads), and sigmoid-activation operators, respectively.

Figure 8: Component 3 of the MIV-head: “multi-block computation and aggregation of logits”, described in Section 3.4 . The block-wise logits are computed using Equation ( 3 ).

dataset	Support size			Class count			Mean shot
\Block [c]2-1Test
	min	mean	max	min	mean	max	min	mean	max
Omniglot	5	89.06	319	5	19.04	45	1	4.70	9.00
Aircraft	5	341.91	497	5	9.92	15	1	36.20	79.80
Birds	5	316.73	494	5	17.40	30	1	19.80	44.00
Textures	5	282.42	498	5	6.02	7	1	46.88	96.80

Table 2: Summary statistics of the (600) test episodes for all MD+ datasets used in our main experiments.

Characteristics	MIV-head (Ours)	TSA	eTT
Image	i) Resize to $224\times 224$
pre-processing	ii) Normalize data using “ImageNet	Identical to Ours	Identical to Ours
	mean & standard deviation”
Backbones	ResNet-50 (supervised & self-supervised)	Identical to Ours	Identical to Ours
Backbones	ViT-S/16(/8) (supervised & self-supervised)	(ResNet-50)	(ViT-S/16)
Modification	No	Yes (add	Yes (add

Table 3: Summary of key model characteristics of the MIV-head (Ours) and the two baselines, TSA and eTT.

\Block [c]3-1Test
dataset	Off-the-shelf pretrained backbone models for $224\times 224$ input resolution
	Supervised ResNet-50			Self-supervised(DINO) ResNet-50
	NCC	TSA	Ours	NCC	TSA	Ours
Omniglot	50.4 $\pm$ 1.4	66.7 $\pm$ 1.4	76.5 $\pm$ 1.2	55.9 $\pm$ 1.3	68.5 $\pm$ 1.4	74.4 $\pm$ 1.2
Aircraft	54.7 $\pm$ 0.9	78.9 $\pm$ 1.1	84.4 $\pm$ 1.0	55.3 $\pm$ 0.9	80.2 $\pm$ 1.1	84.4 $\pm$ 1.0
Birds	81.0 $\pm$ 0.7	84.9 $\pm$ 0.7	87.2 $\pm$ 0.8	60.6 $\pm$ 0.9	78.5 $\pm$ 1.0	81.8 $\pm$ 1.1

Table 4: “Varying-way varying-shot”: comparison of accuracy (in %)

\pm

95% confidence interval between TSA and the MIV-head (Ours) based on all non-ILSVRC datasets in the extended MD benchmark and the same off-the-shelf backbones. A NCC head with no learnable parameters is included for reference, reflecting the classification capability purely from the backbone. The row of “Average (MD)” indicates the average accuracy across the nine original non-ILSVRC MD datasets whereas “Average (MD+)” is the average across the total 17 extended MD datasets. We also conducted a two-sided paired t-test between TSA and Ours, and bolded the higher accuracy when the

p

-value of the

t

-test is

<0.01

(i.e., significant at

99\%

confidence level).

\Block [c]4-1Test
dataset	Off-the-shelf pretrained backbone models for $224\times 224$ input resolution
	Supervised(DeiT) ViT-small			Self-supervised(DINO) ViT-small
	NCC	eTT	Ours	NCC	eTT \tabularnote Due to the “OOM” issue caused by eTT when it was applied to the backbone taking $8\times 8$ input patch-size (patch 8), the closest alternative is eTT based on the “patch 16” backbone.	Ours
		(Patch 16)			(Patch 16)	(Patch 8)	(Patch 16)
Omniglot	54.1 $\pm$ 1.3	69.4 $\pm$ 1.3	74.5 $\pm$ 1.2	61.8 $\pm$ 1.3	77.0 $\pm$ 1.3	78.6 $\pm$ 1.1	77.0 $\pm$ 1.2
Aircraft	54.9 $\pm$ 0.9	78.4 $\pm$ 1.0	79.4 $\pm$ 1.0	62.4 $\pm$ 1.0	84.1 $\pm$ 1.0	86.0 $\pm$ 0.9	83.4 $\pm$ 1.0

Table 5: “Varying-way varying-shot”: comparison of accuracy (in %)

\pm

95% confidence interval between eTT and the MIV-head (Ours) based on all non-ILSVRC datasets in the extended MD benchmark and the same off-the-shelf backbones. A NCC head with no learnable parameters is included for reference, reflecting the classification capability purely from the backbone. The row labeled “Average (MD)” indicates the average accuracy across the nine original non-ILSVRC MD datasets whereas “Average (MD+)” is the average across the total 17 extended MD datasets. We also conducted a two-sided paired t-test between eTT and Ours, and bolded the higher accuracy when the

p

-value of the

t

-test is

<0.01

(i.e., significant at

99\%

confidence level). For the DINO ViT-small backbone, the eTT(Patch-16) results are underlined if they are no better than Ours(Patch-8) model but better than Ours(Patch-16) model.

\Block [c]5-1Test

dataset

Off-the-shelf pretrained backbone models for

224\times 224

input resolution

Supervised

Self-supervised (DINO)

ResNet-50

DeiT-small

ResNet-50

ViT-small

TSA

Ours

eTT

Ours

TSA

Ours

eTT

Ours

Patch-16

Patch-8

Patch-16

Average (MD)

58.5

60.6

59.9

61.4

57.3

56.1

64.0

64.1

62.1

Table 6: “Five-way One-shot”: summary of accuracy (in %), compared between TSA/eTT and the MIV-head (Ours) using the same setting as in Tables 4 and 5 , except that “varying-way varying-shot” test datasets are replaced by “5-way 1-shot” ones. “Average (MD)” indicates the average accuracy across the nine original non-ILSVRC MD datasets, and “Average (MD+)” across the 17 extended MD datasets. Higher accuracy is bolded within each comparison. See Appendix 8.1.1 for details.

Figure 9: Comparison of adaptation cost between eTT and the MIV-head (Ours) based on non-ILSVRC datasets in the extended Meta-Dataset benchmark and the same self-supervised (DINO) backbone. The adaptation cost is measured by GFLOPs (upper panel) and training time (in seconds) per task using the same hardware (lower panel). Ours’ adaptation cost is plotted by light-colored bars, showing that ours’ GFLOPs is typically below $50\%$ , and training time is $50\%$ – $70\%$ , of eTT’s.

\Block [l]2-1Backbone
architecture	\Block [c]2-1Pretraining
data	\Block [c]2-1Pretraining
algorithm		Average (MD)
			Ours	Baseline++
DenseNet-161	ILSVRC	supervised	77.2	68.8
RegNetY-1.6GF	ILSVRC	supervised	77.4	64.3

Table 8: Average accuracies (in %) of the MIV-head (Ours) and Baseline++ with a diverse range of backbones, based on the (non-ILSVRC) MD benchmark. For ViT backbones, we follow the standard naming conventions to denote them as “ViT-[size]/[patch]” where [size] is either “B”(Base) or “L”(Large) representing model sizes, and [patch]

\in\{8,14,16\}

representing the input patch sizes.

Table 16

Table 12: Interactions between the two core components of the MIV-head, “pooling-by-attention” and “number of output blocks

N

”, based on the supervised ResNet-50 backbone and the (non-ILSVRC) MD benchmark. Note that in this analysis, CAP is enabled in the MIV-head but data augmentation is disabled.

Table 13: Interactions between the core components of the MIV-head, “pooling-by-attention + CAP” and “number of output blocks

N

”, based on the supervised ResNet-50 backbone and the (non-ILSVRC) MD benchmark. Note that in this analysis, data augmentation is disabled.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	MIV-head	TSA	eTT	“FiT Head” / Baseline++
	SGD,	Adadelta,	AdamW,	Adam,
Optimizer	momentum=0.9,	$\rho=0.9$ ,	$\epsilon=1\mathrm{e}{-4}$ ,	$\epsilon=1\mathrm{e}{-8}$ ,
	no weight-decay	no weight-decay	weight-decay=0.01	no weight-decay
Learning rate	$LR$ =0.3,	$LR_{\alpha}=0.5$ , $LR_{\beta}=1$	$LR=1\mathrm{e}{-3},$	FiT Head: $0.0035$ , Baseline++: $0.03$
	Component 2: $LR$ ,		feature adaptation: $LR$ ,
	Component 1: $0.05LR$		otherwise: $0.5LR$

Appendix

Table 14: Optimization settings

Table 15: “Five-way One-shot” setting: comparison of accuracy (%)

\pm

95% confidence interval between TSA/eTT and Ours based on all non-ILSVRC datasets in an extended MD benchmark, aligned by the same backbones. Higher accuracy is bolded within each comparison.

Algorithm	TSA	eTT	CTX	ALFA	ProtoNet	BOHB	Ours
Backbone	ResNet-34	ViT-small/16	ResNet-34	4CONV	ResNet-34	ResNet-18	ViT-small/8
Test dataset
Omniglot	82.6 $\pm$ 1.1	78.1 $\pm$ 1.2	82.2 $\pm$ 1.0	61.9 $\pm$ 1.5	68.5 $\pm$ 1.3	67.6 $\pm$ 1.2	78.6 $\pm$ 1.1
Aircraft	80.1 $\pm$ 1.0	79.9 $\pm$ 1.1	79.5 $\pm$ 0.9	63.4 $\pm$ 1.1	58.0 $\pm$ 1.0	54.1 $\pm$ 0.9	86.0 $\pm$ 0.9
CUB	83.4 $\pm$ 0.8	85.9 $\pm$ 0.9	80.6 $\pm$ 0.9	69.8 $\pm$ 1.1	74.1 $\pm$ 0.9	70.7 $\pm$ 0.9	91.8 $\pm$ 0.7
DTD	79.6 $\pm$ 0.7	87.6 $\pm$ 0.6	75.6 $\pm$ 0.6	70.8 $\pm$ 0.9	68.8 $\pm$ 0.8	68.3 $\pm$ 0.8	89.7 $\pm$ 0.6

Appendix

Table 17: More results reported from the CDFSL literature, based on non-ILSVRC datasets in Meta-Dataset (MD). Accuracies(in %) are reported, along with

\pm

95% confidence interval. All methods together with their backbones are included in the first and second rows. Our approach’s results on an off-the-shelf self-supervised (DINO) ViT-small/8 backbone are in the rightmost column. Particularly, in addition to TSA ( Li et al., 2022 ) and eTT ( Xu et al., 2022 ) , “CTX” and “ProtoNet” are from ( Doersch et al., 2020 ) , “ALFA” is the “ALFA+fo-Proto-MAML” model in ( Baik et al., 2020 ) , “BOHB” is from ( Saikia et al., 2020 ) , “/16” and “/8” denote patch-sizes of 16 and 8, respectively, of the input images taken by the corresponding backbones. All approaches except ours use their specially-trained backbones.

Figure 11: Comparison of adaptation cost between TSA and the MIV-head (Ours), based on non-ILSVRC datasets in the extended Meta-Dataset and the same self-supervised (DINO) backbone. The adaptation cost is measured in GFLOPs (upper panel) and average training time per task, in seconds, using the same hardware (lower panel). Ours’ adaptation cost is plotted by light-colored bars in which a percentage relative to the baselines’ cost is also shown. Ours’ GFLOPs is typically below $20\%$ , and training time is $60\%$ – $80\%$ , of TSA’s.

Figure 12: Comparison of adaptation cost between TSA (upper panel), eTT (lower panel), and our MIV-head approach (“Ours”), based on non-ILSVRC datasets in the extended Meta-Dataset and based on the same supervised backbones. The adaptation cost is measured by average training time per task, in seconds, using the same hardware. Our method’s adaptation cost is plotted by light-colored bars in which a percentage relative to the baselines’ cost is also shown—our method’s cost is typically $50\%$ – $70\%$ and $60\%$ – $80\%$ , respectively, of that of eTT and TSA.

Table 18: Accuracy (in %) of the MIV-head (Table 18(a) ) and Baseline++ (Table 18(b) ), with a diverse range of backbones, based on the (non-ILSVRC) MD benchmark. For ViT backbones, we follow the standard naming convention to denote them as “ViT-[size]/[patch]” where [size] is either “B”(Base) or “L”(Large) representing model sizes, and [patch]

\in\{8,14,16\}

representing patch sizes.

dataset	$N=1$		$N=2$		$N=3$
\Block [c]2-1Test
	GFLOPs	Accuracy	GFLOPs	Accuracy	GFLOPs	Accuracy
Omniglot	20440.9	65.4	25110.5	76.5	30683.2	75.4
Aircraft	10665.3	75.8	12693.0	84.4	26093.2	84.6
CUB	21958.6	88.1	26716.7	87.2	43511.6	82.8
DTD	6152.9	87.2	7174.0	89.0	16695.1	88.5

Appendix

Table 19: GFLOPs and accuracy(in %), broken down by MD datasets, used to plot Figure 10 in Section 4.4.1 .

	1	2	3	4
\Block [c]3-1Test
dataset	Full	Excluding	Excluding	Excluding
	CAP	cross-attention	“in-attention skip-	co-excitation
	module	mechanism	connection”(IASC)	(and IASC)
Omniglot	78.6	77.1	77.8	77.9
Aircraft	86.0	84.6	86.1	86.0

Appendix

Table 20: Ablation study of Component 2 (CAP) of the MIV-head based on self-supervised (DINO) ViT-small/8 backbone, by removing some of the three mechanisms within CAP (columns 2–4) from this component (column 1). See detailed description of each column in Appendix 10.2

Utilizing augmented support samples?
Test	Self-supervised(DINO)		Supervised
dataset	ViT-small/8 backbone		ResNet-50 backbone
	Yes	No	Yes	No
Omniglot	78.6	75.8	76.5	73.2
Aircraft	86.0	85.5	84.4	83.6
Birds	91.8	92.0	87.2	86.7

Appendix

Table 21: Study of the impact of data augmentation on the performance of our approach based on supervised ResNet-50 and self-supervised (DINO) ViT-small/8 backbones, by comparing performance of the MIV-head with and without data augmentation.

\Block [c]2-1Test
dataset	ResNet-18 backbones for $84\times 84$ input resolution
	SDL-ResNet-18		Off-the-shelf ResNet-18
	TSA	Ours	TSA	Ours
Omniglot	75.8 $\pm$ 1.2	79.1 $\pm$ 1.1	72.8 $\pm$ 1.3	78.5 $\pm$ 1.1
Aircraft	71.9 $\pm$ 1.1	71.5 $\pm$ 1.0	68.3 $\pm$ 1.1	68.7 $\pm$ 1.0
Birds	73.2 $\pm$ 0.9	68.9 $\pm$ 1.1	62.6 $\pm$ 1.1	61.1 $\pm$ 1.1

Appendix

Table 22: Comparison of accuracy (

\pm

95% confidence interval) between TSA and the MIV-head (Ours) based on all non-ILSVRC datasets in extended MD benchmark, and based on the same ResNet-18 backbones. The row labeled “Average (MD)” indicates the average accuracy across the nine original (non-ILSVRC) MD datasets whereas the row labeled “Average (MD+)” is the average across the total 17 extended MD datasets. We also conducted a two-sided paired t-test between TSA and Ours, and bolded the higher accuracy when the p-value of the t-test is

<0.01

Panel of Figure 13 — Figure 13: Embedding visualizations with t-SNE of the support set (circles), prototype (squares) and query (star) produced by the MIV-head vs. TSA, based on ResNet-50 backbone and the same task from the Aircraft and Omniglot datasets. For the MIV-head, embeddings from the last and second-to-last blocks of the backbone are visualized in the left and right panels, respectively, in 13(a) , 13(c) , 13(e) . Embeddings produced by TSA are visualized in 13(b) , 13(d) , 13(f) . All embeddings are colored according to their ground-truth class labels, with colors specified by the legend. Best viewed in color.

Few-Shot Domain Adaptation

Few-shot domain adaptation transfers models to a new data domain using only a small number of labeled target examples.

See all 53 papers

Sep 20, 2026

Training-Free Spectral Transductive Refinement for Cross-Domain Few-Shot Classification

Few-shot recognition with frozen visual features is especially fragile under domain shift and one-shot supervision, where a single labelled image is an unreliable estimate of its class. We ask how far this fragility can be reduced purely at test time, without retraining the encoder or augmenting the source domain. We present Spectral Transductive Refinement (STR), a training-free transductive inference rule that exploits the geometry of the complete support-query episode. Given frozen embeddings, STR builds a joint k-nearest-neighbour graph, maps the episode into a normalized-Laplacian spectral coordinate system, initializes class representatives from the labelled support, and iteratively refines them using pseudo-labelled queries. We evaluate STR under two protocols. A controlled component study with frozen ResNet-18 features shows that spectral refinement consistently improves over single-prototype spectral initialization across five shifted domains, with the largest gains in the one-shot regime where support estimates are weakest. We then benchmark STR against recent Cross-Domain Few-Shot Learning (CD-FSL) methods using the standard miniImageNet-pretrained ResNet-10 backbone over eight established target domains. Operating entirely at inference time, STR attains the highest 1-shot average among compared methods and remains competitive at 5-shot, rivalling approaches relying on heavy source-domain meta-training augmentations. Because STR is transductive, we report its setting explicitly. Diagnostics attribute its gains to iterative refinement in spectral coordinates rather than added prototype capacity, which remains inactive in our configuration.

Fahim Rahman, S. M. Tanjeeb Meheran Rohan, Md. Taimum Ibne Sayed +2

Department of Computer Science and Engineering Islamic University of Technology Gazipur, Bangladesh

Few-Shot Learning Spectral Methods Few-Shot Image Classification

Ask PDF

Sep 25, 2026

Query-Conditioned Prototype Adaptation for Cross-Domain Few-Shot Learning: Single-Query Inference, Controlled Comparisons, and Failure Modes

Cross-domain few-shot learning requires adapting a classifier to a new visual domain from very few labelled examples without target-time parameter updates. We isolate one question: under a fixed global representation, what does joint query-support adaptation contribute to prototype construction? The Within-Instance Prototypical Transformer (WIPT) implements single-query test-time prototype adaptation by jointly transforming one unlabelled query and the labelled support embeddings, then forming query-specific class means. Using a shared frozen ViT-S/16 encoder, miniImageNet source training, and CUB, EuroSAT and ISIC targets, we replicate the key comparisons across five independent training seeds. In 1-shot evaluation, WIPT improves frozen ProtoNet in every run on CUB (+0.21 percentage points) and EuroSAT (+2.07), but decreases ISIC (-0.22). In 5-shot evaluation, ProtoNet remains strongest overall, while WIPT consistently improves a capacity-matched support-only Transformer on ISIC (+0.99). Joint processing of up to five queries yields no reliable accuracy gain; in a head-only 5-shot benchmark, g = 5 reduces analytical attention-token pairs by 73% and peak allocated memory by 29% relative to g = 1, although latency is non-monotonic. Across all target/shot conditions, WIPT changes uncertain ProtoNet decisions far more than confident ones, and rescue/break decomposition accounts for the observed gains and losses. Source-shift and scorer controls further show that the benefit is not universal. Overall, WIPT provides a streaming-compatible form of test-time prototype adaptation that can improve difficult low-shot cross-domain decisions without target-time optimization.

Rushab Rasik Karania, Tomas Maul

School of Computer Science, University of Nottingham Malaysia, Semenyih, 43500, Selangor, Malaysia

Source-Free Cross-Domain Few-Shot Learning Few-Shot Learning Few-Shot Image Classification Prototype-Based Learning Test-Time Adaptation

Ask PDF

May 28, 2026

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning

Vision-Language Models (VLMs) such as CLIP demonstrate strong zero-shot generalization, but their performance significantly degrades in cross-domain scenarios with scarce target-domain training data (Cross-Domain Few-Shot Learning, CDFSL). In this paper, we focus on the target-domain few-shot finetuning in the CLIP-based CDFSL task. Prevailing finetuning paradigms uniformly align all image patch tokens with their corresponding textual embeddings. However, we find a counterintuitive phenomenon: actively pushing away certain low-similarity image tokens, termed "tail tokens", from their textual embeddings consistently improves target-domain performance. We delve into this phenomenon and provide a novel interpretation: under great domain shifts and scarce training data, the model can hardly extract semantic information from visual inputs; therefore, the common belief of alignment is valid only for tokens already containing sufficient semantic information; for tail tokens, forcing the alignment would lead to excessive overfitting to the scarce training, while breaking the alignment is more useful. Motivated by this, we propose Adaptive Tail-Head Alignment (ATHA), a novel fine-tuning strategy for CLIP that transforms the conventional uniform alignment paradigm to an adaptive alignment paradigm, with both alignment strengthening and weakening. Extensive experiments on four challenging CDFSL benchmarks validate our state-of-the-art performance. Our code is available at https://github.com/shuaiyi308/ATHA.

Shuai Yi, Yixiong Zou, Yuhua Li +1

Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.

Source-Free Cross-Domain Few-Shot Learning Source-Free Domain Adaptation Few-Shot Domain Adaptation VLM Adaptation Few-Shot Learning Vision-Language Alignment

Ask PDF

Abstract

Few-Shot Domain Adaptation

Explore similar work

Training-Free Spectral Transductive Refinement for Cross-Domain Few-Shot Classification

Query-Conditioned Prototype Adaptation for Cross-Domain Few-Shot Learning: Single-Query Inference, Controlled Comparisons, and Failure Modes

Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning