Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection

Adhemar de Senneville; Xavier Bou; Jérémy Anger; Rafael Grompone; Gabriele Facciolo

cs.CVMar 25, 2026

Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection

Authors: Adhemar de Senneville, Xavier Bou, Jérémy Anger, Rafael Grompone, Gabriele Facciolo

Organizations: Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · École Polytechnique, AMIAD, France · Institut Universitaire de France

Abstract

Current Large Vision Language Models (LVLMs) excel at many zero-shot tasks like image captioning, visual question answering and OCR. However, these same models suffer from poor performance at image classification tasks, underperforming against CLIP-based methods. Notably, this gap is surprising because many LVLMs use CLIP-pretrained vision encoders. Yet LVLMs are not inherently limited by CLIP's architecture with independent vision and text encoders. In CLIP, this separation biases classification toward class-name matching rather than joint visual-text reasoning. In this paper we show that, despite their poor raw performance, LVLMs can improve visual feature class separability at inference using prompt conditioning, and LVLMs' internal representations, especially attention heads, can outperform the model itself at zero-shot and few-shot classification. We introduce Head Ensemble Classifiers (HEC) to bridge the performance gap between CLIP-based and LVLM-based classification methods. Inspired by Gaussian Discriminant Analysis, HEC ranks the most discriminative vision and text heads and combines them into a training-free classifier. We show that HEC achieves state-of-the-art performance in few-shot and zero-shot classification across 12 datasets.

Figures & tables

Figure 1: CLIP-based vs HEC (Ours). CLIP-based methods encode class names and support set images independently to construct a zero-shot and a few-shot classifier respectively. Our method keeps the same two-classifier structure. However, both distributions go through a shared LLM decoder which can be conditioned by a text prompt to include guidance on the domain or the classes of the support set. The few-shot classifier (HEC-V) builds on the distribution of a sparse set of heads from the LLM decoder. This subset, which we refer to as vision-heads , is selected using Gaussian Discriminant Analysis [ 6 ] ( Fig. 2 ). The zero-shot classifier (HEC-T) builds on the distribution of another sparse set of heads, which we refer to as text-heads . Similarly to CLIP-based methods, the two classifiers can be combined in a single one (HEC-VT) by adding their output probabilities.

Figure 2: Overview of HEC-V. Given a prompt, we first encode all the images from the support set with the LVLM. We then extract the distribution of attention vectors ( 3 ) for the last token across all heads in every layer. Then, based on a Gaussian Discriminant Analysis [ 6 ] , we rank each head based on its class separability. Lastly, given a query image, we ensemble the predictions of the top $k$ heads for that task by averaging their class probabilities.

Panel of Figure 3 — Figure 3: Experiments to identify where the best classification representation lies in LVLMs. At different locations of Qwen2-VL, we compute linear probing accuracy averaged over a thousand 10-way 4-shot tasks across 10 datasets using Class conditioning. (a) Although vision tokens yield strong accuracy early on, inherited from the CLIP vision transformer, the last token builds better representations by integrating multimodal features from vision and text prompt tokens. (b) For each few-shot setup, a small number of top heads called vision-heads yield better performance than the summary token. (c) Similarly, for each zero-shot setup, a small number of top heads called text-heads yield better performance than the summary token.

Panel of Figure 4 — Figure 4: Top head attention map. We concatenate bird [ 62 ] and aircraft [ 47 ] datasets images horizontally in one support set. We then select the top vision-head for bird classification using the prompt What type of bird is this? and do the same for plane using the prompt What type of plane is this? . The attention map of the bird (left) and plane (right) top vision-head is overlaid on top of the image.

Model	Method	PETS	ESAT	UCF	SUN	CAL	DTD	AIR	FOOD	FLWR	CARS	BIRD	SIGN	AVG
DINOv1	Probing	$81.9$	$\mathbf{81.6}$	$71.8$	$50.8$	$86.9$	$49.6$	$25.5$	$37.9$	$88.1$	$28.3$	$54.2$	$46.3$	$58.6$
DINOv2	Probing	$78.5$	$73.6$	$67.6$	$63.4$	$87.0$	$51.6$	$29.9$	$43.0$	$97.0$	$35.9$	$65.5$	$35.6$	$60.7$
DINOv3	Probing	$88.0$	$77.2$	$82.8$	$72.7$	$95.4$	$63.6$	$56.9$	$74.3$	$\mathbf{99.5}$	$79.4$	$\mathbf{77.3}$	$53.1$	$76.7$
OpenAI CLIP	Probing	$72.9$	$73.6$	$81.5$	$73.2$	$90.2$	$55.8$	$28.2$	$73.4$	$89.5$	$56.9$	$53.6$	$56.5$	$67.1$
OpenCLIP	Probing	$73.0$	$75.6$	$79.5$	$72.1$	$90.4$	$60.3$	$29.6$	$62.9$	$89.0$	$75.0$	$51.2$	$62.7$	$68.4$
DFN	Probing	$84.5$	$80.8$	$79.8$	$73.9$	$94.4$	$62.8$	$40.0$	$77.5$	$96.8$	$85.6$	$66.9$	$68.2$	$75.9$

Table 1: Vision-Few-Shot classification accuracy (%) on 4-shot across 12 datasets without knowing class names. DC indicates Domain prompt conditioning and TC indicates Task prompt conditioning. HEC-V achieves state-of-the-art performance on average and across all datasets except FLWR, BIRD and ESAT. Underline denotes the best LVLM method. Bold denotes the best overall. Methods marked with † do not use hyperparameter tuning.

Model

Method

PETS

ESAT

UCF

SUN

CAL

DTD

AIR

FOOD

FLWR

CARS

BIRD

SIGN

AVG

DFN

Zero-Shot

\mathbf{97.6}

53.0

87.8

\mathbf{97.4}

\mathbf{99.3}

78.4

70.7

\mathbf{96.7}

\mathbf{93.8}

\mathbf{99.6}

\mathbf{96.5}

45.3

84.7

DFN+LLM (Qwen2-VL)

Baseline

84.1

33.6

85.6

94.2

98.4

70.9

62.1

91.2

78.3

91.1

69.7

54.5

76.1

ST [ 53 ]

90.1

41.6

90.5

94.5

98.8

77.3

66.4

92.3

90.0

96.0

78.5

56.8

81.1

HEC-T

\underline{95.2}

\underline{\mathbf{54.0}}

\underline{\mathbf{92.6}}

\underline{97.0}

\underline{99.3}

\underline{\mathbf{84.2}}

\underline{\mathbf{78.7}}

\underline{95.2}

\underline{91.1}

\underline{97.7}

\underline{85.9}

\underline{\mathbf{63.8}}

\underline{\mathbf{86.2}}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}11.1}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}20.4}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}7.0}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}2.8}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}0.9}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}13.3}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}16.6}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}4.0}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}12.9}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}6.7}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}16.2}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}9.3}

{\color[rgb]{0,0.4688,0}\raisebox{0.86108pt}{\scriptsize+}10.1}

SigLIP

Zero-Shot

\mathbf{97.0}

\mathbf{41.1}

87.1

96.8

\mathbf{99.5}

84.1

\mathbf{79.6}

\mathbf{97.1}

\mathbf{95.8}

\mathbf{99.5}

\mathbf{95.5}

47.1

\mathbf{85.0}

Table 2: Zero-Shot classification accuracy (%) on 10-way 0-shot across 12 datasets. Our method HEC-T provides a training-free zero-shot adaptation that outperforms previous baselines and is competitive with CLIP backbones that LVLMs inherit from. HEC-T provides meaningful gain while enabling to zero-shot an unlimited number of classes. Underline denotes the best LVLM method. Bold denotes the best including its CLIP backbone. Green denotes the absolute gain of HEC-T over the LVLM Baseline.

Model	Method	PETS	ESAT	UCF	SUN	CAL	DTD	AIR	FOOD	FLWR	CARS	BIRD	SIGN	AVG
DFN	Zero-Shot † [ 57 ]	$92.0$	$51.6$	$63.4$	$79.5$	$95.6$	$51.1$	$29.6$	$87.2$	$82.0$	$92.1$	$78.0$	$28.1$	$69.2$
	Probing [ 6 ]	$84.9$	$81.6$	$79.9$	$73.7$	$94.4$	$61.4$	$38.0$	$77.0$	$97.3$	$85.2$	$66.7$	$65.9$	$75.5$
	TipAdapter [ 71 ]	$92.3$	$69.2$	$77.4$	$80.5$	$95.8$	$64.4$	$40.2$	$87.2$	$97.0$	$92.7$	$78.7$	$50.0$	$77.1$
	GDA [ 64 ]	$\mathbf{92.8}$	$78.0$	$84.0$	$83.2$	$96.5$	$70.0$	$46.3$	$87.2$	$98.5$	$\mathbf{93.6}$	$80.3$	$67.8$	$81.5$
	ProKeR [ 4 ]	$91.2$	$\mathbf{82.4}$	$83.9$	$82.2$	$\mathbf{97.0}$	$66.2$	$43.1$	$\mathbf{87.8}$	$98.3$	$\mathbf{93.6}$	$\mathbf{81.5}$	$64.6$	$81.0$
DFN+LLM (Qwen2-VL)	Zero-Shot † [ 53 ]	$55.0$	$48.8$	$30.8$	$65.5$	$73.3$	$32.0$	$30.1$	$72.4$	$8.2$	$45.5$	$6.1$	$30.9$	$41.5$

Table 3: Vision-Text-Few-shot classification accuracy (%) on 4-shot across 12 datasets. We report results for CLIP-based and LVLM-based baselines. Combining HEC-T and HEC-V in a single classifier gives state-of-the-art performance, outperforming previous CLIP-based or LVLM-based baselines. Underline denotes the best LVLM-based method. Bold denotes the best overall. Methods marked with † do not use hyperparameter tuning.

Panel of Figure 5 — Figure 5: Ablation studies. Prompt Conditioning (left) and Head Ranking (right).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Top head attention map. We concatenate bird [ 62 ] and aircraft [ 47 ] datasets images horizontally in one support set. We then select the top vision-head for bird classification using the prompt What type of bird is this? and do the same for plane using the prompt What type of plane is this? . The attention map of the bird (left) and plane (right) top vision-head is overlaid on top of the image.

Figure 7: Variation of accuracy after each Attention and MLP block.

Dataset	Domain prompt
PETS	What breed is the animal in this image?
ESAT	What type of remote sensing image does the given image belong to?
UCF	What action is the person performing in this video frame?
SUN	What scene is shown in this image?
CAL	What is the main object in this photo?
DTD	What texture pattern is visible in this image?

Appendix

Table 4: Prompts used.

Category	Method	Acc. ( $\%$ )
Voting	Majority vote	$93.85_{0.41}$
Voting	Weighted vote	$93.92_{0.41}$
Proba	Mean	$94.00_{0.40}$
Proba	Score weights	$94.01_{0.40}$
Proba	Optimal weights	$\mathbf{94.04_{0.42}}$
Logit	Mean	$93.94_{0.41}$

Appendix

Table 5: Comparison of ensemble methods.

Figure 8: Ablation Study of the hyperparameter

\tau

		HEC-T		HEC-V
Heads	Selection Method	Acc.	Gain	Acc.	Gain
Task	HEC Oracle	$88.52_{0.6}$		$93.79_{0.4}$
Domain	HEC Oracle	$88.53_{0.6}$	${\color[rgb]{0,0.4688,0}\uparrow+\mkern 1.0mu0.01}$	$94.18_{0.4}$	${\color[rgb]{0,0.4688,0}\uparrow+\mkern 1.0mu0.39}$
Class	HEC Test-time	$88.45_{0.6}$	${\color[rgb]{0.6289,0,0}\downarrow-\mkern 1.0mu0.08}$	$94.14_{0.4}$	${\color[rgb]{0.6289,0,0}\downarrow-\mkern 1.0mu0.04}$

Appendix

Table 6: Head Selection. We evaluate HEC using different sets of heads. Task heads are the top 20 on ImageNet. Domain heads are the top 20 on the given dataset, and Class heads are selected on the fly on a given support set by our method HEC, without knowing in advance the performance on the query set.

Panel of Figure 9 — Figure 9: Average accuracy of the top 50 heads. Task heads are ranked on ImageNet query set accuracy. Domain heads are ranked on the domain dataset query set accuracy. Class heads are ranked on the 10-way 4-shot current task query set accuracy.

Panel of Figure 10 — Figure 10: Enlarged versions of Figs. 5b and 5c from the main paper.

Figure 11: Impact of

N

-ways on Class and Domain Conditioning Performance on HEC-T

Method	Finedefics [ 22 ]	Idefics2 [ 33 ]	LLaVA-OV [ 35 ]
Zero-Shot	$56.6$	$43.9$	$41.6$
Probing	$82.5$	$80.3$	$81.6$
HEC-VT	$\mathbf{84.6}$	$\mathbf{83.6}$	$\mathbf{84.5}$
	+ 2.1	+ 3.4	+ 2.9

Appendix

Table 8: text-vision-few-shot average accuracy (%) across models. Bold denotes the best method for each model. Green denotes the absolute gain of HEC-VT over Probing.

Large Vision-Language Models

Large vision-language models (LVLMs) connect image and video perception with language generation, reasoning, and interactive tasks.

See all 401 papers

Sep 11, 2025

Image Recognition with Vision and Language Embeddings of VLMs

Vision-language models (VLMs) have enabled strong zero-shot classification through image-text alignment. Yet, their purely visual inference capabilities remain under-explored. In this work, we conduct a comprehensive evaluation of both language-guided and vision-only image classification with a diverse set of dual-encoder VLMs, including both well-established and recent models such as SigLIP 2 and RADIOv2.5. The performance is compared in a standard setup on the ImageNet-1k validation set and its label-corrected variant. The key factors affecting accuracy are analysed, including prompt design, class diversity, the number of neighbours in k-NN, and reference set size. We show that language and vision offer complementary strengths, with some classes favouring textual prompts and others better handled by visual similarity. To exploit this complementarity, we introduce a simple, learning-free fusion method based on per-class precision that improves classification performance. The code is available at: https://github.com/gonikisgo/bmvc2025-vlm-image-recognition.

Illia Volkov, Nikita Kisel, Klara Janouskova +1

Visual Recognition Group, Faculty of Electrical Engineering, Czech Technical University in Prague Prague, Czechia

Image Classification VLM Evaluation Vision-Language Models Zero-Shot Image Classification Multimodal Embedding

Ask PDF

Sep 10, 2025

Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs

Vision language models (VLMs) excel at zero-shot visual classification, but their performance on fine-grained tasks and large hierarchical label spaces is understudied. This paper investigates whether structured, tree-based reasoning can enhance VLM performance. We introduce a framework that decomposes classification into interpretable decisions using decision trees and evaluates it on fine-grained (GTSRB) and coarse-grained (CIFAR-10) datasets. Although the model achieves 98.2% accuracy in understanding the tree knowledge, tree-based reasoning consistently underperforms standard zero-shot prompting. We also explore enhancing the tree prompts with LLM-generated classes and image descriptions to improve alignment. The added description enhances the performance of the tree-based and zero-shot methods. Our findings highlight limitations of structured reasoning in visual classification and offer insights for designing more interpretable VLM systems.

Sary Elmansoury, Islam Mesabah, Gerrit Großmann +3

Department of Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI) · Department of Computer Science, University of Darmstadt · Department of Computer Science, University of Kaiserslautern–Landau (RPTU) +1

VLM Reasoning Image Classification VLM Evaluation Fine-Grained Image Classification

Ask PDF

Jul 1, 2026

AdaBoosting Text Prompts for Vision-Language Models

The classification accuracy of pretrained Vision-Language Models (VLMs) relies on the quality of the text prompts. Handcrafted templates and Large Language Model (LLM)-generated descriptions not only make predictions more interpretable, but also enable reuse of the same prompts across heterogeneous VLMs. Recent works construct task-adapted text prompts with a small number of labeled images. However, existing few-shot text prompting methods do not explicitly focus on misclassified examples during prompt construction, leading to only marginal improvements even as more shots become available. To fully exploit few-shot supervision, we propose Text Prompt Boosting (TPB), an AdaBoost-inspired framework that treats each text-prompt-based classifier as a weak learner and sequentially aggregates them into a strong ensemble by explicitly targeting hard, misclassified examples. Extensive experiments show that TPB preserves task-intrinsic, model-agnostic cues in text space, enabling robust cross-model transfer. Across eleven classification benchmarks, TPB improves accuracy on the source model and preserves shot-driven gains when transferred to larger, more capable VLMs, where existing methods struggle to sustain such improvements.

Seokhee Jin, Changhwan Sung, Sunung Mun +2

KT Corporation, Republic of Korea · Pohang University of Science and Technology (POSTECH), Republic of Korea · National AI Research Lab, Republic of Korea

Vision-Language Models Ensemble Learning VLM Adaptation Few-Shot Image Classification Prompt Learning for VLMs

Ask PDF

Abstract

Large Vision-Language Models

Explore similar work

Image Recognition with Vision and Language Embeddings of VLMs

Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs

AdaBoosting Text Prompts for Vision-Language Models