Vision-Language Models (VLMs), such as CLIP, have significantly advanced zero-shot image recognition. However, their performance remains limited by suboptimal prompt engineering and poor adaptability to target classes. While recent methods attempt to improve prompts through diverse class descriptions, they often rely on heuristic designs, lack versatility, and are vulnerable to outlier prompts. This paper enhances prompt by incorporating class-specific concepts. By treating concepts as latent variables, we rethink zero-shot image classification from a Bayesian perspective, casting prediction as marginalization over the concept space, where each concept is weighted by a prior and a test-image conditioned likelihood. This formulation underscores the importance of both a well-structured concept proposal distribution and the refinement of concept priors. To construct an expressive and efficient proposal distribution, we introduce a multi-stage concept synthesis pipeline driven by LLMs to generate discriminative and compositional concepts, followed by a Determinantal Point Process to enforce diversity. To mitigate the influence of outlier concepts, we propose a training-free, adaptive soft-trim likelihood, which attenuates their impact in a single forward pass. We further provide robustness guarantees and derive multi-class excess risk bounds for our framework. Extensive experiments demonstrate that our method consistently outperforms state-of-the-art approaches, validating its effectiveness in zero-shot image classification. Our code is available at https://github.com/less-and-less-bugs/CGBC.
Figures & tables
Figure 1: Q–Q plots of similarity distributions between all Great White Shark images in ImageNet dataset and CuPL [ 40 ] prompts for this class and Porcupine. Departures from the normal reference line indicate non-normality. Pronounced skewness is taken as ∣skewness∣>0.5 , and heavy tails as excess kurtosis >3 .
Figure 2: Instantiation of our proposed Concept-Guided Bayesian Classification (CGBC) framework. (a) LLM-driven Concept Synthesis Pipeline: Constructs class-aware hard-negative neighborhoods in Step 1, generates discriminative atomic concepts via LLM prompting in Step 2, composes higher-order concepts through random sampling in Step 3, and selects diverse concepts using Determinantal Point Processes (DPPs) in Step 4. (b) Adaptive Soft-Trim Likelihood Estimation: Estimates contamination rates in similarity distributions and downweights outlier concepts that deviate significantly from the median, improving robustness in concept-based image classification.
Figure 3: Top-1 Accuracy difference between discriminative (CGBC Prior) and descriptive concepts on ten datasets.
Figure 4: Averaged top-1 accuracy (%) on eleven datasets using different numbers of concepts per prompt.
Figure 5: Averaged top-1 accuracy (%) on eleven datasets with and without DPP under different prompt budgets.
Figure 6: Illustrative examples of Discriminability in constructing the Concept Proposal Distribution. We multiply the raw similarity values by CLIP’s scaling factor.
Figure 7: Visualization of similarity between prompts enhanced by discriminative and common concepts for the beagle and basset hound classes. We multiply the raw similarity values by CLIP’s scaling factor.
SYSTEM PROMPT:
You are a visual concept proposer tasked with enhancing text descriptions for zero-shot image classification on the test dataset using CLIP.
Given:
- A core class from the test dataset
- The set of other classes in the dataset
Task:
Propose concise, visually discriminative concepts to append to the text description (i.e., “A photo of {core class} with {your concept}”) that help CLIP better distinguish the core class from the other classes.
Table 7: Prompt for discriminative concept generation. The red text represents variables that are dynamically replaced based on the specific datasets and image classes being tested. “Core class” and “Other Classes” correspond to Yi and Li,j , respectively.
SYSTEM PROMPT:
You are a visual concept proposer tasked with enhancing text descriptions for zero-shot image classification on the test dataset using CLIP.
Given:
- A class from the test dataset
Task:
Propose descriptive concepts to append to the text description (i.e., “A photo of {core class} with {your concept}”) that help CLIP better understand and recognize the core class.
Guidelines:
Table 8: Prompt for descriptive concept generation. The red text represents variables that will be replaced based on the specific datasets and image classes being tested. “Core class” will be instantiated by Yi .
Figure 8: Illustrative examples of Compositionality in constructing the Concept Proposal Distribution. We multiply the raw similarity values by CLIP’s scaling factor.
Large-scale vision-language models such as CLIP have achieved remarkable success in zero-shot image recognition, yet their predictions remain largely opaque to human understanding. In contrast, Concept Bottleneck Models provide interpretable intermediate representations by reasoning through human-defined concepts, but they rely on concept supervision and lack the ability to generalize to unseen classes. We introduce EZPC that bridges these two paradigms by explaining CLIP's zero-shot predictions through human-understandable concepts. Our method projects CLIP's joint image-text embeddings into a concept space learned from language descriptions, enabling faithful and transparent explanations without additional supervision. The model learns this projection via a combination of alignment and reconstruction objectives, ensuring that concept activations preserve CLIP's semantic structure while remaining interpretable. Extensive experiments on five benchmark datasets, CIFAR-100, CUB-200-2011, Places365, ImageNet-100, and ImageNet-1k, demonstrate that our approach maintains CLIP's strong zero-shot classification accuracy while providing meaningful concept-level explanations. By grounding open-vocabulary predictions in explicit semantic concepts, our method offers a principled step toward interpretable and trustworthy vision-language models. Code is available at https://github.com/oonat/ezpc.
Onat Ozdemir, Anders Christensen, Stephan Alaniz +2
School of Informatics, University of Edinburgh · Dept. of Computer Eng., Middle East Technical University (METU) · Orbital +8
Prompt learning has become an effective and widely used technique in enhancing vision-language models (VLMs) such as CLIP for various downstream tasks, particularly in zero-shot classification within specific domains. Existing methods typically focus on either learning class-shared prompts for a given domain or generating instance-specific prompts through conditional prompt learning. While these methods have achieved promising performance, they often overlook class-specific knowledge in prompt design, leading to suboptimal outcomes. The underlying reasons are: 1) class-specific prompts offer more fine-grained supervision compared to coarse class-shared prompts, which helps prevent misclassification of data from different classes into a single class; 2) compared to class-specific prompts, instance-specific prompts neglect the richer class-level information across multiple instances, potentially causing data from the same class to be divided into multiple classes. To effectively supplement the class-specific knowledge into existing methods, we propose a plug-and-play Class-Aware Knowledge Injection (CAKI) framework. CAKI comprises two key components, i.e., class-specific prompt generation and query-key prompt matching. The former encodes class-specific knowledge into prompts from few-shot samples that belong to the same class and stores the learned prompts in a class-level knowledge bank. The latter provides a plug-and-play mechanism for each test instance to retrieve relevant class-level knowledge from the knowledge bank and inject such knowledge to refine model predictions. Extensive experiments demonstrate that our CAKI effectively improves the performance of existing methods on base and novel classes. Code is publicly available at this https URL.
Junhui Yin, Nan Pu, Xinyu Zhang +4
University of Science and Technology Beijing, China. · Hefei University of Technology, China. · University of Auckland, New Zealand. +3
Current Large Vision Language Models (LVLMs) excel at many zero-shot tasks like image captioning, visual question answering and OCR. However, these same models suffer from poor performance at image classification tasks, underperforming against CLIP-based methods. Notably, this gap is surprising because many LVLMs use CLIP-pretrained vision encoders. Yet LVLMs are not inherently limited by CLIP's architecture with independent vision and text encoders. In CLIP, this separation biases classification toward class-name matching rather than joint visual-text reasoning. In this paper we show that, despite their poor raw performance, LVLMs can improve visual feature class separability at inference using prompt conditioning, and LVLMs' internal representations, especially attention heads, can outperform the model itself at zero-shot and few-shot classification. We introduce Head Ensemble Classifiers (HEC) to bridge the performance gap between CLIP-based and LVLM-based classification methods. Inspired by Gaussian Discriminant Analysis, HEC ranks the most discriminative vision and text heads and combines them into a training-free classifier. We show that HEC achieves state-of-the-art performance in few-shot and zero-shot classification across 12 datasets.
Adhemar de Senneville, Xavier Bou, Jérémy Anger +2
Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · École Polytechnique, AMIAD, France · Institut Universitaire de France