Zero-shot classification needs efficient label retrieval and fine-grained visual reasoning, yet discriminative and generative vision-language models fail in complementary ways.When CLIP's top-1 prediction is wrong, the correct label often remains in its top-K shortlist, making disambiguation rather than recall the key challenge.Standalone generative models, however, are hindered by large label spaces and unconstrained outputs.This complementarity motivates separating broad candidate retrieval from fine-grained, image-grounded verification.We propose G2D, a training-free framework that uses a generative VLM to verify CLIP-retrieved candidates against the image.Candidate names and CLIP probabilities provide a structured prior for resolving visually similar classes.Fixed confidence routing, entropy-adaptive candidate sizing, and trie-constrained decoding focus generative reasoning on uncertain samples and ensure one valid output for each input at test time.Across eight benchmarks, G2D achieves 68.85% average accuracy, versus 59.35% for CLIP and 63.11% for the standalone VLM.Across seven generator configurations, candidate-set verification improves average accuracy by 1.08--27.42 percentage points.G2D also transfers to DCLIP, WaffleCLIP, and CuPL, supporting a practical interface between discriminative proposal and generative visual reasoning. Code: https://github.com/Harzva/G2D
Figures & tables
Figure 1. Paradigm comparison for zero-shot image classification. A panoramic figure with seven panels. Panels (a)–(d) show discriminant-paradigm pipelines (CLIP, DCLIP, WaffleCLIP, ComparativeCLIP) that rewrite text prompts without seeing the image. Panel (e) shows CLS-RL, a generative paradigm requiring RL training. Panel (f) shows Qwen3-VL applied directly to zero-shot classification, contrasting full-class vs.\ constrained candidate sets. The bottom-right panel shows G2D: a generative VLM constrained to CLIP's top-K, with a CUB200 example where G2D succeeds while CLIP and standalone VLM fail.
Figure 2. Prompt-enriched baseline comparison under the shared zero-shot protocol. Strict G2D (1θ) is compared against representative prompt-enriched CLIP baselines (DCLIP ( Menon and Vondrick, 2023 ) , WaffleCLIP ( Roth et al., 2023 ) , CuPL ( Pratt et al., 2023 ) , and Comp-CLIP ( Lee et al., 2025 ) ) across all eight datasets. A dumbbell plot comparing strict G2D with four prompt-enriched CLIP baselines and vanilla CLIP across eight datasets and their average.
Figure 3. Unified overview of G2D inference and verifier formulation. The figure combines the high-level G2D pipeline with the verifier-centric view. Pure discriminative zero-shot classification provides useful CLIP priors but can make wrong high-confidence decisions, while pure generative zero-shot classification must reason over the full label set and is more prone to answer drift. Within G2D, CLIP first constructs an adaptive shortlist and routes the sample by confidence; when verification is invoked, the verifier receives the image together with a CLIP-derived candidate shortlist and prior scores as an input constraint, and its decoding is restricted to valid class names as an output constraint. This training-free design turns the generative model into a focused visual verifier that corrects errors within CLIP’s candidate space. A unified conceptual figure combining the G2D inference framework and the verifier formulation. The figure contrasts discriminative CLIP ranking, standalone generative classification over the full label set, and G2D's routed verifier that uses a CLIP shortlist and prior scores as input constraints together with constrained decoding over valid labels only.
Figure 4. Benchmark overview and verifier-formulation analysis. (a) CLIP, Qwen-only, and strict G2D (1θ) are compared across eight datasets and their average. (b) The same generators are compared as freeform classifiers (Standalone) and as verifier-only branches over CLIP ViT-B/16 candidates (Verifier = G2D w/o routing); LLaVA-1.5-7B is averaged over its six jointly supported datasets. A two-panel figure. Panel (a) compares CLIP, Qwen-only, and strict one-threshold G2D across eight datasets and the average. Panel (b) compares standalone and verifier accuracy for seven generators, with LLaVA averaged on six supported datasets.
Method
DTD
CUB200
Oxford-Pets
EuroSAT
Food101
Places365
ImageNetV2
ImageNet
Avg.
CLIP ViT-B/16-only
44.27
54.97
87.27
42.30
83.69
34.53
60.93
66.84
59.35
Qwen3-VL-8B (Gene-only)
67.97
44.70
85.75
48.93
84.82
36.38
65.32
71.01
63.11
Verifier-only (no routing)
52.19
59.54
90.00
47.81
87.31
37.11
66.52
73.00
64.19
G2D (1θ) (strict/no-label)
67.97
63.39
90.41
58.09
88.51
38.52
68.41
75.52
68.85
G2D (2θ) (label-calibrated analysis)
67.67
59.68
91.66
51.00
87.90
38.03
69.03
75.29
67.53
Table 1. Strict zero-shot result and controlled attribution. Top-1 accuracy (%). The rows are ordered Gene-only → Verifier-only → fixed routing to isolate full-label generation, candidate-set verification, and routing with the same Qwen3-VL-8B. G2D (1θ) fixes θhigh=0.70 before deployment and uses no labeled validation or test feedback. The selected G2D (2θ) row uses benchmark labels for threshold selection and is shown only as calibrated analysis.
Method
VLM route
Model stack
CLIP-only
0%
∼ 150M; no VLM
Qwen-only
100%
Qwen3-VL-8B
Verifier-only
100%
CLIP + Qwen3-VL-8B
G2D (1θ)
22.9–85.8%
CLIP + Qwen3-VL-8B
Table 2. Deployment ledger. Strict G2D invokes the VLM on 22.9–85.8% of images on EuroSAT, DTD, Oxford-Pets, and CUB200. This is an accuracy–cost trade-off, not a universal speedup claim.
Var.
Cfg.
DTD
CUB200
Oxford-Pets
EuroSAT
Avg.
Qwen-only
67.97
44.70
85.75
48.93
61.84
CLIP
Base
44.27
54.97
87.27
42.30
57.20
R
63.24
58.68
91.66
56.14
67.43
Δ
( +18.97 , -4.73 )
( +3.71 , +13.98 )
( +4.39 , +5.91 )
( +13.84 , +7.21 )
( +10.23 , +5.59 )
NR
64.18
58.61
92.07
56.26
67.78
ΔNR
( +19.91 , -3.79 )
( +3.64 , +13.91 )
( +4.80 , +6.32 )
( +13.96 , +7.33 )
( +10.58 , +5.94 )
Table 3. Discriminator ablation with text-enrichment methods. Four-dataset accuracy (%) with Qwen3-VL-8B-Instruct, fixed θhigh=0.7 , and adaptive K∈[3,10] .
Figure 5. θhigh sensitivity across discriminator variants with Qwen3-VL-8B fixed. The green dashed line marks the shared default θhigh=0.7 ; curves are smooth and their optima mostly lie within 0.7–0.9. A threshold-sensitivity plot over discriminator variants and four datasets. The default threshold at 0.7 is marked by a green dashed line, the curves are smooth, and the best points cluster near 0.7 to 0.9.
Vision-Language Models (VLMs), such as CLIP, have significantly advanced zero-shot image recognition. However, their performance remains limited by suboptimal prompt engineering and poor adaptability to target classes. While recent methods attempt to improve prompts through diverse class descriptions, they often rely on heuristic designs, lack versatility, and are vulnerable to outlier prompts. This paper enhances prompt by incorporating class-specific concepts. By treating concepts as latent variables, we rethink zero-shot image classification from a Bayesian perspective, casting prediction as marginalization over the concept space, where each concept is weighted by a prior and a test-image conditioned likelihood. This formulation underscores the importance of both a well-structured concept proposal distribution and the refinement of concept priors. To construct an expressive and efficient proposal distribution, we introduce a multi-stage concept synthesis pipeline driven by LLMs to generate discriminative and compositional concepts, followed by a Determinantal Point Process to enforce diversity. To mitigate the influence of outlier concepts, we propose a training-free, adaptive soft-trim likelihood, which attenuates their impact in a single forward pass. We further provide robustness guarantees and derive multi-class excess risk bounds for our framework. Extensive experiments demonstrate that our method consistently outperforms state-of-the-art approaches, validating its effectiveness in zero-shot image classification. Our code is available at https://github.com/less-and-less-bugs/CGBC.
Hui Liu, Kecheng Chen, Jialiang Wang +3
City University of Hong Kong · Harbin Institute of Technology · Nanyang Technological University
Current Large Vision Language Models (LVLMs) excel at many zero-shot tasks like image captioning, visual question answering and OCR. However, these same models suffer from poor performance at image classification tasks, underperforming against CLIP-based methods. Notably, this gap is surprising because many LVLMs use CLIP-pretrained vision encoders. Yet LVLMs are not inherently limited by CLIP's architecture with independent vision and text encoders. In CLIP, this separation biases classification toward class-name matching rather than joint visual-text reasoning. In this paper we show that, despite their poor raw performance, LVLMs can improve visual feature class separability at inference using prompt conditioning, and LVLMs' internal representations, especially attention heads, can outperform the model itself at zero-shot and few-shot classification. We introduce Head Ensemble Classifiers (HEC) to bridge the performance gap between CLIP-based and LVLM-based classification methods. Inspired by Gaussian Discriminant Analysis, HEC ranks the most discriminative vision and text heads and combines them into a training-free classifier. We show that HEC achieves state-of-the-art performance in few-shot and zero-shot classification across 12 datasets.
Adhemar de Senneville, Xavier Bou, Jérémy Anger +2
Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · École Polytechnique, AMIAD, France · Institut Universitaire de France
Verifying the authenticity of AI-generated images presents a growing challenge on social media platforms these days. While vision-language models (VLMs) like CLIP outdo in multimodal representation, their capacity for AI-generated image classification is underexplored due to the absence of such labels during the pre-training process. This work investigates whether CLIP embeddings inherently contain information indicative of AI generation. A proposed pipeline extracts visual embeddings using a frozen CLIP model, feeds its embeddings to lightweight networks, and fine-tunes only the final classifier. Experiments on the public CIFAKE benchmark show the performance reaches 95% accuracy without language reasoning. Few-shot adaptation to curated custom with 20% of the data results in performance to 85%. A closed-source baseline (Gemini-2.0) has the best zero-shot accuracy yet fails on specific styles. Notably, some specific image types, such as wide-angle photographs and oil paintings, pose significant challenges to classification. These results indicate previously unexplored difficulties in classifying certain types of AI-generated images, revealing new and more specific questions in this domain that are worth further investigation.
Ziyang Ou
Department of Electrical and Computer Engineering University of Rochester Rochester, NY 14627, USA