G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification

Zehua Hao; Fang Liu; Qinliang Wang; Yaoyang Du; Xinyan Huang; Puhua Chen

cs.CVAug 27, 2026

G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification

Authors: Zehua Hao, Fang Liu, Qinliang Wang, Yaoyang Du, Xinyan Huang, Puhua Chen

Organizations: Xidian University Xi’an, Shaanxi, China

Abstract

Zero-shot classification needs efficient label retrieval and fine-grained visual reasoning, yet discriminative and generative vision-language models fail in complementary ways.When CLIP's top-1 prediction is wrong, the correct label often remains in its top- $K$ shortlist, making disambiguation rather than recall the key challenge.Standalone generative models, however, are hindered by large label spaces and unconstrained outputs.This complementarity motivates separating broad candidate retrieval from fine-grained, image-grounded verification.We propose G2D, a training-free framework that uses a generative VLM to verify CLIP-retrieved candidates against the image.Candidate names and CLIP probabilities provide a structured prior for resolving visually similar classes.Fixed confidence routing, entropy-adaptive candidate sizing, and trie-constrained decoding focus generative reasoning on uncertain samples and ensure one valid output for each input at test time.Across eight benchmarks, G2D achieves 68.85% average accuracy, versus 59.35% for CLIP and 63.11% for the standalone VLM.Across seven generator configurations, candidate-set verification improves average accuracy by 1.08--27.42 percentage points.G2D also transfers to DCLIP, WaffleCLIP, and CuPL, supporting a practical interface between discriminative proposal and generative visual reasoning. Code: https://github.com/Harzva/G2D

Figures & tables

Figure 1. Paradigm comparison for zero-shot image classification. A panoramic figure with seven panels. Panels (a)–(d) show discriminant-paradigm pipelines (CLIP, DCLIP, WaffleCLIP, ComparativeCLIP) that rewrite text prompts without seeing the image. Panel (e) shows CLS-RL, a generative paradigm requiring RL training. Panel (f) shows Qwen3-VL applied directly to zero-shot classification, contrasting full-class vs.\ constrained candidate sets. The bottom-right panel shows G2D: a generative VLM constrained to CLIP's top-K, with a CUB200 example where G2D succeeds while CLIP and standalone VLM fail.

Figure 2. Prompt-enriched baseline comparison under the shared zero-shot protocol. Strict G2D

(1\theta)

is compared against representative prompt-enriched CLIP baselines (DCLIP ( Menon and Vondrick, 2023 ) , WaffleCLIP ( Roth et al., 2023 ) , CuPL ( Pratt et al., 2023 ) , and Comp-CLIP ( Lee et al., 2025 ) ) across all eight datasets. A dumbbell plot comparing strict G2D with four prompt-enriched CLIP baselines and vanilla CLIP across eight datasets and their average.

Figure 3. Unified overview of G2D inference and verifier formulation. The figure combines the high-level G2D pipeline with the verifier-centric view. Pure discriminative zero-shot classification provides useful CLIP priors but can make wrong high-confidence decisions, while pure generative zero-shot classification must reason over the full label set and is more prone to answer drift. Within G2D, CLIP first constructs an adaptive shortlist and routes the sample by confidence; when verification is invoked, the verifier receives the image together with a CLIP-derived candidate shortlist and prior scores as an input constraint, and its decoding is restricted to valid class names as an output constraint. This training-free design turns the generative model into a focused visual verifier that corrects errors within CLIP’s candidate space. A unified conceptual figure combining the G2D inference framework and the verifier formulation. The figure contrasts discriminative CLIP ranking, standalone generative classification over the full label set, and G2D's routed verifier that uses a CLIP shortlist and prior scores as input constraints together with constrained decoding over valid labels only.

Figure 4. Benchmark overview and verifier-formulation analysis. (a) CLIP, Qwen-only, and strict G2D

(1\theta)

are compared across eight datasets and their average. (b) The same generators are compared as freeform classifiers (Standalone) and as verifier-only branches over CLIP ViT-B/16 candidates (Verifier = G2D w/o routing); LLaVA-1.5-7B is averaged over its six jointly supported datasets. A two-panel figure. Panel (a) compares CLIP, Qwen-only, and strict one-threshold G2D across eight datasets and the average. Panel (b) compares standalone and verifier accuracy for seven generators, with LLaVA averaged on six supported datasets.

Method	DTD	CUB200	Oxford-Pets	EuroSAT	Food101	Places365	ImageNetV2	ImageNet	Avg.
CLIP ViT-B/16-only	44.27	54.97	87.27	42.30	83.69	34.53	60.93	66.84	59.35
Qwen3-VL-8B (Gene-only)	67.97	44.70	85.75	48.93	84.82	36.38	65.32	71.01	63.11
Verifier-only (no routing)	52.19	59.54	90.00	47.81	87.31	37.11	66.52	73.00	64.19
G2D $(1\theta)$ (strict/no-label)	67.97	63.39	90.41	58.09	88.51	38.52	68.41	75.52	68.85
G2D $(2\theta)$ (label-calibrated analysis)	67.67	59.68	91.66	51.00	87.90	38.03	69.03	75.29	67.53

Table 1. Strict zero-shot result and controlled attribution. Top-1 accuracy (%). The rows are ordered Gene-only

\rightarrow

Verifier-only

\rightarrow

fixed routing to isolate full-label generation, candidate-set verification, and routing with the same Qwen3-VL-8B. G2D

(1\theta)

fixes

\theta_{\mathrm{high}}=0.70

before deployment and uses no labeled validation or test feedback. The selected G2D

(2\theta)

row uses benchmark labels for threshold selection and is shown only as calibrated analysis.

Method	VLM route	Model stack
CLIP-only	0%	$\sim$ 150M; no VLM
Qwen-only	100%	Qwen3-VL-8B
Verifier-only	100%	CLIP + Qwen3-VL-8B
G2D $(1\theta)$	22.9–85.8%	CLIP + Qwen3-VL-8B

Table 2. Deployment ledger. Strict G2D invokes the VLM on 22.9–85.8% of images on EuroSAT, DTD, Oxford-Pets, and CUB200. This is an accuracy–cost trade-off, not a universal speedup claim.

Var.	Cfg.	DTD	CUB200	Oxford-Pets	EuroSAT	Avg.
Qwen-only		67.97	44.70	85.75	48.93	61.84
CLIP	Base	44.27	54.97	87.27	42.30	57.20
	R	63.24	58.68	91.66	56.14	67.43
	$\Delta$	( +18.97 , -4.73 )	( +3.71 , +13.98 )	( +4.39 , +5.91 )	( +13.84 , +7.21 )	( +10.23 , +5.59 )
	NR	64.18	58.61	92.07	56.26	67.78
	$\Delta_{NR}$	( +19.91 , -3.79 )	( +3.64 , +13.91 )	( +4.80 , +6.32 )	( +13.96 , +7.33 )	( +10.58 , +5.94 )

Table 3. Discriminator ablation with text-enrichment methods. Four-dataset accuracy (%) with Qwen3-VL-8B-Instruct, fixed

\theta_{high}{=}0.7

, and adaptive

K{\in}[3,10]

Figure 5.

\theta_{high}

sensitivity across discriminator variants with Qwen3-VL-8B fixed. The green dashed line marks the shared default

\theta_{high}{=}0.7

; curves are smooth and their optima mostly lie within 0.7–0.9. A threshold-sensitivity plot over discriminator variants and four datasets. The default threshold at 0.7 is marked by a green dashed line, the curves are smooth, and the best points cluster near 0.7 to 0.9.

Zero-Shot Image Classification

Zero-shot image classification uses semantic descriptions and pretrained visual representations to recognize image classes without labeled examples.

See all 49 papers

Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognition
Mar 9, 2026Hui Liu, Kecheng Chen, Jialiang Wang +3Zero-Shot Image Classification Zero-Shot Learning
City University of Hong Kong · Harbin Institute of Technology · Nanyang Technological University
Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection
Mar 25, 2026Adhemar de Senneville, Xavier Bou, Jérémy Anger +2Large Vision-Language Models Zero-Shot Image Classification
Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · École Polytechnique, AMIAD, France · Institut Universitaire de France
CLIP Embeddings for AI-Generated Image Detection: A Few-Shot Study with Lightweight Classifier
May 15, 2025Ziyang OuAI-Generated Image Detection
Department of Electrical and Computer Engineering University of Rochester Rochester, NY 14627, USA

Mar 9, 2026J/K move · Enter open · S save

Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognition

Hui Liu, Kecheng Chen, Jialiang Wang, Xianming Liu+2

City University of Hong Kong · Harbin Institute of Technology · Nanyang Technological University

Zero-Shot Image Classification Zero-Shot Learning Bayesian Inference

Open paper AskPDF arXiv

Vision-Language Models (VLMs), such as CLIP, have significantly advanced zero-shot image recognition. However, their performance remains limited by suboptimal prompt engineering and poor adaptability to target classes. While recent methods attempt to improve prompts through diverse class descriptions, they often rely on heuristic designs, lack versatility, and are vulnerable to outlier prompts. This paper enhances prompt by incorporating class-specific concepts. By treating concepts as latent variables, we rethink zero-shot image classification from a Bayesian perspective, casting prediction as marginalization over the concept space, where each concept is weighted by a prior and a test-image conditioned likelihood. This formulation underscores the importance of both a well-structured concept proposal distribution and the refinement of concept priors. To construct an expressive and efficient proposal distribution, we introduce a multi-stage concept synthesis pipeline driven by LLMs to generate discriminative and compositional concepts, followed by a Determinantal Point Process to enforce diversity. To mitigate the influence of outlier concepts, we propose a training-free, adaptive soft-trim likelihood, which attenuates their impact in a single forward pass. We further provide robustness guarantees and derive multi-class excess risk bounds for our framework. Extensive experiments demonstrate that our method consistently outperforms state-of-the-art approaches, validating its effectiveness in zero-shot image classification. Our code is available at https://github.com/less-and-less-bugs/CGBC.

Abstract

Zero-Shot Image Classification

Explore similar work

Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognition

Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection

CLIP Embeddings for AI-Generated Image Detection: A Few-Shot Study with Lightweight Classifier