Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs
Authors: Mohammad Jalali, Azim Ospanov, Amin Gohari, Farzan Farnia
Organizations: Department of Computer Science and Engineering, The Chinese University of Hong Kong · Department of Information Engineering, The Chinese University of Hong Kong
Generative models guided by text prompts are widely evaluated for fidelity and prompt alignment, yet their ability to produce diverse outputs remains underexplored. Existing diversity metrics such as Vendi and RKE, which are based on the von Neumann and Rényi entropies of kernel matrices, were developed for unconditional models and cannot distinguish prompt-induced from model-induced variability. We address this gap by introducing Conditional-Vendi and Conditional-RKE, diversity measures derived from the conditional entropy of positive semidefinite matrices. These scores isolate model-induced diversity in prompt-guided generation, with Conditional-RKE enjoying an O(1/n) convergence rate. For Conditional-Vendi, we introduce a truncated-spectrum approximation that yields scalable and consistent estimates. Experiments on text-to-image, image-captioning, and LLM tasks show that the conditional scores recover ground-truth diversity orderings and can also guide diffusion models toward more diverse samples. The codebase is available at https://github.com/mjalali/conditional-vendi.
Figures & tables
Figure 1: Illustration of prompt-induced diversity (left), where variation comes from different prompts, and model-induced diversity (right), where variation arises from the generator itself. Unconditional image diversity scores, Vendi and RKE, assign higher diversity to the right side, overlooking prompt effects. Our proposed Conditional-RKE and Conditional-Vendi account for prompts and assign higher diversity to the left side.
Figure 2: Statistical convergence of Conditional-Vendi and RKE scores with different sample sizes on the MS-COCO validation dataset with a finite-dimensional (normalized) linear kernel and infinite-dimensional Gaussian (RBF) kernel.
Figure 3: Statistical convergence of Conditional-Vendi scores with different sample sizes on data generated by SDXL using MS-COCO validation set prompts with finite-dimension cosine similarity and infinite-dimension Gaussian kernel. DINOv2 and CLIP embeddings are used for image and text modalities, respectively.
Figure 4: Conditional and Information Vendi and RKE score comparison across text-to-image models. We clustered MS-COCO prompts into k groups and generated images for each cluster center. Within each cluster, we paired prompts with identical images. The results show increasing diversity and stronger correlation as the number of clusters grows, indicating that clusters become more relevant and diverse with finer partitioning.
Figure 5: Evaluated Conditional-Vendi and Vendi scores on animal samples generated by SDXL. (Left Plot) We do not specify the animal types in the prompt (Right Plot) we specify the animal types in the prompt.
Figure 6: Quantifying image diversity of PixArt- α –generated outputs for the top-2 prompt clusters (from 10,000 GPT-4o–generated prompts).
Guidance Method
CLIPScore ↑
KD ×102↓
Cond-Vendi DINOv2 ↑
Vendi DINOv2 ↑
In-batch Sim. ×102↓
Vendi Latent
30.41
35.40
29.85
309.45
80.06
Conditional-Vendi Latent
30.47
29.14
32.45
325.20
78.02
Table 1: Quantitative comparison of guidance methods on Stable Diffusion XL
Figure 7: Qualitative comparison of Conditional-Vendi score guidance vs. Vendi score guidance using SD-XL.
Figure 8: Measuring Conditional-Vendi and Information-Vendi for text-to-video models
Method
T=0.4
T=0.7
T=1.0
T=1.3
Conditional-Vendi
45.38
46.73
48.26
49.60
Conditional-RKE
41.45
44.67
46.29
48.45
Table 2: Conditional Vendi and RKE Scores evaluated for Llama 2 with different temperature parameters.
Figure 9: Conditional-Vendi and Information-Vendi of image-captioning models for 3 image types
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Qualitative comparison of Conditional-Vendi score guidance vs. Vendi score guidance using SD-XL.
Figure 11: Qualitative comparison of Conditional-Vendi score guidance vs. Vendi score guidance using PixArt- Σ .
Guidance Method
CLIPScore ↑
KD ×102↓
Cond-Vendi DINOv2 ↑
Vendi DINOv2 ↑
In-batch Sim. ×102↓
Vendi CLIP
29.63
38.14
26.28
261.73
83.26
Vendi Latent
30.39
36.20
28.95
298.15
81.50
Conditional-Vendi Latent
30.44
29.80
31.50
312.80
79.45
Appendix
Table 3: Quantitative comparison of guidance methods on PixArt
Figure 12: Comparing Conditional-Vendi and Information-Vendi on 2-D Gaussian Distribution. We used 1000 pair of points and used a Gaussian Kernel with bandwidth σ=0.6 .
Figure 13: Substituting images generated from models trained on MS- dataset.
Figure 14: Quantifying image diversity for different clusters of text prompts. Images are generated using the Stable Diffusion XL model.
Figure 15: Quantifying image diversity for different clusters of text prompts. Images are generated using the PixArt- α model.
Figure 16: Quantifying image diversity for different clusters of text prompts. Images are generated using the Flux model.
Figure 17: Evaluated Conditional-Vendi, Vendi, Conditional-RKE, and RKE scores on animal samples generated by PixArt Σ . (Left Plot) We do not specify the animal types in the prompt (Right Plot) we specify the animal types in the prompt.
Figure 18: Comparing Conditional-Vendi with Vendi on different fruit types generated by Stable Diffusion-XL.
Figure 19: Comparing Conditional-Vendi with Vendi on different fruit types generated by PixArt- Σ .
Figure 20: Statistical convergence of Conditional-Vendi scores with different sample sizes on data generated by Stable Diffusion-XL and Kandinsky using MS-COCO validation set prompts with finite-dimension cosine similarity and infinite-dimension Gaussian kernel. DINOv2 and CLIP embeddings are used for image and text modalities, respectively.
Figure 21: Conditional-Vendi and Information-Vendi of image-captioning models for 3 image types
Figure 22: Llama 2 Samples generated with different temperature parameters.
Method
T=0.4
T=0.7
T=1.0
T=1.3
Conditional-Vendi
40.82
42.82
44.03
48.23
Conditional-RKE
38.42
41.82
43.16
45.93
Appendix
Table 4: Conditional Vendi and RKE Scores evaluated for Gemma 3 with different temperature parameters.
Method
T=0.4
T=0.7
T=1.0
T=1.3
Conditional-Vendi
41.02
45.93
49.93
51.60
Conditional-RKE
39.43
41.74
47.27
49.82
Appendix
Table 5: Conditional Vendi and RKE Scores evaluated for Phi 4 Mini with different temperature parameters.
Figure 23: Measuring Conditional-Vendi and Information-Vendi for text-to-video models
Figure 24: Conditional and Information Vendi and RKE score comparison across text-to-image models. We clustered MS-COCO prompts into k groups and generated images for each cluster center. Within each cluster, we paired prompts with identical images. The results show increasing diversity and stronger correlation as the number of clusters grows, indicating that clusters become more relevant and diverse with finer partitioning.
Figure 25: 3 clusters of MS-COCO generated samples
The expansion of generative AI and LLM services underscores the growing need for adaptive mechanisms to select an appropriate available model to respond to a user's prompts. Recent works have proposed offline and online learning formulations to identify the optimal generative AI model for an input prompt, based solely on maximizing prompt-based fidelity evaluation scores, e.g., CLIP-Score in text-to-image generation. However, such fidelity-based selection methods overlook the diversity of generated outputs, and hence, they can fail to address potential diversity shortcomings in the generated responses. In this paper, we introduce the Diversity-Aware Kernelized Upper Confidence Bound (DAK-UCB) method as a contextual bandit algorithm for the online selection of generative models with diversity considerations. The proposed DAK-UCB method incorporates both fidelity and diversity-related metrics into the selection process. We design this framework based on prompt-aware diversity score functions that decompose to a two-sample-based expectation over prompt-output pairs in the previous generation rounds. Specifically, we illustrate the application of our framework using joint kernel distance and kernel entropy measures. Our experimental results demonstrate the effectiveness of DAK-UCB in promoting diversity-aware model selection while maintaining fidelity in the generations for a sequence of prompts. The code is available at https://github.com/Donya-Jafari/DAK-UCB.
Donya Jafari, Farzan Farnia
Sharif University of Technology · The Chinese University of Hong Kong
Deep generative models have achieved great success in producing high-quality samples, making them a central tool across machine learning applications. Beyond sample quality, an important yet less systematically studied question is whether trained generative models faithfully capture the diversity of the underlying data distribution. In this work, we address this question by directly comparing the diversity of samples generated by state-of-the-art models with that of test samples drawn from the target data distribution, using recently proposed reference-free entropy-based diversity scores, Vendi and RKE. Across multiple benchmark datasets, we find that test data consistently attains substantially higher Vendi and RKE diversity scores than the generated samples, suggesting a systematic downward diversity bias in modern generative models. To understand the origin of this bias, we analyze the finite-sample behavior of entropy-based diversity scores and show that their expected values increase with sample size, implying that diversity estimated from finite training sets could inherently underestimate the diversity of the true distribution. As a result, optimizing the generators to minimize divergence to empirical data distributions would induce a loss of diversity. Finally, we discuss potential diversity-aware regularization and guidance strategies based on Vendi and RKE as principled directions for mitigating this bias, and provide empirical evidence suggesting their potential to improve the results.
Farzan Farnia, Mohammad Jalali, Azim Ospanov
Department of Computer Science and Engineering, The Chinese University of Hong Kong.
We introduce Gacha Decoding, an inference-time method for eliciting diverse language model generations that scales with model capability. Across open-ended domains (in-the-wild chat, creative writing, planning for image generation, and protein design), Gacha Decoding significantly outperforms existing generation diversity approaches at equal quality (up to 2.4x Vendi over the next-best prior approach), reaching the same number of high-quality modes with over an order of magnitude fewer samples (11.0x) and discovering novel modes that no other approach surfaces. Our key insight is to treat diversity as an instruction-following problem: rather than relying on the LM's token entropy, we combine its instruction-following capability with randomness from an external RNG tool to scalably identify and realize distinct modes of the response space. This approach of "planning with dice" enables Gacha to invert the long-observed tension between diversity and model capability. As the underlying LM becomes a better instruction follower, diversity under Gacha Decoding consistently improves--even as its token entropy and diversity under prior approaches decline. Together, our results highlight that instruction following, rather than token entropy alone, can drive generation diversity.
Scott Geng, Yufei Zhang, Joseph Lee +3
University of Washington · Meta Superintelligence Labs