Diversity is a critical property of text corpora, with direct implications for training data curation, test set design, and model generalization. However, its measurement remains fragmented and poorly understood. Existing embedding-based diversity measures vary widely in how they are computed and can yield substantially different diversity scores on the same data, yet no systematic comparison of their empirical behaviour exists. We make two contributions toward addressing this gap. First, we present a taxonomy of 21 embedding-based diversity measures, organizing them into four families based on their underlying computations. Second, we introduce EmbDivBench, a benchmark that evaluates these measures along three conceptually distinct components (variety, balance and disparity) on both simulated and natural text data. Although Mean Pairwise Distance is the most commonly used measure, our experiments show that it is outperformed by several alternatives on the three diversity components, with Graph Entropy being the best choice overall. Lastly, we distil these findings into practical recommendations for diversity measure selection.
Figures & tables
Figure 1: The three components of diversity in EmbDivBench . In each panel, the left configuration has lower diversity and the right has higher diversity along the corresponding component. Variety counts the number of groups present; Balance measures how evenly points are distributed across groups; Disparity captures how far apart the groups lie in the embedding space.
Distance-based Aggregates pairwise distances; no constructed structures.
Mean Pairwise Dist. Sum Pairwise Dist. Energy Diameter 2 2 2 Although the names of Diameter and Bottleneck suggest geometric constructs, they are simply the maximal and minimal pairwise distance, respectively; we therefore place them, together with their variants (Sum Diameter, Sum Bottleneck), in the distance-based family. Bottleneck Sum Diameter Sum Bottleneck Chamfer Dist.
Geometry-based Quantifies spatial properties of embeddings.
Convex Hull Vol. (2D) Span (Centroid) Span (Medoid) Cluster Inertia Geo. Mean of Std. Log-Determinant
Distribution-based Quantifies distributional properties of embeddings.
Vendi Score Rényi Kernel Ent. DCScore Bins Entropy (UMAP) MagArea
Graph-based Summarizes a graph built over the embeddings.
MST Dispersion Graph Entropy
Table 1: Taxonomy of embedding-based diversity measures into four families.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Illustration behind the four measure families on a toy vector space.
Distribution
Description
Uniform
Equal counts per category
Slightly skewed
Top 20% categories take 40% of points
Mildly skewed
Top 20% categories take 60% of points
Zipfian
Zipfian decay over categories
Strongly skewed
Top 1 category takes 50%; next 4 take 30%
Appendix
Table 2: The five balance levels, ordered from most balanced (top) to least balanced (bottom).
Figure 4: Incremental construction across the three components, illustrated with small example values. Each dataset is a hierarchy of super-categories (thick outer box) containing categories (thin inner column) containing data points (dot). The total dataset size n is conserved across levels in every panel. Variety ( k=2→5 , with m=1 fixed): each kept category’s data points are truncated to a prefix, and 3 new categories are added. Balance (uniform → mildly skewed; k=5 and m=1 both fixed): the category set is held constant; data points are redistributed from low-frequency to high-frequency categories. Disparity ( m=1→3 , with k=6 fixed): the kept super-category retains a fixed-order prefix of 2 of its 6 categories (the other 4 are left out), and 2 new super-categories are added. Blue marks elements kept from the lower diversity level, orange marks elements newly added, and grey marks elements that were in the lower level but are left out at the higher level; this colour coding applies at all three levels of the hierarchy.
Domain
Topic
Example sentence
Government by country
National legislatures
A general election is an electoral process to choose most or all members of a governing body at the same time.
After being elected the Prime Minister appoints the cabinet ministers and announces them to the Riksdag.
Parliaments by country
Article 16.4.2° requires that the newly elected Dáil Éireann must convene no later than 30 days after the election.
Humanities
Anthropology
A horse culture is a tribal group or community whose day-to-day life revolves around the herding and breeding of horses.
Engineering disciplines
Materials science
Air and vacuum have high reluctance, while easily magnetized materials such as soft iron have low reluctance.
Appendix
Table 3: Examples from the Wikipedia Domains/Topics hierarchy. Sentences within the same topic share subject matter; sentences within the same domain but different topics are related but distinguishable; sentences from different domains are semantically distant.
Measure
Hyperparameters
Distance-based
Mean Pairwise Dist.
—
Sum Pairwise Dist.
—
Energy
γ=1.0 , ϵ=10−12
Diameter
—
Bottleneck
—
Appendix
Table 4: Hyperparameter settings for all 21 measures. “—” means the measure takes no further parameters. Measures differ in whether they operate on cosine distances or on a cosine-similarity kernel. More details can be found in our repository at https://github.com/nlpsoc/EmbDivBench
Sentence
Assigned topic
Comment
match — exact matches
He also contributed a foreword to white supremacist commentator Stefan Molyneux’s book Against the Gods .
New Atheism
The topic is explicit in the sentence itself.
match — neighbouring topics
In German, for example, the pentagraph tzsch represents the [t] sound of the English digraph ch , and indeed is found in the English word Nietzschean .
Orthography
Orthography and Linguistic terminology both make sense, but they lie close enough in the embedding space.
match — requires background knowledge
Quickly was founded in California in 1996 and started its trademark licensing program at the same time.
Tea culture
Resolvable only by recognizing Quickly as a Taiwanese bubble tea chain.
Appendix
Table 5: Illustrative examples from the manual audit, grouped by the reason for each label. Examples are selected for illustration and are not proportional to the frequency of each case.
Level
match
mismatch
Noise
Variety
k=10
44
6
0.12
k=20
40
10
0.20
k=30
36
14
0.28
k=40
39
11
0.22
k=50
40
10
0.20
Appendix
Table 6: Per-level annotation counts (50 sentences per level) and noise rate ( mismatch over annotated sentences) for the three components of the natural text tier.
Figure 5: Per-component mean Spearman ρ (5 seeds) on the simulated data (GMM) tier across embedding dimensions d∈{256,384,1024,2048,4096} for the Variety (V), Balance (B) and Disparity (D) components; the smaller italic number below each ρ is the across-seed standard deviation, shown when it exceeds 0.01 . Columns are grouped and coloured by the four measure families (Distance-, Geometry-, Distribution-, Graph-based). As in Fig. 2 , the two rank-equivalent measures are given in parentheses (Sum Pairwise Dist., Sum Bottleneck).
Table 7: Compute cost and asymptotic complexity of measures. Relative cost is reported as a multiple of mean pairwise distance on identical inputs ( n=10000 , d=384 ); within each family, rows are ordered by ascending cost.
There is growing evidence that data diversity is crucial for developing fair and robust NLP models. However, current approaches to measure diversity remain inconsistent and fragmented: While there exist a number of tools for measuring the lexical diversity of texts, researchers lack standardized tools for quantifying diversity based on embeddings. Embedding-based diversity measures are highly flexible: They work with any embedding model and any data that can be embedded, and are thus applicable to many notions of diversity. With emb-diversity, we provide a comprehensive embedding-based diversity measurement tool, spanning a broad range of measures. We demonstrate its potential for several use cases: measuring the stylistic, semantic, language and speaker diversity of datasets. https://github.com/nlpsoc/emb-diversity/
Cantao Su, Menan Velayuthan, Esther Ploeger +2
Department of Information & Computing Sciences, Utrecht University · Utrecht, The Netherlands
The concept of diversity received increasing attention in natural language processing (NLP) in recent years. It became an advocated property of datasets and systems, with many measures used to quantify it, but also many cross-paper inconsistencies. To address this fragmentation, we take inspiration from other scientific fields where diversity has been more thoroughly conceptualized. We build upon (Sarkar, 2010) and (Stirling, 2007), two unified cross-domain frameworks inspired from ecology, economy and other domains. The first one advocates explicit justification of diversity endorsement. The latter distinguishes three dimensions of diversity: variety, balance, and disparity. Surveying over 300 recent diversity-related papers from ACL Anthology, we propose an NLP-specific framework with 4 perspectives: why diversity is important, what diversity is measured on, where it is measured, and how. We find that in NLP richness (the simplest variety) and distance averaging (the simplest disparity) are the most prevalent measure families, and that pure balance measures are nearly absent. Moreover, half of the measures used in NLP do not conform to Stirling's framework. Our analysis increases comparability of approaches to diversity in NLP, allowing us to formulate recommendations for the field. We recommend that the field systematically use a unified terminology, and that it engage in debates about the desirable properties of diversity measures as well as about the normative assumptions behind their use.
Louis Estève, Marie-Catherine de Marneffe, Nurit Melnik +2
Université Paris-Saclay, CNRS, LISN · FNRS - UCLouvain · The Open University of Israel +1
Generative models guided by text prompts are widely evaluated for fidelity and prompt alignment, yet their ability to produce diverse outputs remains underexplored. Existing diversity metrics such as Vendi and RKE, which are based on the von Neumann and Rényi entropies of kernel matrices, were developed for unconditional models and cannot distinguish prompt-induced from model-induced variability. We address this gap by introducing Conditional-Vendi and Conditional-RKE, diversity measures derived from the conditional entropy of positive semidefinite matrices. These scores isolate model-induced diversity in prompt-guided generation, with Conditional-RKE enjoying an O(1/n) convergence rate. For Conditional-Vendi, we introduce a truncated-spectrum approximation that yields scalable and consistent estimates. Experiments on text-to-image, image-captioning, and LLM tasks show that the conditional scores recover ground-truth diversity orderings and can also guide diffusion models toward more diverse samples. The codebase is available at https://github.com/mjalali/conditional-vendi.
Mohammad Jalali, Azim Ospanov, Amin Gohari +1
Department of Computer Science and Engineering, The Chinese University of Hong Kong · Department of Information Engineering, The Chinese University of Hong Kong