cs.CLOct 17, 2026
SaveMeasuring Data Diversity with Embeddings: A Taxonomy and Benchmark for Diversity Measures
Organizations: Department of Information & Computing Sciences, Utrecht University Utrecht, The Netherlands
Abstract
Diversity is a critical property of text corpora, with direct implications for training data curation, test set design, and model generalization. However, its measurement remains fragmented and poorly understood. Existing embedding-based diversity measures vary widely in how they are computed and can yield substantially different diversity scores on the same data, yet no systematic comparison of their empirical behaviour exists. We make two contributions toward addressing this gap. First, we present a taxonomy of 21 embedding-based diversity measures, organizing them into four families based on their underlying computations. Second, we introduce EmbDivBench, a benchmark that evaluates these measures along three conceptually distinct components (variety, balance and disparity) on both simulated and natural text data. Although Mean Pairwise Distance is the most commonly used measure, our experiments show that it is outperformed by several alternatives on the three diversity components, with Graph Entropy being the best choice overall. Lastly, we distil these findings into practical recommendations for diversity measure selection.
Figures & tables
Figure 1: The three components of diversity in EmbDivBench . In each panel, the left configuration has lower diversity and the right has higher diversity along the corresponding component. Variety counts the number of groups present; Balance measures how evenly points are distributed across groups; Disparity captures how far apart the groups lie in the embedding space.
| Distance-based Aggregates pairwise distances; no constructed structures. | Mean Pairwise Dist. Sum Pairwise Dist. Energy Diameter 2 2 2 Although the names of Diameter and Bottleneck suggest geometric constructs, they are simply the maximal and minimal pairwise distance, respectively; we therefore place them, together with their variants (Sum Diameter, Sum Bottleneck), in the distance-based family. Bottleneck Sum Diameter Sum Bottleneck Chamfer Dist. |
|---|---|
| Geometry-based Quantifies spatial properties of embeddings. | Convex Hull Vol. (2D) Span (Centroid) Span (Medoid) Cluster Inertia Geo. Mean of Std. Log-Determinant |
| Distribution-based Quantifies distributional properties of embeddings. | Vendi Score Rényi Kernel Ent. DCScore Bins Entropy (UMAP) MagArea |
| Graph-based Summarizes a graph built over the embeddings. | MST Dispersion Graph Entropy |
Table 1: Taxonomy of embedding-based diversity measures into four families.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Illustration behind the four measure families on a toy vector space.
| Distribution | Description |
|---|---|
| Uniform | Equal counts per category |
| Slightly skewed | Top 20% categories take 40% of points |
| Mildly skewed | Top 20% categories take 60% of points |
| Zipfian | Zipfian decay over categories |
| Strongly skewed | Top 1 category takes 50%; next 4 take 30% |
Table 2: The five balance levels, ordered from most balanced (top) to least balanced (bottom).
Figure 4: Incremental construction across the three components, illustrated with small example values. Each dataset is a hierarchy of super-categories (thick outer box) containing categories (thin inner column) containing data points (dot). The total dataset size is conserved across levels in every panel. Variety ( , with fixed): each kept category’s data points are truncated to a prefix, and 3 new categories are added. Balance (uniform mildly skewed; and both fixed): the category set is held constant; data points are redistributed from low-frequency to high-frequency categories. Disparity ( , with fixed): the kept super-category retains a fixed-order prefix of 2 of its 6 categories (the other 4 are left out), and 2 new super-categories are added. Blue marks elements kept from the lower diversity level, orange marks elements newly added, and grey marks elements that were in the lower level but are left out at the higher level; this colour coding applies at all three levels of the hierarchy.
| Domain | Topic | Example sentence |
|---|---|---|
| Government by country | National legislatures | A general election is an electoral process to choose most or all members of a governing body at the same time. |
| After being elected the Prime Minister appoints the cabinet ministers and announces them to the Riksdag. | ||
| Parliaments by country | Article 16.4.2° requires that the newly elected Dáil Éireann must convene no later than 30 days after the election. | |
| Humanities | Anthropology | A horse culture is a tribal group or community whose day-to-day life revolves around the herding and breeding of horses. |
| Engineering disciplines | Materials science | Air and vacuum have high reluctance, while easily magnetized materials such as soft iron have low reluctance. |
Table 3: Examples from the Wikipedia Domains/Topics hierarchy. Sentences within the same topic share subject matter; sentences within the same domain but different topics are related but distinguishable; sentences from different domains are semantically distant.
| Measure | Hyperparameters |
|---|---|
| Distance-based | |
| Mean Pairwise Dist. | — |
| Sum Pairwise Dist. | — |
| Energy | , |
| Diameter | — |
| Bottleneck | — |
Table 4: Hyperparameter settings for all 21 measures. “—” means the measure takes no further parameters. Measures differ in whether they operate on cosine distances or on a cosine-similarity kernel. More details can be found in our repository at https://github.com/nlpsoc/EmbDivBench
| Sentence | Assigned topic | Comment |
|---|---|---|
| match — exact matches | ||
| He also contributed a foreword to white supremacist commentator Stefan Molyneux’s book Against the Gods . | New Atheism | The topic is explicit in the sentence itself. |
| match — neighbouring topics | ||
| In German, for example, the pentagraph tzsch represents the [t] sound of the English digraph ch , and indeed is found in the English word Nietzschean . | Orthography | Orthography and Linguistic terminology both make sense, but they lie close enough in the embedding space. |
| match — requires background knowledge | ||
| Quickly was founded in California in 1996 and started its trademark licensing program at the same time. | Tea culture | Resolvable only by recognizing Quickly as a Taiwanese bubble tea chain. |
Table 5: Illustrative examples from the manual audit, grouped by the reason for each label. Examples are selected for illustration and are not proportional to the frequency of each case.
| Level | match | mismatch | Noise |
| Variety | |||
| 44 | 6 | 0.12 | |
| 40 | 10 | 0.20 | |
| 36 | 14 | 0.28 | |
| 39 | 11 | 0.22 | |
| 40 | 10 | 0.20 | |
Table 6: Per-level annotation counts (50 sentences per level) and noise rate ( mismatch over annotated sentences) for the three components of the natural text tier.
Figure 5: Per-component mean Spearman (5 seeds) on the simulated data (GMM) tier across embedding dimensions for the Variety (V), Balance (B) and Disparity (D) components; the smaller italic number below each is the across-seed standard deviation, shown when it exceeds . Columns are grouped and coloured by the four measure families (Distance-, Geometry-, Distribution-, Graph-based). As in Fig. 2 , the two rank-equivalent measures are given in parentheses (Sum Pairwise Dist., Sum Bottleneck).
Table 7: Compute cost and asymptotic complexity of measures. Relative cost is reported as a multiple of mean pairwise distance on identical inputs ( , ); within each family, rows are ordered by ascending cost.