The concept of diversity received increasing attention in natural language processing (NLP) in recent years. It became an advocated property of datasets and systems, with many measures used to quantify it, but also many cross-paper inconsistencies. To address this fragmentation, we take inspiration from other scientific fields where diversity has been more thoroughly conceptualized. We build upon (Sarkar, 2010) and (Stirling, 2007), two unified cross-domain frameworks inspired from ecology, economy and other domains. The first one advocates explicit justification of diversity endorsement. The latter distinguishes three dimensions of diversity: variety, balance, and disparity. Surveying over 300 recent diversity-related papers from ACL Anthology, we propose an NLP-specific framework with 4 perspectives: why diversity is important, what diversity is measured on, where it is measured, and how. We find that in NLP richness (the simplest variety) and distance averaging (the simplest disparity) are the most prevalent measure families, and that pure balance measures are nearly absent. Moreover, half of the measures used in NLP do not conform to Stirling's framework. Our analysis increases comparability of approaches to diversity in NLP, allowing us to formulate recommendations for the field. We recommend that the field systematically use a unified terminology, and that it engage in debates about the desirable properties of diversity measures as well as about the normative assumptions behind their use.
Figures & tables
Figure 1: ACL Anthology papers from 1990 to 2024 with “diversity” or “diverse” in their title or abstract.
ARR area
#
Dialogue & Interactive Systems
19
Discourse & Pragmatics
1
Generation
27
Information Extraction
1
Language Modeling
8
Machine Learning for NLP
31
Table 1: Number of papers per ARR area in our corpus.
Figure 2: High-level stages in standard machine learning pipelines (rectangles for processes, ellipses for states), with numbers of papers quantifying diversity. Papers with two-stage quantification indicated with *.
D
NLP diversity measures, conforming to Stirling (2007)
Variety
dRichness (77) • number of categories Vylomova et al. (2020) , Dist-n and Dist-S Park et al. (2023) , vocabulary size Cegin et al. (2023) , number of unique words Panagiaris et al. (2020) , richness Lion-Bouton et al. (2022) • ave. # categories per text Ye et al. (2021) ; Byrne et al. (2019)
Balance
dBalance (3) • Hill-based E 2,1 evenness Lion-Bouton et al. (2022) • Zipfian coefficient Hong et al. (2020) ; Zhang et al. (2023)
Variety-balance
dEntropy (21) • entropy Joshi et al. (2020) ; Keleg and Magdy (2023) • sem-ent Han et al. (2022) • syntax/morphology typological index Samardzic et al. (2024)
dVBOth (3) • Herfindahl-Herschman index Dunn et al. (2020) • Gini-index Raza et al. (2022) • Simpson’s D index Kosmajac and Keselj (2019)
Disparity
dAvePairwise (52) • Self-BLEU Han et al. (2021) , pairwise BLEU E et al. (2023) , average pairwise BLEU Shao et al. (2022) , 100-BLEU Awasthi et al. (2022) , BLEU Aji et al. (2021) • BLEU-based discrepancy Shu et al. (2019) , 1-BLEU Hu et al. (2019) • average BLEU-based discrepancy Shu et al. (2019) • i-BLEU Burchell et al. (2022) • Self-BLEU-n Hwang et al. (2023) , BLEU-n Weir et al. (2020) • ave. pairwise cosine Kim et al. (2024) , ADC Hadifar et al. (2023) , average USE similarity Dopierre et al. (2021) , or dispersion of sentence embeddings Guo et al. (2024) • average syntactic lang2vec Poelman et al. (2024) • self-entailment Narayan et al. (2022) • average tree edit distance Hu et al. (2019) ; Cegin et al. (2023) , parse tree edit distance Jegadeesan et al. (2021) • mean tree kernel diff. Burchell et al. (2022) , ave. graph kernel dist. Guo et al. (2024) , DStree Bawden et al. (2020) • i-chrF (character n-gram F-score) Burchell et al. (2022) • pairwise average SBERT dissimilarity Weir et al. (2020) • Self-BERTScore Narayan et al. (2022) • Self-BLEURT Kwon et al. (2021) • average of unique words in common Bawden et al. (2020) • average word-level Jaccard Aji et al. (2021) , mean Jaccard n-gram similarity Li et al. (2022) , or average intersection over union Jegadeesan et al. (2021) • average pairwise lexical token overlap Yadav et al. (2024) • average reverse Jaccard of n-grams Larson et al. (2019) • average word-error rate Jegadeesan et al. (2021) • inverse rank-biased overlap Li et al. (2023) • gate diversity Lai et al. (2020a)
dVolume (3) • mean of ellipsoid radii across all axes Lai et al. (2020b) • squared volume of vectors’ geometry Yang et al. (2024) • convex hull volume Yu et al. (2022)
Table 2: Stirling’s dimensions ( D ) and NLP families (in bold) of diversity measures and sample articles using them. Number of papers per family in parentheses.
NLP diversity measures, non-conforming to Stirling (2007)
dRelOverlap (9) • MS-Jaccard Alihosseini et al. (2019) • Jaccard minmax Samardzic et al. (2024) • % novel Schüz et al. (2021) ; Park et al. (2023) • vocabulary usage Liu et al. (2024b) , % coverage Schüz et al. (2021) • number of novel Murahari et al. (2019) • overlap Zhou et al. (2021) • proportion of times a ground truth token was found in a model’s nucleus Sultan et al. (2020) • F1 between model-generated categories in the reference and in the predictions Huang et al. (2023)
dRelDistrib (5) • Fréchet BERT Distance Alihosseini et al. (2019) • Bhattacharyya Alihosseini et al. (2019) • Jensen-Shannon Divergence Xia et al. (2024) • data cross-entropy Kumar et al. (2022b) • average frequency rank with respect to the training set Schüz et al. (2021) • binary unfair rate, unfair error rate, area under curve Zhang et al. (2024b)
dRelDist (8) • Self-BLEU Cao and Wan (2020) • recall Gao et al. (2019) • Max-BLEU Tu et al. (2019) • sentiment diversity in recommendation systems: Senti MRR , Senti@5, Senti@10 Wu et al. (2020a)
dTTR (43) • Distinct-n Ippolito et al. (2019) , Dist-n Liu et al. (2024b) , topic uniqueness Li et al. (2023) • TTRn Schüz et al. (2021) , proportion of distinct n-grams Nishida et al. (2024) ; Song et al. (2024) • Measure of Textual Lexical Diversity Fu and Nederhof (2021) ; Lee et al. (2022) • Moving-Average Type–Token Ratio Lee et al. (2022) ; Wu et al. (2024) , Mean Segmental Type-Token Ratio Chang et al. (2022) • product of n-gram repetition proportions Chen et al. (2023a) • proportion of duplicates Bahuleyan and El Asri (2020)
dHuman (12) • ranking Eo et al. (2023) ; Park et al. (2023) ; Lahoti et al. (2023) ; Lee et al. (2023) • scale Diao et al. (2021) ; Narayan et al. (2022) ; Kim et al. (2023b) ; Yoon and Bak (2023)
dOther (12) • Mean IDF Stasaski et al. (2020) • max-gap Lahoti et al. (2023) • classifier ensemble diversity measures: average Q statistic, ratio errors, negative double fault, disagreement measure, correlation coefficient Kobayashi et al. (2022) • R-QAE Lee et al. (2020) • average length Panagiaris et al. (2020) • frequency of low-frequency words Han et al. (2022) • demand-weighted utility Khanuja et al. (2023) • F1 of a baseline coreference solver Zhukova et al. (2022) • Honoré’s R, Sichel’s S, Yule’s K Kosmajac and Keselj (2019)
Table 3: Families (in bold) of diversity measures in NLP non-conforming to Stirling’s framework. Relative quantification in the upper part. Number of papers per family in parentheses.
There is growing evidence that data diversity is crucial for developing fair and robust NLP models. However, current approaches to measure diversity remain inconsistent and fragmented: While there exist a number of tools for measuring the lexical diversity of texts, researchers lack standardized tools for quantifying diversity based on embeddings. Embedding-based diversity measures are highly flexible: They work with any embedding model and any data that can be embedded, and are thus applicable to many notions of diversity. With emb-diversity, we provide a comprehensive embedding-based diversity measurement tool, spanning a broad range of measures. We demonstrate its potential for several use cases: measuring the stylistic, semantic, language and speaker diversity of datasets. https://github.com/nlpsoc/emb-diversity/
Cantao Su, Menan Velayuthan, Esther Ploeger +2
Department of Information & Computing Sciences, Utrecht University · Utrecht, The Netherlands
Diversity is a critical property of text corpora, with direct implications for training data curation, test set design, and model generalization. However, its measurement remains fragmented and poorly understood. Existing embedding-based diversity measures vary widely in how they are computed and can yield substantially different diversity scores on the same data, yet no systematic comparison of their empirical behaviour exists. We make two contributions toward addressing this gap. First, we present a taxonomy of 21 embedding-based diversity measures, organizing them into four families based on their underlying computations. Second, we introduce EmbDivBench, a benchmark that evaluates these measures along three conceptually distinct components (variety, balance and disparity) on both simulated and natural text data. Although Mean Pairwise Distance is the most commonly used measure, our experiments show that it is outperformed by several alternatives on the three diversity components, with Graph Entropy being the best choice overall. Lastly, we distil these findings into practical recommendations for diversity measure selection.
Cantao Su, Anna Wegmann, Esther Ploeger +1
Department of Information & Computing Sciences, Utrecht University Utrecht, The Netherlands
This position paper argues that recent progress with diversity in NLP is disproportionately concentrated on a small number of areas surrounding fairness. We further argue that this is the result of a number of incentives, biases, and barriers which come together to disenfranchise marginalized researchers in non-fairness fields, or to move them into fairness-related fields. We substantiate our claims with an investigation into the demographics of NLP researchers by subfield, using our research to support a number of recommendations for ensuring that all areas within NLP can become more inclusive and equitable. In particular, we highlight the importance of breaking down feedback loops that reinforce disparities, and the need to address geographical and linguistic barriers that hinder participation in NLP research.