A survey of diversity quantification in natural language processing: The why, what, where and how
Organizations: Université Paris-Saclay, CNRS, LISN · FNRS - UCLouvain · The Open University of Israel · Heidelberg University
Abstract
The concept of diversity received increasing attention in natural language processing (NLP) in recent years. It became an advocated property of datasets and systems, with many measures used to quantify it, but also many cross-paper inconsistencies. To address this fragmentation, we take inspiration from other scientific fields where diversity has been more thoroughly conceptualized. We build upon (Sarkar, 2010) and (Stirling, 2007), two unified cross-domain frameworks inspired from ecology, economy and other domains. The first one advocates explicit justification of diversity endorsement. The latter distinguishes three dimensions of diversity: variety, balance, and disparity. Surveying over 300 recent diversity-related papers from ACL Anthology, we propose an NLP-specific framework with 4 perspectives: why diversity is important, what diversity is measured on, where it is measured, and how. We find that in NLP richness (the simplest variety) and distance averaging (the simplest disparity) are the most prevalent measure families, and that pure balance measures are nearly absent. Moreover, half of the measures used in NLP do not conform to Stirling's framework. Our analysis increases comparability of approaches to diversity in NLP, allowing us to formulate recommendations for the field. We recommend that the field systematically use a unified terminology, and that it engage in debates about the desirable properties of diversity measures as well as about the normative assumptions behind their use.
Figures & tables
| ARR area | # |
|---|---|
| Dialogue & Interactive Systems | 19 |
| Discourse & Pragmatics | 1 |
| Generation | 27 |
| Information Extraction | 1 |
| Language Modeling | 8 |
| Machine Learning for NLP | 31 |
| NLP diversity measures, conforming to Stirling (2007) | |
|---|---|
| Variety | dRichness (77) • number of categories Vylomova et al. (2020) , Dist-n and Dist-S Park et al. (2023) , vocabulary size Cegin et al. (2023) , number of unique words Panagiaris et al. (2020) , richness Lion-Bouton et al. (2022) • ave. # categories per text Ye et al. (2021) ; Byrne et al. (2019) |
| Balance | dBalance (3) • Hill-based E 2,1 evenness Lion-Bouton et al. (2022) • Zipfian coefficient Hong et al. (2020) ; Zhang et al. (2023) |
| Variety-balance | dEntropy (21) • entropy Joshi et al. (2020) ; Keleg and Magdy (2023) • sem-ent Han et al. (2022) • syntax/morphology typological index Samardzic et al. (2024) |
| dVBOth (3) • Herfindahl-Herschman index Dunn et al. (2020) • Gini-index Raza et al. (2022) • Simpson’s D index Kosmajac and Keselj (2019) | |
| Disparity | dAvePairwise (52) • Self-BLEU Han et al. (2021) , pairwise BLEU E et al. (2023) , average pairwise BLEU Shao et al. (2022) , 100-BLEU Awasthi et al. (2022) , BLEU Aji et al. (2021) • BLEU-based discrepancy Shu et al. (2019) , 1-BLEU Hu et al. (2019) • average BLEU-based discrepancy Shu et al. (2019) • i-BLEU Burchell et al. (2022) • Self-BLEU-n Hwang et al. (2023) , BLEU-n Weir et al. (2020) • ave. pairwise cosine Kim et al. (2024) , ADC Hadifar et al. (2023) , average USE similarity Dopierre et al. (2021) , or dispersion of sentence embeddings Guo et al. (2024) • average syntactic lang2vec Poelman et al. (2024) • self-entailment Narayan et al. (2022) • average tree edit distance Hu et al. (2019) ; Cegin et al. (2023) , parse tree edit distance Jegadeesan et al. (2021) • mean tree kernel diff. Burchell et al. (2022) , ave. graph kernel dist. Guo et al. (2024) , DStree Bawden et al. (2020) • i-chrF (character n-gram F-score) Burchell et al. (2022) • pairwise average SBERT dissimilarity Weir et al. (2020) • Self-BERTScore Narayan et al. (2022) • Self-BLEURT Kwon et al. (2021) • average of unique words in common Bawden et al. (2020) • average word-level Jaccard Aji et al. (2021) , mean Jaccard n-gram similarity Li et al. (2022) , or average intersection over union Jegadeesan et al. (2021) • average pairwise lexical token overlap Yadav et al. (2024) • average reverse Jaccard of n-grams Larson et al. (2019) • average word-error rate Jegadeesan et al. (2021) • inverse rank-biased overlap Li et al. (2023) • gate diversity Lai et al. (2020a) |
| dVolume (3) • mean of ellipsoid radii across all axes Lai et al. (2020b) • squared volume of vectors’ geometry Yang et al. (2024) • convex hull volume Yu et al. (2022) |
| NLP diversity measures, non-conforming to Stirling (2007) |
|---|
| dRelOverlap (9) • MS-Jaccard Alihosseini et al. (2019) • Jaccard minmax Samardzic et al. (2024) • % novel Schüz et al. (2021) ; Park et al. (2023) • vocabulary usage Liu et al. (2024b) , % coverage Schüz et al. (2021) • number of novel Murahari et al. (2019) • overlap Zhou et al. (2021) • proportion of times a ground truth token was found in a model’s nucleus Sultan et al. (2020) • F1 between model-generated categories in the reference and in the predictions Huang et al. (2023) |
| dRelDistrib (5) • Fréchet BERT Distance Alihosseini et al. (2019) • Bhattacharyya Alihosseini et al. (2019) • Jensen-Shannon Divergence Xia et al. (2024) • data cross-entropy Kumar et al. (2022b) • average frequency rank with respect to the training set Schüz et al. (2021) • binary unfair rate, unfair error rate, area under curve Zhang et al. (2024b) |
| dRelDist (8) • Self-BLEU Cao and Wan (2020) • recall Gao et al. (2019) • Max-BLEU Tu et al. (2019) • sentiment diversity in recommendation systems: Senti MRR , Senti@5, Senti@10 Wu et al. (2020a) |
| dTTR (43) • Distinct-n Ippolito et al. (2019) , Dist-n Liu et al. (2024b) , topic uniqueness Li et al. (2023) • TTRn Schüz et al. (2021) , proportion of distinct n-grams Nishida et al. (2024) ; Song et al. (2024) • Measure of Textual Lexical Diversity Fu and Nederhof (2021) ; Lee et al. (2022) • Moving-Average Type–Token Ratio Lee et al. (2022) ; Wu et al. (2024) , Mean Segmental Type-Token Ratio Chang et al. (2022) • product of n-gram repetition proportions Chen et al. (2023a) • proportion of duplicates Bahuleyan and El Asri (2020) |
| dHuman (12) • ranking Eo et al. (2023) ; Park et al. (2023) ; Lahoti et al. (2023) ; Lee et al. (2023) • scale Diao et al. (2021) ; Narayan et al. (2022) ; Kim et al. (2023b) ; Yoon and Bak (2023) |
| dOther (12) • Mean IDF Stasaski et al. (2020) • max-gap Lahoti et al. (2023) • classifier ensemble diversity measures: average Q statistic, ratio errors, negative double fault, disagreement measure, correlation coefficient Kobayashi et al. (2022) • R-QAE Lee et al. (2020) • average length Panagiaris et al. (2020) • frequency of low-frequency words Han et al. (2022) • demand-weighted utility Khanuja et al. (2023) • F1 of a baseline coreference solver Zhukova et al. (2022) • Honoré’s R, Sichel’s S, Yule’s K Kosmajac and Keselj (2019) |