Who Do Perspectivist Models Predict Well? Annotator Agreement, Sparsity, and Performance

London Lowmanstone; Vaagishwaran Sivakumar; Anwesha Samaddar; Akul Mundada; Harnoor Kaur; Maria Gini

cs.CLOct 15, 2026

Who Do Perspectivist Models Predict Well? Annotator Agreement, Sparsity, and Performance

Authors: London Lowmanstone, Vaagishwaran Sivakumar, Anwesha Samaddar, Akul Mundada, Harnoor Kaur, Maria Gini

Organizations: University of Minnesota, Twin Cities

Abstract

Successful completion of perspectivist natural language processing (NLP) annotation tasks requires algorithms to be aligned across diverse perspectives in order to predict how different humans will rate text. This work studies how the performance of such algorithms varies for annotators who behave differently than others, either by providing annotations that diverge from others' responses or by providing fewer annotations altogether. To aid with characterizing model performance, we curate the largest collection to date of perspectivist NLP datasets with ordinal annotations. After evaluating multiple classes of perspectivist algorithms across each of the datasets, we model the relationship between individual annotator idiosyncrasy and algorithm performance. In general, we find that algorithms perform better on annotators who tend to agree with others, but that annotator response rate does not have a statistically significant positive correlation with algorithm performance.

Figures & tables

Figure 1: Depiction of the dataset split process. The figure displays 20 observed annotations from an illustrative pool of 60; cell numbers indicate positions in the shuffled order. The final test set contains the last 3 remaining annotations (5%), and the training set contains 57 annotations.

Hypothesis term	Coef.	95% CI	One-sided $p$	Holm-adjusted $p$
Disagreement ( $\beta_{D}$ on $Z^{D}_{da}$ )	0.025	[0.016, 0.033]	$\mathbf{1.90\times 10^{-8}}$	$\mathbf{3.80\times 10^{-8}}$
Sparsity ( $\beta_{S}$ on $Z^{S}_{da}$ )	0.001	[-0.001, 0.003]	0.158	0.158

Table 1: Results for the regression shown in Eq. 1 with cluster-robust standard errors. Statistically significant results are in bold .

Panel of Figure 2 — Figure 2: Dataset-family block bootstrap results from 10,000 samples each. Top: histograms displaying the estimated regression slope across samples. Left: disagreement effect. Right: sparsity effect. Bottom: 95% bootstrap confidence intervals and positive coefficient rates. Positive coefficient rate denotes the proportion of bootstrap samples in which the coefficient estimate was positive. The orange vertical line highlights the median value, and the black vertical line highlights a value of 0. Our hypotheses predicted that both distributions would lie significantly to the right of the black line.

Figure 3: Forest plot from fitting the regression model from Equation 1 with one family of algorithms left out of the analysis. The figure displays the disagreement effect (left) and sparsity effect (right). Error bars indicate the 95% CI.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Algorithm	Family	#Param
Example Mean	Mean	0
Global Mean	Mean	0
User Mean	Mean	0
KMF	KMF	672,815
MASDA-Categorical	MASDA	23,868,264
MASDA-Ordinal	MASDA	23,868,222

Appendix

Table 2: Algorithms used in the experiments, grouped by source family, with maximum parameter counts across the experiment datasets.

Figure 4: Soft-label cross-entropy scores of the original MASDA model and our ordinal and categorical MASDA variants on the ArMIS ( Almanea and Poesio, 2022 ) and HS-Brexit ( Akhtar et al., 2021 ) datasets. Lower scores indicate better performance. Scores for our variants were computed on held-out annotations, while the original MASDA scores are those reported in the original paper and were not recomputed using our train/test splits.

Dataset	Source	N	M	Ratings/ Example	Label Meaning
(Other) $\times$ 6 VALENCE	AffectiveText	987 987	6 6	6.0 6.0	Scale from absent (0) to maximum (100) Highly negative (0), Highly positive (200)
Cockamamie Gobbledegook	HuBi	10000	351	3.6	Not funny (0), Funny (1)
(Other) $\times$ 8 EMOT. AROUSAL VALENCE	HuBi/ Personal Bias	100 100 100	500 500 500	8.5 8.5 8.5	Scale from low (3) to high (7) Scale from weak (3) to strong (7) Scale from negative (0) to positive (6)
Wiki-Aggression Wiki-Attack Wiki-Toxicity	HuBi/ Wikipedia Detox	7303 7303 7101	488 488 490	1.8 1.8 1.6	Friendly/neutral (0), Aggressive (1) Not attack/harassment (0), Att./harass. (1) Healthy/neutral (0), Toxic (1)
SChem SChem5Labels	NLPerspectives	400 7861	100 102	50.0 5.0	No one believes (0), Occasionally believed (1), Controversial (2), Common belief (3), Universally true (4)
SBIC	NLPerspectives	10000	204	2.5	Not offensive (0), Maybe (1), Offensive (2)

Appendix

Table 3: Information and statistics on the 46 datasets used in our experiment. The statistics include the number of unique text instances (N), the number of unique annotators (M), and the average number of ratings per unique text. Within each source section, “(Other)

\times

K” represents K datasets with identical statistics and label meanings.

Dataset Source	Dataset(s)	Terms
AffectiveText	ANGER DISGUST FEAR JOY SADNESS SURPRISE VALENCE	Public Release
HuBi	Cockamamie Gobbledegook	Public Release
HuBi/Personal Bias	ANGER ANTICIPATION DISGUST EMOT. AROUSAL FEAR JOY SADNESS SURPRISE TRUST VALENCE	Public Release
HuBi/Wikipedia Detox ( Wulczyn et al., 2017b ; Wulczyn et al., 2017c ) ( Thain et al., 2017 )	Wiki-Aggression Wiki-Attack Wiki-Toxicity	CC0 1.0 CC0 1.0 CC0 1.0
NLPerspectives	SChem SChem5Labels SBIC GHC Sentiment	CC BY-SA 4.0 CC BY-SA 4.0 CC BY 4.0 CC BY 4.0 CC0 1.0
NLPerspectives/Stanford Politeness Corpus	Politeness	CC BY 4.0

Appendix

Table 4: Publicly stated licenses, access restrictions, or availability terms for English-language dataset sources used in our experiments.

Panel of Figure 5 — Figure 5: Smoothed density curves of annotation values across datasets with train data (top left), test data (top right), and both train and test data combined (bottom).

Figure 6: Distribution of agreement and annotator response rate across all datasets. Datasets with the same source and annotation patterns are grouped together when graphing response rate.

Panel of Figure 7 — Figure 7: Forest plots of per-algorithm normalized mean squared error (top left), normalized root mean squared error (top right), and a bar chart of normalized mean squared error averaged over each algorithm family (bottom). The error bars in the forest plots show bootstrapped 95% confidence intervals based on 1,000 resamples of individual datasets.

Figure 8: Per-model normalized root mean squared error performance on each dataset. Datasets are ordered from top to bottom in order of average performance across algorithms. Algorithms are ordered from left to right by average performance across datasets. Here, we use NRMSE (as opposed to NMSE) for better interpretability.

Figure 9: Average per-model-family normalized root mean squared error performance on each dataset. Similar to Figure 8 , datasets are ordered from top to bottom by performance across algorithm families, algorithms are ordered from left to right by average performance across datasets, and we use NRMSE for stronger interpretability.

Effect	Coef.	95% CI	Two-sided $p$	Holm-adjusted $p$
Disagreement ( $\beta_{D}$ on $Z^{D}_{da}$ )	0.025	[0.016, 0.033]	$\mathbf{3.80\times 10^{-8}}$	$\mathbf{7.61\times 10^{-8}}$
Sparsity ( $\beta_{S}$ on $Z^{S}_{da}$ )	0.001	[-0.001, 0.003]	0.317	0.317

Appendix

Table 5: Two-sided post hoc regression results for the disagreement and sparsity effects in the regression shown in Equation 1 in the main paper. The main hypothesis tests shown in Table 1 in the main paper use the predetermined one-sided tests for

\beta_{D}>0

and

\beta_{S}>0

; these two-sided results are reported only to characterize the direction and robustness of the estimated effects. Results which are statistically significant are shown in bold .

Annotator Disagreement

Annotator disagreement studies why human labels differ, how to measure and model that variation, and how it should shape AI training and evaluation.

See all 86 papers

Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP
Jan 14, 2026Yinuo Xu, David JurgensAnnotator Disagreement
University of Michigan
Modeling Human Perspectives with Socio-Demographic Representations
Apr 20, 2026Leixin Zhang, Cagri ColtekinContrastive Learning Annotator Disagreement
University of Tübingen
Human Label Variation in Implicit Discourse Relation Recognition
Feb 26, 2026Frances Yung, Daniil Ignatev, Merel Scholman +2Annotator Disagreement Label Distribution Learning
Saarland University, Saarland Informatics Campus, Germany · Utrecht University, Utrecht, The Netherlands · Queen Mary University of London, London, UK

Jan 14, 2026J/K move · Enter open · S save

Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP

Yinuo Xu, David Jurgens

University of Michigan

Annotator Disagreement

Open paper AskPDF arXiv

Annotator disagreement is widespread in NLP, particularly for subjective and ambiguous tasks such as toxicity detection and stance analysis. While early approaches treated disagreement as noise to be removed, recent work increasingly models it as a meaningful signal reflecting variation in interpretation and perspective. This survey provides a unified view of disagreement-aware NLP methods. We first present a domain-agnostic taxonomy of the sources of disagreement spanning data, task, and annotator factors. We then synthesize modeling approaches using a common framework defined by prediction targets and pooling structure, highlighting a shift from consensus learning toward explicitly modeling disagreement, and toward capturing structured relationships among annotators. We review evaluation metrics for both predictive performance and annotator behavior, and noting that most fairness evaluations remain descriptive rather than normative. We conclude by identifying open challenges and future directions, including integrating multiple sources of variation, developing disagreement-aware interpretability frameworks, and grappling with the practical tradeoffs of perspectivist modeling.

Abstract

Annotator Disagreement

Explore similar work

Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP

Modeling Human Perspectives with Socio-Demographic Representations

Human Label Variation in Implicit Discourse Relation Recognition