Successful completion of perspectivist natural language processing (NLP) annotation tasks requires algorithms to be aligned across diverse perspectives in order to predict how different humans will rate text. This work studies how the performance of such algorithms varies for annotators who behave differently than others, either by providing annotations that diverge from others' responses or by providing fewer annotations altogether. To aid with characterizing model performance, we curate the largest collection to date of perspectivist NLP datasets with ordinal annotations. After evaluating multiple classes of perspectivist algorithms across each of the datasets, we model the relationship between individual annotator idiosyncrasy and algorithm performance. In general, we find that algorithms perform better on annotators who tend to agree with others, but that annotator response rate does not have a statistically significant positive correlation with algorithm performance.
Figures & tables
Figure 1: Depiction of the dataset split process. The figure displays 20 observed annotations from an illustrative pool of 60; cell numbers indicate positions in the shuffled order. The final test set contains the last 3 remaining annotations (5%), and the training set contains 57 annotations.
Hypothesis term
Coef.
95% CI
One-sided p
Holm-adjusted p
Disagreement ( βD on ZdaD )
0.025
[0.016, 0.033]
1.90×10−8
3.80×10−8
Sparsity ( βS on ZdaS )
0.001
[-0.001, 0.003]
0.158
0.158
Table 1: Results for the regression shown in Eq. 1 with cluster-robust standard errors. Statistically significant results are in bold .
Figure 2: Dataset-family block bootstrap results from 10,000 samples each. Top: histograms displaying the estimated regression slope across samples. Left: disagreement effect. Right: sparsity effect. Bottom: 95% bootstrap confidence intervals and positive coefficient rates. Positive coefficient rate denotes the proportion of bootstrap samples in which the coefficient estimate was positive. The orange vertical line highlights the median value, and the black vertical line highlights a value of 0. Our hypotheses predicted that both distributions would lie significantly to the right of the black line.
Figure 3: Forest plot from fitting the regression model from Equation 1 with one family of algorithms left out of the analysis. The figure displays the disagreement effect (left) and sparsity effect (right). Error bars indicate the 95% CI.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Algorithm
Family
#Param
Example Mean
Mean
0
Global Mean
Mean
0
User Mean
Mean
0
KMF
KMF
672,815
MASDA-Categorical
MASDA
23,868,264
MASDA-Ordinal
MASDA
23,868,222
Appendix
Table 2: Algorithms used in the experiments, grouped by source family, with maximum parameter counts across the experiment datasets.
Figure 4: Soft-label cross-entropy scores of the original MASDA model and our ordinal and categorical MASDA variants on the ArMIS ( Almanea and Poesio, 2022 ) and HS-Brexit ( Akhtar et al., 2021 ) datasets. Lower scores indicate better performance. Scores for our variants were computed on held-out annotations, while the original MASDA scores are those reported in the original paper and were not recomputed using our train/test splits.
Dataset
Source
N
M
Ratings/ Example
Label Meaning
(Other) × 6 VALENCE
AffectiveText
987 987
6 6
6.0 6.0
Scale from absent (0) to maximum (100) Highly negative (0), Highly positive (200)
Cockamamie Gobbledegook
HuBi
10000
351
3.6
Not funny (0), Funny (1)
(Other) × 8 EMOT. AROUSAL VALENCE
HuBi/ Personal Bias
100 100 100
500 500 500
8.5 8.5 8.5
Scale from low (3) to high (7) Scale from weak (3) to strong (7) Scale from negative (0) to positive (6)
No one believes (0), Occasionally believed (1), Controversial (2), Common belief (3), Universally true (4)
SBIC
NLPerspectives
10000
204
2.5
Not offensive (0), Maybe (1), Offensive (2)
Appendix
Table 3: Information and statistics on the 46 datasets used in our experiment. The statistics include the number of unique text instances (N), the number of unique annotators (M), and the average number of ratings per unique text. Within each source section, “(Other) × K” represents K datasets with identical statistics and label meanings.
Dataset Source
Dataset(s)
Terms
AffectiveText
ANGER DISGUST FEAR JOY SADNESS SURPRISE VALENCE
Public Release
HuBi
Cockamamie Gobbledegook
Public Release
HuBi/Personal Bias
ANGER ANTICIPATION DISGUST EMOT. AROUSAL FEAR JOY SADNESS SURPRISE TRUST VALENCE
Public Release
HuBi/Wikipedia Detox ( Wulczyn et al., 2017b ; Wulczyn et al., 2017c ) ( Thain et al., 2017 )
Wiki-Aggression Wiki-Attack Wiki-Toxicity
CC0 1.0 CC0 1.0 CC0 1.0
NLPerspectives
SChem SChem5Labels SBIC GHC Sentiment
CC BY-SA 4.0 CC BY-SA 4.0 CC BY 4.0 CC BY 4.0 CC0 1.0
NLPerspectives/Stanford Politeness Corpus
Politeness
CC BY 4.0
Appendix
Table 4: Publicly stated licenses, access restrictions, or availability terms for English-language dataset sources used in our experiments.
Figure 5: Smoothed density curves of annotation values across datasets with train data (top left), test data (top right), and both train and test data combined (bottom).
Figure 6: Distribution of agreement and annotator response rate across all datasets. Datasets with the same source and annotation patterns are grouped together when graphing response rate.
Figure 7: Forest plots of per-algorithm normalized mean squared error (top left), normalized root mean squared error (top right), and a bar chart of normalized mean squared error averaged over each algorithm family (bottom). The error bars in the forest plots show bootstrapped 95% confidence intervals based on 1,000 resamples of individual datasets.
Figure 8: Per-model normalized root mean squared error performance on each dataset. Datasets are ordered from top to bottom in order of average performance across algorithms. Algorithms are ordered from left to right by average performance across datasets. Here, we use NRMSE (as opposed to NMSE) for better interpretability.
Figure 9: Average per-model-family normalized root mean squared error performance on each dataset. Similar to Figure 8 , datasets are ordered from top to bottom by performance across algorithm families, algorithms are ordered from left to right by average performance across datasets, and we use NRMSE for stronger interpretability.
Effect
Coef.
95% CI
Two-sided p
Holm-adjusted p
Disagreement ( βD on ZdaD )
0.025
[0.016, 0.033]
3.80×10−8
7.61×10−8
Sparsity ( βS on ZdaS )
0.001
[-0.001, 0.003]
0.317
0.317
Appendix
Table 5: Two-sided post hoc regression results for the disagreement and sparsity effects in the regression shown in Equation 1 in the main paper. The main hypothesis tests shown in Table 1 in the main paper use the predetermined one-sided tests for βD>0 and βS>0 ; these two-sided results are reported only to characterize the direction and robustness of the estimated effects. Results which are statistically significant are shown in bold .
Annotator disagreement is widespread in NLP, particularly for subjective and ambiguous tasks such as toxicity detection and stance analysis. While early approaches treated disagreement as noise to be removed, recent work increasingly models it as a meaningful signal reflecting variation in interpretation and perspective. This survey provides a unified view of disagreement-aware NLP methods. We first present a domain-agnostic taxonomy of the sources of disagreement spanning data, task, and annotator factors. We then synthesize modeling approaches using a common framework defined by prediction targets and pooling structure, highlighting a shift from consensus learning toward explicitly modeling disagreement, and toward capturing structured relationships among annotators. We review evaluation metrics for both predictive performance and annotator behavior, and noting that most fairness evaluations remain descriptive rather than normative. We conclude by identifying open challenges and future directions, including integrating multiple sources of variation, developing disagreement-aware interpretability frameworks, and grappling with the practical tradeoffs of perspectivist modeling.
Humans often hold different perspectives on the same issues. In many NLP tasks, annotation disagreement can reflect valid subjective perspectives. Modeling annotator perspectives and understanding their relationship with other human factors, such as socio-demographic attributes, have received increasing attention. Prior work typically focuses on single demographic factors or limited combinations. However, in real-world settings, annotator perspectives are shaped by complex social contexts, and finer-grained socio-demographic attributes can better explain human perspectives. In this work, we propose Socio-Contrastive Learning, a method that jointly models annotator perspectives while learning socio-demographic representations. Our method provides an effective approach for the fusion of socio-demographic features and textual representations to predict annotator perspectives, outperforming standard concatenation-based methods. The learned representations further enable analysis and visualization of how demographic factors relate to variation in annotator perspectives. Our code is available at GitHub: https://github.com/Leixin-Zhang/Socio_Contrastive_Learning
There is growing recognition that many NLP tasks lack a single ground truth, as human judgments reflect diverse perspectives. To capture this variation, models have been developed to predict full annotation distributions rather than majority labels, while perspectivist models aim to reproduce the interpretations of individual annotators. In this work, we compare these approaches on Implicit Discourse Relation Recognition (IDRR), a highly ambiguous task where disagreement often arises from cognitive complexity rather than ideological bias. Our experiments show that existing annotator-specific models perform poorly in IDRR unless ambiguity is reduced, whereas models trained on label distributions yield more stable predictions. Further analysis indicates that frequent cognitively demanding cases drive inconsistency in human interpretation, posing challenges for perspectivist modeling in IDRR.
Frances Yung, Daniil Ignatev, Merel Scholman +2
Saarland University, Saarland Informatics Campus, Germany · Utrecht University, Utrecht, The Netherlands · Queen Mary University of London, London, UK