When humans label subjective content, they disagree, and that disagreement is not noise. It reflects genuine differences in perspective shaped by annotators' social identities and lived experiences. Yet standard practice still flattens these judgments into a single majority label, and recent LLM-based approaches fare no better: we show that prompted large language models, even with chain-of-thought reasoning, fail to recover the structure of human disagreement. We introduce DiADEM, a neural architecture that learns "how much each demographic axis matters" for predicting who will disagree and on what. DiADEM encodes annotators through per-demographic projections governed by a learned importance vector α, fuses annotator and item representations via complementary concatenation and Hadamard interactions, and is trained with a novel item-level disagreement loss that directly penalizes mispredicted annotation variance. On the DICES conversational-safety and VOICED political-offense benchmarks, DiADEM substantially outperforms both the LLM-as-a-judge and neural model baselines across standard and perspectivist metrics, achieving strong disagreement tracking (r=0.75 on DICES). The learned α weights reveal that race and age consistently emerge as the most influential demographic factors driving annotator disagreement across both datasets. Our results demonstrate that explicitly modeling who annotators are not just what they label is essential for NLP systems that aim to faithfully represent human interpretive diversity.
Figures & tables
Figure 1: Block Diagram of DiADEM Encoder and Decoder architecture
Standard Metrics
Soft Prediction
Perspectivist
Model
Acc.
F1 mac
F1 wt
κ
MCC
JSD ↓
MD ↓
ER ↓
ECE ↓
DiADEM
0.7337
0.4116
0.6980
0.2450
0.2643
0.0446
0.7795
0.4911
0.0391
DisCo
0.6292
0.3261
0.6002
0.0030
0.0030
0.0809
0.9010
0.7036
0.0710
LeWiDi
0.6562
0.2641
0.5200
0.0000
0.0000
0.0254 ∗
0.8954
0.6323
0.0377
LLM baselines
GPT-4o-mini
0.6613
0.3706
0.6457
0.1391
0.1419
0.3381
0.6761
0.6194
0.3387
Table 1: DICES standard, soft, and perspectivist metrics (annotator-level, 3-class). Bold = best overall; underline = best among LLM baselines. ∗ LeWiDi achieves artificially low JSD by predicting only “No” ( κ=0 , zero recall on Unsure / Yes ); this constitutes degenerate collapse, not genuine distributional alignment.
Standard Metrics
Soft Prediction
Perspectivist
Model
Acc.
F1 mac
F1 wt
κ
MCC
JSD ↓
MD ↓
ER ↓
ECE ↓
DiADEM
0.7646
0.5587
0.7025
0.1826
0.2616
0.0923
0.4125
0.2292
0.0129
DisCo
0.7385
0.4248
0.6274
0.0000
0.0000
0.1765
0.4989
0.2495
0.2615
LeWiDi
0.2615
0.2073
0.1084
0.0000
0.0000
0.6560
1.5011
0.7505
0.7385
LLM-as-a-judge baselines
GPT-4o-mini
0.5394
0.5041
0.5682
0.0563
0.0629
0.3218
0.9210
0.4605
0.4606
Table 2: VOICED standard, soft, and perspectivist metrics (affiliation-matched, binary). Bold = best overall; underline = best among LLM baselines.
Standard Metrics
Soft Prediction
Perspectivist
Model
Acc.
F1 mac
F1 wt
κ
MCC
JSD ↓
MD ↓
ER ↓
ECE ↓
DiADEM
0.6779
0.3802
0.6283
0.1726
0.1931
0.0229
0.8266
0.5895
0.0484
DisCo
0.6628
0.3737
0.6178
0.1486
0.1620
0.0266
0.8471
0.6198
0.0470
LeWiDi
0.6599
0.3887
0.6277
0.1821
0.1887
0.0233
0.8830
0.6258
0.0214
Table 3: DICES standard, soft, and perspectivist metrics, split by item (annotator-level, 3-class). Bold = best per column.
Standard Metrics
Soft Prediction
Perspectivist
Model
Acc.
F1 mac
F1 wt
κ
MCC
JSD ↓
MD ↓
ER ↓
ECE ↓
DiADEM
0.8000
0.5574
0.7438
0.1690
0.2411
0.0250
0.2070
0.1911
0.0144
DisCo
0.7835
0.5731
0.7435
0.1751
0.2032
0.0338
0.5910
0.2089
0.0260
LeWiDi
0.7792
0.4591
0.6957
0.0152
0.0339
0.0429
0.6271
0.2103
0.0479
Table 4: VOICED. standard, soft, and perspectivist metrics, split by item (binary). Bold = best per column.
Annotator-level
Item-level
Demographic
DICES
VOICED
DICES
VOICED
Age
0.1965
0.2336
0.1935
0.1964
Education
0.1940
0.2208
0.1962
0.2194
Gender
0.1948
0.1548
0.1944
0.1981
Locale
0.1982
–
0.1988
–
Political Affiliation
–
0.1530
–
0.1823
Table 5: Learned α weights across demographic groups for each dataset by split
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: DICES item split: confusion matrix and per-class F1/support.
Figure 3: DICES item split: disagreement calibration (variance and entropy).
Figure 4: DICES annotator split: confusion matrix and per-class F1/support.
Figure 5: DICES annotator split: disagreement calibration (variance and entropy).
Figure 6: VOICED item split: confusion matrix and per-class F1/support.
Figure 7: VOICED item split: disagreement calibration (variance and entropy).
Figure 8: VOICED annotator split: confusion matrix and per-class F1/support.
Figure 9: VOICED annotator split: disagreement calibration (variance and entropy).
Dataset
Model
Var Pearson
Var Spearman
Ent Pearson
Ent Spearman
DICES
DiADEM (by-annotator)
0.7544
0.7406
0.7292
0.7176
DICES
DisCo baseline (by-annotator)
0.0122
0.0228
0.0027
0.0147
DICES
LeWiDi (DisCo)
0.7048
0.6524
0.7074
0.6495
VOICES
DiADEM (by-annotator)
0.1878
0.1928
0.1803
0.1909
VOICES
DisCo baseline (by-annotator)
–
–
-0.0015
0.0089
VOICES
LeWiDi (DisCo)
0.0471
0.0425
0.0371
0.0420
Appendix
Table 6: Distribution-level disagreement correlation coefficients (higher is better).
Annotator disagreement is increasingly recognized as a meaningful property of subjective tasks, rather than merely noise to be removed. Predicting which items are likely to be contested could help allocate annotation effort and route difficult cases to human review. We propose predicting disagreement from interpretable semantic properties of the text. An LLM answers a set of binary questions covering harmful-content characteristics, disagreement-related cues such as borderline severity and missing context, and language-sensitive cues such as local slurs. A classifier predicts disagreement from these answers, and we analyze which questions contribute most to the predictions. Across four datasets in English, German, and Turkish and three open-weight LLMs, our structured question features outperform direct prompting by 10.5 ROC-AUC points and the strongest text-based baselines by 5.0 points. A compact subset of eight questions performs within 1.5 ROC-AUC points of the full twenty-question set. In leave-one-dataset-out evaluation, models retain roughly 90-95% of their in-domain ROC-AUC. Across models and datasets, questions about derogation, missing context, and decision boundaries serve as the strongest predictors, suggesting that what makes harmful content contested is partly observable in the text.
Irem Ecem Yelkanat, Faeze Ghorbanpour, Alexander Fraser
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning (MCML)
Hate speech detection is inherently subjective: people from different demographic groups perceive the same content very differently. Collecting enough annotations from multiple demographic groups is costly and difficult to scale. Persona-conditioned Large Language Models (models prompted to adopt a specific demographic identity) have been proposed as a way to simulate diverse perspectives at scale. But do they actually reflect how different groups disagree? We evaluate three aspects of human social judgement: (i) whether personas from different groups disagree in human-like ways (inter-group disagreement), (ii) whether they become more sensitive when content targets their own identity (in-group sensitivity), and (iii) whether they can accurately predict how another group would react (vicarious prediction). Our results show that no model consistently captures all three dimensions, and performance is highly model-dependent and does not emerge reliably from minimal identity prompts alone. However, vicarious prompting with Llama 3.1 yields the highest cross-group agreement in most demographic axes and provides the closest overall approximation to human disagreement patterns, indicating that this configuration may provide a more reliable setting for automatic annotation aligned with human judgements.
Paloma Piot, Javier Parapar
Information Retrieval Lab CITIC Research Centre Universidade da Coruña
Annotator disagreement is widespread in NLP, particularly for subjective and ambiguous tasks such as toxicity detection and stance analysis. While early approaches treated disagreement as noise to be removed, recent work increasingly models it as a meaningful signal reflecting variation in interpretation and perspective. This survey provides a unified view of disagreement-aware NLP methods. We first present a domain-agnostic taxonomy of the sources of disagreement spanning data, task, and annotator factors. We then synthesize modeling approaches using a common framework defined by prediction targets and pooling structure, highlighting a shift from consensus learning toward explicitly modeling disagreement, and toward capturing structured relationships among annotators. We review evaluation metrics for both predictive performance and annotator behavior, and noting that most fairness evaluations remain descriptive rather than normative. We conclude by identifying open challenges and future directions, including integrating multiple sources of variation, developing disagreement-aware interpretability frameworks, and grappling with the practical tradeoffs of perspectivist modeling.