Machine learning models are often trained on labeled data produced by multiple human annotators. When those annotators disagree, their labels are commonly collapsed into a single majority vote, which discards information about the underlying disagreement. We examine the cost of this practice for cyberbullying severity, a subjective judgment on which annotators disagree heavily. In the corpus we study, all 5 annotators assigned the same severity label to only 4.1% of comments. Turning 5 judgments into one label hides this disagreement and obscures how much the annotators differed in their assessments. In this paper, we ask whether a model can preserve this information by predicting the distribution of annotator judgments instead of a single majority label. We find 3 main results. First, accuracy against the majority label cannot tell competent models apart: 2 models with similar accuracy can differ greatly in how well they match the distribution of annotator judgments. Second, predicting the full distribution requires little extra effort. Calibrating an existing model, which adjusts its output probabilities after training, captures most of the improvement without retraining, across 3 datasets covering severity, abuse, and irony, while training on label distributions adds a small further gain. Third, we examine a dataset that includes annotator demographics, where a single shared model has one advantage and one limit. The advantage is that a model predicting the distribution of judgments matches each demographic group's judgments significantly better than a model trained on majority labels, so both groups gain. The limit is that it does not narrow the gap in performance between the groups, because one shared model cannot encode which group is right on a given comment. Preserving annotator disagreement is therefore a low-cost step toward fairer moderation, though not a complete one.
Figures & tables
Figure 1. Why a majority vote loses information. (a) A comment labeled mild by 3 annotators, moderate by 1 and severe by 1 becomes the single label mild, and the other 2 judgments are lost. (b) The share of the 9,553 comments in each agreement pattern. Only 4.1% are unanimous, so 95.9% have at least one annotator who disagrees with the rest.
Figure 2. Study design. Annotated comments are summarized as a majority label and as a label distribution (the colors of the annotator icons and bars stand for the severity levels mild, moderate, and severe). Three experiments follow, one for each research question of the introduction. Experiments 1 and 2 use the study set, and Experiment 3 uses EPIC, a second dataset that records annotator gender (our own dataset does not record annotator identities). Experiment 2 repeats its comparison on further datasets (not shown).
Label pattern
Mild
Moderate
Severe
Total (%)
5–0 unanimous
121
192
79
392 (4.1)
4–1 strong
1,009
914
142
2,065 (21.6)
3–2 weak
1,407
1,237
90
2,734 (28.6)
3–1–1 plurality
1,304
993
117
2,414 (25.3)
Split ≤2
1,296
593
59
1,948 (20.4)
All CB comments
5,137 (53.8%)
3,929 (41.1%)
487 (5.1%)
9,553 (100)
Table 1. The 9,553 study-set comments in Insta-CTSR by majority severity level and 5-annotator label pattern, with the percentage of all 9,553 comments in parentheses. A label pattern gives the number of annotators who chose each label, from the most to the least common. For example, 3–2 means that 3 annotators chose one label and 2 chose another, and 3–1–1 means that 3 annotators chose one label and the other 2 each chose a different label. Split ≤2 means that no label was selected by more than 2 annotators.
System
What it does
Learns from
Flat 3-class
Softmax head over mild, moderate, severe; the reference model
Majority label
CORAL head, CORN head
Output layer that respects the order mild < moderate < severe
Majority label
EMD loss
Penalizes the distance between predicted and target cumulative distributions
Majority label
Pairwise CORAL
Learns severity from pairwise comparisons
Majority label
Hierarchical
2 binary decisions: mild vs. more severe, then moderate vs. severe
Majority label
Level-1 only (mild vs rest)
Only the first of those 2 decisions
Majority label
Table 2. The systems we compare and what each learns from. The first 10 rows, with CORAL and CORN counted separately, form the 11 systems of the accuracy comparison. The last 3 rows are added for the fidelity comparison. Only the 2 distribution-trained systems see the annotators’ label distributions.
Model
Macro-F1
Level-1 only (mild vs rest)
0.452 ±0.012
Pairwise CORAL
0.587 ±0.020
CORAL head
0.591 ±0.032
No context encoder
0.652 ±0.016
EMD loss
0.665 ±0.031
CORN head
0.674 ±0.026
Table 3. Macro-F1 (mean ± SD over 10 folds) for each system. Every system except the distribution-trained one learns from the majority label. The 5 strongest systems fall between 0.68 and 0.70, so macro-F1 barely separates them. For reference, a single annotator compared with the majority label of the other 4 annotators reaches 0.53, and the 2 zero-shot language models (single runs) are shown at the bottom.
Step (applied to the matched twin)
TV
Brier
Log
% gap
Raw (trained on majority labels)
0.318
0.294
1.244
0
+ temperature, no label distributions
0.263
0.182
0.836
50
+ majority map, no label distributions
0.237
0.164
0.798
58
+ label-distribution map (12 parameters)
0.233
0.144
0.764
67
+ distribution training
0.228
0.137
0.755
70
Sampling-noise floor
–
0.069
–
100
Table 4. Where the fit to the label distributions comes from. Each step adds the information named to the step above it, starting from the matched twin, the hierarchical model trained on the majority label, and moving toward the sampling-noise floor. TV is total variation, Brier is the Brier score, and Log is the log score. Each measures how far the predicted distribution is from the human label distribution, so lower is better. The last column is the percentage of the closable Brier gap ( 0.294→0.069 ) that each step reaches. Temperature scaling closes half the gap using no label distributions. A small map fitted to the label distributions closes most of the rest without retraining. Training on the label distributions adds a little more.
Model
TV
Brier
Log score
F1
Hierarchical (majority label, matched twin)
0.318 ±.009
0.294 ±.019
1.244 ±.045
0.685 ±.019
+ label calibration
0.233 ±.016
0.144 ±.019
0.764 ±.024
0.682 ±.026
Label smoothing, ϵ=0.1 (softened majority label)
0.319 ±.008
0.271 ±.016
1.116 ±.023
0.690 ±.025
+ label calibration
0.231 ±.017
0.143 ±.018
0.761 ±.025
0.683 ±.025
SORD (softened majority label)
0.280 ±.016
0.167 ±.016
0.847 ±.012
0.695 ±.027
+ label calibration
0.231 ±.018
0.143 ±.019
0.760 ±.025
0.693 ±.024
Table 5. Fidelity to the human label distribution (10-fold means ± SD, lower is better except F1, with the best mean in bold). F1 is macro-F1 of the most-probable label, reported for the calibrated rows as well because calibration can change which label is most probable, though here it rarely does. Label calibration is a Dirichlet map fitted on training-fold label distributions and applied cross-fold.
Corpus
Task
+ temperature
+ label-distribution map
+ distribution training
Insta-CTSR (ours)
Severity, 3-class
50
67
70
EPIC
Irony, binary
63
63
67
ConvAbuse
Abuse, 5-class
27
22
32
Table 6. The decomposition across 3 disagreement corpora, as the percentage of the closable Brier gap reached after temperature scaling with no label distributions, a map fitted to the label distributions, and training on the label distributions. The replication values are means over 5 seeds, with a per-seed SD of at most 4 points. Temperature scaling recovers most of the gap everywhere. The further gain from the label-distribution map appears only on Insta-CTSR. Training adds a small consistent step on all 3.
Model (EPIC)
Men’s labels
Women’s labels
Majority-trained
0.283
0.318
+ label calibration
0.212
0.249
Label-preserving
0.205
0.244
Table 7. How well each model reproduces each group’s judgments on EPIC, scored as Brier against the label distribution of men and of women separately (lower is better, means over 10 folds and 5 seeds). Preserving the label distribution improves both groups significantly over the calibrated majority baseline, but the gap between them does not narrow (0.037 against 0.038, difference not significant).
Approach
Pros
Cons
One shared model (tested here)
Improves the fit to both groups (0.007 Brier for men, 0.005 for women). Needs only label counts, with no annotator identities. Adds only a calibration step. Favors neither group.
Cannot encode which group is right on a given comment. Leaves the gap between the groups unchanged (0.037 against 0.038). Represents the pooled split, so differences between groups stay hidden.
Per-annotator or jury models (not tested here)
Can represent each annotator’s or group’s judgment and let the designer choose whose judgment to follow ( Davani et al., 2022 ; Gordon et al., 2022 ) .
Need annotator identities, which this corpus does not record. Leave the choice of whose judgment prevails to the designer.
Table 8. One shared model compared with per-annotator and jury models. The first row summarizes our EPIC results. The second row describes approaches from prior work that we did not test.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
Macro-F1
QWK
MAE
Off-by-1
DeBERTa-v3-base
0.652
0.730
0.240
0.970
+ context ( K=5 )
0.650
0.734
0.244
0.971
HateBERT
0.648
0.726
0.245
0.972
Twitter-RoBERTa
0.645
0.725
0.242
0.972
RoBERTa-base
0.641
0.714
0.253
0.968
ToxicBERT
0.634
0.711
0.251
0.967
Appendix
Table 9. Backbone comparison under one shared training configuration (fine-tuned flat classifier, 4-class scope, mean over the session-level folds). DeBERTa-v3-base is best on macro-F1, MAE and cross-fold variance and ties for best QWK. The prior-art Pairwise+CORAL ordinal competitor is shown for reference. The 5-comment context window changes QWK by +0.005 at higher variance. The 0.617–0.652 macro-F1 range previews the accuracy ceiling of Section 4 , “ ”.
Figure 3. (a) Reliability diagram. Gray circles ( ∙ ) show the flat 3-class baseline, blue squares ( ■ ) its temperature-scaled version and orange diamonds ( ⧫ ) the distribution-trained model. The dotted line is perfect calibration. The raw baseline sits far below the diagonal, so its confidence overstates its accuracy, while the scaled baseline and the distribution-trained model both track the diagonal. (b) Total variation to the human label distribution per fold under the same label-fitted calibration for both systems, with lines connecting the same fold. The distribution-trained model ( ∙ ) is below its hard-label twin ( ∙ ) in 9 of 10 folds. The full comparison is in Table 5 and Figure 4 .
Figure 4. Paired per-fold differences, the distribution-trained model with label calibration minus each label-free baseline with the same calibration, on (a) Brier score, (b) log score and (c) total variation. Diamonds ( ⧫ ) are mean differences over 10 folds with bootstrap 95% intervals. The dashed line marks zero and the annotation gives the number of folds in which the distribution-trained model is lower. Axis magnitudes are unsigned, with orange tick labels marking the side where the distribution-trained model is better and blue tick labels the side where the baseline is better. Every interval lies on the distribution-trained model’s side.
Figure 5. (a) Model entropy against human label entropy for all 9,553 comments, where darker cells hold more comments. The 2 signals are nearly uncorrelated ( ρ=0.12 ). (b) Error rate by severity-label pattern. Gray bars ( ) are clear patterns and orange bars ( ) are contested patterns, those where no single severity level holds 4 of the 5 annotator labels, where the error rate is 1.9 times the clear-pattern rate.
Figure 6. Predicted distributions against the 5 human labels for 3 comments: (a) unanimously severe, “@user you are really shtty and go die”; (b) split 2-3-0, “tell that fool @user to lay off the haterade. Get a life…”; (c) split 1-3-1, “A little fggish much @user”. Dark bars ( ) show the human label distribution, orange bars ( ) the prediction of the distribution-trained model and blue bars ( ) the temperature-scaled flat baseline. On the unanimous comment all 3 agree. On the split labels the scaled baseline concentrates its mass on a single class while the distribution-trained model follows the shape of the human distribution.
Hate speech annotation pipelines routinely collapse annotator disagreement into majority vote labels before training. We show that this aggregation is not neutral: 42.6% of all annotator disagreement in HateXplain concentrates specifically at the hate/offensive boundary, a pattern consistent with annotators applying different thresholds for where hate begins (chi-squared = 135.199, df = 2, p < 0.0001). Both a hard-label BERT model (Model A) and a soft-label model (Model B) drop 22 percentage points in accuracy from agreed posts (~80%) to disagreement posts (~58%), confirmed at p < 0.0001. A per-annotator multi-head model (Model C) widens this gap further to 28 points while collapsing offensive disagreement accuracy to 0.245. Critically, Model A expresses significantly higher confidence on boundary case errors than Model C (0.710 vs. 0.495, p < 0.0001), meaning standard evaluation metrics will not detect the failure. Three downstream interventions of increasing sophistication all fail to recover boundary accuracy. We argue the problem is structural. Majority vote presents a contested judgment as ground truth, and models inherit that false certainty. The intervention must be upstream in annotation design.
When humans label subjective content, they disagree, and that disagreement is not noise. It reflects genuine differences in perspective shaped by annotators' social identities and lived experiences. Yet standard practice still flattens these judgments into a single majority label, and recent LLM-based approaches fare no better: we show that prompted large language models, even with chain-of-thought reasoning, fail to recover the structure of human disagreement. We introduce DiADEM, a neural architecture that learns "how much each demographic axis matters" for predicting who will disagree and on what. DiADEM encodes annotators through per-demographic projections governed by a learned importance vector α, fuses annotator and item representations via complementary concatenation and Hadamard interactions, and is trained with a novel item-level disagreement loss that directly penalizes mispredicted annotation variance. On the DICES conversational-safety and VOICED political-offense benchmarks, DiADEM substantially outperforms both the LLM-as-a-judge and neural model baselines across standard and perspectivist metrics, achieving strong disagreement tracking (r=0.75 on DICES). The learned α weights reveal that race and age consistently emerge as the most influential demographic factors driving annotator disagreement across both datasets. Our results demonstrate that explicitly modeling who annotators are not just what they label is essential for NLP systems that aim to faithfully represent human interpretive diversity.
Samay U. Shetty, Tharindu Cyril Weerasooriya, Deepak Pandita +1
When annotators disagree on a label, the disagreement itself carries signal - and the number of annotators needed to capture it depends on the evaluation metric. We fine-tune NLI models on label distributions subsampled from ChaosNLI, a dataset providing 100 independent annotator judgments per item, and identify metric-dependent saturation. In our 3-class NLI setting, entropy correlation - whether the model identifies which items elicit disagreement - requires N ~ 20-50 annotators to converge, while distributional match (KL divergence) saturates by N ~ 10 (87-95% of improvement across five model seeds); the specific thresholds are estimates for this setting, and it is the phenomenon, not the constants, that we expect to generalize. This finding rests on a prior observation: soft labels carry item-specific signal that label smoothing cannot replicate. Across five smoothing intensities, entropy correlation clusters at r ~ 0.45-0.49, while soft labels reach r = 0.643 (p < 0.001); per-item analysis traces this gap to smoothing's inability to distinguish ambiguous items from clear ones. The soft label advantage replicates across two architectures (DeBERTa, RoBERTa) and a non-NLI-pretrained baseline; differential saturation additionally appears in an exploratory cross-domain evaluation on content safety. These results suggest that annotation budgets should be informed by the target evaluation metric rather than set uniformly.