Calculating inter-annotator agreement is an established practice in NLP, and disagreement-aware work increasingly retains the distribution of human judgments rather than reducing annotations to a single gold label. Keeping the distribution, however, does not explain why annotators disagree. This paper examines the types of disagreement that occur in emotion annotation, starting from multilabel annotations of 1,677 sentence-level units from Ernest Hemingway's The Old Man and the Sea and testing the patterns they reveal against individual-level annotations from BRIGHTER and GoEmotions. Across the three datasets, we find three main recurring disagreement structures. Emotion categories overlap systematically; annotators differ both in how broadly they apply labels and in how they treat particular category boundaries, while sparse participation and explicit abstention can look like disagreement. Where intensity is rated, disagreement over whether an emotion is present can be separated from disagreement over its intensity once recognized. Similar aggregate disagreement can therefore arise from qualitatively different annotation processes. Hard aggregation removes most of this structure, while annotator vote-share representations keep marginal label frequencies but not the co-occurrence, annotator-specific, or procedural patterns that produced them. We argue that disagreement should be understood in terms of its underlying patterns before it is summarized, aggregated, or used as a modeling target, and propose a vocabulary for disagreement structure.
Figures & tables
Dataset
Size
Emotions
Annotators / raters
Annotation format
Role in this study
TOMATS ( Öhman and Koufakou, 2026b )
1,677 text units
8 core emotions
3 annotators
Up to three labels per item; contextual literary annotation; overall intensity field; explicit no-emotion option
Motivating case for semantic overlap, boundary structure, and procedural sparsity
BRIGHTER ( Muhammad et al., 2025 )
5,655 text units in English
6 dimensions
5–30 ratings per item
Individual 0–3 ordinal rating on every emotion dimension
Tests structured co-occurrence and separates presence from conditional-intensity disagreement
GoEmotions ( Demszky et al., 2020 )
211,225 individual annotation rows
27 emotions + neutral
82 raters
Fine-grained multilabel annotation with persistent rater IDs
Tests general response style, pair-specific boundaries, and procedural/abstention behavior
Table 1: Overview of the three emotion-annotation resources used in this study
Annotator
Items with primary emotion
Secondary labels
Tertiary labels
A1
1,593
1,258
443
A2
184
13
3
A3
1,369
1,200
566
Table 2: Annotation coverage by annotator. Reproduced from ( Öhman and Koufakou, 2026b )
Signature
Observable pattern
Plausible interpretation
What aggregation obscures
Semantic overlap
Recurrent cross-annotator emotion pairings
Multiple or blended readings; overlap in the taxonomy
Which alternatives regularly compete or co-occur
Boundary structure
Different within-annotator co-occurrence patterns
Different implicit category boundaries or labeling strategies
Which annotators treat which categories as compatible
Procedural sparsity
High rates of blank or thin annotation
Nonresponse, limited protocol engagement, or a restrictive labeling style
Whether an absence represents disagreement or lack of engagement
Table 3: Three disagreement signatures identified in the TOMATS annotations
Figure 1: Cross-annotator emotion association between A1 and A3, expressed as lift. Each cell shows P(A3includescolumn∣A1includesrow)/P(A3includescolumn) . Values above 1 indicate that A3 selected the column emotion more often than expected when A1 selected the row emotion; values below 1 indicate less frequent co-selection than expected. Cell labels report raw lift values, while color intensity is based on log 2 (lift) and centered at 1.
Figure 2: Structured co-occurrence among BRIGHTER English emotion categories. Each cell reports co-occurrence lift within individual judgments. Values above 1 indicate more joint presence than expected from the marginal frequencies of the two emotions, values below 1 indicate less joint presence than expected, and diagonal cells are omitted.
Figure 3: Within-annotator emotion co-occurrence lift for Annotators 1 and 3. Values above 1 indicate that a pair of emotions was selected together more often than expected from the annotator’s marginal use of the two categories; values below 1 indicate less co-occurrence than expected. Diagonal cells are omitted. Cell labels report raw lift values, and the color scale is centered at lift = 1.
Figure 4: GoEmotions: item-adjusted label cardinality by rater, ordered from most restrictive to most expansive.
Figure 5: GoEmotions: pair-specific semantic-boundary residuals for six neighboring emotion pairs, after controlling for each rater’s overall annotation breadth.
Pair
Condition
Macro-F1
Micro-F1
A1–A2
Full
0.0788
0.0882
A1–A2
Filtered
0.3949
0.4857
A1–A3
Full
0.3794
0.3952
A1–A3
Filtered
0.4899
0.5187
A2–A3
Full
0.0829
0.0899
A2–A3
Filtered
0.3797
0.4664
Table 4: Pairwise multilabel agreement on the full TOMATS corpus and the A2-filtered subset
Figure 6: BRIGHTER English disagreement by emotion category. Panel (a) shows mean disagreement over emotion presence, measured as binary entropy over ratings of 0 versus greater than 0. Panel (b) shows mean conditional-intensity disagreement among annotators assigning positive ratings, measured as normalized mean pairwise absolute distance and restricted to item–emotion combinations with at least three positive ratings. The panels use separate y-axes because the two measures are not commensurate in absolute magnitude.
Disagreement dimension
TOMATS
BRIGHTER
GoEmotions
Structured category compatibility
Recurrent cross-annotator alternatives (anger–disgust; fear–anticipation in one direction)
Structured within-judgment lift; strongest for happiness–social warmth
Not tested separately
General response style
A3 averages 2.21 core labels per labelled item versus 2.06 for A1; not cleanly separable from boundary style
Not identifiable without persistent rater IDs
Item-adjusted cardinality ranges from −0.235 to +1.936 across 76 raters
Pair-specific boundary behavior
Different within-annotator association patterns (e.g. anger–joy); not cleanly separable from general breadth
Not identifiable without persistent rater IDs
Merging/separating residuals remain after controlling for general breadth
Emotion annotation is inherently subjective, yet most NLP pipelines still assume "gold" labels, typically produced by majority voting, and treat annotator variation as noise. In this paper, we present a multilabel emotion annotation case study and use it to examine how annotator behavior and aggregation choices affect both agreement estimates and downstream emotion classifiers. Rather than collapsing disagreement into a single label, we represent targets as soft vote-share labels (including an intensity-weighted variant) and evaluate models using both thresholded metrics (macro-/micro-F1) and probabilistic alignment (Bernoulli cross-entropy SoftBCE), alongside data-derived disagreement diagnostics. Across annotation regimes, we show that disagreement is structured and leaves measurable traces in model behavior: hard labels may maximize F1 metrics, while soft supervision yields predictions that better reflect empirical annotator variance and uncertainty. Our results provide practical guidance for designing, aggregating, and evaluating multilabel emotion datasets when multiple interpretations are plausible.
Emily Öhman, Anna Koufakou
Waseda University, Tokyo, Japan · Florida Gulf Coast University, Fort Myers FL, USA
When humans label subjective content, they disagree, and that disagreement is not noise. It reflects genuine differences in perspective shaped by annotators' social identities and lived experiences. Yet standard practice still flattens these judgments into a single majority label, and recent LLM-based approaches fare no better: we show that prompted large language models, even with chain-of-thought reasoning, fail to recover the structure of human disagreement. We introduce DiADEM, a neural architecture that learns "how much each demographic axis matters" for predicting who will disagree and on what. DiADEM encodes annotators through per-demographic projections governed by a learned importance vector α, fuses annotator and item representations via complementary concatenation and Hadamard interactions, and is trained with a novel item-level disagreement loss that directly penalizes mispredicted annotation variance. On the DICES conversational-safety and VOICED political-offense benchmarks, DiADEM substantially outperforms both the LLM-as-a-judge and neural model baselines across standard and perspectivist metrics, achieving strong disagreement tracking (r=0.75 on DICES). The learned α weights reveal that race and age consistently emerge as the most influential demographic factors driving annotator disagreement across both datasets. Our results demonstrate that explicitly modeling who annotators are not just what they label is essential for NLP systems that aim to faithfully represent human interpretive diversity.
Samay U. Shetty, Tharindu Cyril Weerasooriya, Deepak Pandita +1
We propose a novel method to conceptually decompose an existing annotation into separate levels, allowing the analysis of inter-annotators disagreement in each level separately. We suggest two distinct strategies in order to actualize this approach: a theoretically-driven one, in which the researcher defines a decomposition based on prior knowledge of the annotation task, and an exploration-based one, in which many possible decompositions are inductively computed and presented to the researcher for interpretation and evaluation. Utilizing a recently constructed dataset for narrative analysis as our use-case, we apply each of the two strategies to demonstrate the potential of our approach in testing hypotheses regarding the sources of annotation disagreements, as well as revealing latent structures and relations within the annotation task. We conclude by suggesting how to extend and generalize our approach, as well as use it for other purposes.
Effi Levi, Ari Rappoport, Shaul R. Shenhav
Department of Computer Science, The Hebrew University of Jerusalem · Department of Political Science, The Hebrew University of Jerusalem