cs.CLNov 1, 2026

The Structure of Disagreement in Emotion Annotation

Authors: Emily Öhman, Anna Koufakou, Eugenia Diegoli

Organizations: Waseda University, Japan · Florida Gulf Coast University, USA · Ca’ Foscari University of Venice, Italy

Abstract

Calculating inter-annotator agreement is an established practice in NLP, and disagreement-aware work increasingly retains the distribution of human judgments rather than reducing annotations to a single gold label. Keeping the distribution, however, does not explain why annotators disagree. This paper examines the types of disagreement that occur in emotion annotation, starting from multilabel annotations of 1,677 sentence-level units from Ernest Hemingway's The Old Man and the Sea and testing the patterns they reveal against individual-level annotations from BRIGHTER and GoEmotions. Across the three datasets, we find three main recurring disagreement structures. Emotion categories overlap systematically; annotators differ both in how broadly they apply labels and in how they treat particular category boundaries, while sparse participation and explicit abstention can look like disagreement. Where intensity is rated, disagreement over whether an emotion is present can be separated from disagreement over its intensity once recognized. Similar aggregate disagreement can therefore arise from qualitatively different annotation processes. Hard aggregation removes most of this structure, while annotator vote-share representations keep marginal label frequencies but not the co-occurrence, annotator-specific, or procedural patterns that produced them. We argue that disagreement should be understood in terms of its underlying patterns before it is summarized, aggregated, or used as a modeling target, and propose a vocabulary for disagreement structure.

Figures & tables

Explore similar work

CardsList
  1. Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

    Jun 19, 2026Emily Öhman, Anna KoufakouSoft-Label LearningAnnotator Disagreement

  2. Learning Who Disagrees: Demographic Importance Weighting for Modeling Annotator Distributions with DiADEM

    Apr 9, 2026Samay U. Shetty, Tharindu Cyril Weerasooriya, Deepak Pandita +1Annotator DisagreementLabel Distribution Learning

  3. A Decomposition-Based Approach for Evaluating and Analyzing Inter-Annotator Disagreement

    Jun 11, 2022Effi Levi, Ari Rappoport, Shaul R. ShenhavAnnotator Disagreement