Annotator Disagreement

Ask about this topic

Overview

Annotator disagreement research examines how judgments vary across people and items, and whether variation comes from ambiguity, task difficulty, annotator perspectives, or annotation conditions. Common approaches include agreement metrics, statistical and Bayesian models of annotators, demographic or perspective-aware models, and training on soft label distributions rather than majority votes. The work matters for subjective tasks such as emotion recognition, content moderation, and medical assessment, where a single gold label can hide uncertainty and distort model calibration or benchmark rankings. A central tension is deciding when to preserve disagreement as meaningful variation and when to resolve it through clearer definitions or adjudication, especially when annotation budgets are limited.

Latest papers

CardsList

91 papers13 papers in the last four weeks, up 63% on the four weeks before. 0.1% of all new papers.

Jun 11, 2022

A Decomposition-Based Approach for Evaluating and Analyzing Inter-Annotator Disagreement

We propose a novel method to conceptually decompose an existing annotation into separate levels, allowing the analysis of inter-annotators disagreement in each level separately. We suggest two distinct strategies in order to actualize this approach: a theoretically-driven one, in which the researcher defines a decomposition based on prior knowledge of the annotation task, and an exploration-based one, in which many possible decompositions are inductively computed and presented to the researcher for interpretation and evaluation. Utilizing a recently constructed dataset for narrative analysis as our use-case, we apply each of the two strategies to demonstrate the potential of our approach in testing hypotheses regarding the sources of annotation disagreements, as well as revealing latent structures and relations within the annotation task. We conclude by suggesting how to extend and generalize our approach, as well as use it for other purposes.

Effi Levi, Ari Rappoport, Shaul R. Shenhav

Department of Computer Science, The Hebrew University of Jerusalem · Department of Political Science, The Hebrew University of Jerusalem

Annotator Disagreement
AskPDF