Annotator Disagreement

Annotator disagreement research examines how judgments vary across people and items, and whether variation comes from ambiguity, task difficulty, annotator perspectives, or annotation conditions. Common approaches include agreement metrics, statistical and Bayesian models of annotators, demographic or perspective-aware models, and training on soft label distributions rather than majority votes. The work matters for subjective tasks such as emotion recognition, content moderation, and medical assessment, where a single gold label can hide uncertainty and distort model calibration or benchmark rankings. A central tension is deciding when to preserve disagreement as meaningful variation and when to resolve it through clearer definitions or adjudication, especially when annotation budgets are limited.