Overview

Annotator disagreement research examines how judgments vary across people and items, and whether variation comes from ambiguity, task difficulty, annotator perspectives, or annotation conditions. Common approaches include agreement metrics, statistical and Bayesian models of annotators, demographic or perspective-aware models, and training on soft label distributions rather than majority votes. The work matters for subjective tasks such as emotion recognition, content moderation, and medical assessment, where a single gold label can hide uncertainty and distort model calibration or benchmark rankings. A central tension is deciding when to preserve disagreement as meaningful variation and when to resolve it through clearer definitions or adjudication, especially when annotation budgets are limited.

Latest papers

CardsList

102 papers13 papers in the last four weeks, up 30% on the four weeks before. 0.1% of all new papers.

  1. What Majority Labels Hide: Annotator Disagreement and Model Errors in Multi-Dialect Arabic Sexism Detection

    Nov 18, 2026Alaa Bazaid, Arkaitz ZubiagaAnnotator Disagreementarabic language model evaluation

  2. When accuracy hides risk: risk-controlled confirmation of the critical view of safety under expert disagreement

    Nov 18, 2026Bowen Xiang, Xiao Yu, Mingze XuSurgical Video UnderstandingAnnotator Disagreement

  3. A Scalable Multi Task Learning Framework for Modeling Demographic Disagreements in Annotation Tasks

    Nov 16, 2026Soumyajit Gupta, Maria De-Arteaga, Matthew LeaseAnnotator DisagreementMulti-Task Learning

  4. Calibrate, Don't Retrain: Decomposing the Value of Annotator Disagreement in Cyberbullying Severity

    Nov 11, 2026Arslan Bisharat, Katelyn Skees, Manuel Sandoval Madrigal +5Annotator DisagreementClassification

  5. Measuring Definition Sensitivity in LLM Annotation of Social Science Concepts

    Nov 9, 2026Bogdan Mamaev, Qizhou Wang, Christopher Leckie +1LLM-Assisted AnnotationPrompt Sensitivity

  6. When Disagreement Labels Do Not Identify F1 Rankings

    Nov 1, 2026Roberto Fernández-Barrios, Iker Pastor-López, Asier González-Santocildes +1Pairwise RankingPartial Identification

  7. The Structure of Disagreement in Emotion Annotation

    Nov 1, 2026Emily Öhman, Anna Koufakou, Eugenia DiegoliAnnotator DisagreementInter-Rater Reliability

  8. Single-Annotation, Multi-Hypothesis Segmentation with Semantic-Spatial Residual Banks

    Oct 29, 2026Armin Dadras, Robert Sablatnig, Franziska Proksa +1Image SegmentationSemantic Segmentation

  9. The Anatomy of Socio-Demographics in Pluralistic Offensiveness Modeling: When and Where It Helps, and the Role of Reasoning

    Oct 27, 2026Yifan Wang, Leixin Zhang, Jinyi Mu +4User ModelingAnnotator Disagreement

  10. A Contextual Measure of Ambiguity in Prescriptive Text

    Oct 26, 2026Sally Dong, Luke Thorburn, Andrew Konya +1Annotator DisagreementPolicy Evaluation

  11. Knowing When to Escalate: Disagreement Patterns for Uncertainty-Aware Content Moderation

    Oct 26, 2026Michał Wawer, Jarosław A. ChudziakContent ModerationReasoning Trace Analysis

  12. Interpretive Variation and Discourse Structure in Detecting Enthymemes in Controversial Political Discourse

    Oct 22, 2026Martial Pastor, Nelleke OostdijkAnnotator DisagreementArgument Mining

  13. Label Authority in Speech Emotion Recognition: Intended and Perceived Ground Truth in Emozionalmente

    Oct 19, 2026Marco Giordano, Stefano Giacomelli, Claudia Rinaldi +1Speech Emotion RecognitionAnnotator Disagreement

  14. Who Do Perspectivist Models Predict Well? Annotator Agreement, Sparsity, and Performance

    Oct 15, 2026London Lowmanstone, Vaagishwaran Sivakumar, Anwesha Samaddar +3Annotator Disagreement

  15. What Does Disagreement Mean? A Semantic Identifiability Frontier for Pairwise Preference Learning

    Oct 14, 2026Manoj Saravanan, Rohit Kumar Salla, Shrikar Reddy KotaPairwise Preference LearningAnnotator Disagreement

  16. Decomposing and Predicting Annotation Disagreement in Harmful Content Across Languages

    Oct 13, 2026Irem Ecem Yelkanat, Faeze Ghorbanpour, Alexander FraserAnnotator DisagreementContent Moderation

  17. Activating the Annotator: Operationalising Lived Experience in Data Annotation

    Oct 8, 2026Sheza Munir, Ramaravind Kommiya Mothilal, Syed Ishtiaque AhmedHuman-in-the-Loop AnnotationAnnotator Disagreement

  18. Language-model ratings of depression reflect the rater more than the patient

    Oct 6, 2026Baihan LinInter-Rater ReliabilityAnnotator Disagreement

  19. Boundaries Agree, Labels Do Not: Intra-Annotator Dynamics as a Kind of Training Data

    Oct 3, 2026Marharyta ShvetsAnnotator DisagreementLLM Reliability

  20. QuanReview: Offline, Auditable Reconciliation of Human and LLM Span Annotations

    Sep 28, 2026Matteo Musacchio, Juan Cruz Giner Pulero, Isabel Castañeda +4Annotator DisagreementLLM-Assisted Annotation

  21. How Many Humans Are 32 LLM Judges Worth?

    Sep 18, 2026Chao Li, Yingying Yu, Yunfeng LiAnnotator DisagreementLLM-as-a-Judge

  22. Predicting Human Disagreement for Calibrated Dynamic Facial Expression Recognition

    Sep 15, 2026Yiming Wang, Frederick W. B. Li, Jingyun WangAnnotator DisagreementUncertainty Quantification

  23. Uncertainty-Aware Sea-Ice Type Mapping with Multiple Ice Charts

    Sep 8, 2026Samira Alkaee Taleghan, Younghyun Koo, Andrew P. Barrett +1Remote Sensing Image UnderstandingAnnotator Disagreement