Annotator Disagreement

Ask about this topic

Overview

Annotator disagreement research examines how judgments vary across people and items, and whether variation comes from ambiguity, task difficulty, annotator perspectives, or annotation conditions. Common approaches include agreement metrics, statistical and Bayesian models of annotators, demographic or perspective-aware models, and training on soft label distributions rather than majority votes. The work matters for subjective tasks such as emotion recognition, content moderation, and medical assessment, where a single gold label can hide uncertainty and distort model calibration or benchmark rankings. A central tension is deciding when to preserve disagreement as meaningful variation and when to resolve it through clearer definitions or adjudication, especially when annotation budgets are limited.

Latest papers

CardsList

91 papers13 papers in the last four weeks, up 63% on the four weeks before. 0.1% of all new papers.

  1. STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems

    May 4, 2026Akash Bonagiri, Gerard Janno Anderias, Saee Patil +6Annotator DisagreementAI Agent Evaluation

    Department of Computer Science, University of California, Davis, Davis, CA, USA.

  2. Quantifying and Predicting Disagreement in Graded Human Ratings

    May 1, 2026Leixin Zhang, Çağrı ÇöltekinAnnotator Disagreement

    University of Tübingen, Germany

  3. Are You the A-hole? A Fair, Multi-Perspective Ethical Reasoning Framework

    Apr 30, 2026Sheza Munir, Ahanaf Rodoshi, Sumin Lee +3Annotator DisagreementNeuro-Symbolic Reasoning

    Computer Science, University of Toronto

  4. LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps

    Apr 30, 2026Keito Inoshita, Xiaokang Zhou, Akira Kawai +1LLM EvaluationAnnotator Disagreement

    Faculty of Business and Commerce, Kansai University, Osaka, Japan · Faculty of Business Data Science, Kansai University, Osaka, Japan · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan +2

  5. Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement

    Apr 24, 2026Wataru Hirota, Tomoki Taniguchi, Tomoko Ohkuma +6Human Preference EvaluationAnnotator Disagreement

    1Stockmark Inc · 2Asahi Kasei Corporation · National Institute of Informatics +1

  6. Fine-Grained Perspectives: Modeling Explanations with Annotator-Specific Rationales

    Apr 23, 2026Olufunke O. Sarumi, Charles Welch, Daniel BraunAnnotator DisagreementNatural Language Inference

    Marburg University, Marburg, Germany · McMaster University, Hamilton, Ontario, Canada

  7. Structured Disagreement in Health-Literacy Annotation: Epistemic Stability, Conceptual Difficulty, and Agreement-Stratified Inference

    Apr 21, 2026Olga Kellert, Sriya Kondury, Candice Koo +2Inter-Rater ReliabilityAnnotator Disagreement

    Arizona State University

  8. Modeling Human Perspectives with Socio-Demographic Representations

    Apr 20, 2026Leixin Zhang, Cagri ColtekinContrastive LearningAnnotator Disagreement

    University of Tübingen

  9. Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains

    Apr 19, 2026Finn Schmidt, Jan Philip Wahle, Terry Ruas +1Annotator DisagreementAutomated Evaluation

    University of Göttingen, Germany

  10. Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks

    Apr 18, 2026Nisrine Rair, Alban Goupil, Valeriu Vrabie +1Annotator Disagreement

    CReSTIC, Université de Reims Champagne-Ardenne, Reims, France · Chochoy Conseil, Reims, France

  11. IYKYK (But AI Doesn't): Automated Content Moderation Does Not Capture Communities' Heterogeneous Attitudes Towards Reclaimed Language

    Apr 17, 2026Christina Chance, Rebecca Pattichis, Arjun Subramonian +4Social Media AnalysisAnnotator Disagreement

    University of California Los Angeles, USA · Independent Researcher, USA

  12. Learning Who Disagrees: Demographic Importance Weighting for Modeling Annotator Distributions with DiADEM

    Apr 9, 2026Samay U. Shetty, Tharindu Cyril Weerasooriya, Deepak Pandita +1Annotator DisagreementLabel Distribution Learning

    Rochester Institute of Technology

  13. From Ground Truth to Measurement: A Statistical Framework for Human Labeling

    Apr 8, 2026Robert Chew, Stephanie Eckman, Christoph Kern +1Annotator DisagreementHuman-in-the-Loop Annotation

    Department of AI, Data, and Product Engineering, RTI International · Social Data Science Center, University of Maryland · Department of Statistics, LMU Munich +1

  14. Beyond the Mean: Modelling Annotation Distributions in Continuous Affect Prediction

    Apr 8, 2026Kosmas Pinitas, Ilias MaglogiannisAffective ComputingAnnotator Disagreement

    Department of Digital Systems, University of Piraeus

  15. When AI Agents Disagree Like Humans: Reasoning Trace Analysis for Human-AI Collaborative Moderation

    Apr 4, 2026Michał Wawer, Jarosław A. ChudziakHuman-AI Decision MakingAnnotator Disagreement

    Faculty of Electronics and Information Technology Warsaw University of Technology, Warsaw, Poland

  16. Evaluating Interactive 2D Visualization as a Sample Selection Strategy for Biomedical Time-Series Data Annotation

    Mar 27, 2026Einari Vaaras, Manu Airaksinen, Okko RäsänenHuman-in-the-Loop AnnotationActive Learning

    Signal Processing Research Centre, Tampere University, Tampere, 33720, Finland · BABA Center, Department of Physiology, University of Helsinki, Helsinki, 00029, Finland

  17. When Annotators Agree but Labels Disagree: The Projection Problem in Stance Detection

    Mar 25, 2026Bowen ZhangStance DetectionAnnotator Disagreement

    Shenzhen Technology University, Shenzhen, China

  18. Where Experts Disagree, Models Fail: Detecting Implicit Legal Citations in French Court Decisions

    Mar 24, 2026Avrile Floro, Tamara Dhorasoo, Soline Pellez +1Annotator DisagreementLegal Citation Verification

    UPHF · Télécom Paris, Institut Polytechnique de Paris · Université Polytechnique Hauts-de-France

  19. Confidence Calibration under Ambiguous Ground Truth

    Mar 24, 2026Linwei Tao, Haoyang Luo, Minjing Dong +1Post-Hoc CalibrationAnnotator Disagreement

    L. Tao and C. Xu are with the School of Computer Science, University of Sydney, Sydney, NSW 2006, Australia. · H. Luo and M. Dong are with the Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China.

  20. Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks

    Mar 22, 2026Navya Mehrotra, Adam Visokay, Kristina GligorićAnnotator DisagreementLLM-Assisted Annotation

    University of Washington

  21. ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation

    Mar 22, 2026Smitha Muthya Sudheendra, Jaideep SrivastavaLLM-Assisted AnnotationHuman-in-the-Loop Annotation

    University of Minnesota, Twin Cities

  22. Deep EM with Hierarchical Latent Label Modelling for Multi-Site Prostate Lesion Segmentation

    Mar 15, 2026Wen Yan, Yipei Wang, Shiqi Huang +5Medical Image SegmentationLearning with Noisy Labels

    UCL Hawkes Institute; Department of Medical Physics and Biomedical Engineering, University College London, Gower St, London WC1E 6BT, U.K. · Centre for Medical Imaging, Division of Medicine, University College London, Gower St, London WC1E 6BT, U.K. · Division of Surgery and Interventional Science, University College London, 10 Pond St, London NW3 2PS, U.K. +2

  23. Annotation Entropy Predicts Per-Example Learning Dynamics in LoRA Fine-Tuning

    Mar 12, 2026Brady SteeleParameter-Efficient Fine-TuningAnnotator Disagreement

    Georgia Institute of Technology

  24. Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation

    Mar 6, 2026Joseph JamesInter-Rater ReliabilityAnnotator Disagreement

    Department of Computer Science, The University of Sheffield Sheffield, United Kingdom

  25. From Variance to Invariance: Qualitative Content Analysis for Narrative Graph Annotation

    Mar 2, 2026Junbo Huang, Max Weinig, Ulrich Fritsche +1Annotator DisagreementInter-Rater Reliability

    Department of Computer Science, University of Hamburg · Department of Socioeconomics, University of Hamburg · AI and Explanability Group, Leuphana University of Lüneburg

  26. Decomposing Physician Disagreement in HealthBench

    Feb 26, 2026Satya Borgohain, Roy MariathasAnnotator DisagreementClinical Language Model Evaluation

  27. Human Label Variation in Implicit Discourse Relation Recognition

    Feb 26, 2026Frances Yung, Daniil Ignatev, Merel Scholman +2Annotator DisagreementLabel Distribution Learning

    Saarland University, Saarland Informatics Campus, Germany · Utrecht University, Utrecht, The Netherlands · Queen Mary University of London, London, UK

  28. CREMD: Crowd-Sourced Emotional Multimodal Dogs Dataset

    Feb 17, 2026Jinho Baek, Houwei Cao, Kate BlackwellEmotion RecognitionMultimodal Emotion Recognition

    dept. of Computer Science, New York Institute of Technology, New York, USA

  29. Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP

    Jan 14, 2026Yinuo Xu, David JurgensAnnotator Disagreement

    University of Michigan

  30. Labels have Human Values: Value Calibration of Subjective Tasks

    Jan 10, 2026Mohammed Fayiz Parappan, Ricardo HenaoAnnotator DisagreementAI Alignment

    Department of Electrical and Computer Engineering Duke University