Annotator Disagreement

Ask about this topic

Momentum

5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 56

All topics
CardsList

  1. Language-model ratings of depression reflect the rater more than the patient

    Oct 6, 2026Baihan LinInter-Rater ReliabilityAnnotator Disagreement

    Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA +2

  2. The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)

    Oct 5, 2026Feilian HuangInter-Rater ReliabilityAnnotator Disagreement

    Independent researcher

  3. QuanReview: Offline, Auditable Reconciliation of Human and LLM Span Annotations

    Sep 28, 2026Matteo Musacchio, Juan Cruz Giner Pulero, Isabel Castañeda +4Annotator DisagreementLLM-Assisted Annotation

    Universidad de San Andrés, Buenos Aires, Argentina · ISI Foundation, Turin, Italy · CONICET, Buenos Aires, Argentina +1

  4. Decomposing Error and Style in Automated Clinical Coding

    Sep 21, 2026Han-Chin Shing, Jack Moriarty, Ryan Ware +6Annotator DisagreementAutomated Medical Coding

    Amazon

  5. How Many Humans Are 32 LLM Judges Worth?

    Sep 18, 2026Chao Li, Yingying Yu, Yunfeng LiAnnotator DisagreementLLM-as-a-Judge

    Tsinghua University · University College London · Hebei University of Economics and Business

  6. Predicting Human Disagreement for Calibrated Dynamic Facial Expression Recognition

    Sep 15, 2026Yiming Wang, Frederick W. B. Li, Jingyun WangAnnotator DisagreementUncertainty Quantification

    Department of Computer Science, Durham University

  7. Uncertainty-Aware Sea-Ice Type Mapping with Multiple Ice Charts

    Sep 8, 2026Samira Alkaee Taleghan, Younghyun Koo, Andrew P. Barrett +1Remote Sensing Image UnderstandingAnnotator Disagreement

    University of Colorado Denver Denver, Colorado, USA · National Snow and Ice Data Center (NSIDC), CIRES, University of Colorado Boulder Boulder, Colorado, USA

  8. Aggregate Disambiguation Systems

    Aug 31, 2026José María Lago, Albert Castellana, Edgars NemšeAnnotator DisagreementConfidence Region Estimation

    GenLayer Labs Research

  9. Definitional Sensitivity in Media Bias Detection: A Multi-Definition Dataset and Benchmark

    Aug 24, 2026Martin Wessel, Timo Spinde, Jürgen Pfeffer +1Annotator Disagreement

    Technical University of Munich · Center for Digital Technology and Management (CDTM) · National Institute of Informatics (NII), Tokyo +1

  10. Learning Sexism Detection Using Multi-Agent Perspectivist Preference Optimization

    Aug 4, 2026Hadi Mohammadi, Tina Shahedi, Robert A. Bagheri +2Annotator DisagreementPreference Optimization

    Department of Methodology and Statistics, Utrecht University, The Netherlands · Department of Information and Computing Sciences, Utrecht University, The Netherlands · Field Technology Innovations (FTI) & Robotics, Wageningen University & Research, The Netherlands

  11. Selection Shapes the Boundary: A Preregistered Replication of Monotonicity and Label Agreement in Unselected NLI Populations

    Jul 21, 2026Haram ChoiAnnotator DisagreementSelection Bias

    University of Bremen

  12. How Much Human Label Variation Does Formal Semantic Structure Explain? Group-Level Effects and Item-Level Ceilings in NLI

    Jul 17, 2026Haram ChoiAnnotator DisagreementNatural Language Inference

    University of Bremen

  13. Ensemble Diversity Optimization for Subjective Supervision

    Jul 9, 2026Xia Cui, Ziyi Huang, N. R. AbeynayakeAnnotator DisagreementEnsemble Learning

    School of Computing and Mathematics, Manchester Metropolitan University, Manchester, UK. · School of Computer Science, Hubei University, Wuhan, China.

  14. A Hybrid Framework for Song Lyric Annotation Based on Human-LLM Alignment

    Jun 28, 2026Rashini Liyanarachchi, Frank Tran, Md Mahmudul Hasan +2Annotator DisagreementLLM-Assisted Annotation

    School of Computer Science and Engineering, University of New South Wales, Sydney, Australia

  15. Majority Vote Silences Minority Values: Annotator Disagreement at the Hate/Offensive Boundary in HateXplain

    Jun 27, 2026Joshua Muhumuza, Joab Ezra Agaba, Mercy AmiyoAnnotator DisagreementHate Speech Detection

    Makerere University, Kampala, Uganda.

  16. Learning from Annotation Uncertainty: Entropy-Aware Curriculum for Speech Emotion Recognition

    Jun 25, 2026Zahra Omidi, John H. L. HansenAnnotator DisagreementLabel Distribution Learning

    Center for Robust Speech Systems, The University of Texas at Dallas, USA

  17. Introducing corpora Hlava Cor and Hlava AD: Human Label Variation in Coreference and Discourse Relations

    Jun 24, 2026Anna Nedoluzhko, Šárka Zikánová, Jiří Mírovský +2Inter-Rater ReliabilityCoreference Resolution

    Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Prague, Czech Republic

  18. Learning Moral Diversity: Modelling Individual Perspectives in Moral Classification of Texts

    Jun 22, 2026Yi Ren, Lewis Mitchell, Matthew RoughanAnnotator DisagreementText Classification

    School of Mathematical Sciences, Adelaide University

  19. Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

    Jun 19, 2026Emily Öhman, Anna KoufakouSoft-Label LearningAnnotator Disagreement

    Waseda University, Tokyo, Japan · Florida Gulf Coast University, Fort Myers FL, USA

  20. Event-Aligned Analysis of Multi-Rater Pain Assessments Using Continuous Wearable Physiology

    Jun 11, 2026Saba A. Farahani, Elahe Khatibi, Thomas D. Hughes +3Inter-Rater ReliabilityAnnotator Disagreement

    University of California, Irvine · Irvine, CA, USA

  21. A Resource for Enthymeme Detection in Controversial Political Discourse

    Jun 10, 2026Martial Pastor, Nelleke OostdijkAnnotator DisagreementHuman-in-the-Loop Annotation

    Centre for Language Studies, Radboud University, Nijmegen

  22. Factions Within, Uncertain Across: Within-Document Reader Sub-Groups in Social Highlighting

    Jun 10, 2026Kazuki Nakayashiki, Keisuke WatanabeInter-Rater ReliabilityAnnotator Disagreement

    Glasp Inc.

  23. The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction

    Jun 1, 2026Mirko Lai, Alessandra Urbinati, Simona Frenda +2Annotator DisagreementConformal Prediction

    Heriot-Watt University, Edinburgh, Scotland · aequa-tech, Torino, Italy · Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USA +2

  24. Bayesian Spectral Emotion Transition Discovery from Multi-Annotator Disagreement

    Jun 1, 2026Keito Inoshita, Takato UenoAnnotator DisagreementEmotion Recognition in Conversations

    Faculty of Business and Commerce, Kansai University, Suita, Japan · Data Science and AI Innovation Promotion Center, Shiga University, Hikone, Japan · Graduate School of Data Science, Shiga University, Hikone, Japan

  25. Disagreeing Rationales: Rethinking Classification and Explainability Evaluation in Hate Speech Detection

    May 29, 2026Benedetta Muscato, Beiduo Chen, Gizem Gezici +2Annotator DisagreementHate Speech Detection

    Scuola Normale Superiore, Italy · Munich Center for Machine Learning, Germany · UMaiNLP, LMU Munich, Germany

  26. Metric-Dependent Annotation Saturation for Learning from Label Distributions

    May 28, 2026Guneet KohliSoft-Label LearningAnnotator Disagreement

    Apple

  27. When Models Disagree: Rethinking LLM Evaluation for Public Comment Analysis

    May 27, 2026Aisha Najera, Alvin Moon, Vedant Srinivasan +1LLM EvaluationAnnotator Disagreement

    AI Lab, Princeton University · Engineering and Applied Sciences, RAND Corporation · Science, Technology, and International Affairs, Georgetown University

  28. Temporal Simultaneity Predicts Annotation Quality in Sentiment Corpora

    May 26, 2026Idris Abdulmumin, Mokgadi Penelope Matloga, Tadesse Destaw Belay +5Annotator DisagreementSentiment Analysis

    Data Science for Social Impact, University of Pretoria · Instituto Politécnico Nacional · Department of African Languages, University of Pretoria +1

  29. A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays

    May 25, 2026Pawitsapak Akarajaradwong, Wuttikrai Lertprasertphakorn, Chompakorn Chaksangchaichot +1Inter-Rater ReliabilityAnnotator Disagreement

    VISAI AI · Independent Researcher · Vidyasirimedhi Institute of Science and Technology

  30. Uncertainty Decomposition via Cyclical SG-MCMC and Soft-label Learning for Subjective NLP

    May 23, 2026Keito Inoshita, Takato UenoSoft-Label LearningBayesian Neural Networks

    Faculty of Business and Commerce, Kansai University, Suita, Japan · Data Science and AI Innovation Research Promotion Center, Shiga University, Hikone, Japan · Graduate School of Data Science, Shiga University, Hikone, Japan