Annotator Disagreement

Ask about this topic

Momentum

5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 56

All topics
CardsList

  1. Language-model ratings of depression reflect the rater more than the patient

    Oct 6, 2026Baihan LinInter-Rater ReliabilityAnnotator Disagreement

    Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA +2

  2. The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)

    Oct 5, 2026Feilian HuangInter-Rater ReliabilityAnnotator Disagreement

    Independent researcher

  3. QuanReview: Offline, Auditable Reconciliation of Human and LLM Span Annotations

    Sep 28, 2026Matteo Musacchio, Juan Cruz Giner Pulero, Isabel Castañeda +4Annotator DisagreementLLM-Assisted Annotation

    Universidad de San Andrés, Buenos Aires, Argentina · ISI Foundation, Turin, Italy · CONICET, Buenos Aires, Argentina +1

  4. Decomposing Error and Style in Automated Clinical Coding

    Sep 21, 2026Han-Chin Shing, Jack Moriarty, Ryan Ware +6Annotator DisagreementAutomated Medical Coding

    Amazon

  5. How Many Humans Are 32 LLM Judges Worth?

    Sep 18, 2026Chao Li, Yingying Yu, Yunfeng LiAnnotator DisagreementLLM-as-a-Judge

    Tsinghua University · University College London · Hebei University of Economics and Business

  6. Predicting Human Disagreement for Calibrated Dynamic Facial Expression Recognition

    Sep 15, 2026Yiming Wang, Frederick W. B. Li, Jingyun WangAnnotator DisagreementUncertainty Quantification

    Department of Computer Science, Durham University

  7. Uncertainty-Aware Sea-Ice Type Mapping with Multiple Ice Charts

    Sep 8, 2026Samira Alkaee Taleghan, Younghyun Koo, Andrew P. Barrett +1Remote Sensing Image UnderstandingAnnotator Disagreement

    University of Colorado Denver Denver, Colorado, USA · National Snow and Ice Data Center (NSIDC), CIRES, University of Colorado Boulder Boulder, Colorado, USA

  8. Aggregate Disambiguation Systems

    Aug 31, 2026José María Lago, Albert Castellana, Edgars NemšeAnnotator DisagreementConfidence Region Estimation

    GenLayer Labs Research

  9. Definitional Sensitivity in Media Bias Detection: A Multi-Definition Dataset and Benchmark

    Aug 24, 2026Martin Wessel, Timo Spinde, Jürgen Pfeffer +1Annotator Disagreement

    Technical University of Munich · Center for Digital Technology and Management (CDTM) · National Institute of Informatics (NII), Tokyo +1

  10. Learning Sexism Detection Using Multi-Agent Perspectivist Preference Optimization

    Aug 4, 2026Hadi Mohammadi, Tina Shahedi, Robert A. Bagheri +2Annotator DisagreementPreference Optimization

    Department of Methodology and Statistics, Utrecht University, The Netherlands · Department of Information and Computing Sciences, Utrecht University, The Netherlands · Field Technology Innovations (FTI) & Robotics, Wageningen University & Research, The Netherlands

  11. Selection Shapes the Boundary: A Preregistered Replication of Monotonicity and Label Agreement in Unselected NLI Populations

    Jul 21, 2026Haram ChoiAnnotator DisagreementSelection Bias

    University of Bremen

  12. How Much Human Label Variation Does Formal Semantic Structure Explain? Group-Level Effects and Item-Level Ceilings in NLI

    Jul 17, 2026Haram ChoiAnnotator DisagreementNatural Language Inference

    University of Bremen

  13. Ensemble Diversity Optimization for Subjective Supervision

    Jul 9, 2026Xia Cui, Ziyi Huang, N. R. AbeynayakeAnnotator DisagreementEnsemble Learning

    School of Computing and Mathematics, Manchester Metropolitan University, Manchester, UK. · School of Computer Science, Hubei University, Wuhan, China.

  14. A Hybrid Framework for Song Lyric Annotation Based on Human-LLM Alignment

    Jun 28, 2026Rashini Liyanarachchi, Frank Tran, Md Mahmudul Hasan +2Annotator DisagreementLLM-Assisted Annotation

    School of Computer Science and Engineering, University of New South Wales, Sydney, Australia

  15. Majority Vote Silences Minority Values: Annotator Disagreement at the Hate/Offensive Boundary in HateXplain

    Jun 27, 2026Joshua Muhumuza, Joab Ezra Agaba, Mercy AmiyoAnnotator DisagreementHate Speech Detection

    Makerere University, Kampala, Uganda.

  16. Learning from Annotation Uncertainty: Entropy-Aware Curriculum for Speech Emotion Recognition

    Jun 25, 2026Zahra Omidi, John H. L. HansenAnnotator DisagreementLabel Distribution Learning

    Center for Robust Speech Systems, The University of Texas at Dallas, USA

  17. Introducing corpora Hlava Cor and Hlava AD: Human Label Variation in Coreference and Discourse Relations

    Jun 24, 2026Anna Nedoluzhko, Šárka Zikánová, Jiří Mírovský +2Inter-Rater ReliabilityCoreference Resolution

    Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Prague, Czech Republic

  18. Learning Moral Diversity: Modelling Individual Perspectives in Moral Classification of Texts

    Jun 22, 2026Yi Ren, Lewis Mitchell, Matthew RoughanAnnotator DisagreementText Classification

    School of Mathematical Sciences, Adelaide University

  19. Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

    Jun 19, 2026Emily Öhman, Anna KoufakouSoft-Label LearningAnnotator Disagreement

    Waseda University, Tokyo, Japan · Florida Gulf Coast University, Fort Myers FL, USA

  20. Event-Aligned Analysis of Multi-Rater Pain Assessments Using Continuous Wearable Physiology

    Jun 11, 2026Saba A. Farahani, Elahe Khatibi, Thomas D. Hughes +3Inter-Rater ReliabilityAnnotator Disagreement

    University of California, Irvine · Irvine, CA, USA

  21. A Resource for Enthymeme Detection in Controversial Political Discourse

    Jun 10, 2026Martial Pastor, Nelleke OostdijkAnnotator DisagreementHuman-in-the-Loop Annotation

    Centre for Language Studies, Radboud University, Nijmegen

  22. Factions Within, Uncertain Across: Within-Document Reader Sub-Groups in Social Highlighting

    Jun 10, 2026Kazuki Nakayashiki, Keisuke WatanabeInter-Rater ReliabilityAnnotator Disagreement

    Glasp Inc.

  23. The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction

    Jun 1, 2026Mirko Lai, Alessandra Urbinati, Simona Frenda +2Annotator DisagreementConformal Prediction

    Heriot-Watt University, Edinburgh, Scotland · aequa-tech, Torino, Italy · Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USA +2

  24. Bayesian Spectral Emotion Transition Discovery from Multi-Annotator Disagreement

    Jun 1, 2026Keito Inoshita, Takato UenoAnnotator DisagreementEmotion Recognition in Conversations

    Faculty of Business and Commerce, Kansai University, Suita, Japan · Data Science and AI Innovation Promotion Center, Shiga University, Hikone, Japan · Graduate School of Data Science, Shiga University, Hikone, Japan

  25. Disagreeing Rationales: Rethinking Classification and Explainability Evaluation in Hate Speech Detection

    May 29, 2026Benedetta Muscato, Beiduo Chen, Gizem Gezici +2Annotator DisagreementHate Speech Detection

    Scuola Normale Superiore, Italy · Munich Center for Machine Learning, Germany · UMaiNLP, LMU Munich, Germany

  26. Metric-Dependent Annotation Saturation for Learning from Label Distributions

    May 28, 2026Guneet KohliSoft-Label LearningAnnotator Disagreement

    Apple

  27. When Models Disagree: Rethinking LLM Evaluation for Public Comment Analysis

    May 27, 2026Aisha Najera, Alvin Moon, Vedant Srinivasan +1LLM EvaluationAnnotator Disagreement

    AI Lab, Princeton University · Engineering and Applied Sciences, RAND Corporation · Science, Technology, and International Affairs, Georgetown University

  28. Temporal Simultaneity Predicts Annotation Quality in Sentiment Corpora

    May 26, 2026Idris Abdulmumin, Mokgadi Penelope Matloga, Tadesse Destaw Belay +5Annotator DisagreementSentiment Analysis

    Data Science for Social Impact, University of Pretoria · Instituto Politécnico Nacional · Department of African Languages, University of Pretoria +1

  29. A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays

    May 25, 2026Pawitsapak Akarajaradwong, Wuttikrai Lertprasertphakorn, Chompakorn Chaksangchaichot +1Inter-Rater ReliabilityAnnotator Disagreement

    VISAI AI · Independent Researcher · Vidyasirimedhi Institute of Science and Technology

  30. Uncertainty Decomposition via Cyclical SG-MCMC and Soft-label Learning for Subjective NLP

    May 23, 2026Keito Inoshita, Takato UenoSoft-Label LearningBayesian Neural Networks

    Faculty of Business and Commerce, Kansai University, Suita, Japan · Data Science and AI Innovation Research Promotion Center, Shiga University, Hikone, Japan · Graduate School of Data Science, Shiga University, Hikone, Japan

  31. Calibrating Probabilistic Object Detectors with Annotator Disagreement

    May 23, 2026Zhi Qin Tan, Owen Addison, Yunpeng LiAnnotator DisagreementObject Detection

    Faculty of Dentistry, Oral & Craniofacial Sciences, King’s College London, London, United Kingdom

  32. Improving Labeling Consistency with Detailed Constitutional Definitions and AI-Driven Evaluation

    May 22, 2026Konstantin Berlin, Adam SwandaAnnotator DisagreementLanguage Model Safety Evaluation

    Cisco AI Defense

  33. Same Target, Different Basins: Hard vs. Soft Labels for Annotator Distributions

    May 20, 2026Mirerfan Gheibi, Gashin GhazizadehAnnotator DisagreementLabel Distribution Learning

    Independent Researcher.

  34. Predicting Disagreement with Human Raters in LLM-as-a-Judge Difficulty Assessment without Using Generation-Time Probability Signals

    May 12, 2026Yo EharaAnnotator DisagreementLLM-as-a-Judge

    Tokyo Gakugei University

  35. Aligning LLM Uncertainty with Human Disagreement in Subjectivity Analysis

    May 11, 2026Junyu Lu, Deyi Ji, Xuanyi Liu +5Annotator DisagreementLLM Alignment

    School of Computer Science and Technology, Dalian University of Technology, China · Tencent · Peking University +1

  36. Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks

    May 11, 2026Tadesse Destaw Belay, Ibrahim Said Ahmad, Idris Abdulmumin +6Annotator DisagreementClustering

    1Instituto Politécnico Nacional · University of Wisconsin–Stevens Point · University of Pretoria +3

  37. Parser agreement and disagreement in L2 Korean UD: Implications for human-in-the-loop annotation

    May 7, 2026Hakyung Sung, Gyu-Ho ShinAnnotator DisagreementMorphosyntactic Tagging

    Psychology, Rochester Institute of Technology · Linguistics, University of Illinois Chicago

  38. Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variation

    May 7, 2026Maximilian Maurer, Maximilian Linde, Gabriella LapesaAnnotator DisagreementHuman-in-the-Loop Annotation

    1GESIS - Leibniz Institute for the Social Sciences · 2Heinrich-Heine University Düsseldorf

  39. Understanding Annotator Safety Policy with Interpretability

    May 6, 2026Alex Oesterling, Donghao Ren, Yannick Assogba +4Annotator DisagreementHuman-in-the-Loop Annotation

    Harvard University, Cambridge, MA, USA · Apple, Seattle, WA, USA · Apple, Cambridge, MA, USA +1

  40. Annotation Quality in Aspect-Based Sentiment Analysis: A Case Study Comparing Experts, Students, Crowdworkers, and Large Language Model

    May 5, 2026Niklas Donhauser, Jakob Fehle, Nils Constantin Hellwig +3Annotator DisagreementLLM-Assisted Annotation

    Media Informatics Group, University of Regensburg, Regensburg, Germany · Information Science Group, University of Regensburg, Regensburg, Germany

  41. STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems

    May 4, 2026Akash Bonagiri, Gerard Janno Anderias, Saee Patil +6Annotator DisagreementAI Agent Evaluation

    Department of Computer Science, University of California, Davis, Davis, CA, USA.

  42. Quantifying and Predicting Disagreement in Graded Human Ratings

    May 1, 2026Leixin Zhang, Çağrı ÇöltekinAnnotator Disagreement

    University of Tübingen, Germany

  43. Are You the A-hole? A Fair, Multi-Perspective Ethical Reasoning Framework

    Apr 30, 2026Sheza Munir, Ahanaf Rodoshi, Sumin Lee +3Annotator DisagreementNeuro-Symbolic Reasoning

    Computer Science, University of Toronto

  44. LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps

    Apr 30, 2026Keito Inoshita, Xiaokang Zhou, Akira Kawai +1LLM EvaluationAnnotator Disagreement

    Faculty of Business and Commerce, Kansai University, Osaka, Japan · Faculty of Business Data Science, Kansai University, Osaka, Japan · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan +2

  45. Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement

    Apr 24, 2026Wataru Hirota, Tomoki Taniguchi, Tomoko Ohkuma +6Human Preference EvaluationAnnotator Disagreement

    1Stockmark Inc · 2Asahi Kasei Corporation · National Institute of Informatics +1

  46. Fine-Grained Perspectives: Modeling Explanations with Annotator-Specific Rationales

    Apr 23, 2026Olufunke O. Sarumi, Charles Welch, Daniel BraunAnnotator DisagreementNatural Language Inference

    Marburg University, Marburg, Germany · McMaster University, Hamilton, Ontario, Canada

  47. Structured Disagreement in Health-Literacy Annotation: Epistemic Stability, Conceptual Difficulty, and Agreement-Stratified Inference

    Apr 21, 2026Olga Kellert, Sriya Kondury, Candice Koo +2Inter-Rater ReliabilityAnnotator Disagreement

    Arizona State University

  48. Modeling Human Perspectives with Socio-Demographic Representations

    Apr 20, 2026Leixin Zhang, Cagri ColtekinContrastive LearningAnnotator Disagreement

    University of Tübingen

  49. Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains

    Apr 19, 2026Finn Schmidt, Jan Philip Wahle, Terry Ruas +1Annotator DisagreementAutomated Evaluation

    University of Göttingen, Germany

  50. Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks

    Apr 18, 2026Nisrine Rair, Alban Goupil, Valeriu Vrabie +1Annotator Disagreement

    CReSTIC, Université de Reims Champagne-Ardenne, Reims, France · Chochoy Conseil, Reims, France

  51. IYKYK (But AI Doesn't): Automated Content Moderation Does Not Capture Communities' Heterogeneous Attitudes Towards Reclaimed Language

    Apr 17, 2026Christina Chance, Rebecca Pattichis, Arjun Subramonian +4Social Media AnalysisAnnotator Disagreement

    University of California Los Angeles, USA · Independent Researcher, USA

  52. Where Experts Disagree, Models Fail: Detecting Implicit Legal Citations in French Court Decisions

    Mar 24, 2026Avrile Floro, Tamara Dhorasoo, Soline Pellez +1Annotator DisagreementLegal Citation Verification

    UPHF · Télécom Paris, Institut Polytechnique de Paris · Université Polytechnique Hauts-de-France

  53. Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks

    Mar 22, 2026Navya Mehrotra, Adam Visokay, Kristina GligorićAnnotator DisagreementLLM-Assisted Annotation

    University of Washington

  54. Labels have Human Values: Value Calibration of Subjective Tasks

    Jan 10, 2026Mohammed Fayiz Parappan, Ricardo HenaoAnnotator DisagreementAI Alignment

    Department of Electrical and Computer Engineering Duke University

  55. Decomposing LLM-Judge Uncertainty to Target Expert Labels

    Date pendingRyan LailAnnotator DisagreementLLM-as-a-Judge

    Composo AI

  56. When Does a Laugh Begin? Structured Annotator Disagreement in Temporal Laughter Localization

    Date pendingEyal Hanania, Daniel Arkushin, Naveh Ayal +4Inter-Rater ReliabilityAnnotator Disagreement

    WSC Sports