Inter-Rater Reliability

Ask about this topic

Overview

Inter-rater reliability studies whether different annotators give consistent judgments on the same items, a key concern when labels are used to train or evaluate AI systems. Researchers use measures such as Cohen’s and Fleiss’ kappa, Krippendorff’s alpha, and the intraclass correlation coefficient, choosing among them based on the task, label scale, missing data, and class balance. Other approaches model rater-specific tendencies, aggregate multiple labels, or represent disagreement directly in settings such as essay scoring, medical image segmentation, and LLM evaluation. A central challenge is distinguishing unreliable judgment from genuine ambiguity: high agreement does not establish validity, and disagreement may reveal unclear criteria, difficult cases, or meaningful differences in interpretation.

Latest papers

CardsList

45 papers6 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

  1. Multi-Label Agreement: A Flexible Framework for Inter-Annotator Reliability

    Oct 27, 2026Tim Menzner, Jochen L. LeidnerInter-Rater Reliability

    Center for Responsible Artificial Intelligence (CRAI), Coburg University of Applied Sciences

  2. Quantifying Annotation Agreement in Object Detection with Optimal Transport

    Oct 14, 2026Kirill Prokhorov, Alexandr A. KalininInter-Rater ReliabilityObject Detection

    Openface.io, Vilnius, Lithuania · Broad Institute of MIT and Harvard, Cambridge, MA, USA

  3. Can We Believe AI Peer Review?

    Oct 14, 2026Zongrui Yang, Haoyuan Li, Zhongsheng Wang +2Scholarly Peer ReviewAutomated Peer Review

    University of Auckland Auckland, New Zealand

  4. Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans

    Oct 9, 2026Sangdo Han, Moonsoo KimLLM-as-a-JudgeRubric-Based Evaluation

    CEO Business School

  5. Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts

    Oct 7, 2026Himarsha R. Jayanetti, Sivakanesan Dhanushkanda, Shuai Hao +2Sentiment AnalysisSocial Media Analysis

    Old Dominion University, Norfolk, Virginia, USA

  6. Language-model ratings of depression reflect the rater more than the patient

    Oct 6, 2026Baihan LinInter-Rater ReliabilityAnnotator Disagreement

    Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA +2

  7. The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)

    Oct 5, 2026Feilian HuangInter-Rater ReliabilityAnnotator Disagreement

    Independent researcher

  8. LLM Judge Validation Under Sparse Overlap: From Inference to Design

    Sep 25, 2026Junxuan Li, Arko Mukherjee, Soumyabrata PalInter-Rater ReliabilityLLM Evaluation

    Adobe · Adobe Research

  9. LENS-GRF: Permutation-Invariant Lesion Evidence Network with Gated Residual Fusion for Acne Severity Grading and Multi-Rater Clinical Oracle Analysis

    Sep 25, 2026Muhammad Muhtasim Shahriar, M. F. MridhaSet TransformerAutomated Grading

    Department of Computer Science, International Islamic University Chittagong (IIUC), Chittagong, Bangladesh · Department of Computer Science, American International University-Bangladesh (AIUB), Dhaka, Bangladesh

  10. Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features: Three Studies on a Turkish Corpus

    Sep 12, 2026Levent BulutLLM-Assisted AnnotationInter-Rater Reliability

    Independent Researcher

  11. What an odour descriptor corpus can and cannot measure: valence, attenuation, and the ceiling of the public record

    Sep 11, 2026Stylianos Kampakis, Fabio RovaiInter-Rater ReliabilityMolecular Property Prediction

    Tesseract Academy, London, UK

  12. Bridging Network Psychometrics and Artificial Intelligence: An Ising-Potts Model with LLM-Derived Weights

    Sep 8, 2026Matthias von DavierInter-Rater ReliabilityPsychometric Validation

  13. When Does a Laugh Begin? Structured Annotator Disagreement in Temporal Laughter Localization

    Sep 6, 2026Eyal Hanania, Daniel Arkushin, Naveh Ayal +4Inter-Rater ReliabilityAnnotator Disagreement

    WSC Sports

  14. LLM Judges as Raters: A Pre-Registered Audit of Severity, Halo, Reliability, and Version Instability in LLM Essay Scoring on Public Corpora

    Aug 30, 2026Veerendra Kumar SunkavalliLLM-as-a-JudgeLLM Reliability

    Independent Researcher

  15. Evaluating the Effects of Inter-Observer and Model Variability on Radiological Peritoneal Cancer Index Assessment

    Aug 28, 2026Savvas Saragiotis, Pieter C. Gort, Lotte J. S. Fleurkens-Ewals +5Medical Image Segmentation3D Medical Image Segmentation

    Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands · Catharina Hospital Eindhoven, Eindhoven, The Netherlands

  16. Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons

    Aug 10, 2026Kaustubh Shivshankar Shejole, Tanish Agarwal, Arpit Agarwal +1Pairwise Preference LearningReward Modeling

    IIT Bombay, Mumbai, India – 400076

  17. Consensus Measures for Unstructured Biomedical Text Annotations

    Aug 4, 2026Pascal Wullschleger, Christian Kreis, Martin A. Walter +2Inter-Rater ReliabilitySemantic Textual Similarity

    Hamilton Institute, Maynooth University, Ireland · University of Lucerne, Switzerland · Lucerne School of Computer Science and IT, Switzerland

  18. Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

    Aug 1, 2026William CabanInter-Rater ReliabilityAI Agent Evaluation

    Red Hat, Inc., Raleigh, North Carolina, USA · Alma Mater Europaea University, Vienna, Austria

  19. AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling

    Jul 20, 2026Ju Chen, Sijia Xu, Jun Feng +2Inter-Rater ReliabilityNoisy-Label Learning

    Hohai University, China · University of New South Wales, Australia · Southeast University, China +1

  20. A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents

    Jul 8, 2026A. Sayyad, J. Emmons, S. Jones +2Audio-Language Model EvaluationVoice Agent Evaluation

    Salesforce Applied AI Research, eVerse team

  21. Introducing corpora Hlava Cor and Hlava AD: Human Label Variation in Coreference and Discourse Relations

    Jun 24, 2026Anna Nedoluzhko, Šárka Zikánová, Jiří Mírovský +2Inter-Rater ReliabilityCoreference Resolution

    Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Prague, Czech Republic

  22. The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel

    Jun 22, 2026Tarek GaraInter-Rater ReliabilityLLM-Assisted Annotation

    *Independent researcher.

  23. Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

    Jun 17, 2026Justin D. Norman, Michael U. Rivera, D. Alex HughesInter-Rater ReliabilityLLM Evaluation

    UC Berkeley School of Information 102 South Hall Drive, Berkeley CA, 94708

  24. Attention-Based Prototype Calibration for Multi-Rater Few-Shot Medical Image Segmentation

    Jun 15, 2026Truong Vu, Minh Khoi Ho, Yutong XiePrototype-Based LearningInter-Rater Reliability

    Mohamed bin Zayed University of Artificial Intelligence

  25. Event-Aligned Analysis of Multi-Rater Pain Assessments Using Continuous Wearable Physiology

    Jun 11, 2026Saba A. Farahani, Elahe Khatibi, Thomas D. Hughes +3Inter-Rater ReliabilityAnnotator Disagreement

    University of California, Irvine · Irvine, CA, USA

  26. Factions Within, Uncertain Across: Within-Document Reader Sub-Groups in Social Highlighting

    Jun 10, 2026Kazuki Nakayashiki, Keisuke WatanabeInter-Rater ReliabilityAnnotator Disagreement

    Glasp Inc.

  27. The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment

    Jun 2, 2026Sourabrata Mukherjee, Hamna Hamna, Kalika Bali +1Inter-Rater ReliabilityMultilingual Language Model Evaluation

    Microsoft Research

  28. Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated

    May 30, 2026Rashid MushkaniVLM EvaluationInter-Rater Reliability

    Université de Montréal, Montréal, Québec, Canada · Mila – Québec AI Institute, Montréal, Québec, Canada

  29. A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays

    May 25, 2026Pawitsapak Akarajaradwong, Wuttikrai Lertprasertphakorn, Chompakorn Chaksangchaichot +1Inter-Rater ReliabilityAnnotator Disagreement

    VISAI AI · Independent Researcher · Vidyasirimedhi Institute of Science and Technology

  30. Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

    May 25, 2026Delip Rao, Chris Callison-BurchInter-Rater ReliabilityLLM Evaluation

    University of Pennsylvania