Overview

Inter-rater reliability studies whether different annotators give consistent judgments on the same items, a key concern when labels are used to train or evaluate AI systems. Researchers use measures such as Cohen’s and Fleiss’ kappa, Krippendorff’s alpha, and the intraclass correlation coefficient, choosing among them based on the task, label scale, missing data, and class balance. Other approaches model rater-specific tendencies, aggregate multiple labels, or represent disagreement directly in settings such as essay scoring, medical image segmentation, and LLM evaluation. A central challenge is distinguishing unreliable judgment from genuine ambiguity: high agreement does not establish validity, and disagreement may reveal unclear criteria, difficult cases, or meaningful differences in interpretation.

Latest papers

CardsList

48 papers3 papers in the last four weeks, down 62% on the four weeks before. 0.0% of all new papers.

  1. The Structure of Disagreement in Emotion Annotation

    Nov 1, 2026Emily Öhman, Anna Koufakou, Eugenia DiegoliAnnotator DisagreementInter-Rater Reliability

  2. Multi-Label Agreement: A Flexible Framework for Inter-Annotator Reliability

    Oct 27, 2026Tim Menzner, Jochen L. LeidnerInter-Rater Reliability

  3. Quantifying Annotation Agreement in Object Detection with Optimal Transport

    Oct 14, 2026Kirill Prokhorov, Alexandr A. KalininInter-Rater ReliabilityObject Detection

  4. Can We Believe AI Peer Review?

    Oct 14, 2026Zongrui Yang, Haoyuan Li, Zhongsheng Wang +2Scholarly Peer ReviewAutomated Peer Review

  5. Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts

    Oct 7, 2026Himarsha R. Jayanetti, Sivakanesan Dhanushkanda, Shuai Hao +2Sentiment AnalysisSocial Media Analysis

  6. Language-model ratings of depression reflect the rater more than the patient

    Oct 6, 2026Baihan LinInter-Rater ReliabilityAnnotator Disagreement

  7. LLM Judge Validation Under Sparse Overlap: From Inference to Design

    Sep 25, 2026Junxuan Li, Arko Mukherjee, Soumyabrata PalInter-Rater ReliabilityLLM Evaluation

  8. LENS-GRF: Permutation-Invariant Lesion Evidence Network with Gated Residual Fusion for Acne Severity Grading and Multi-Rater Clinical Oracle Analysis

    Sep 25, 2026Muhammad Muhtasim Shahriar, M. F. Mridha, Md. Rakibul IslamSet TransformerAutomated Grading

  9. Evaluating the Effects of Inter-Observer and Model Variability on Radiological Peritoneal Cancer Index Assessment

    Aug 28, 2026Savvas Saragiotis, Pieter C. Gort, Lotte J. S. Fleurkens-Ewals +5Medical Image Segmentation3D Medical Image Segmentation

  10. Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons

    Aug 10, 2026Kaustubh Shivshankar Shejole, Tanish Agarwal, Arpit Agarwal +1Pairwise Preference LearningReward Modeling

  11. Consensus Measures for Unstructured Biomedical Text Annotations

    Aug 4, 2026Pascal Wullschleger, Christian Kreis, Martin A. Walter +2Inter-Rater ReliabilitySemantic Textual Similarity

  12. Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

    Aug 1, 2026William CabanInter-Rater ReliabilityAI Agent Evaluation

  13. AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling

    Jul 20, 2026Ju Chen, Sijia Xu, Jun Feng +2Inter-Rater ReliabilityNoisy-Label Learning

  14. A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents

    Jul 8, 2026A. Sayyad, J. Emmons, S. Jones +2Audio-Language Model EvaluationVoice Agent Evaluation

  15. Introducing corpora Hlava Cor and Hlava AD: Human Label Variation in Coreference and Discourse Relations

    Jun 24, 2026Anna Nedoluzhko, Šárka Zikánová, Jiří Mírovský +2Inter-Rater ReliabilityCoreference Resolution

  16. Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

    Jun 17, 2026Justin D. Norman, Michael U. Rivera, D. Alex HughesInter-Rater ReliabilityLLM Evaluation

  17. Attention-Based Prototype Calibration for Multi-Rater Few-Shot Medical Image Segmentation

    Jun 15, 2026Truong Vu, Minh Khoi Ho, Yutong XiePrototype-Based LearningInter-Rater Reliability

  18. Event-Aligned Analysis of Multi-Rater Pain Assessments Using Continuous Wearable Physiology

    Jun 11, 2026Saba A. Farahani, Elahe Khatibi, Thomas D. Hughes +3Inter-Rater ReliabilityAnnotator Disagreement

  19. Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated

    May 30, 2026Rashid MushkaniVLM EvaluationInter-Rater Reliability