Inter-Rater Reliability

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 31

All topics

Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts
Oct 7, 2026Himarsha R. Jayanetti, Sivakanesan Dhanushkanda, Shuai Hao +2Sentiment Analysis Social Media Analysis
Old Dominion University, Norfolk, Virginia, USA
Language-model ratings of depression reflect the rater more than the patient
Oct 6, 2026Baihan LinInter-Rater Reliability Annotator Disagreement
Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA +2
The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)
Oct 5, 2026Feilian HuangInter-Rater Reliability Annotator Disagreement
Independent researcher
LLM Judge Validation Under Sparse Overlap: From Inference to Design
Sep 25, 2026Junxuan Li, Arko Mukherjee, Soumyabrata PalInter-Rater Reliability LLM Evaluation
Adobe · Adobe Research
What an odour descriptor corpus can and cannot measure: valence, attenuation, and the ceiling of the public record
Sep 14, 2026Stylianos Kampakis, Fabio RovaiInter-Rater Reliability Molecular Property Prediction
Tesseract Academy, London, UK
Bridging Network Psychometrics and Artificial Intelligence: An Ising-Potts Model with LLM-Derived Weights
Sep 8, 2026Matthias von DavierInter-Rater Reliability Psychometric Validation
Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons
Aug 10, 2026Kaustubh Shivshankar Shejole, Tanish Agarwal, Arpit Agarwal +1Pairwise Preference Learning Reward Modeling
IIT Bombay, Mumbai, India – 400076
Consensus Measures for Unstructured Biomedical Text Annotations
Aug 4, 2026Pascal Wullschleger, Christian Kreis, Martin A. Walter +2Inter-Rater Reliability Semantic Textual Similarity
Hamilton Institute, Maynooth University, Ireland · University of Lucerne, Switzerland · Lucerne School of Computer Science and IT, Switzerland
AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling
Jul 20, 2026Ju Chen, Sijia Xu, Jun Feng +2Inter-Rater Reliability Noisy-Label Learning
Hohai University, China · University of New South Wales, Australia · Southeast University, China +1
A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents
Jul 8, 2026A. Sayyad, J. Emmons, S. Jones +2Audio-Language Model Evaluation Voice Agent Evaluation
Salesforce Applied AI Research, eVerse team
Introducing corpora Hlava Cor and Hlava AD: Human Label Variation in Coreference and Discourse Relations
Jun 24, 2026Anna Nedoluzhko, Šárka Zikánová, Jiří Mírovský +2Inter-Rater Reliability Coreference Resolution
Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Prague, Czech Republic
The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel
Jun 22, 2026Tarek GaraInter-Rater Reliability LLM-Assisted Annotation
*Independent researcher.
Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
Jun 17, 2026Justin D. Norman, Michael U. Rivera, D. Alex HughesInter-Rater Reliability LLM Evaluation
UC Berkeley School of Information 102 South Hall Drive, Berkeley CA, 94708
Attention-Based Prototype Calibration for Multi-Rater Few-Shot Medical Image Segmentation
Jun 15, 2026Truong Vu, Minh Khoi Ho, Yutong XiePrototype-Based Learning Inter-Rater Reliability
Mohamed bin Zayed University of Artificial Intelligence
Event-Aligned Analysis of Multi-Rater Pain Assessments Using Continuous Wearable Physiology
Jun 11, 2026Saba A. Farahani, Elahe Khatibi, Thomas D. Hughes +3Inter-Rater Reliability Annotator Disagreement
University of California, Irvine · Irvine, CA, USA
Factions Within, Uncertain Across: Within-Document Reader Sub-Groups in Social Highlighting
Jun 10, 2026Kazuki Nakayashiki, Keisuke WatanabeInter-Rater Reliability Annotator Disagreement
Glasp Inc.
The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment
Jun 2, 2026Sourabrata Mukherjee, Hamna Hamna, Kalika Bali +1Inter-Rater Reliability Multilingual Language Model Evaluation
Microsoft Research
Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated
May 30, 2026Rashid MushkaniVLM Evaluation Inter-Rater Reliability
Université de Montréal, Montréal, Québec, Canada · Mila – Québec AI Institute, Montréal, Québec, Canada
A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays
May 25, 2026Pawitsapak Akarajaradwong, Wuttikrai Lertprasertphakorn, Chompakorn Chaksangchaichot +1Inter-Rater Reliability Annotator Disagreement
VISAI AI · Independent Researcher · Vidyasirimedhi Institute of Science and Technology
Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
May 25, 2026Delip Rao, Chris Callison-BurchInter-Rater Reliability LLM Evaluation
University of Pennsylvania
Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling
May 13, 2026Deepak Pandita, Flip Korn, Chris Welty +1Inter-Rater Reliability LLM Evaluation
Rochester Institute of Technology · Google Research
Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement
May 7, 2026Jessica Huynh, Alfredo Gomez, Athiya Deviyani +3Inter-Rater Reliability LLM-as-a-Judge
Carnegie Mellon University · Google Research
Multi-Rater Calibrated Segmentation Models
May 4, 2026Meritxell Riera-Marín, Javier García López, Júlia Rodríguez-Comas +2Inter-Rater Reliability Image Segmentation
aSycai Technologies SL, Scientific and Technical Department, Barcelona, Spain · bBCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain · cInstitució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, Spain +1
Assessing Pancreatic Ductal Adenocarcinoma Vascular Invasion: the PDACVI Benchmark
Apr 30, 2026M. Riera-Marín, O. K. Sikha, J. Rodríguez-Comas +23Inter-Rater Reliability Medical Image Analysis
BCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain · Universitätsklinikum Erlangen, Department of Radiology of the Uniklinikum Erlangen (UKER), Erlangen, Germany · University Hospital Erlangen, Imaging Science Institute, Erlangen, Germany +10
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
Apr 23, 2026Abel YagubyanInter-Rater Reliability LLM Evaluation
Independent Researcher
Structured Disagreement in Health-Literacy Annotation: Epistemic Stability, Conceptual Difficulty, and Agreement-Stratified Inference
Apr 21, 2026Olga Kellert, Sriya Kondury, Candice Koo +2Inter-Rater Reliability Annotator Disagreement
Arizona State University
Has Automated Essay Scoring Reached Sufficient Accuracy? Deriving Achievable QWK Ceilings from Classical Test Theory
Apr 21, 2026Masaki UtoInter-Rater Reliability Educational Assessment
The University of Electro-Communications, Tokyo, Japan
Honest and Reliable Evaluation and Expert Equivalence Testing of Automated Neonatal Seizure Detection
Aug 6, 2025Jovana Kljajic, John M. O'Toole, Robert Hogan +1Inter-Rater Reliability Automated Evaluation
Faculty of Technical Sciences, University of Novi Sad, 21000 Novi Sad, Serbia · CergenX Ltd, Dublin, Ireland
Hierarchical Bayesian Crowdsourcing with Item Difficulty
May 29, 2024Seong Woo Han, Ozan Adıgüzel, Bob CarpenterInter-Rater Reliability Latent Variable Models
University of Pennsylvania · Columbia University · Center for Computational Mathematics, Flatiron Institute
Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation
Date pendingWilliam CabanInter-Rater Reliability AI Agent Evaluation
Red Hat, Inc., Raleigh, North Carolina, USA · Alma Mater Europaea University, Vienna, Austria

Oct 7, 2026, cs.CLJ/K move · Enter open · S save

Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts

Himarsha R. Jayanetti, Sivakanesan Dhanushkanda, Shuai Hao, Michael L. Nelson+1

Old Dominion University, Norfolk, Virginia, USA

Sentiment Analysis Social Media Analysis Inter-Rater Reliability LLM Evaluation

Open paper AskPDF arXiv

Social media is a rich source of real-time public sentiment, but widely used sentiment analysis tools are often applied without understanding their limitations. In this study, we evaluate the inter-rater reliability of three bespoke sentiment analysis tools (TextBlob, VADER, and Twitter-roBERTa-base) and three large language models (LLMs: Qwen3-32B, GPT-OSS-120B, Llama-4-Maverick-17B) against six human raters across 100 tweets. We measured agreement using two statistical measures: Cohen's kappa for pairwise comparisons and Fleiss' kappa for multiple raters. Even among the human raters, our results showed only fair agreement, highlighting the subjectivity of sentiment analysis. Higher agreement was observed under the binary sentiment classification (negative vs. non-negative and positive vs. non-positive) than under the three-class classification across both humans and automated tools. The Twitter-roBERTa-base model showed the strongest alignment with human ratings, outperforming both bespoke sentiment tools and LLMs, particularly in distinguishing negative versus non-negative sentiment. LLMs showed substantial agreement among themselves and moderate to substantial alignment with humans, performing better in positive vs. non-positive classifications. Our findings underscore that domain-specific fine-tuning remains crucial for reliable social media sentiment analysis, and human-centered evaluation remains essential for establishing gold-standard labels.

Momentum

Related topics

Latest papers 31