Inter-Annotator Agreement

Ask about this topic

Momentum

9 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 58

All topics
CardsList

  1. Making COMET Comparable Across Scripts: Diagnosis and Correction of Tokeniser-Induced Script Bias in Indic MT Evaluation

    Oct 6, 2026G. L. John Salvin, Swapnil HingmireMachine Translation QualityUnicode U+2014

    Indian Institute of Technology Palakkad · Mehta Family School of Data Science and Artificial Intelligence Indian Institute of Technology Palakkad, Kerala, India

  2. COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal

    Oct 5, 2026Baihan LinPsychometric PropertiesInter-Annotator Agreement

    Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA +2

  3. The Review Lottery: Calibrating an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)

    Oct 5, 2026Feilian HuangPeer ReviewInter-Annotator Agreement

    Independent researcher

  4. LLM-Assisted Discovery of Typed Semantic Links for Ontology Network Construction

    Oct 1, 2026Nouha Hayouni, Sheeba Samuel, Alsayed AlgergawyOntologySemantic Relationships

    Chair of Data and Knowledge Engineering, University of Passau, Germany · Distributed and Self-organizing Systems, Chemnitz University of Technology, Germany

  5. Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression Severity

    Sep 30, 2026Xinkai ChenDepressionSeverity

    Independent Researcher

  6. Decomposing Error and Style in Automated Clinical Coding

    Sep 21, 2026Han-Chin Shing, Jack Moriarty, Ryan Ware +6International Classification Of Disease CodingCode Quality

    Amazon

  7. An Analysis of Training-Free Self-Reported Confidence in Language Models

    Sep 17, 2026Lukas Meyer, Sofia Rossi, Wei Chen +2Confidence CalibrationLarge Language Model Reliability

    DreamAI

  8. Reproducibility is not construct validity: LLM measurement of institutionally situated communication

    Sep 17, 2026Veronika Batzdorfer, Carlo Romano Marcello Alessandro SantagiustinaInter-Annotator AgreementLarge Language Model Evaluation

    KIT · Department of Sociology and Computational Sociology, Karlsruhe Institute of Technology (KIT), Douglasstr. 24, 76133 Karlsruhe, Germany · ALMAnaCH, médialab, Sciences Po +2

  9. The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability

    Sep 17, 2026Michael Hernandez, Tian ZhaoPass-Rate EvaluationCode Generation

    University of Wisconsin–Milwaukee

  10. What an odour descriptor corpus can and cannot measure: valence, attenuation, and the ceiling of the public record

    Sep 14, 2026Stylianos Kampakis, Fabio RovaiSmellsInter-Annotator Agreement

    Tesseract Academy, London, UK

  11. Vision-Language Models for Criterion-Level Grading of Handwritten Examinations in Outcome-Based Education

    Sep 13, 2026Asif Hasan Tonmoy, Saad Ahmed, Md Khalid Syfullah +1GradingInter-Annotator Agreement

  12. Rubric-Aligned Disentangled Evaluation of Human Simultaneous Interpreting

    Sep 11, 2026Ziyu Zhang, Satoshi NakamuraInter-Annotator AgreementMulti-Dimensional Evaluation

    School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Loop Area Institute, China

  13. RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching

    Sep 1, 2026Charles Corbière, Léo Machado, Aubin Charley +3Radiology Report GenerationInter-Annotator Agreement

    Raidium, Paris, France · Department of Radiology, Fondation Ophtalmologique Adolphe de Rothschild, Paris, France · Service de Radiologie Interventionnelle, Département d’Anesthésie, Chirurgie et Imagerie Interventionnelle (DACI), Gustave Roussy, Villejuif, France

  14. Augmenting Interviewer Judgments of Patient Experience with Automatic Language Analysis

    Aug 31, 2026Aowen Shi, Michal Balazia, Danilo Postin +4InterviewsClinician Trust

    INRIA Universit´e Cˆote d’Azur, Valbonne, France · Carl von Ossietzky University of Oldenburg, Germany · German Research Center for Artificial Intelligence, Saarbrücken, Germany +1

  15. DICS: Exploring Data Intrinsic Consistency for Visual Instruction Selection

    Aug 31, 2026Yuyang Hong, Jinhui Guo, Jiaqi Gu +6Vision-Language AlignmentInstruction

    School of Artificial Intelligence, University of Chinese Academy of Sciences · 2MAIS, Institute of Automation, Chinese Academy of Sciences · 3Alibaba Token Hub, Alibaba Group

  16. LiNC: Lightweight Noise Correction via Per-Sample Trust and Gaussian Mixture Modeling

    Aug 4, 2026Abhishek Moturu, Babak Taati, Anna GoldenbergNoisy LabelsMedical Imaging Datasets

    Department of Computer Science University of Toronto The Hospital for Sick Children UHN KITE Research Institute T-CAIREM Vector Institute · Department of Computer Science Institute of Biomedical Engineering University of Toronto Rehabilitation Sciences Institute UHN KITE Research Institute Vector Institute · Department of Computer Science Department of Laboratory Medicine and Pathobiology University of Toronto The Hospital for Sick Children T-CAIREM Vector Institute

  17. Consensus Measures for Unstructured Biomedical Text Annotations

    Aug 4, 2026Pascal Wullschleger, Christian Kreis, Martin A. Walter +2Inter-Annotator AgreementMedical Ontology

    Hamilton Institute, Maynooth University, Ireland · University of Lucerne, Switzerland · Lucerne School of Computer Science and IT, Switzerland

  18. Language Models Agree With Each Other, Not With Readers

    Jul 31, 2026Kazuki Nakayashiki, Keisuke WatanabeInter-Annotator AgreementHuman Judgment

    Glasp Inc.

  19. Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth

    Jul 30, 2026Alex Liu, Lief Esbenshade, Michael Xiao +4Llm-As-A-JudgeQualitative Research

    University of Washington · Colleague AI/ · Colleague AI

  20. SciFigAlign: Scoring Scientific Figures by Fine-tuned Alignment of Visuals with Manuscript Evidence

    Jul 29, 2026Chuanzhi Xu, Zihan Deng, Huiqi Liang +4Scientific FigurePeer Review

    The University of Sydney · The University of Hong Kong · Indiana University +2

  21. Quality Action Assurance: Multimodal Verification of Examiner Claims in VR OSCEs

    Jul 21, 2026Harry Rogers, Sally Shiels, Ashley Tomlinson +5Reference-Guided Multimodal In-Context VerificationClinician Trust

    Department of Engineering Science, University of Oxford, Oxford, UK · Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK · Department for Continuing Education, University of Oxford, Oxford, UK

  22. Graph Feedback Controls Consensus and Clique Formation in Open-Weight Language-Model Populations

    Jul 13, 2026Samer Saab, Chaouki AbdallahMulti-Agent Large Language Model SystemsInter-Annotator Agreement

    Lebanese American University Byblos, Lebanon · Electrical and Computer Engineering Department Lebanese American University Byblos, Lebanon

  23. JobHop v2: A Large-Scale Career Trajectory Dataset from Unstructured Resumes

    Jul 13, 2026Iman Johary, Guillaume Bied, Alexandru C. Mara +1JobsContinuation

    AIDA-IDLab, Department of Electronics and Information Systems, Ghent University, Ghent, Belgium

  24. Dilemmadata: On the Interoperability of Heterogeneous Roman Numeral Datasets

    Jun 30, 2026Johannes Hentschel, Emmanouil Karystinaios, Gerhard Widmer +1HarmonizationInter-Annotator Agreement

    1Linz Computational Music Analysis Research Group (LCMA), Anton Bruckner University, Austria · Institute of Computational Perception (CP), Johannes Kepler University Linz, Austria

  25. Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support

    Jun 29, 2026Mizanur Rahman, Abeer Badawi, Elahe Rahimi +4Mental HealthJudges

    1York University, Ontario, Canada · 2Vector Institute, Ontario, Canada · 3Connected Minds, Ontario, Canada +1

  26. Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs

    Jun 26, 2026Manuel PitaCorrectnessInter-Annotator Agreement

    Artificial Intelligence, Social Interaction and Complexity Laboratory CICANT, Universidade Lusófona 376, Campo Grande, 1700-097 Lisbon, Portugal

  27. Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert Adjudication

    Jun 25, 2026Jerome Marston, Tino Kreutzer, Salomé Garnier +3Humanitarian ReportsInter-Annotator Agreement

    Kobo Inc · McGill University · Harvard University

  28. Interpretable Probabilistic Medical Image Segmentation via Gaussian Process with Explicit Modelling of Annotation Bias and Variability

    Jun 22, 2026Qi Li, Yuliang Huang, Shaheer U. Saeed +7Semi-Supervised Medical Image SegmentationInter-Annotator Agreement

    UCL Hawkes Institute, Department of Medical Physics and Biomedical Engineering, University College London, London, U.K. · Centre for Bioengineering, Digital Environment Research Institute, School of Engineering and Materials Science, Queen Mary University of London, London, U.K. · Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, U.K. +4

  29. CAPRA: Scaling Feedback on Software Architecture Deliverables with a Multi-Agent LLM System

    Jun 17, 2026Marco Becattini, Niccolò Caselli, Matteo Minin +2Code QualityFeedback

    Department of Information Engineering, University of Florence, Florence, Italy

  30. A Two-Stage Statistical Framework for Evaluating Associative Interference in Large Language Models

    Jun 12, 2026Achraf Cohen, Andrew KincaidLarge Language Model BiasInter-Annotator Agreement

    Department of Mathematics and Statistics University of West Florida

  31. Creative Integration: A Decidable Criterion of Creativity

    Jun 11, 2026Yoshinori NomuraCreativityIntelligence

    Mirage Mountain Technologies

  32. YTClickbait21K: Human-Annotated Multimodal Dataset for YouTube Clickbait Detection Across Diverse Channels and Content Categories

    Jun 10, 2026Md. Minhazul Islam, Md. Tanbeer Jubaer, Amith Khandakar +5Multimodal DatasetContent Moderation

    Department of Computer Science and Engineering, Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh · Department of Electrical Engineering, Qatar University, Doha, Qatar · Department of Civil and Environmental Engineering, Qatar University, Doha, Qatar +1

  33. Contemporary AI lacks the imagination to diverge or negate in science

    Jun 6, 2026Honglin Bao, Siyang Wu, Xiao Liu +3Scientific DiscoveryArtificial Intelligence Evaluation

    Data Science Institute, University of Chicago · Knowledge Lab, University of Chicago

  34. Automatic Extraction of Structured Information from Brain MRI Reports Using an Open-Weight Large Language Model

    Jun 5, 2026Kaouther Mouheb, Amos Pomp, Antoine Manenti +9Radiology Report GenerationMedical Vision-Language Models

    Department of Radiology & Nuclear Medicine, Erasmus MC, Rotterdam, the Netherlands · Department of Epidemiology, Erasmus MC, Rotterdam, the Netherlands · Department of Electrical and Electronics Engineering, ENSEEIHT, Toulouse, France +3

  35. RadSEM: A Finding-by-Finding Metric for Clinical Consistency in Radiology Reports

    Jun 3, 2026Zhenhong Yang, Zhuoyun Liu, Jintao Fei +4Radiology Report GenerationReport

    JDH Algo, JD Health International Inc., China · Department of Big Data in Health Science, The First Affiliated Hospital of Wenzhou Medical University, China · Zhejiang Engineering Research Center for Hospital Emergency and Process Digitization, Wenzhou, China +1

  36. ACAT: A Collaborative Platform for Efficient Aspect-Based Sentiment Dataset Annotation

    Jun 2, 2026Ana-Maria Luisa Mocanu, Ciprian-Octavian Truica, Elena-Simona ApostolSentimentInter-Annotator Agreement

    1National University of Science and Technology POLITEHNICA Bucharest, Romania · 2Academy of Romanian Scientists, Ilfov 3, Bucharest, 050044, Romania

  37. PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges

    May 29, 2026Swastik Roy, Rajkumar Pujari, Tharindu Kumarage +5Llm-As-A-JudgeJudges

    Amazon AGI

  38. Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm

    May 27, 2026Yuming, Huang, Yao Liu +3SubjectivityMulti-Dimensional Evaluation

    Rapheal · Cylingo team

  39. Temporal Simultaneity Predicts Annotation Quality in Sentiment Corpora

    May 26, 2026Idris Abdulmumin, Mokgadi Penelope Matloga, Tadesse Destaw Belay +5SentimentInter-Annotator Agreement

    Data Science for Social Impact, University of Pretoria · Instituto Politécnico Nacional · Department of African Languages, University of Pretoria +1

  40. A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays

    May 25, 2026Pawitsapak Akarajaradwong, Wuttikrai Lertprasertphakorn, Chompakorn Chaksangchaichot +1Llm-As-A-JudgeLarge Language Model Judges

    VISAI AI · Independent Researcher · Vidyasirimedhi Institute of Science and Technology

  41. Fine-Tuning Vision-Language Models for Understanding Current Damage and Scoring Priority with Quality Guard Agent

    May 24, 2026Takato YasunoRecent Vision-Language ModelsStructural Damage

  42. Cohesion-6K: An Arabic Dataset for Analyzing Social Cohesion and Conflict in Online Discourse

    May 21, 2026Aisha Ali Al-Athba, Wajdi ZaghouaniArabicComputational Social Science

    Hamad Bin Khalifa University, Northwestern University in Qatar Doha, Qatar

  43. CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs

    May 15, 2026Kamil Guttmann, Zofia Fraś, Artur Nowakowski +1Machine Translation QualityMachine Translation

    Laniqo, Pozna´n, Poland · Faculty of Mathematics and Computer Science, Adam Mickiewicz University, Pozna´n, Poland

  44. Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation

    May 13, 2026Abdelrahman Eldesokey, Merey Ramazanova, Ahmad Sait +4Inter-Annotator Agreement

    King Abdullah University of Science and Technology (KAUST) · Thuwal, Saudi Arabia

  45. Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?

    May 12, 2026Yichen Feng, Yuetai Li, Chunjiang Liu +14AestheticsEvaluation Benchmarks

  46. Nürnberg NLP at PsyDefDetect: Multi-Axis Voter Ensembles for Psychological Defence Mechanism Classification

    May 8, 2026Philipp Steigerwald, Eric Rudolph, Jens AlbrechtPsychologyNatural Language Processing

    Technische Hochschule Nürnberg Georg Simon Ohm

  47. Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement

    May 7, 2026Jessica Huynh, Alfredo Gomez, Athiya Deviyani +3RubricsInter-Annotator Agreement

    Carnegie Mellon University · Google Research

  48. The First Token Knows: Single-Decode Confidence for Hallucination Detection

    May 6, 2026Mina GabrielSemantic EntropySelf-Consistency

    Department of Computer and Information Sciences Temple University Philadelphia, PA 19122, USA

  49. Raising the Ceiling: Better Empirical Fixation Densities for Saliency Benchmarking

    May 5, 2026Susmit Agrawal, Jannis Hollman, Matthias KümmererSaliencyDensity Ratio Estimation

    Tübingen AI Center, University of Tübingen, Germany

  50. Multi-Rater Calibrated Segmentation Models

    May 4, 2026Meritxell Riera-Marín, Javier García López, Júlia Rodríguez-Comas +2Semi-Supervised Medical Image SegmentationInter-Annotator Agreement

    aSycai Technologies SL, Scientific and Technical Department, Barcelona, Spain · bBCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain · cInstitució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, Spain +1

  51. Case-Specific Rubrics for Clinical AI Evaluation: Methodology, Validation, and LLM-Clinician Agreement Across 823 Encounters

    Apr 27, 2026Aaryan Shah, Andrew Hines, Alexia Downs +6Clinician TrustArtificial Intelligence Evaluation

    1Canvas Medical, San Francisco, CA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · 3XPC (X Primary Care) +2

  52. Phase-Separated Complex Hilbert PCA on Markerless 3D Pose Estimation Data: A Global Phase Network and Its Extension to a Continuous Field on the Body Surface

    Apr 27, 2026Hiromitsu Goto, Tao Tao, Zheng-Lin ChiaPrincipal Component AnalysisKinematic Chain

    Faculty of Information Engineering, Kanazawa Gakuin University, Kanazawa, Japan

  53. A Benchmark Suite of Reddit-Derived Datasets for Mental Health Detection

    Apr 25, 2026Khalid Hasan, Jamil SaquerMental HealthDepression

    Missouri State University, Springfield, MO, USA

  54. MedSkillAudit: A Domain-Specific Audit Framework for Medical Research Agent Skills

    Apr 22, 2026Yingyong Hou, Xinyuan Lao, Huimei Wang +10Inter-Annotator AgreementClinician Trust

    Department of Pathology, Zhongshan Hospital, Fudan University, Shanghai, China · 1AIPOCH PTE. LTD., Singapore

  55. IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text

    Apr 21, 2026Rajveer Singh PallFinance BenchmarksLegal Natural Language Processing Benchmarks

    Gyan Ganga Institute of Technology and Sciences, Jabalpur, India

  56. Multilingual Training and Evaluation Resources for Vision-Language Models

    Apr 20, 2026Daniela Baiamonte, Elena Fano, Matteo Gabburo +3Vision-Language Foundation ModelsInter-Annotator Agreement

    Aithlas

  57. Sentiment Analysis of German Sign Language Fairy Tales

    Apr 17, 2026Fabrizio Nunnari, Siddhant Jain, Patrick GebhardSentimentSign Language Translation

    German Research Center for Artificial Intelligence (DFKI) Saarland Informatics Campus D3 2, 66123 Saarbrücken, Germany

  58. The Hitchhikers Guide to Rubric Quality Understanding and Enrichment

    Apr 1, 2026Ankit Aich, Zhengyang Qi, Charles Dickens +7Rubric-Based ScoringRubrics

    Snorkel AI · University of Wisconsin–Madison