Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation
Organizations: Department of Computer Science, The University of Sheffield Sheffield, United Kingdom
Abstract
Human annotation remains the foundation of reliable and interpretable data in Natural Language Processing (NLP). As annotation and evaluation tasks continue to expand, from categorical labelling to segmentation, subjective judgment, and continuous rating, measuring agreement between annotators has become increasingly more complex. This paper outlines how inter-annotator agreement (IAA) has been conceptualised and applied across NLP and related disciplines, describing the assumptions and limitations of common approaches. We organise agreement measures by task type and discuss how factors such as label imbalance and missing data influence reliability estimates. In addition, we highlight best practices for clear and transparent reporting, including the use of confidence intervals and the analysis of disagreement patterns. The paper aims to serve as a guide for selecting and interpreting agreement measures, promoting more consistent and reproducible human annotation and evaluation in NLP.
Figures & tables
| Metric | Data Type | Missing Data? | Number of Annotators? | Chance-Corrected? | Sensitive to Imbalance? | Limitations |
|---|---|---|---|---|---|---|
| Percentage Agreement | Nominal | – | Pairwise | – | High | Overestimates reliability. |
| Bennett’s S | Nominal | – | Two or more | ✓ | – | Assumes uniform label distribution. |
| Cohen’s | Nominal | – | Two only | ✓ | High | Unstable when class frequencies or rater biases differ. |
| Fleiss’ | Nominal | – | Three or more | ✓ | High | Requires equal numbers of ratings per item. |
| Krippendorff’s | Nominal / Ordinal / Interval / Ratio | ✓ | Two or more | ✓ | Moderate | Complex computation; interpretation can vary by distance metric. |
| Gwet’s AC1/AC2 | Nominal / Ordinal | ✓ | Two or more | ✓ | Low | Less common in literature; parameter selection affects results. |
Inter-Rater Reliability
Inter-rater reliability measures how consistently different people or models judge the same items, accounting for chance and task-specific disagreement.
See all 45 papers