Multi-Label Agreement: A Flexible Framework for Inter-Annotator Reliability

Tim Menzner; Jochen L. Leidner

cs.CLOct 27, 2026

Multi-Label Agreement: A Flexible Framework for Inter-Annotator Reliability

Authors: Tim Menzner, Jochen L. Leidner

Organizations: Center for Responsible Artificial Intelligence (CRAI), Coburg University of Applied Sciences

Abstract

Reliable evaluation of inter-annotator agreement (IAA) is important for assessing the consistency of annotation processes. While established IAA metrics such as Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha are commonly applied in single-label settings, multi-label tasks require additional decisions about how full label sets, joint selections, and joint omissions should be compared. In this work, we present Multi-Label Agreement (MLA), a flexible framework for evaluating agreement in multi-label annotation tasks. MLA preserves the full binary label vectors assigned to each item and uses average pairwise Hamming agreement, thereby accounting for both joint selections and joint omissions. Its pooled-marginal form is not a mathematically independent coefficient: it is a chance-disagreement-weighted average of label-wise Fleiss/Scott-type coefficients and is closely related to Krippendorff's alpha with normalized Hamming distance. The framework supports pooled-marginal, Gwet AC1, and fixed-baseline normalizations, yields item-level, label-level, and corpus-level results, and includes a conditional minimum-agreement diagnostic. Through a worked example and Monte Carlo simulations, we show how the resulting values depend on label prevalence, the declared label inventory, and the selected model of expected agreement. We also provide a Python implementation. MLA thus offers a transparent and interpretable framework for multi-label agreement analysis without claiming that one normalization is universally preferable.

Figures & tables

Sentence	Annotator 1	Annotator 2	Annotator 3
1	Faulty Analogy Bias [0,0,0,0,0,0,0,0,0,0, 0,0,0,1,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0]	Faulty Analogy Bias [0,0,0,0,0,0,0,0,0,0, 0,0,0,1,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0]	Faulty Analogy Bias [0,0,0,0,0,0,0,0,0,0, 0,0,0,1,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0]
2	Political Bias, Projection Bias [0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0, 0,1,1,0,0,0,0,0,0,0,0]	Speculation Bias, Unsubstantiated Claims Bias, Word Choice Bias [0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,1,0,1,0,1]	Unsubstantiated Claims Bias, Word Choice Bias [0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,1,0,1]
3	Ambiguous Attribution Bias, Cherry Picking Bias, Political Bias [0,1,0,0,1,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0, 0,1,0,0,0,0,0,0,0,0,0]	Anecdotal Evidence Bias [0,0,1,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0]	Ambiguous Attribution Bias [0,1,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0]
4	Intergroup Bias, Opinionated Bias, Political Bias, Word Choice Bias [0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,1,0,0,0, 1,1,0,0,0,0,0,0,0,0,1]	Ambiguous Attribution Bias, Intergroup Bias, Political Bias, Projection Bias, Unsubstantiated Claims Bias [0,1,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,1,0,0,0, 0,1,1,0,0,0,0,0,1,0,0]	Ambiguous Attribution Bias, Opinionated Bias, Projection Bias, Word Choice Bias [0,1,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0, 1,0,1,0,0,0,0,0,0,0,1]

Table 1: Assigned bias labels and corresponding 31-bit vectors by sentence and annotator.

Label universe	$A_{o}$	$A_{ml}^{(F)}$	$A_{ml}^{(ac1)}$	$A_{ml}^{(pb)}$	$A_{o,\min}^{(\mathbf{p})}$
All 31 labels	0.914	0.232	0.903	0.828	0.849
11 selected labels	0.758	0.232	0.646	0.515	0.576

Table 2: Effect of the declared label universe in the worked example. Removing never-selected labels leaves the pooled-marginal coefficient unchanged but changes the AC1 and fixed-baseline summaries.

label $j$	$p_{j}$	$A_{o,j}$	$A_{ml,j}^{(F)}$	$A_{ml,j}^{(ac1)}$	$A_{ml,j}^{(pb)}$
2 (Ambiguous Attribution Bias)	0.333	0.667	0.250	0.400	0.333
3 (Anecdotal Evidence Bias)	0.083	0.833	$-0.091$	0.803	0.667
5 (Cherry Picking Bias)	0.083	0.833	$-0.091$	0.803	0.667
14 (Faulty Analogy Bias)	0.250	1.000	1.000	1.000	1.000
17 (Intergroup Bias)	0.167	0.833	0.400	0.769	0.667
21 (Opinionated Bias)	0.167	0.833	0.400	0.769	0.667

Table 3: Label-wise agreement statistics for the 11 labels assigned at least once.

Scenario	$A_{ml}^{(F)}$	$A_{ml}^{(ac1)}$	$A_{ml}^{(pb)}$	MASI	Aug. $\kappa$	boot-F1	$A_{m}$
Perf. Agr. + High Prev.	1.000	1.000	1.000	1.000	0.042	1.000	1.000
Perf. Agr. + Mod. Prev.	1.000	1.000	1.000	1.000	0.171	1.000	1.000
Perf. Agr. + Low Prev.	1.000	1.000	1.000	1.000	0.381	1.000	1.000
High Agr. + High Prev.	0.524	0.711	0.641	0.870	0.030	0.523	0.426
High Agr. + Mod. Prev.	0.634	0.647	0.641	0.817	0.101	0.619	0.495
High Agr. + Low Prev.	0.523	0.710	0.640	0.693	0.164	0.487	0.424

Table 4: Representative agreement metrics (averaged over

n\!\in\!\{3,5,10\}

N\!\in\!\{20,50,100\}

k\!\in\!\{5,10,25\}

, 20 repeats, 100 bootstrap iterations for boot-F1)

Figure 1: Analytic prevalence sensitivity. Left: annotators draw independently from $\mathrm{Bernoulli}(p)$ . Right: annotators copy a latent label and independently flip it with probability $\varepsilon=0.1$ . The pooled-marginal coefficient is calibrated to the independent generator; AC1 and PABAK use different baselines.

Inter-Rater Reliability

Inter-rater reliability measures how consistently different people or models judge the same items, accounting for chance and task-specific disagreement.

See all 45 papers

Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation
Mar 6, 2026Joseph JamesInter-Rater Reliability Annotator Disagreement
Department of Computer Science, The University of Sheffield Sheffield, United Kingdom
Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework
Jun 19, 2026Emily Öhman, Anna KoufakouSoft-Label Learning Annotator Disagreement
Waseda University, Tokyo, Japan · Florida Gulf Coast University, Fort Myers FL, USA
AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling
Jul 20, 2026Ju Chen, Sijia Xu, Jun Feng +2Inter-Rater Reliability Noisy-Label Learning
Hohai University, China · University of New South Wales, Australia · Southeast University, China +1

Mar 6, 2026J/K move · Enter open · S save

Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation

Joseph James

Department of Computer Science, The University of Sheffield Sheffield, United Kingdom

Inter-Rater Reliability Annotator Disagreement

Open paper AskPDF arXiv

Human annotation remains the foundation of reliable and interpretable data in Natural Language Processing (NLP). As annotation and evaluation tasks continue to expand, from categorical labelling to segmentation, subjective judgment, and continuous rating, measuring agreement between annotators has become increasingly more complex. This paper outlines how inter-annotator agreement (IAA) has been conceptualised and applied across NLP and related disciplines, describing the assumptions and limitations of common approaches. We organise agreement measures by task type and discuss how factors such as label imbalance and missing data influence reliability estimates. In addition, we highlight best practices for clear and transparent reporting, including the use of confidence intervals and the analysis of disagreement patterns. The paper aims to serve as a guide for selecting and interpreting agreement measures, promoting more consistent and reproducible human annotation and evaluation in NLP.

Abstract

Inter-Rater Reliability

Explore similar work

Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation

Quality and Agreement in Multilabel Emotion Annotation: A Case Study and Evaluation Framework

AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modeling