A Decomposition-Based Approach for Evaluating and Analyzing Inter-Annotator Disagreement

Effi Levi; Ari Rappoport; Shaul R. Shenhav

cs.CLJun 11, 2022

A Decomposition-Based Approach for Evaluating and Analyzing Inter-Annotator Disagreement

Authors: Effi Levi, Ari Rappoport, Shaul R. Shenhav

Organizations: Department of Computer Science, The Hebrew University of Jerusalem · Department of Political Science, The Hebrew University of Jerusalem

Abstract

We propose a novel method to conceptually decompose an existing annotation into separate levels, allowing the analysis of inter-annotators disagreement in each level separately. We suggest two distinct strategies in order to actualize this approach: a theoretically-driven one, in which the researcher defines a decomposition based on prior knowledge of the annotation task, and an exploration-based one, in which many possible decompositions are inductively computed and presented to the researcher for interpretation and evaluation. Utilizing a recently constructed dataset for narrative analysis as our use-case, we apply each of the two strategies to demonstrate the potential of our approach in testing hypotheses regarding the sources of annotation disagreements, as well as revealing latent structures and relations within the annotation task. We conclude by suggesting how to extend and generalize our approach, as well as use it for other purposes.

Figures & tables

	# Sentences	Comp.	Res.	Suc.
$A_{1}$	2,209	1,083	448	312
$A_{2}$	1,135	504	262	137
$A_{3}$	1,074	569	296	168

Table 1: Annotation statistics: the number of annotated sentences (out of a total of 2,209) and the number of sentences annotated with each of the three narrative elements ( Complication , Resolution and Success ), for each of the three annotators.

PPA	Comp.	Res.	Suc.	Average
$A_{1}A_{2}$	88.7%	88.2%	92.9%	89.9%
$A_{1}A_{3}$	90.5%	86.8%	92.7%	90.0%
Average	89.6%	87.5%	92.8%	90.0%
$\bm{\kappa}$	Comp.	Res.	Suc.	Average
$A_{1}A_{2}$	0.77	0.65	0.68	0.70
$A_{1}A_{3}$	0.81	0.64	0.72	0.72

Table 2: Overall inter-annotator agreement

	Plot	Comp.	Res.	Suc.	Average
$A_{1}A_{2}$	0.71	0.91	0.77	0.80	0.83
$A_{1}A_{3}$	0.69	0.91	0.73	0.84	0.83
Average	0.70	0.91	0.75	0.82	0.83

Table 3: Inter-annotator agreement, factored into the two decomposed annotation levels: whether the sentence contains a narrative plot (“ Plot ”), and which narrative elements the sentence contains, given an agreement on Plot (“ Comp. ”, “ Res. ” and “ Suc. ”)

Encoding	Comp.	Res.	Suc.
000
001
010
011
100
101

Table 4: 3-bit Encoding for all the possible label combinations in NEAT

			2 nd Level
$\bm{S_{1}}$	$\bm{S_{2}}$	1 st Level	Complication	Resolution	Success	Average
101	000,001,010,011,100,110,111	0.35	0.80	0.66	0.72	0.72
010,101	000,001,011,100,110,111	0.42	0.83	0.74	0.76	0.78
010	000,001,011,100,101,110,111	0.45	0.83	0.73	0.74	0.77
010,111	000,001,011,100,101,110	0.48	0.84	0.74	0.76	0.78
010,101,111	000,001,011,100,110	0.48	0.84	0.73	0.80	0.79

Table 5: Top ten decompositions with the lowest first-level inter-annotator agreement scores.

\bm{S_{1}}

and

\bm{S_{2}}

: the decomposition; 1 st Level : inter-annotator agreement (

\kappa

) on

C_{S_{1}}

; Complication , Resolution and Success : inter-annotator agreement (

\kappa

) on each element given an agreement on

C_{S_{1}}

; Average : second-level inter-annotator agreement, averaged over the three narrative elements. All the statistics were averaged over the two annotator pairs.

			2 nd Level
$\bm{S_{1}}$	$\bm{S_{2}}$	1 st Level	Complication	Resolution	Success	Average
000,011,101,110	001,010,100,111	0.55	0.94	0.88	0.90	0.91
001,010,100	000,011,101,110,111	0.57	0.94	0.89	0.87	0.90
000,101,110	001,010,011,100,111	0.57	0.94	0.85	0.89	0.89
000,010,101,110	001,011,100,111	0.64	0.93	0.79	0.97	0.89
000,011,110,111	001,010,100,101	0.57	0.94	0.90	0.85	0.89

Table 6: Top ten decompositions with highest second-level inter-annotator agreement scores.

\bm{S_{1}}

and

\bm{S_{2}}

: the decomposition; 1 st Level : inter-annotator agreement (

\kappa

) on

C_{S_{1}}

; Complication , Resolution and Success : inter-annotator agreement (

\kappa

) on each element given an agreement on

C_{S_{1}}

; Average : second-level inter-annotator agreement, averaged over the three narrative elements. All the statistics were averaged over the two annotator pairs.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

			2 nd Level
$\bm{S_{1}}$	$\bm{S_{2}}$	1 st Level	Complication	Resolution	Success	Avergae
000	001,010,011,100,101,110,111	0.70	0.91	0.75	0.82	0.83
001	000,010,011,100,101,110,111	0.60	0.80	0.68	0.81	0.76
010	000,001,011,100,101,110,111	0.45	0.83	0.73	0.74	0.77
011	000,001,010,100,101,110,111	0.57	0.80	0.67	0.75	0.74
100	000,001,010,011,101,110,111	0.74	0.92	0.78	0.73	0.81

Appendix

Table 7: A full summary of the results of the procedure described in Section 4.1 .

\bm{S_{1}}

and

\bm{S_{2}}

: the decomposition; 1 st Level : inter-annotator agreement (

\kappa

) on

C_{S_{1}}

; Complication , Resolution and Success : inter-annotator agreement (

\kappa

) on each element given an agreement on

C_{S_{1}}

; Average : second-level inter-annotator agreement, averaged over the three narrative elements. All the statistics were averaged over the two annotator pairs.

Annotator Disagreement

Annotator disagreement studies why human labels differ, how to measure and model that variation, and how it should shape AI training and evaluation.

See all 91 papers

Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP
Jan 14, 2026Yinuo Xu, David JurgensAnnotator Disagreement
University of Michigan
Structured Disagreement in Health-Literacy Annotation: Epistemic Stability, Conceptual Difficulty, and Agreement-Stratified Inference
Apr 21, 2026Olga Kellert, Sriya Kondury, Candice Koo +2Inter-Rater Reliability Annotator Disagreement
Arizona State University
Task Decomposition for Efficient Annotation
Jun 23, 2026Nupoor Gandhi, Emma StrubellHuman-in-the-Loop Annotation Task Decomposition
Carnegie Mellon University

Jan 14, 2026J/K move · Enter open · S save

Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP

Yinuo Xu, David Jurgens

University of Michigan

Annotator Disagreement

Open paper AskPDF arXiv

Annotator disagreement is widespread in NLP, particularly for subjective and ambiguous tasks such as toxicity detection and stance analysis. While early approaches treated disagreement as noise to be removed, recent work increasingly models it as a meaningful signal reflecting variation in interpretation and perspective. This survey provides a unified view of disagreement-aware NLP methods. We first present a domain-agnostic taxonomy of the sources of disagreement spanning data, task, and annotator factors. We then synthesize modeling approaches using a common framework defined by prediction targets and pooling structure, highlighting a shift from consensus learning toward explicitly modeling disagreement, and toward capturing structured relationships among annotators. We review evaluation metrics for both predictive performance and annotator behavior, and noting that most fairness evaluations remain descriptive rather than normative. We conclude by identifying open challenges and future directions, including integrating multiple sources of variation, developing disagreement-aware interpretability frameworks, and grappling with the practical tradeoffs of perspectivist modeling.

Abstract

Annotator Disagreement

Explore similar work

Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP

Structured Disagreement in Health-Literacy Annotation: Epistemic Stability, Conceptual Difficulty, and Agreement-Stratified Inference

Task Decomposition for Efficient Annotation