From Variance to Invariance: Qualitative Content Analysis for Narrative Graph Annotation

Junbo Huang; Max Weinig; Ulrich Fritsche; Ricardo Usbeck

cs.CLMar 2, 2026

From Variance to Invariance: Qualitative Content Analysis for Narrative Graph Annotation

Authors: Junbo Huang, Max Weinig, Ulrich Fritsche, Ricardo Usbeck

Organizations: Department of Computer Science, University of Hamburg · Department of Socioeconomics, University of Hamburg · AI and Explanability Group, Leuphana University of Lüneburg

Abstract

Narratives in news discourse play a critical role in shaping public understanding of economic events, such as inflation. Annotating and evaluating these narratives in a structured manner remains a key challenge for Natural Language Processing (NLP). In this work, we introduce a narrative graph annotation framework that integrates principles from qualitative content analysis (QCA) to prioritize annotation quality by reducing annotation errors. We present a dataset of inflation narratives annotated as directed acyclic graphs (DAGs), where nodes represent events and edges encode causal relations. To evaluate annotation quality, we employed a $6\times3$ factorial experimental design to examine the effects of narrative representation (six levels) and distance metric type (three levels) on inter-annotator agreement (Krippendorrf's $α$ ), capturing the presence of human label variation (HLV) in narrative interpretations. Our analysis shows that (1) lenient metrics (overlap-based distance) overestimate reliability, and (2) locally-constrained representations (e.g., one-hop neighbors) reduce annotation variability. Our annotation and implementation of graph-based Krippendorrf's $α$ are open-sourced. The annotation framework and evaluation results provide practical guidance for NLP research on graph-based narrative annotation under HLV.

Figures & tables

Figure 1: Example of narrative graph annotation under human label variation.

Figure 2: Flowchart of iterative QCA research procedure

Panel of Figure 3 — Figure 3: Visual example of six types of narrative representations. (a-c) represent the categorical representation of narratives, where (a) includes all annotated events causing and affected by Inflation , (b) includes only events directly causing Inflation and (c) includes a set of relations ( Increases or Decreases ) from (e). (d-f) represent the graph-based representation of narratives, where (d) includes all annotated events and their relations from and to Inflation , (e) includes only annotated events directly causing Inflation , as well as their relations, and (f) includes only annotated events directly and indirectly causing Inflation , as well as their relations.

Narrative Representation		Lenient	Moderate	Strict	$\Delta_{lm}$	$\Delta_{ms}$	$\Delta_{ls}$	$\mu(\|\cdot\|)$	$sdt(\|\cdot\|)$
		Distance Metric Type			$\Delta_{lm}$	$\Delta_{ms}$	$\Delta_{ls}$	$\mu(\|\cdot\|)$	$sdt(\|\cdot\|)$
Category	All Events	0.868	0.481	0.244	0.387	0.237	0.624	2.764	1.809
	Adj. Events	0.717	0.424	0.285	0.293	0.139	0.432	2.112	1.452
	Relations	0.642	0.513	0.483	0.129	0.030	0.159	1.220	0.415
Graph	Full Story	0.678	0.320	0.127	0.358	0.193	0.551	2.783	2.070
	Adj. Story	0.702	0.441	0.202	0.261	0.239	0.500	2.143	1.567

Table 1: Effects of narrative representation and distance metric type on Krippendorff’s

\alpha

for narrative extraction task.

\Delta

denotes the difference in

\alpha

between lenient, moderate and strict metrics.

|\cdot|

indicates the average set cardinality (i.e., the number of events, relations, or triples per annotation).

\mu(\cdot)

and

std(\cdot)

denote mean and standard deviation operation.

Figure 4: Distribution of documents across publication years. The sampling strategy focuses on inflation-peak years: 1990, 1991, 1999, 2001, 2007, 2008, 2021, 2022 and 2023.

Figure 5: Average word count of documents across publication years.

Figure 6: Proportion of Articles with Full Annotators Agreement per Label.

Category	Subcategory	Explanation
Demand	Government Spending	Adjustments in government spending (e.g., stimulus payments)
	Monetary Policy	Monetary policy by the Federal Reserve or other Central Banks
	Pent-up Demand	Reopening of the economy and the associated higher incomes, new spending opportunities, and optimism about the future
	Demand Shift	Shift of demand across sectors (particularly increases in durables)
	Demand (Residuals)	Increase in demand that cannot be attributed to the other channels
Supply	Supply Chain Issues	Disruption of global supply chains

Table 2: Events and Explanations in the Category System.

Annotator Disagreement

Annotator disagreement studies why human labels differ, how to measure and model that variation, and how it should shape AI training and evaluation.

See all 91 papers

From Repetition to Recognition: Inductive Discovery of Disinformation Narratives
Sep 10, 2026Max Upravitelev, Veronika Solopova, Jing Yang +4Unsupervised Clustering
Technische Universität Berlin · German Research Center for Artificial Intelligence (DFKI) · BIFOLD – Berlin Institute for the Foundations of Learning and Data +2
A Decomposition-Based Approach for Evaluating and Analyzing Inter-Annotator Disagreement
Jun 11, 2022Effi Levi, Ari Rappoport, Shaul R. ShenhavAnnotator Disagreement
Department of Computer Science, The Hebrew University of Jerusalem · Department of Political Science, The Hebrew University of Jerusalem
Developing a Guideline for the Labovian-Structural Analysis of Oral Narratives in Japanese
Mar 31, 2026Amane Watahiki, Tomoki Doi, Akari Kikuchi +5
The University of Tokyo, Tokyo, Japan · Riken, Tokyo, Japan · Kobe University, Hyogo, Japan +2

Sep 10, 2026J/K move · Enter open · S save

From Repetition to Recognition: Inductive Discovery of Disinformation Narratives

Max Upravitelev, Veronika Solopova, Jing Yang, Charlott Jakob+3

Technische Universität Berlin · German Research Center for Artificial Intelligence (DFKI) · BIFOLD – Berlin Institute for the Foundations of Learning and Data+2

Unsupervised Clustering

Open paper AskPDF arXiv

In disinformation datasets, narratives are often understood as recurring interpretive patterns that group texts under narrative labels. Recent work formalized narrative mining as inductively inferring narrative labels from corpora, but its evaluation stays tied to predefined taxonomies, a closed-world setting that cannot capture narratives absent from the reference labels. We introduce a three-tier evaluation framework for unsupervised narrative label generation: recovery (against a corpus's own taxonomy), mining (against external label sets), and discovery (without predefined labels). Applying it, we compare clustering-based and graph-community-based pipelines across seven disinformation datasets, with human validation of discovery on two. The two families are complementary under automated metrics, but in a corpus with two prominent topics, clustering can reduce one topic to 2% of generated labels while graph-based pipelines stay balanced. Discovery validation also reveals many singletons (narrative labels derived from single claims, 30-62% of graph outputs), which clustering cannot produce. Annotators confirm many as recognizable disinformation narratives, suggesting that in open-world discovery the repetition assumed by narrative mining may be recognized outside the corpus, not within it. We release human-validated narrative candidate labels for the Climate Obstruction and PolyNarrative datasets to support taxonomy development and dataset extension.

Abstract

Annotator Disagreement

Explore similar work

From Repetition to Recognition: Inductive Discovery of Disinformation Narratives

A Decomposition-Based Approach for Evaluating and Analyzing Inter-Annotator Disagreement

Developing a Guideline for the Labovian-Structural Analysis of Oral Narratives in Japanese