Narrative extraction allows us to identify online hate narratives, supporting the construction of rigorous detection systems. Existing computational approaches, however, are limited in precision as they rely on semantic representations, which tend to capture only surface-level meaning. To detect more precise and interpretable narratives, we present an extraction pipeline that represents narratives as entity-evaluation pairs. Narratives are extracted using a Large Language Model (LLM) reasoning process that extends Aspect-Based Sentiment Analysis, identifying the aspect, classifying its judgement type as the basis for evaluation, and deriving the evaluation accordingly. Extracted narratives are then clustered using Leiden, following which clusters are resolved to an intended level of granularity through an LLM-guided refinement process. We illustrate this narrative pipeline with English Reddit comments from 2024 that criticize Taylor Swift, analyzing a representative cluster that exhibits hate speech patterns to demonstrate its interpretive value.
Figures & tables
Figure 1: Narrative Pipeline Logic.
Judgement Type
Definition
Ethics
Judgements about morality, integrity, honesty, sincerity, kindness, hypocrisy, humility, abuse of power or trust, or harm to others. Some examples include being genuine, dishonest, having double standards, lacking integrity, insincerity.
Capability
Judgements about competence, skill, intelligence, knowledge, ability to deliver, or producing high quality work. This would include characteristics related to the job of the person, e.g., a musician/singer who produces good quality music.
Appearance
Judgements which are about looks and physical appearance. Examples include being beautiful, sexy, handsome, unattractive.
Table 1: Judgement Type Definitions in Our Narrative Extraction Prompt
Feature
Definition
Aspect
The specific characteristic of the entity that is being discussed
Judgement
The underlying basis on which an evaluation is made
Evaluation
Attitudes towards the entity’s behaviour or actions
Sentiment
Sentiment of evaluation towards the entity
Summary
Summarize the evaluation relating to the entity’s aspect
Table 2: Summary of Extracted Outputs in Narrative Detection
Entity (%)
Aspect within accurate
Evaluation within accurate
entities (%)
aspect and judgement (%)
98.9
75.0
92.5
Table 3: Results for Entity, Aspect and Evaluation.
School of Software, Dalian University of Technology · School of Computer Science and Technology, Dalian University of Technology · School of Computing Technologies, RMIT University