A Multi-Stage Agentic Framework for Effective Counter-Narrative Generation and Refinement

Carmel Kronfeld; Sharva Gogawale; Tetsuro Kobayashi; Irad Ben-Gal

cs.CLSep 12, 2026

A Multi-Stage Agentic Framework for Effective Counter-Narrative Generation and Refinement

Authors: Carmel Kronfeld, Sharva Gogawale, Tetsuro Kobayashi, Irad Ben-Gal

Organizations: School of Industrial & Intelligent Systems Engineering, Tel Aviv University · School of Electrical and Computer Engineering, Tel Aviv University · School of Political Science and Economics, Waseda University

Abstract

The rapid diffusion of hate speech and misinformation on social networks challenges democratic societies, since direct suppression efforts may deepen polarization, fuel public distrusts, and strengthen extremist narratives. LLM-driven counter-narratives (CNs) offer a promising way to reduce those risks, yet their effectiveness depends on rhetorical and stylistic choices that remain poorly understood. We present a multi-stage agent-based framework for generating, refining, and evaluating CNs, applied to pro-Russian hate and misinformation narratives on the war with Ukraine and adaptable to other domains. A pilot experiment with human evaluators identifies effective technique style pairings, such as repetition with emotional framing enhancing persuasiveness. Building on these insights, we introduce a multi-agent refinement process that iteratively improves CNs for persuasiveness, emotional engagement, and shareability. After human validation confirmed improvement, an automated safety analysis shows that our refined CNs match or improve on expert-written counterspeech. A simulated experiment then shows that they reduce the perceived strength of pro-Russian narratives and consistently outperform a vanilla LLM baseline, highlighting a pathway toward scalable, narrative-specific interventions against hate speech and misinformation. Code and data accompanying this work are publicly available at https://github.com/carmelkron/inlg2026-counter-narratives.

Figures & tables

Figure 1: The proposed multi-stage framework.

KPI	Rhetorical technique	Writing style
Persuasiveness	Repetition	Emotional
Engagement Emotional	Fear mongering	Empathic
Shareability	Card stacking	Metaphorical

Table 1: Best combination of rhetorical technique and writing style per KPI

Panel of Figure 2 — Figure 2: Refinement curves for Group 1 (NATO / Western Aggression & Broken Promises) across the three KPIs. Black lines represent group means; blue bars indicate $\pm 1$ standard deviation.

Comparison	p-value (Holm-corrected)
Kruskal-Wallis (overall)	$p=5.47\times 10^{-13}$
3 vs. 2	$p=3.93\times 10^{-10}$
3 vs. 1	$p=2.66\times 10^{-9}$
6 vs. 2	$p=6.82\times 10^{-5}$
6 vs. 1	$p=1.79\times 10^{-4}$
3 vs. 4	$p=0.036$

Table 2: Statistical testing of peak persuasiveness scores across meta-narrative groups.

Figure 3: Peak persuasiveness scores achieved, grouped by meta-narrative. Each box represents the distribution of peak values across all narratives in that group.

Figure 4: Improvement deltas for persuasiveness across narrative groups. Each box represents the distribution of deltas across claims in that group.

KPI	Accuracy	Cohen’s $\kappa$
Persuasiveness	81.87%	0.638
Emotional Engagement	80.63%	0.606
Shareability	83.75%	0.675

Table 3: High-difference pair alignment with gold labels (combined,

n=160

per KPI).

Source	KPI	Tox.	Ins.	Thr.	Off.
CONAN		$.076\pm.122$	$.004\pm.013$	$.001\pm.001$	$.745\pm.103$
MT-CONAN		$.062\pm.141$	$.006\pm.035$	$.001\pm.003$	$.745\pm.127$
Pre-refin.	P	$.130\pm.172$	$.005\pm.016$	$.004\pm.006$	$.687\pm.135$
	EE	$.132\pm.120$	$.003\pm.003$	$.005\pm.006$	$.691\pm.079$
	S	$.012\pm.020$	$.001\pm.001$	$.000\pm.000$	$.714\pm.121$
Refined	P	$.060\pm.104$	$.002\pm.004$	$.001\pm.003$	$.720\pm.108$

Table 4: Safety metric scores (mean

\pm

std).

N=180

for baselines;

N=60

per KPI for generated CNs. KPI: P = Persuasiveness; EE = Emotional Engagement; S = Shareability. Tox. = Toxicity; Ins. = Insult; Thr. = Threat; Off. = Offensiveness.

	Control	Vanilla	Refined	KW p-value
Persuasiveness	91.38	56.95	49.82
Emotional
Engagement	87.50	56.07	33.27
Shareability	87.80	58.93	39.18

Table 5: Average KPI scores across groups and Kruskal-Wallis (KW) results. Significance codes: *

p<0.05

, **

p<0.01

, ***

p<0.001

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Evaluator	Rank-biserial $r$
	(0 = no assoc., 1 = perfect)
Ev1	0.774
Ev2	0.760
Ev3	0.784
Ev4	0.783
Ev5	0.652

Appendix

Table 6: Rank-biserial correlation per evaluator (mean over 3 base claims and 3 KPIs).

KPI	Mean $W$
Persuasiveness	0.376
Emotional Eng.	0.374
Shareability	0.237
Overall	0.329

Appendix

Table 7: Kendall’s

W

(on BT-implied rankings) per KPI, averaged over 3 base claims.

Text Generation

Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.

See all 423 papers

Assisted Counterspeech Writing at the Crossroads of Hate Speech and Misinformation
May 21, 2026Genoveffa Martone, Helena Bonaldi, Marco GueriniKnowledge Augmentation for Language Models Text Generation
Fondazione Bruno Kessler, Italy · Università Cattolica del Sacro Cuore, Italy
CATCH-ME if you RAG: a dataset of Contextually Annotated multi-Turn Counterspeech against Hate and Misinformation Exchanges
Jun 18, 2026Helena Bonaldi, Genoveffa Martone, Marco GueriniContent Moderation Text Generation
Fondazione Bruno Kessler, Italy · Università Cattolica del Sacro Cuore, Italy
Counter with Evidence! A Multi-Agent Memory Efficient Reasoning Framework for Hate Category Informed Counterspeech Generation
Aug 24, 2026Sujoy Nath, Aswini Kumar, Tanmoy ChakrabortyText Generation Evidence-Grounded Generation
Indian Institute of Technology Delhi, India

May 21, 2026J/K move · Enter open · S save

Assisted Counterspeech Writing at the Crossroads of Hate Speech and Misinformation

Genoveffa Martone, Helena Bonaldi, Marco Guerini

Fondazione Bruno Kessler, Italy · Università Cattolica del Sacro Cuore, Italy

Knowledge Augmentation for Language Models Text Generation

Open paper AskPDF arXiv

Hate speech and misinformation frequently co-occur online, amplifying prejudice and polarization. Given their scale, using Large Language Models (LLMs) to assist expert counterspeech (CS) writing has gained interest, yet prior work has addressed these phenomena separately. We bridge this gap by studying CS generation in contexts where both hate and misinformation co-occur. We test three knowledge-driven generation strategies: first we prompt an LLM with fact-checkers' guidelines and fact-checking articles; secondly, with NGOs' guidelines and reports; thirdly, we create a mixed strategy that combines guidelines and documents from both. 23 experts revise the generated CS, which are assessed via human and automatic metrics. While LLMs produce adequate CS in 40% of cases, expert edits substantially improve naturalness, exhaustiveness, and adherence to guidelines. Based on the post-edited CS, the mixed strategy proves to be the most effective in crowdsourcing evaluation, pairing strong factual correction with stereotype mitigation and empathetic engagement. We release a dataset of hateful and misinformed claims with expert-verified CS and supporting knowledge.

Abstract

Text Generation

Explore similar work

Assisted Counterspeech Writing at the Crossroads of Hate Speech and Misinformation

CATCH-ME if you RAG: a dataset of Contextually Annotated multi-Turn Counterspeech against Hate and Misinformation Exchanges

Counter with Evidence! A Multi-Agent Memory Efficient Reasoning Framework for Hate Category Informed Counterspeech Generation