Counter with Evidence! A Multi-Agent Memory Efficient Reasoning Framework for Hate Category Informed Counterspeech Generation

Sujoy Nath; Aswini Kumar; Tanmoy Chakraborty

cs.CLAug 24, 2026

Counter with Evidence! A Multi-Agent Memory Efficient Reasoning Framework for Hate Category Informed Counterspeech Generation

Authors: Sujoy Nath, Aswini Kumar, Tanmoy Chakraborty

Organizations: Indian Institute of Technology Delhi, India

Abstract

Counterspeech effectively neutralizes the impact of online hate. Although prior work explores automated counterspeech generation, it largely emphasizes stylistic control while treating hate speech as homogeneous, overlooking that distinct forms of abuse require fundamentally different counterspeech strategies. To address this gap, we introduce FIRE (Factuality Informed Multi-Agent Reasoning Framework) that first decomposes hate speech into one of the five distinct categories (misinformation, stereotype, conspiracy, dehumanizing, non-factual), and then maps it to a targeted counterspeech style. To facilitate FIRE, we curate FactualCS, a novel dataset of $4,784$ instances that provides the annotations regarding hate categories, reasoning traces, and evidence mappings, which are critical elements for grounded generation that are missing in prior work. A comprehensive evaluation across $28$ baseline configurations demonstrates that FIRE significantly surpasses existing methods, despite using compact agents ( $<$ 2B). FIRE achieves a $\sim$ $12 \%$ and $\sim$ $11 \%$ improvements in factual and category-specific accuracy respectively, while simultaneously reducing toxicity by $\sim$ $11 \%$ relative to the strongest baselines. Further human evaluation confirms that responses generated by FIRE are significantly preferred over the strongest baselines, underscoring its effectiveness for real-world deployment. These findings show that decomposing the underlying intent of hate speech is essential for generating safe, effective, and contextually precise counterspeech.

Figures & tables

Figure 1: A qualitative comparison demonstrating the efficacy of FIRE against baselines; our framework generates superior quality response by accurately identifying the specific characteristics of the target hate speech.

Target	SH	MH	CH	DH	NH	Total
Women	178	12	3	166	285	644
Men	29	3	1	14	15	62
LGBT+	117	34	2	110	137	400
Muslim	271	101	22	90	99	583
Jews	167	35	96	109	36	443
IMGT	278	275	15	103	166	837

Table 1: Distribution of Hate Speech Types Across Targeted Groups: Stereotype Hate (SH), Misinformation Hate (MH), Conspiracy Hate (CH), Dehumanization Hate (DH), Non-factual Hate (NH) 3 3 3 OR: Other Religions, AP: Affected Person, PoC: Person of Color

Figure 2: An overview of the proposed FIRE framework that follows the two-phase approach. The Memory Module utilizes a trained contrasting encoder ( $w_{\theta}$ ) to retrieve top- $k$ exemplars ( $E$ ) and compute a purity score ( $\rho$ ). In Phase 1 (HEAL), the Hate Speech Analyst (HSA) leverages these exemplars to predict structured latent variables ( $g$ , $t$ , $r$ , $q$ ). If the detected hate type contains factual claims ( $t\in\mathcal{T}_{\text{factual}}$ ), a Web Search Tool is triggered to retrieve supporting evidence ( $\mathcal{E}$ ). In Phase 2 (CARE), the Counterspeech Generator (CSG) synthesizes the final response by conditioning on the original input, the structured analysis from HSA, the retrieved evidence, and the stylistic patterns from the memory exemplars.

Model	P/A	ROUGE $\uparrow$			BS $\uparrow$	M $\uparrow$	R $\downarrow$	CoSIM $\uparrow$	T $\downarrow$	Nov $\uparrow$	Div $\uparrow$	CatAcc $\uparrow$	FSc $\uparrow$
		R1	R2	RL
GPS	–	$0.103$	$0.032$	$0.129$	$0.831$	$0.179$	$0.257$	$0.472$	$0.035$	$0.489$	$0.643$	$0.422$	$0.467$
DialoGPT	–	$0.191$	$0.087$	$0.188$	$0.863$	$0.210$	$0.231$	$0.518$	$0.021$	$0.618$	$0.765$	$0.563$	$0.529$
CoARL	–	$0.207$	$0.101$	$0.202$	$0.865$	$0.213$	$0.212$	$0.549$	$0.019$	$0.649$	$0.788$	$0.604$	$0.579$
HiPPrO	–	$0.235$	$0.104$	$0.217$	$0.879$	$0.227$	$0.198$	$0.574$	$0.018$	$0.651$	$0.809$	$0.622$	$0.605$
BART Large	ZS	$0.063$	$0.018$	$0.072$	$0.823$	$0.083$	$0.274$	$0.318$	$0.041$	$0.487$	$0.742$	$0.108$	$0.198$

Table 2: Comparing FIRE with baselines across various evaluation metrics. Here, ↑ ( resp. ↓) denotes that higher ( resp. lower) is better. Bold ( resp. underline ) indicates the best ( resp. second-ranked) performance. Here P/A represents different strategies used: ZS for Zero Shot Prompting, FS for Few Shot Prompting, RB for Retrieval-Based and SFT represent the Supervised Finetuning.

Models on comparison	Metrics
	ICS $\uparrow$	Ad $\uparrow$	CoRl $\uparrow$	ArgE $\uparrow$
FIRE vs Llama-3.1-8b (SFT)	$0.91$	$0.89$	$0.86$	$0.93$
FIRE vs HiPPrO	$0.96$	$0.94$	$0.95$	$0.97$
FIRE vs CoARL	$0.98$	$0.96$	$0.96$	$0.98$

Table 3: Results of the human evaluation study, where responses generated by FIRE are shown against those produced by (a) Llama-3.1-8b-instruct, (b) HiPPrO, and (c) CoARL. The results are reported in terms of Win Rate % , indicating the % of instances where FIRE outperforms the respective baselines.

Type	Content
Hate Speech	No muslims are compatible with any Western society.
Model	Generated Counterspeech
Llama-3.1-8b-instruct	Muslims are a diverse group and integration varies across countries . Many Muslims contribute positively to Western societies.
FIRE (Ours)	A Muslim leader like London Mayor Sadiq Khan has integrated into Western societies while upholding Islamic values, demonstrating that compatibility is possible through mutual respect and dialogue . Diversity enriches societies, and stereotypes ignore the complexity of individual experiences. Understanding Islamophobia requires recognizing that cultural and religious differences foster mutual growth , not division.

Table 4: Error analysis between the response generated by our method FIRE vs the best baseline.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	Annotator 1	Annotator 2	Annotator 3	Annotator 4	Annotator 5	Annotator 6
Annotator 1	1.0000	0.9220	0.9500	0.9260	0.9420	0.9540
Annotator 2	0.9220	1.0000	0.9440	0.9240	0.9240	0.9520
Annotator 3	0.9500	0.9440	1.0000	0.9480	0.9520	0.9760
Annotator 4	0.9260	0.9240	0.9480	1.0000	0.9280	0.9560
Annotator 5	0.9420	0.9240	0.9520	0.9280	1.0000	0.9560
Annotator 6	0.9540	0.9520	0.9760	0.9560	0.9560	1.0000

Appendix

Table 5: Inter-annotator Agreement Coefficients for 500 samples in Pilot Annotation Phase

	Annotator 1	Annotator 2	Annotator 3	Annotator 4	Annotator 5	Annotator 6
Annotator 1	1.0000	0.8860	0.9200	0.9000	0.9120	0.9300
Annotator 2	0.8860	1.0000	0.9140	0.8900	0.8900	0.9240
Annotator 3	0.9200	0.9140	1.0000	0.9240	0.9280	0.9540
Annotator 4	0.9000	0.8900	0.9240	1.0000	0.9000	0.9300
Annotator 5	0.9120	0.8900	0.9280	0.9000	1.0000	0.9300
Annotator 6	0.9300	0.9240	0.9540	0.9300	0.9300	1.0000

Appendix

Table 6: Inter-annotator Agreement Coefficients in Main Annotation Phase for Hate Speech Type, Reasoning, Target Group, Query, and collected evidence

	Annotator 1	Annotator 2	Annotator 3	Annotator 4	Annotator 5	Annotator 6
Annotator 1	1.0000	0.8960	0.9180	0.9080	0.9140	0.9240
Annotator 2	0.8960	1.0000	0.9180	0.9000	0.8980	0.9200
Annotator 3	0.9180	0.9180	1.0000	0.9180	0.9240	0.9380
Annotator 4	0.9080	0.9000	0.9180	1.0000	0.9060	0.9240
Annotator 5	0.9140	0.8980	0.9240	0.9060	1.0000	0.9220
Annotator 6	0.9240	0.9200	0.9380	0.9240	0.9220	1.0000

Appendix

Table 7: Inter-annotator Agreement Coefficients for generated counterspeech

HS Type	Description	Example	Ideal Response Type	Factual?
Misinformation	Relies on false or misleading factual claims about a group	Immigrants increase crime rates.	Fact-based rebuttal using data, statistics, authoritative sources	Yes
Conspiracy	Misinformation mixed with large-scale conspiracy framing	Jews control the world’s banks.	Debunk narrative patterns, use cognitive inoculation strategies	Yes
Stereotype	Cultural or historical generalizations applied to a group	Women can’t handle leadership.	Reframing, factual correction, highlighting counterexamples	Yes
Dehumanizing	Use of animalistic or violent metaphors	Refugees are parasites.	Moral, legal, and social norm reminders	No
Non-factual	Hatred or disgust without factual claims	Muslims are disgusting.	Empathy-based counterspeech, reinforcing social norms	No

Appendix

Table 8: Hate Speech Types based on Factuality, Response Strategies, and Factual Rebuttal Requirements

Metric	T-statistic	$p$ -value	Significant	Outperform
FIRE vs LLaMA-3.1-8B-Instruct (SFT)
ROUGE-1	-9.41	$2.05E{-19}$	Yes	No
ROUGE-2	-9.75	$1.28E{-20}$	Yes	No
ROUGE-L	-11.19	$5.63E{-26}$	Yes	No
BERTScore	-11.33	$1.48E{-26}$	Yes	No
METEOR	-7.35	$8.80E{-13}$	Yes	No

Appendix

Table 9: Statistical comparison of FIRE against LLaMA-3.1-8B-Instruct under supervised fine-tuning (SFT) and few-shot (FS) settings using paired t-tests. Positive T-statistic values indicate better performance by FIRE. Statistical significance is determined at

p<0.05

Model	Params	Peak VRAM	Latency
FIRE (Sequential)	2 $\times$ 1.7B	$\sim$ 4GB	2.5–3.0s
Llama-3.1 8B-Instruct	8B	$\sim$ 16GB	2.0–2.8s
Qwen-2.5 7B-Instruct	7B	$\sim$ 16GB	1.8–2.5s
Mistral 7B-Instruct	7B	$\sim$ 16GB	1.7–2.4s

Appendix

Table 10: Computational efficiency profiling comparing peak memory requirements and inference latency between FIRE and single-pass baselines.

Condition	Samples (%)	Accuracy
High Confidence ( $\mathcal{P}{\geq}0.62$ )	88.4% (426)	73.2%
Ambiguous ( $\mathcal{P}{<}0.62$ )	11.6% (56)	48.2%
Overall	100% (482)	70.3%

Appendix

Table 11: Distribution and accuracy breakdown of HSA predictions based on the purity score threshold.

Source Category	Retrieval Frequency
Institutional / Academic	58%
Mainstream Informational	34%
Social-Media-Style	6%
Potentially Low-Quality	2%

Appendix

Table 12: Manual audit of 100 retrieved web search instances categorized by source reliability.

Explore similar work

Sep 15, 2026cs.CL

Deconstructing Stereotypes: Scope-Conditioned Generation for Effective Multilingual Counterspeech

Counterspeech (CS) - direct responses that counter online Hate Speech (HS) using reasoning and alternative viewpoints - has emerged as an alternative to content removal. Current automatic CS generation methods, however, frequently produce generic, ineffective replies that fail to target the implicit stereotypes behind HS. To bridge this gap, we propose a novel scope-conditioned generation framework that explicitly integrates structured stereotype characteristics into Large Language Models prompts. We validate our approach on a novel, human-curated dataset annotated in English, Italian, and Spanish. Extensive evaluations show that stereotype-conditioned prompting substantially outperforms generic baselines across all three languages, obtaining significant gains in factuality, specificity, cogency, and effectiveness for both explicit and implicit implied stereotypes.

Greta Damo, Elias Urios Alacreu, Elena Cabrio +2

1Universit´e Cˆote d’Azur, CNRS, INRIA, I3S, France · 2PRHLT Research Center, Universitat Polit`ecnica de Val`encia, Valencia, Spain · 3ValgrAI Valencian Graduate School and Research Network of Artificial Intelligence, Spain

Controllable Text Generation Prompt Engineering Text Generation

Ask PDF

Jun 18, 2026cs.CL

CATCH-ME if you RAG: a dataset of Contextually Annotated multi-Turn Counterspeech against Hate and Misinformation Exchanges

Online hate speech and misinformation frequently overlap, yet NLP research has mainly treated them in isolation. While LLMs represent a scalable solution for assisting humans in the generation of counterspeech for both threats, zero-shot models frequently generate repetitive and vague responses, underscoring the need for high-quality examples to steer model generation. However, existing counterspeech datasets against the overlap of hate and misinformation are scarce and limited to single-turn English dialogues, while real-life interactions span across multiple turns and languages. To bridge this gap, we introduce the first large-scale, expert-curated, multilingual dataset of dialogues tackling the intersection of hate and misinformation. To ensure factual grounding, the dialogues are also anchored in verified external knowledge (i.e., fact-checking articles and NGO reports) and include document- and chunk-level span annotations, making it directly applicable for RAG systems. Covering five languages and targeting hate directed at seven marginalized groups, this novel resource enables the training and evaluation of more persuasive, factually grounded counterspeech models.

Helena Bonaldi, Genoveffa Martone, Marco Guerini

Fondazione Bruno Kessler, Italy · Università Cattolica del Sacro Cuore, Italy

Content Moderation Text Generation

Ask PDF

May 21, 2026cs.CL

Assisted Counterspeech Writing at the Crossroads of Hate Speech and Misinformation

Hate speech and misinformation frequently co-occur online, amplifying prejudice and polarization. Given their scale, using Large Language Models (LLMs) to assist expert counterspeech (CS) writing has gained interest, yet prior work has addressed these phenomena separately. We bridge this gap by studying CS generation in contexts where both hate and misinformation co-occur. We test three knowledge-driven generation strategies: first we prompt an LLM with fact-checkers' guidelines and fact-checking articles; secondly, with NGOs' guidelines and reports; thirdly, we create a mixed strategy that combines guidelines and documents from both. 23 experts revise the generated CS, which are assessed via human and automatic metrics. While LLMs produce adequate CS in 40% of cases, expert edits substantially improve naturalness, exhaustiveness, and adherence to guidelines. Based on the post-edited CS, the mixed strategy proves to be the most effective in crowdsourcing evaluation, pairing strong factual correction with stereotype mitigation and empathetic engagement. We release a dataset of hateful and misinformed claims with expert-verified CS and supporting knowledge.

Genoveffa Martone, Helena Bonaldi, Marco Guerini

Fondazione Bruno Kessler, Italy · Università Cattolica del Sacro Cuore, Italy

Knowledge Augmentation for Language Models Text Generation

Ask PDF