Beating Hateful Narratives on Social Media with Facts: Retrieval-Augmented Counter-Speech Generation
Organizations: Université Côte d’Azur, CNRS, Inria, I3S, France
Abstract
Online hate speech threatens social cohesion by amplifying discrimination and harmful narratives against marginalized communities. While counter-speech (CS) - fact-based, respectful responses to hate - offers a constructive alternative to content removal, its large-scale generation remains constrained by factual reliability and the risk of generic responses. We address these challenges with a Retrieval-Augmented Generation (RAG) pipeline grounded in the first large-scale, curated knowledge base built from authoritative international and regional human rights sources, including the United Nations Digital Library, EUR-Lex, and the EU Agency for Fundamental Rights. The framework covers eight groups frequently targeted by online abuse. We systematically evaluate all pipeline components demonstrating that each design choice is empirically motivated, combining automatic metrics, LLM-based judgments, and human evaluation with comparisons against strong baselines. Results show that RAG improves factual grounding, diversity, and perceived effectiveness over No-RAG and performs favorably against existing systems. Our human evaluation further reveals differences between in-group and out-group perceptions of CS, with RAG narrowing but not eliminating this gap. These findings highlight the potential of knowledge grounded CS for scalable social media interventions, while underscoring the need to evaluate such systems with and for the communities they are intended to serve.
Figures & tables
| Keyword | EU # | EU wds | UN # | UN wds |
|---|---|---|---|---|
| Disabled | 20 | 17,980 | 1,718 | 9,705 |
| Human rights | 72 | 35,070 | 13,396 | 10,048 |
| Jews | 29 | 25,111 | 178 | 5,707 |
| LGBT+ | 22 | 26,953 | 164 | 5,998 |
| Migrants | 61 | 25,948 | 3,784 | 8,748 |
| Muslim | 11 | 21,340 | 406 | 3,272 |
| Model | BLEU | METEOR | ROUGE-L | BERTScore | Distinct-1 | Distinct-2 | RR | Safety | Contr. | Ent. | Neutral | Similarity |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No Retrieval (No-RAG) | ||||||||||||
| Llama | .0115 .0004 | .1638 .0020 | .1164 .0015 | .8575 .0004 | .0196 | .1238 | .0034 | .9867 .0010 | - | - | - | - |
| CommandR | .0118 .0004 | .1432 .0020 | .1163 .0016 | .8590 .0004 | .0233 | .1462 | .0002 | .9884 .0010 | - | - | - | - |
| Mistral | .0116 .0004 | .1435 .0018 | .1121 .0015 | .8575 .0004 | .0210 | .1059 | .2668 | .9924 .0007 | - | - | - | - |
| GPT | .0105 .0003 | .1494 .0018 | .1124 .0014 | .8580 .0004 | .0156 | .1037 | .0018 | .9934 .0006 | - | - | - | - |
| BM25 Retrieval | ||||||||||||
| Panel A: JudgeLM RAG wins vs No RAG | |||
|---|---|---|---|
| Model | BM25 | SentBERT | BGE-M3 |
| Mistral | 3011 | 3103 | 3557 |
| Llama | 2576 | 2805 | 3532 |
| GPT | 3524 | 3941 | 4223 |
| CommandR | 2490 (lost) | 2620 | 3125 |
| Panel B: JudgeLM RAG wins vs MT-Co | |||
| Method | Rel. | Fact. | Cog. | Corr. | Eff. | Best |
|---|---|---|---|---|---|---|
| MT-Co | 2.07 0.74 | 1.38 0.67 | 1.67 0.73 | 2.28 0.82 | 48 | 7 |
| Llama NoRAG | 2.72 0.59 | 2.14 0.78 | 2.49 0.65 | 2.77 0.56 | 171 | 33 |
| Llama RAG | 2.66 0.64 | 2.50 0.66 | 2.53 0.69 | 2.75 0.57 | 197 | 67 |
| Com.R NoRAG | 2.63 0.62 | 1.85 0.75 | 2.25 0.71 | 2.75 0.58 | 120 | 7 |
| Com.R RAG | 2.75 0.55 | 2.25 0.74 | 2.48 0.63 | 2.74 0.57 | 172 | 26 |
| Mistral NoRAG | 2.58 0.67 | 1.77 0.77 | 2.11 0.75 | 2.69 0.63 | 109 | 11 |
| Panel A: Average ratings by dimension and RAG status | ||
|---|---|---|
| Dimension / RAG | In Mean | Out Mean |
| Cogency (No RAG) | 2.10 0.77 | 2.32 0.69 |
| Cogency (RAG) | 2.43 0.71 | 2.54 0.57 |
| Correctness (No RAG) | 2.58 0.72 | 2.77 0.54 |
| Correctness (RAG) | 2.71 0.62 | 2.80 0.49 |
| Factuality (No RAG) | 1.71 0.79 | 2.01 0.73 |
| Competitor | Target | Total | Wins (ours) | % Wins |
| vs Wilk et al. (2025) | DISABLED | 36 | 25 | 69.4 |
| JEWS | 204 | 102 | 50.0 | |
| LGBT+ | 164 | 105 | 64.0 | |
| MIGRANTS | 310 | 176 | 56.8 | |
| MUSLIMS | 518 | 286 | 55.2 | |
| POC | 109 | 58 | 53.2 |
| Group | Setting | Specificity | Evidence | Conciseness | Persuasiveness | Grammaticality | CS Tokens | Input Tokens |
|---|---|---|---|---|---|---|---|---|
| 0 | 4.51 0.54 | 0.00 0.00 | 4.43 0.45 | 4.73 0.31 | 5.00 0.00 | 34.88 | – | |
| 1 | 4.73 0.44 | 4.34 1.12 | 3.79 0.41 | 4.76 0.31 | 5.00 0.00 | 80.70 | – | |
| 2 | 4.74 0.48 | 4.31 1.09 | 3.74 0.44 | 4.73 0.32 | 5.00 0.00 | 84.08 | – | |
| 3 | 4.79 0.47 | 4.42 0.95 | 3.74 0.46 | 4.75 0.33 | 5.00 0.00 | 89.29 | – | |
| 4 | 4.79 0.44 | 4.48 0.81 | 3.68 0.50 | 4.79 0.30 | 4.98 0.11 | 92.16 | – | |
| 5 | 4.84 0.44 | 4.64 0.62 | 3.70 0.47 | 4.84 0.31 | 5.00 0.00 | 95.38 | – |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Target | Representative document | Year |
|---|---|---|
| Jews | History under attack: holocaust denial and distortion on social media (UN Digital Library) | 2022 |
| LGBT+ | EU LGBTIQ Survey III, LGBTIQ Equality at a Crossroads - Progress and Challenges (FRA) | 2024 |
| Migrants | Directive 2013/33/EU, laying down standards for the reception of applicants for international protection (EUR-Lex) | 2013 |
| Muslims | Being Muslim in the EU - Experiences of Muslims, EU Survey on Immigrants and Descendants of Immigrants (FRA) | 2024 |
| Women | Sex-based violence against women and girls: new frontiers and emerging issues, report of the Special Rapporteur on Violence against Women and Girls, Its Causes and Consequences (UN Digital Library) | 2025 |
| POC | Addressing Racism in Policing (FRA) | 2024 |
| No-RAG vs Retrieval: Per-sample Automatic Metrics | |||||
|---|---|---|---|---|---|
| Model | Metric | Friedman | BM25 | SBERT | BGE |
| Llama | BLEU | ||||
| METEOR | |||||
| ROUGE-L | |||||
| BERTScore F1 | |||||
| Safety | |||||
| Model | Setting | Tokens | Est. FLOPs | GPU-time | Energy | CO 2 e | Cost |
|---|---|---|---|---|---|---|---|
| Llama-3.1-8B | No-RAG | 585,407 | 1.6 h | 0.63 kWh | 0.25 kg | – | |
| Llama-3.1-8B | RAG (avg/retriever) | 10,960,856 | 42.2 h | 16.9 kWh | 6.76 kg | – | |
| Mistral-7B | No-RAG | 533,977 | 1.1 h | 0.45 kWh | 0.18 kg | – | |
| Mistral-7B | RAG (avg/retriever) | 5,007,558 | 6.4 h | 2.56 kWh | 1.02 kg | – | |
| CommandR-7B | No-RAG | 514,684 | 1.3 h | 0.51 kWh | 0.20 kg | – | |
| CommandR-7B | RAG (avg/retriever) | 5,021,287 | 8.3 h | 3.19 kWh | 1.28 kg | – |
| LLM / RAG | Metric | In-group | Out-group |
|---|---|---|---|
| CommandR No RAG | Cogency | 2.44 0.59 | 2.17 0.74 |
| Correctness | 2.91 0.28 | 2.67 0.66 | |
| Factuality | 2.05 0.65 | 1.76 0.77 | |
| Relevance | 2.83 0.41 | 2.55 0.68 | |
| CommandR RAG | Cogency | 2.51 0.55 | 2.46 0.66 |
| Correctness | 2.81 0.42 | 2.71 0.62 |
| Setting | Metric | Rel. | Fact. | Cog. | Corr. | max | Holm |
|---|---|---|---|---|---|---|---|
| No-RAG | BLEU | .240 | .056 | .188 | .362 ∗ | .362 | .125 |
| METEOR | .002 | -.014 | .118 | .058 | .118 | 1.000 | |
| ROUGE-L | .076 | .011 | .154 | .216 | .216 | 1.000 | |
| BERTScore | .032 | -.106 | -.002 | .151 | .151 | 1.000 | |
| RAG | BLEU | .255 | -.210 | -.038 | -.131 | .255 | 1.000 |
| METEOR | .034 | .185 | .132 | -.085 | .185 | 1.000 |
| Target Group | UN-Only (%) | EU-Only (%) | Hybrid (%) | Total EU (%) |
|---|---|---|---|---|
| DISABLED | 90.00 | 0.00 | 10.00 | 10.00 |
| JEWS | 70.00 | 10.00 | 20.00 | 30.00 |
| LGBT+ | 50.00 | 10.00 | 40.00 | 50.00 |
| MIGRANTS | 70.00 | 0.00 | 30.00 | 30.00 |
| MUSLIMS | 0.00 | 40.00 | 60.00 | 100.00 |
| POC | 90.00 | 0.00 | 10.00 | 10.00 |
Text Generation
Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.
See all 445 papers