cs.CLNov 3, 2026

Evaluating LLMs as Judges of Counterspeech Quality Against Online Hate Speech

Authors: Johnny Alali, Nicolás Benjamín Ocampo, Davide Ceolin

Organizations: Vrije Universiteit Amsterdam, Amsterdam, The Netherlands · Centrum Wiskunde & Informatica, Amsterdam, The Netherlands

Abstract

Prolonged exposure to hate speech on social media can cause lasting psychological distress among users. Counterspeech has emerged as an effective, non-censorial mitigation strategy, but its deployment requires rigorous quality assessment. Manual assessment is unscalable due to the large volume of content on the Web and imposes severe emotional strain on human annotators. Large Language Models (LLMs) offer a scalable alternative, yet concerns remain regarding their reliability. In this paper, we investigate the extent to which LLMs can effectively perform this task. We test 12 open-weight LLMs of different sizes and families under zero-shot, few-shot, and chain-of-thought prompting, to approximate pointwise and pairwise human expert assessment of counterspeech quality across four dimensions: relevance, coherence, aggressiveness, and suitableness. We find that (i) LLMs show low agreement and moderate rank correlations with human experts: our best configuration, Qwen 2.5 72B on coherence, achieves up to a linear-weighted Cohen's κl=0.19κ_l = 0.19 and Spearman's ρ=0.42ρ= 0.42 in pointwise evaluation and an unweighted Cohen's κ=0.25κ= 0.25 in pairwise evaluation. (ii) Converting pointwise scores into pairwise labels substantially improves human--LLM agreement for selected configurations. For same-instance coherence, Qwen 2.5 72B under zero-shot prompting achieves κ=0.35κ= 0.35 with converted labels, compared with κ=0.25κ= 0.25 when judging pairs directly, and Mixtral 8×\times7B under few-shot prompting improves from κ=0.07κ= 0.07 to κ=0.34κ= 0.34. (iii) LLMs agree more with one another than with human experts, particularly in pairwise evaluation, where the strongest model pair, OLMo 2 32B and Llama 3.3 70B on coherence, reaches substantial agreement with κ=0.76κ= 0.76, whereas their agreement with the experts in the same configuration does not exceed κ=0.21κ= 0.21.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Attribute-Based Diagnosis of LLM Alignment with Hate Speech Annotations

    May 26, 2026Mohammad Amine Jradi, Faeze Ghorbanpour, Alexander FraserLLM EvaluationLLM Alignment

  2. Assisted Counterspeech Writing at the Crossroads of Hate Speech and Misinformation

    May 21, 2026Genoveffa Martone, Helena Bonaldi, Marco GueriniKnowledge Augmentation for Language ModelsText Generation

  3. Evaluating the Reliability and Fidelity of Automated Judgment Systems of Large Language Models

    Mar 23, 2026Tom Biskupski, Stephan KleberLLM-as-a-JudgeAutomated Evaluation