Prolonged exposure to hate speech on social media can cause lasting psychological distress among users. Counterspeech has emerged as an effective, non-censorial mitigation strategy, but its deployment requires rigorous quality assessment. Manual assessment is unscalable due to the large volume of content on the Web and imposes severe emotional strain on human annotators. Large Language Models (LLMs) offer a scalable alternative, yet concerns remain regarding their reliability. In this paper, we investigate the extent to which LLMs can effectively perform this task. We test 12 open-weight LLMs of different sizes and families under zero-shot, few-shot, and chain-of-thought prompting, to approximate pointwise and pairwise human expert assessment of counterspeech quality across four dimensions: relevance, coherence, aggressiveness, and suitableness. We find that (i) LLMs show low agreement and moderate rank correlations with human experts: our best configuration, Qwen 2.5 72B on coherence, achieves up to a linear-weighted Cohen's κl=0.19 and Spearman's ρ=0.42 in pointwise evaluation and an unweighted Cohen's κ=0.25 in pairwise evaluation. (ii) Converting pointwise scores into pairwise labels substantially improves human--LLM agreement for selected configurations. For same-instance coherence, Qwen 2.5 72B under zero-shot prompting achieves κ=0.35 with converted labels, compared with κ=0.25 when judging pairs directly, and Mixtral 8×7B under few-shot prompting improves from κ=0.07 to κ=0.34. (iii) LLMs agree more with one another than with human experts, particularly in pairwise evaluation, where the strongest model pair, OLMo 2 32B and Llama 3.3 70B on coherence, reaches substantial agreement with κ=0.76, whereas their agreement with the experts in the same configuration does not exceed κ=0.21.
Figures & tables
Figure 1. Overview of the evaluation settings. In setting (a), a single counterspeech item for one hateful instance is evaluated on a Likert scale. In settings (b) and (c), two items are compared: in (b) on the same hateful instance, while in (c) on two different instances matched by target group and counterspeech strategy.
Dimension
n
1
2
3
4
5
Relevance
1,389
296
811
217
65
0
Coherence
1,397
612
712
57
16
0
Aggressiveness
1,092
91
225
568
194
14
Suitableness
1,386
693
542
151
–
–
Table 1. Expert score distribution in the pointwise evaluation set (1,400 instances). Suitableness uses a 1–3 scale.
Same-instance
Cross-instance
Dimension
n
A
B
Tie
n
A
B
Tie
Relevance
441
104
106
231
6,978
1,884
2,007
3,087
Coherence
450
115
111
224
7,047
1,626
1,703
3,718
Aggressiveness
263
68
66
129
5,963
1,890
1,826
2,247
Suitableness
447
102
104
241
6,890
2,050
1,850
2,990
Table 2. Number of pairwise comparisons per dimension and distribution of reference labels.
Family
Model
Size
Group
Identifier
Llama
Llama 3.1 8B
8B
Small
Llama-3.1-8B-Instruct
Llama 3.3 70B
70B
Large
Llama-3.3-70B-Instruct
OLMo 2
OLMo 2 7B
7B
Small
OLMo-2-1124-7B-Instruct
OLMo 2 32B
32B
Medium
OLMo-2-0325-32B-Instruct
Qwen 2.5
Qwen 2.5 7B
7B
Small
Qwen2.5-7B-Instruct
Qwen 2.5 72B
72B
Large
Qwen2.5-72B-Instruct
Table 3. Evaluated open-weight LLMs. For the Mixtral mixture-of-experts models, size groups are based on active parameters.
Relevance
Coherence
Aggressiveness
Suitableness
Model
Prompt
κl
ρ
κl
ρ
κl
ρ
κl
ρ
OLMo 7B
Zero-shot
.026
.065
.077
.264
.001
.014
.000
–
Few-shot
.060
.048
.058
.201
− .025
− .067
− .098
− .142
CoT
.062
.125
.143
.320
.000
–
.000
–
Qwen 7B
Zero-shot
.070
.122
.161
.378
.064
.307
− .021
− .037
Few-shot
.037
.067
.149
.320
.063
.296
− .056
− .101
Table 4. Pointwise human–LLM agreement. Avg.: group mean; ρ is averaged over models for which it is defined. –: same score for every instance. Bold: highest κl per dimension. Majority: most frequent expert score per dimension. Random (95% UB): upper end of the range covering 95% of the κl values from 2,000 runs of random scoring; higher values are unlikely to arise by chance.
Relevance
Coherence
Aggressiveness
Suitableness
Model
Prompt
κ
κpw
κ
κpw
κ
κpw
κ
κpw
OLMo 7B
Zero-shot
.080
.040
.159
.216
.010
− .003
− .014
.000
Few-shot
.008
.150
− .003
.217
− .002
.069
− .001
− .049
CoT
.091
.086
.099
.264
.056
.000
− .001
.000
Qwen 7B
Zero-shot
.073
.017
.192
.305
.010
.176
− .019
.036
Few-shot
.056
.097
.178
.251
.029
.169
− .030
.036
Table 5. Same-instance comparisons: pairwise human–LLM agreement. Bold: highest κ and κpw per dimension. Avg.: group mean. Majority and Random (95% UB): as in Table 4 .
Relevance
Coherence
Aggressiveness
Suitableness
Model
Prompt
κ
κpw
κ
κpw
κ
κpw
κ
κpw
OLMo 7B
Zero-shot
.029
.012
− .025
.037
.011
− .003
− .011
.000
Few-shot
− .001
− .012
.000
− .046
.002
− .008
.000
.002
CoT
− .004
.045
− .006
.029
.028
.000
− .005
.000
Qwen 7B
Zero-shot
.038
.070
.042
.037
.039
.129
.011
.025
Few-shot
.033
.059
.036
.034
− .016
.119
− .023
.054
Table 6. Cross-instance comparisons: pairwise human–LLM agreement. Bold: highest κ and κpw per dimension. Avg.: group mean. Majority and Random (95% UB): as in Table 4 .
Relevance
Coherence
Aggressiveness
Suitableness
Pointwise
n
κl
ρ
n
κl
ρ
n
κl
ρ
n
κl
ρ
All
1389
.024
.198
1397
.136
.368
1092
.042
.273
1386
.031
.053
Muslims
460
.013
.148
463
.132
.363
393
.012
.099
459
.057
.112
Women
224
.023
.207
226
.083
.250
168
.069
.418
225
− .025
− .049
Black people
173
.030
.126
176
.149
.491
89
.069
.264
175
− .024
− .042
LGBTQ+
151
.056
.354
151
.165
.433
113
.065
.344
149
.049
.073
Table 8. Human–LLM agreement of OLMo 2 32B per target group under zero-shot prompting. n : instances or pairs annotated for the dimension. Bold: highest κl or κ per dimension and setting.
Table 10. Model size within families: configurations in which the larger model has higher/lower κ , with Benjamini–Hochberg adjusted p -values in parentheses. PW: pointwise.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.