Annotator disagreement is increasingly recognized as a meaningful property of subjective tasks, rather than merely noise to be removed. Predicting which items are likely to be contested could help allocate annotation effort and route difficult cases to human review. We propose predicting disagreement from interpretable semantic properties of the text. An LLM answers a set of binary questions covering harmful-content characteristics, disagreement-related cues such as borderline severity and missing context, and language-sensitive cues such as local slurs. A classifier predicts disagreement from these answers, and we analyze which questions contribute most to the predictions. Across four datasets in English, German, and Turkish and three open-weight LLMs, our structured question features outperform direct prompting by 10.5 ROC-AUC points and the strongest text-based baselines by 5.0 points. A compact subset of eight questions performs within 1.5 ROC-AUC points of the full twenty-question set. In leave-one-dataset-out evaluation, models retain roughly 90-95% of their in-domain ROC-AUC. Across models and datasets, questions about derogation, missing context, and decision boundaries serve as the strongest predictors, suggesting that what makes harmful content contested is partly observable in the text.
Figures & tables
ID
Short description
Content questions (C)
q1
Protected group referenced
q2
Negative or derogatory description
q3
Slurs or stereotypes
q4
Dehumanization or vilification
q5
Blaming or scapegoating
Table 1: Question IDs and short descriptions grouped into content, boundary, and language/context questions.
MHS
HateXplain
Turkish
DeTox
ROC
PR
F1
MCC
ROC
PR
F1
MCC
ROC
PR
F1
MCC
ROC
PR
F1
MCC
Gemma 3 27B
Direct prompting
63.8
50.5
50.8
23.2
58.1
54.4
43.8
14.0
68.0
58.9
58.9
34.0
64.3
27.3
35.7
19.2
LaBSE
68.3
53.8
62.2
24.9
60.2
57.8
57.2
14.6
69.6
61.2
64.0
29.9
74.6
41.3
63.9
28.6
mE5
70.1
56.2
63.7
28.4
60.1
57.8
57.1
14.4
69.7
61.5
63.9
29.1
75.6
42.2
64.3
30.3
Qwen-emb
70.2
56.3
63.7
28.5
61.2
59.1
57.7
15.9
67.9
60.1
62.6
26.1
74.5
42.7
64.2
28.6
C
70.6
44.7
60.9
30.0
60.0
55.2
56.1
13.9
68.7
61.0
65.2
32.7
76.0
40.7
63.0
36.3
Table 2: Disagreement prediction (%), three outer folds. LaBSE, mE5, and Qwen-emb are multilingual-embedding baselines. C, B, L are content, boundary, and language/context question groups; conf. adds answer confidences. Best per column within each LLM block in bold , second best underlined .
Figure 1: Held-out test PR-AUC as a function of the number of selected questions.
Figure 2: Per-question TreeSHAP importance, as each question’s share of the total attribution within a dataset, for the three LLMs. Questions are grouped into content (q), boundary (d), and language/context (m).
In-domain
Transfer
Rel.
LLM
ROC
PR
MCC
ROC
PR
MCC
ROC
PR
MHS
Gemma
74.7
61.3
37.0
68.6
53.5
23.9
91.8
87.3
Mistral
76.1
63.1
39.3
68.1
54.0
23.4
89.5
85.6
Qwen
76.3
63.3
38.6
70.2
55.8
26.6
92.0
88.1
Xplain
Gemma
64.6
60.9
21.0
60.2
56.6
14.7
93.2
93.0
Mistral
65.5
61.7
22.4
61.5
58.0
16.1
93.8
94.0
Table 3: Leave-one-dataset-out transfer. In-domain models are trained and evaluated on the held-out dataset with cross-validation; transfer models are trained on the other three datasets without any explicit language or dataset features. Rel. is the ratio of transfer to in-domain performance (%). All values in %.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
MHS
HateXplain
Turkish
DeTox
Language
English
English
Turkish
German
License
CC-BY 4.0
MIT
CC BY-NC-SA 4.0
GPL-3.0
Label classes
3
3
3
2
Comments
29,453
19,192
60,308
9,689
Total annotations
135,450
57,687
301,540
28,908
Unique annotators
7,912
253
—
6
Appendix
Table 4: Dataset statistics. Disagreement rate is the fraction of comments with non-unanimous annotations. Krippendorff’s α is the nominal alpha over the label distributions. Annotator identifiers are not available for the Turkish dataset, so annotator-level statistics are omitted.
Gemma 3
Mistral S. 3.2
Qwen 3.6
Params
27B
24B
27B
Developer
Google
Mistral AI
Alibaba
License
Gemma
Apache 2.0
Apache 2.0
Context
128K
128K
262K
Appendix
Table 5: The three LLMs used to answer the questions.
Group
ID
Question
Content
C1
Does the text reference a protected group (e.g., based on race, religion, gender, sexual orientation, or origin)?
C2
Does the text describe the group negatively or derogatorily?
C3
Does the text use slurs or stereotypes?
C4
Does the text dehumanize or vilify the group?
C5
Does the text blame or scapegoat the group?
C6
Does the text call for excluding or discriminating against the group?
Appendix
Table 6: Full question set. Question IDs map to the result tables as C1–C10 = q1–q10, B1–B4 = d1–d4, B5 = d5, and L1–L5 = m1–m5.
LLM
Features
ROC
PR
F1 m
Gemma 3
Gold majority label
64.0
54.0
63.1
Gold + LLM output
72.2
61.8
66.2
Question-based (ours)
72.8
61.6
66.3
Mistral
Gold majority label
64.0
54.0
63.1
Gold + LLM output
69.1
59.1
66.2
Question-based (ours)
72.9
61.6
66.4
Appendix
Table 7: Combined-trained comparison against label-based controls (%). The gold majority label is a privileged input not available at inference time. ROC and PR are ROC-AUC and PR-AUC; F1 m is macro-F1. Best per block in bold .
LLM
Dataset
Full (20)
Compact (8)
ROC / PR
ROC / PR
Gemma 3 27B
MHS
74.7 ± 0.7 / 61.3 ± 0.9
73.8 ± 0.8 / 60.1 ± 1.5
HateXplain
64.6 ± 1.2 / 60.9 ± 0.9
63.8 ± 1.1 / 60.2 ± 1.5
Turkish
73.4 ± 0.2 / 64.7 ± 0.2
71.8 ± 0.3 / 62.8 ± 0.4
DeTox
81.6 ± 0.1 / 48.7 ± 1.1
80.2 ± 0.4 / 46.6 ± 0.4
Mistral Small 3.2 24B
MHS
76.1 ± 0.5 / 63.1 ± 0.7
75.8 ± 0.4 / 63.2 ± 0.7
Appendix
Table 8: Mean ± fold standard deviation (three outer folds) for the full 20-question and compact 8-question confidence models (ROC-AUC / PR-AUC, %).
Votes
Target
RF P
RF
Direct
E1
3 hatespeech
0
0.163
0
0
E2
1 normal / 2 offensive
1
0.695
1
0
E3
2 normal / 1 offensive
1
0.297
0
1
Appendix
Table 9: The three worked instances: annotator votes, disagreement target, random forest disagreement probability and prediction at threshold 0.49 , and the direct-prompting prediction.
Heriot-Watt University, Edinburgh, Scotland · aequa-tech, Torino, Italy · Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USA +2