To Jev or Not? Evaluating the Accuracy and Efficiency of Structured Decision Models for Hate-Speech Moderation

Demetris Paschalides; George Pallis; Marios D. Dikaiakos

cs.CLOct 2, 2026

To Jev or Not? Evaluating the Accuracy and Efficiency of Structured Decision Models for Hate-Speech Moderation

Authors: Demetris Paschalides, George Pallis, Marios D. Dikaiakos

Organizations: University of Cyprus, Computer Science Department, Nicosia, Cyprus

Abstract

The scale of online content makes hate-speech moderation challenging, while Large Language Models (LLMs) enable harmful material to be produced and adapted more easily. Moderation therefore requires efficient classifiers that can accommodate different definitions of hate speech. Recent structured decision models accept natural-language criteria and select among specified answers, raising the question of whether they can meet these requirements without task-specific training. We present HATEDECIDE, an evaluation of six decision-model configurations on four hate-speech datasets against specialized moderation, zero-shot, commercial, and supervised baselines. We examine whether supplying a dataset's definition, or decomposing it into multiple questions, improves classification, and we measure their latency and cost. We find that commercial LLMs significantly outperform all decision models on only one dataset. Supplying definitions changes up to 28% of predictions without consistently improving classification, and decomposition significantly improves performance in only 20% of the comparisons. On a diagnostic set of test cases, the best hosted decision model comes within 1.6 macro-F1 points of the best commercial LLM at approximately 97% lower inference cost. These results identify opportunities for inexpensive moderation, while showing that explicit criteria and additional questions do not reliably improve classification.

Figures & tables

Figure 1: Overview of HateDecide . Out-of-the-box models classify texts directly (A), with or without the dataset’s definition, or through definition-specific decomposition (B), while supervised classifiers are trained on one dataset and tested within and across datasets. Each paired comparison is evaluated against gold annotations on identical texts.

Dataset	Text source	Train	Val.	Test (hate / non-hate)	Additional labels and tasks
Dynamic [ 26 ]	Texts written by annotators to challenge hate-speech classifiers.	28,070	3,770	4,120 (2,268 / 1,852)	Six conduct classes across 3,356 test texts.
MHS [ 20 ]	Comments from YouTube, Reddit, and Twitter.	26,101	3,245	3,203 (782 / 2,421)	-
HateXplain [ 16 ]	Posts from Twitter and Gab.	15,346	1,920	1,924 (594 / 1,330)	Three-class labels retain 548 offensive and 782 normal posts.
HateCheck [ 19 ]	Texts testing abilities such as recognizing threats and counter-speech.	-	-	3,728 (2,563 / 1,165)	Referenced group on 3,436 texts and general/directed hate on 2,563 texts.

Table 1: Datasets after label aggregation and deduplication. HateXplain combines offensive and normal as non-hate .

	Dynamic	MHS	HateXplain (proxy)	HateCheck
Target	Vulnerable, marginalized, or discriminated- against groups; attacks based solely on being male, white, or heterosexual excluded	A person or group	A social group or its members	A protected group, or an individual because of membership
Identities	Open-ended, e.g., ethnicity, religion, gender, sexuality, immigration status, disability, class	Actual or perceived innate characteristics	Unspecified social groups	Protected characteristics, incl. age, disability, gender identity, pregnancy, race, religion, sex, sexual orientation
Conduct	Derogation, animosity, dehumanization, threats, and support for hateful entities	Prejudice-driven hostile or malicious language	Hatred, or intent to demean, humiliate, or insult	Abuse
Stance	Intentional, reporting or rejecting excluded	Not stated	Context decides, in-group use and quoted lyrics not hate	Counterspeech, negated hate, reclaimed slurs non-hateful
Questions	Scope, intent, conduct type	Targeting, abuse	Targeting, abuse	Targeting, abuse (same target)
Hate if	Intent $\land$ (support $\lor$ (scope $\land$ hostile conduct))	Targeting $\land$ abuse	Targeting $\land$ abuse	Targeting $\land$ abuse

Table 2: Dataset-specific definitions and derived decomposition rules. Full definitions appear in Appendix A . In the rules,

\land

means AND and

\lor

means OR. Scope, intent, targeting, and abuse denote affirmative component answers. Support denotes selection of the support category, hostile conduct denotes animosity, derogation, dehumanization, or threatening.

Model	Access	Backbone
Structured Decision Models
Jev	API	Undisclosed
Laya	Local	ModernBERT-large
Decider	Local	Qwen, 2B
SemIf/Qwen3.5-2B	Local	Qwen3.5-2B
SemIf/Qwen3.5-4B	Local	Qwen3.5-4B

Table 3: Evaluated models, access modes, and backbones.

Model	A	B	$\Delta$	A	B	$\Delta$	A	B	$\Delta$	A	B	$\Delta$
	Dynamic ( $N{=}4{,}120$ )			MHS ( $N{=}3{,}203$ )			HateXplain ( $N{=}1{,}924$ )			HateCheck ( $N{=}3{,}728$ )
Jev	.826	.843	$+$ 1.6	.691	.682	$-$ 0.9	.738	.729	$-$ 0.9	.960	.957	$-$ 0.4
Laya	.655	.534	$-$ 12.1	.695	.732	$+$ 3.7	.733	.775	$+$ 4.2	.701	.690	$-$ 1.1
Decider	.586	.552	$-$ 3.3	.801	.707	$-$ 9.3	.677	.706	$+$ 2.8	.730	.390	$-$ 34.0
SemIf/Qwen3.5-2B	.479	.534	$+$ 5.6	.220	.578	$+$ 35.8	.416	.456	$+$ 4.0	.483	.411	$-$ 7.2
SemIf/Qwen3.5-4B	.674	.655	$-$ 1.9	.719	.681	$-$ 3.8	.688	.695	$+$ 0.7	.782	.754	$-$ 2.8

Table 4: Direct (A) and decomposed (B) binary macro-F1 on the full test sets. Both formulations classify the same texts and receive the same dataset-specific definition, together with the decomposition rule and component glossary, so each difference

\Delta

(percentage points) is paired. Bold

\Delta

indicates an adjusted interval that excludes zero.

Figure 2: Binary macro-F1 of supervised classifiers trained on the row dataset and tested on the column dataset without adaptation. Outlined cells mark within-dataset evaluation. HateCheck is evaluation-only.

Figure 3: Change in binary macro-F1 (percentage points) from no definition to the dataset-specific definition, on identical texts. Asterisks mark changes whose adjusted interval excludes zero (Section 3.4 ).

Model	Conduct (Dyn.)	Group (HC)	Direction (HC)
Jev	.538	.989	.682
Laya	.192	.929	.694
Decider	.313	.959	.610
SemIf/Qwen3.5-2B	.196	.939	.269
SemIf/Qwen3.5-4B	.406	.962	.594
Bespoke-Nimble, NF4	.324	.971	.567

Table 5: Attribute macro-F1 with defined labels: Dynamic’s six conduct classes, and HateCheck’s referenced group and general-versus-directed hate.

Figure 4: Change in hate recall against change in hate precision from direct to decomposed classification of the same texts, for each model on Dynamic, MHS, and HateXplain. Colours mark gains and losses in macro-F1 (Table 4 ).

Figure 5: Macro-F1 against median complete-decision latency (a) and hosted cost per 1K texts (b) on identical HateCheck examples (

N=3{,}728

). Connected markers show direct (A) and decomposed (B) classification for five models. Both horizontal axes are logarithmic. Local monetary costs are omitted. Results for all evaluated models appear in Appendix H .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Checkpoint or identifier	Precision	Input limit	Decision
Jev	jev-1.13.0 (hosted API)	-	-	Choice question; option scores
Laya	1c5edc17	FP32, BF16 autocast	1,024 tokens (2,048 for B)	Option scores with the released temperature scaling
Decider	b37f7e1b	BF16	512 tokens	Option-letter scores, scoring temperature 1.3
SemIf/Qwen3.5-2B	Qwen 15852e8c , SemIf ca3ba65f	BF16	4,096 tokens	Next-letter probabilities
SemIf/Qwen3.5-4B	Qwen 851bf6e8 , SemIf ca3ba65f	BF16	4,096 tokens	Next-letter probabilities
Bespoke-Nimble	adapter 594dfdcf , Qwen3.5-9B c2022362	NF4, BF16 compute	2,048 tokens	Letter-code probabilities

Appendix

Table 6: Model revisions and inference settings. Hash prefixes identify the pinned checkpoints; hosted identifiers do not guarantee unchanged weights.

Model	Ch.	Co.	Br.	$\Delta$ F1	Ch.	Co.	Br.	$\Delta$ F1	Ch.	Co.	Br.	$\Delta$ F1	Ch.	Co.	Br.	$\Delta$ F1
	Dynamic				MHS				HateXplain				HateCheck
Jev	8.7	4.1	4.6	$-$ 0.5	10.6	5.2	5.4	$-$ 2.7	7.9	6.1	1.8	$+$ 3.9	2.8	1.4	1.4	$+$ 0.1
Laya	11.6	3.8	7.8	$-$ 4.4	8.0	5.1	2.9	$+$ 1.1	8.6	5.9	2.7	$+$ 2.7	8.0	2.5	5.5	$-$ 2.8
Decider	9.5	6.1	3.4	$+$ 3.3	4.5	1.8	2.7	$-$ 1.0	12.8	3.7	9.1	$-$ 4.7	15.6	11.4	4.1	$+$ 5.8
SemIf/Qwen3.5-2B	19.1	7.9	11.2	$+$ 0.7	7.1	3.0	4.2	$-$ 1.2	27.9	22.4	5.5	$+$ 19.7	6.3	2.8	3.5	$+$ 1.5
SemIf/Qwen3.5-4B	20.9	7.1	13.8	$-$ 7.2	16.3	11.6	4.7	$+$ 4.1	20.4	16.1	4.3	$+$ 10.3	11.6	4.6	7.1	$-$ 0.8

Appendix

Table 7: Prediction-level effect of the dataset-specific definition relative to no definition, as percentages of texts: predictions that change (Ch.), change from incorrect to correct (Co.), and change from correct to incorrect (Br.).

\Delta

F1 is the change in macro-F1, in percentage points. For HateXplain, the definition is the Davidson proxy.

	Binary $\Delta$ F1 (pp)				HX-3 macro-F1
Model	Dyn.	MHS	HX	HC	Gen.	Dav.
Jev	$-$ 0.1	$-$ 2.9	$+$ 5.3	$+$ 0.0	.522	.532
Laya	$-$ 3.2	$+$ 0.6	$+$ 2.5	$-$ 2.3	.582	.570
Decider	$+$ 3.6	$-$ 4.7	$-$ 3.3	$+$ 5.4	.527	.506
SemIf/Qwen3.5-2B	$-$ 1.9	$+$ 17.5	$+$ 23.6	$+$ 4.4	.277	.279
SemIf/Qwen3.5-4B	$-$ 8.3	$-$ 2.6	$+$ 11.0	$-$ 1.5	.405	.411

Appendix

Table 8: Effect of the generic definition. Left: change in binary macro-F1 from no definition to the generic definition, on the benchmark population (descriptive). Right: HateXplain three-class macro-F1 under the generic definition (Gen.) and the Davidson proxy (Dav.), on 1,000 posts.

	Median ms		95th percentile ms		Texts/s		USD / 1,000 texts
Model	A	B	A	B	A	B	A	B
Jev	335.6	342.9	536.9	622.1	2.56	2.37	0.033	0.054
Laya	48.3	52.5	92.6	93.9	18.68	17.49	-	-
Decider	133.6	134.6	292.5	264.9	6.31	6.45	-	-
SemIf/Qwen3.5-2B	125.3	250.5	272.1	540.4	6.56	3.28	-	-
SemIf/Qwen3.5-4B	183.7	343.8	325.4	555.6	4.82	2.63	-	-

Appendix

Table 9: Complete-decision timing and recorded hosted cost on HateCheck (

N{=}3{,}728

) for direct (A) and decomposed (B) classification. Dashes denote unassigned local monetary cost, not zero cost.

	F1			Offensive as
Model	Hate	Off.	Normal	hate	normal
Laya	.703	.316	.692	46%	32%
GPT-6 Sol	.694	.448	.671	42%	14%
TF-IDF+LR	.737	.473	.706	-	-
DistilBERT	.722	.448	.717	-	-

Appendix

Table 10: Per-class F1 on HateXplain’s three-class task (1,000 posts; 309 hate, 285 offensive, 406 normal), and the share of offensive posts labeled hate or normal.

Model	Dataset	A	B	A*	B $-$ A*
Laya	Dynamic	.655	.534	.650	$-$ 11.7 †
	MHS	.695	.732	.696	$+$ 3.6 †
	HateXplain	.733	.775	.795	$-$ 2.1
SemIf/Qwen3.5-4B	Dynamic	.674	.655	.718	$-$ 6.3 †
	MHS	.719	.681	.709	$-$ 2.8 †
	HateXplain	.688	.695	.737	$-$ 4.2

Appendix

Table 11: Direct (A), decomposed (B), and threshold-tuned direct classification (A*) macro-F1. B

-

A* is in percentage points. † Adjusted interval excludes zero (18 comparisons). ∗ 800 texts per dataset.

Hate Speech Detection

Hate Speech Detection identifies hateful or abusive language, its targets and intent, to support safer and more accountable content moderation.

See all 99 papers

Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation
Oct 23, 2026Karan YadavContent Moderation LLM Evaluation
Google
Attribute-Based Diagnosis of LLM Alignment with Hate Speech Annotations
May 26, 2026Mohammad Amine Jradi, Faeze Ghorbanpour, Alexander FraserLLM Evaluation LLM Alignment
School of Computation, Information and Technology, TU Munich Munich Center for Machine Learning (MCML)
Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation
Sep 3, 2026Danting Zhang, Bei Peng, Robert LoftinLLM Safety Benchmarks LLM Evaluation
1Independent Researcher · University of Sheffield

Oct 23, 2026J/K move · Enter open · S save

Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation

Karan Yadav

Google

Content Moderation LLM Evaluation Language Model Auditing Confidence Estimation in Language Models Language Model Calibration Annotator Disagreement

Open paper AskPDF arXiv

Content moderation compresses a graded, contested human judgment into a binary act (leave up / take down) over billions of items, at a scale where running a frontier LLM on every post is uneconomical. A new class of frozen "System One" models returns typed, calibrated decisions instead of text at ~USD 0.000025 per call and ~100 ms. We run the first independent audit of one such model on 142,658 human-labeled comments, and benchmark it against 14 frontier reasoning models (the GPT-5.x, Claude, Gemini 3, and Grok families) on a shared probe set. Two findings stand out. First, the cheap model is accuracy-competitive: its zero-shot classification AUROC (0.83) is not statistically separable (paired bootstrap, n=56; a failure to reject parity, not proof of equivalence) from 13 of the 14 frontier systems while costing 1-3 orders of magnitude less, and it is no worse calibrated than several of them. Second, "confidence" conflates two uncertainties that come apart: human disagreement (where annotators split on satire and coded language) and decision risk (where the automatic action is costly-wrong). Under an asymmetric moderation loss these call for opposite escalation signals, so no single confidence number serves both. Because the model prices extra questions at nearly zero, decomposing the judgment into parallel sub-questions and reading their spread recovers a human-disagreement signal that the model's native confidence misses -- but this remedy is specific: it helps the weakest, disagreement-blind models and only when the decomposition aligns with the sources of disagreement; a specialist toxicity model and the strongest LLMs do not need it, and the effect is distribution-sensitive. We will release the instrument, the cached runs, and the analysis.

Abstract

Hate Speech Detection

Explore similar work

Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation

Attribute-Based Diagnosis of LLM Alignment with Hate Speech Annotations

Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation