Toward Generalized Cross-Lingual Hateful Language Detection with Web-Scale Data and Ensemble LLM Annotations

Dang H. Dang; Jelena Mitrovi; Michael Granitzer

cs.CLMar 18, 2026

Toward Generalized Cross-Lingual Hateful Language Detection with Web-Scale Data and Ensemble LLM Annotations

Authors: Dang H. Dang, Jelena Mitrovi, Michael Granitzer

Organizations: University of Passau, Chair of Data Science

Abstract

We study whether large-scale unlabelled web data and LLM-based synthetic annotations can improve multilingual hate speech detection. Starting from texts crawled via OpenWebSearch.eu~(OWS) in four languages (English, German, Spanish, Vietnamese), we pursue two complementary strategies. First, we apply continued pre-training to BERT models by continuing masked language modelling on unlabelled OWS texts before supervised fine-tuning, and show that this yields an average macro-F1 gain of approximately 3% over standard baselines across sixteen benchmarks, with stronger gains in low-resource settings. Second, we use four open-source LLMs (Mistral-7B, Llama3.1-8B, Gemma2-9B, Qwen2.5-14B) to produce synthetic annotations through three ensemble strategies: mean averaging, majority voting, and a LightGBM meta-learner. The LightGBM ensemble consistently outperforms the other strategies. Fine-tuning on these synthetic labels substantially benefits a small model (Llama3.2-1B: +11% pooled F1), but provides only a modest gain for the larger Qwen2.5-14B (+0.6%). Our results indicate that the combination of web-scale unlabelled data and LLM-ensemble annotations is the most valuable for smaller models and low-resource languages.

Figures & tables

Panel of Figure 1 — Figure 1: End-to-end pipeline. Left (BERT track) : OWS texts are used for domain-adaptive continued pre-training; models are then fine-tuned on human labels. Right (LLM track) : four LLMs generate token-probability annotations; three ensemble strategies produce synthetic labels used to fine-tune smaller LLMs. The dashed arrow shows the cross-track path: ${\approx}$ 46K Qwen2.5-14B -annotated texts augment the BERT training data (7-Set + Synth.). Both tracks are evaluated on the same sixteen test sets. LGB = LightGBM.

Dataset	Train		Test
	Inst#	% Hate	Inst#	% Hate	Lang	Ref
HateXplain	15,299	0.59	3,846	0.59	eng	( Mathew et al., 2022 )
Sexism	10,904	0.13	2,632	0.13	eng	( Samory et al., 2021 )
Covid	1,282	0.19	971	0.20	eng	( He et al., 2022 )
US_election	1,283	0.12	1,117	0.13	eng	( 2021 )
HateEval-eng	9,000	0.42	1,000	0.43	eng	( Basile et al., 2019 )

Table 1: Sixteen human-labelled hate-speech datasets across four languages. Inst# = number of instances; % Hate = fraction labelled hateful. Datasets in bold form the 7-Set training subset.

Task / Model	Time	VRAM	Params
Annotation (240K, batch 128)
Mistral-7B	4 h	13 GB	–
Llama3.1-8B	4 h 45 m	23 GB	–
Gemma2-9B	6 h 30 m	43 GB	–
Qwen2.5-14B	7 h	42 GB	–
LoRA Fine-Tuning

Table 2: Cost per task on a single NVIDIA A6000 (48 GB).

Table 3: Macro-F1 (%) for BERT , HateBERT , and four OWS-BERT variants across six training configurations. Bold-underlined = best per row. Darker-shaded cells = in-domain results (training language matches test language). Per-language averages show the OWS – BERT delta (

\Delta

) on the OWS column.

	#	Base	Hum.	Mean	Vote	LGB	H+LGB
Llama3.2-1B
EN	7	51.7	59.7	61.5	61.9	67.3	75.4
DE	5	44.6	42.7	55.6	54.3	64.4	57.8
VI	1	47.1	59.3	58.5	57.6	58.3	66.1
ES	3	33.0	46.5	36.7	35.2	48.1	47.4
7-Set	7	50.2	60.2	62.6	62.2	66.9	69.7

Table 4: Pooled macro-F1 (%) by language and training split for LoRA-tuned LLMs. # = number of test sets; 7-Set / Rest = training vs. held-out datasets. Scores are computed on the pooled predictions within each group, not averaged across datasets. The classification threshold is the mean predicted probability of the Hate class, not the default 0.5. Full per-dataset breakdown in Table 8 (Appendix).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
objective	binary
metric	binary_logloss
boosting_type	gbdt
num_leaves	34
learning_rate	0.05
feature_fraction	0.9

Appendix

Table 5: LightGBM training configuration for the ensemble meta-learner.

	deu	eng	vie	All
	125 617	108 375	6 655	240 647
Per-model mean $P(\mathrm{Hate})$
Qwen2.5-14B	0.027	0.021	0.031	0.024
Gemma2-9B	0.107	0.039	0.113	0.080
Llama3.1-8B	0.002	0.001	0.001	0.011
Mistral-7B	0.026	0.015	0.023	0.021

Appendix

Table 6: Per-model and ensemble statistics of the 240K annotation pool, broken down by language. “% Hate” = fraction of texts with

P(\mathrm{Hate})>0.5

(per-model) or positive label (ensemble).

	deu	eng	vie	spa	All
	26 084	11 635	6 871	1 350	45 940
Qwen2.5-14B label counts
Neutral	22 702	4 043	6 341	1 143	34 229
Hate	3 284	3 943	474	149	7 850
Offensive	98	3 649	56	58	3 861
Qwen2.5-14B label %

Appendix

Table 7: Label distribution of the 46K BERT augmentation set, annotated by Qwen2.5-14B , broken down by language.

Table 8: Full per-dataset accuracy and macro-F1 (%) of Llama3.2-1B (top) and Qwen2.5-14B (bottom) under six training conditions. Base = zero-shot; Human = 7-Set human labels; Mean / Voting / LGB = synthetic-only; Human+LGB = combined. Signed values (

\pm

) are relative to Human . Bold = best F1 per row. The classification threshold is the mean predicted probability of the Hate class, not the default 0.5.

Hate Speech Detection

Hate Speech Detection identifies hateful or abusive language, its targets and intent, to support safer and more accountable content moderation.

See all 98 papers

Comparison of Modern Multilingual Text Embedding Techniques for Hate Speech Detection Task
Apr 16, 2026Evaldas Vaiciukynas, Paulius Danenas, Linas Ablonskis +5Sentence Embeddings Text Embedding Evaluation
Department of Information Systems, Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania · Centre of Information Systems Design Technologies, Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania
Efficient Adaptation of LLMs for Hate Speech Detection in Low-Resource Languages: A Comparative Study on Roman Urdu
Aug 6, 2026Toneema Zubair, Muhammad Junaid Asif, Faisal Kamiran +2LLM Evaluation Hate Speech Detection
Department of Computer Sciences Information Technology University Lahore (54000), Pakistan · Artificial Intelligence Technology Centre National Centre for Physics Islamabad (44000), Pakistan
ToxSyn-PT: A Synthetic Fine-Grained Dataset of Minority-Targeted Toxic Language in Portuguese
Jun 11, 2025Iago Alves Brito, Julia Soares Dollis, Fernanda Bufon Farber +2Synthetic Data Generation Hate Speech Detection
Advanced Knowledge Center for Immersive Technologies (AKCIT) · Federal University of Goiás

Apr 16, 2026J/K move · Enter open · S save

Comparison of Modern Multilingual Text Embedding Techniques for Hate Speech Detection Task

Evaldas Vaiciukynas, Paulius Danenas, Linas Ablonskis, Algirdas Sukys+4

Department of Information Systems, Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania · Centre of Information Systems Design Technologies, Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania

Sentence Embeddings Text Embedding Evaluation Hate Speech Detection Low-Resource Language Processing Multilingual Text Classification

Open paper AskPDF arXiv

Online hate speech and abusive language pose a growing challenge for content moderation, especially in multilingual settings and for low-resource languages such as Lithuanian. This paper investigates to what extent modern multilingual sentence embedding models can support accurate hate speech detection in Lithuanian, Russian, and English, and how their performance depends on downstream modeling choices and feature dimensionality. We introduce LtHate, a new Lithuanian hate speech corpus derived from news portals and social networks, and benchmark six modern multilingual encoders (potion, gemma, bge, snow, jina, e5) on LtHate, RuToxic, and EnSuperset using a unified Python pipeline. For each embedding, we train both a one class HBOS anomaly detector and a two class CatBoost classifier, with and without principal component analysis (PCA) compression to 64-dimensional feature vectors. Across all datasets, two class supervised models consistently and substantially outperform one class anomaly detection, with the best configurations achieving up to 80.96% accuracy and AUC ROC of 0.887 in Lithuanian (jina), 92.19% accuracy and AUC ROC of 0.978 in Russian (e5), and 77.21% accuracy and AUC ROC of 0.859 in English (e5 with PCA). PCA compression preserves almost all discriminative power in the supervised setting, while showing some negative impact for the unsupervised anomaly detection case. These results demonstrate how modern multilingual sentence embeddings combined with gradient boosted decision trees provide robust soft-computing solutions for multilingual hate speech detection applications.

Abstract

Hate Speech Detection

Explore similar work

Comparison of Modern Multilingual Text Embedding Techniques for Hate Speech Detection Task

Efficient Adaptation of LLMs for Hate Speech Detection in Low-Resource Languages: A Comparative Study on Roman Urdu

ToxSyn-PT: A Synthetic Fine-Grained Dataset of Minority-Targeted Toxic Language in Portuguese