cs.CLAug 26, 2026
SaveFrom Specialization to Generalization: Instruction-tuned LLMs for Robust Harmful Content Mitigation
Organizations: School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
Abstract
Large language models (LLMs) demonstrate impressive performance across a wide range of general NLP tasks; however, their effectiveness in sensitive domains, such as hate speech detection, remains less clear. Prior studies comparing prompted LLMs with state-of-the-art encoder-based models (e.g., BERT variants (Roy et al., 2023; Dönmez et al., 2024)) have shown only marginal gains, suggesting that LLMs may not excel in hate speech detection or mitigation. In this work, we revisit this question through the lens of instruction tuning. By thoroughly unifying 36 English hate speech datasets spanning multiple labeling schemes, we fine-tune a generalist LLM, based on Qwen3 (Qwen Team, 2025), specifically for hate speech mitigation. Our results demonstrate not only state-of-the-art performance on in-domain benchmarks but also substantial improvements in cross-domain and cross-lingual generalization--areas where encoder-based specialist classifiers often struggle.
Figures & tables
Figure 1: In this work, we would like to close the gap in understanding if instruction-tuned general-purpose LLMs (i.e., HIPPO, right) are more robust and precise detectors of hate speech than specially-tuned LMs (i.e., BERT and Llama, left).
| Dataset | Attribution | Train | Test | Domain | Label Type | (Sub)Tasks |
| CAD | Vidgen et al. (2021) | 13423 | Multilabel | 1 | ||
| CAT-LARGE | Pavlopoulos et al. (2020) | 19870 | Binary | 1 | ||
| CONAN | Chung et al. (2019) | 408 | Synthetic | Generation | 1 | |
| ConvAbuse | Cercas Curry et al. (2021) | 4013 | Chatbot | Hierarchical | 4 | |
| DialoCONAN | Bonaldi et al. (2022) | 1299 | Synthetic | Generation | 1 | |
| EAP | Vidgen et al. (2020) | 19989 | Multiclass | 1 |
Table 1: List of datasets used and the number of examples, domain, and type of labels. Domains include Twitter ( ), Reddit ( ), Gab ( ), Facebook ( ), Wikipedia ( ), YouTube ( ), and Civil Comments (CC), among others.
| Dataset | Type | HIPPO | Indiv. | GPT5 | Best | Best Arch. | Best Source |
| AMI18 - A | Binary | 81.0 | 74.8 | 70.9 | 71.0 | BERT | Muti and Barrón-Cedeño (2022) |
| AMI18 - B | Binary x2 | 73.4 | 64.3 | 44.4 | 42.9 | BERT | Pamungkas et al. (2020b) |
| EDOS - A | Binary | 83.7 | 85.0 | 73.0 | 88.4 | PaLM | Sorensen et al. (2023) |
| EDOS - B | Multiclass | 71.2 | 65.5 | 41.9 | 73.3 | PaLM | Sorensen et al. (2023) |
| EDOS - C | Multiclass | 51.8 | 47.5 | 41.0 | 56.1 | DeBERTa-v3 | Zhou (2023) |
| HASOC19 - A | Binary | 76.7 | 79.0 | 72.7 | 79.5 | RoBERTa † | Kovács et al. (2021) |
Table 2: Comparison of our unified model ( HIPPO , based on Qwen3-4B-Instruct-2507 ) to individually trained models (Indiv.), GPT5-mini, and to the best performing models in literature. Bold values denote the best results per dataset. denotes ensembles.
| Dataset | 0.6B | 4B | 32B | Best |
| AMI18 - A | 28.1 | 81.0 | 82.9 | 71.0 |
| AMI18 - B | 26.0 | 73.4 | 75.5 | 42.9 |
| EDOS - A | 23.5 | 83.7 | 87.0 | 88.4 |
| EDOS - B | 42.0 | 71.2 | 74.7 | 73.3 |
| EDOS - C | 27.5 | 51.8 | 57.9 | 56.1 |
| HASOC19 - A | 22.2 | 76.7 | 76.6 | 79.5 |
Table 3: Effectiveness of instruction tuning on varying model sizes, compared to the SOTA. Blue values indicate scores that outperform the SOTA. Bold values denote the best results per dataset.
| Dataset | Zero-shot | LOO | Full |
| AMI18 - A | 59.8 | 78.4 | 81.0 |
| AMI18 - B † | 41.6 | 42.1 | 73.4 |
| EDOS - A | 66.7 | 76.0 | 83.7 |
| EDOS - B ‡ | 37.9 | 43.4 | 71.2 |
| EDOS - C ‡ | 20.2 | 27.6 | 51.8 |
| HASOC19 - A | 71.5 | 70.3 | 76.7 |
Table 4: Comparison of zero-shot to leave-one-out (LOO) and full HIPPO (i.e., unified) training. denotes tasks that do not have equivalent tasks in training of Qwen3-4B . denotes partial equivalence: AMI18-B and HatEval19-B are composed of 2 tasks, both of which include 1 task that has no similar task in training. Blue values indicate scores that outperform the zero-shot baseline. Bold values denote the best results per dataset.
Figure 2: Cross-lingual and functional performance of the zero-shot, minimally-trained, and fully-trained HIPPO . Note: Unlike for English, HASOC19 does not have a Task C for German and Hindi.
Figure 3: Performance of the fully-trained HIPPO on all languages (ordered by performance) of HateCheck, per functional task. Above the black line are hateful functionalities, below are non-hateful. Only functionalities shared across all languages are shown.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
| Batch size | 64 |
| Learning rate | 1e-4 |
| Weight decay | 1e-3 |
| Warmup ratio | 1e-2 |
| Epochs | 3 |
| LoRA rank | 32 |
Table 5: Hyperparameters used in training.
Figure 4: Top-level label distribution of all datasets.
Figure 5: Fine-grained label distribution of the hierarchical datasets.
| Resource | License | Homepage |
| CAD | CC BY 4.0 | https://github.com/dongpng/cad_naacl2021 |
| CAT-LARGE | Apache-2.0 | https://github.com/ipavlopoulos/context_toxicity |
| CONAN | Research Only | https://github.com/marcoguerini/CONAN |
| ConvAbuse | CC BY 4.0 | https://github.com/amandacurry/convabuse |
| DialoCONAN | Research Only | https://github.com/marcoguerini/CONAN |
| EAP | CC BY 4.0 | https://zenodo.org/records/3816667 |
Table 6: Overview of the licenses associated with each resource utilized in this work for experiments.
| Dataset | Examples | Label |
| AMI18 Subtask 1 | There are some wins that will make u go hysterical. Some makes you happy but same time you are sad | Non-misogyny |
| Shut the fck up you fcking monkey jew btch cnt ngger dck l*cking | Misogyny | |
| he thing is if a woman speaks out about rape on Nigerian twitter.The backlash would be shocking. It’s not good for the mind. Hence silence | Non-misogyny | |
| EDOS Subtask 2 | MRA stuff like men saying they wished they hadn’t had daughters, calling ALL American women wh*res, etc. Evidence if this? | Threats |
| Is this ASD (anti sl*t defence) or what? How do I prevent this? Spend more time with her before and after. | Derogation | |
| Not our fault she chose to date an abusive sc*mbag. | Animosity |
Table 7: Examples from a few classification datasets used in our work for experiments. We included multiple subtasks and domains, covering just high-level classification of hate/non-hate as well as other more fine-grained labels.
| Dataset | Examples |
| ToxicSpans | Input Text : your an idot, this is a tax based on a lie Toxic Span : idot |
| Input Text : Have you walked around Denver lately , or Civic Center Park ? These poor homeless people are pgs ! Toxic Span : pgs | |
| Input Text : Mron. Illegals have no Constitutional rights Toxic Span : Mron | |
| ParaDetox | Input Text : the police dont have time for this cr*p. Detoxified : The police have no time for this. |
| Input Text : its a crock of sh*t , and you know it . Detoxified : It’s senseless, you know | |
| Input Text : i didnt vote for the liar Detoxified : I didn’t vote for him |
Table 8: Examples from a few generation datasets used in our work for experiments. We included not only classification, but generation tasks showing that instruction-tuned model can be used for more variety of proactive ways for hate speech mitigation.
| Dataset | Llama3.2 | Phi4-mini | Qwen3 | HIPPO |
| AMI18 - A | 78.7 | 75.7 | 79.2 | 81.0 |
| AMI18 - B | 70.3 | 54.8 | 72.5 | 73.4 |
| EDOS - A | 83.5 | 83.5 | 84.6 | 83.7 |
| EDOS - B | 70.0 | 61.9 | 67.5 | 71.2 |
| EDOS - C | 43.8 | 41.4 | 47.5 | 51.8 |
| HASOC19 - A | 77.6 | 74.5 | 78.1 | 76.7 |
Table 9: Comparison of model types. We can observe that Qwen3 family models ( Qwen3 and HIPPO ) perform better and more stable in comparison to other same-sized models from other families.
Figure 6: Confusion matrices for all classification-based tasks in the test set. Predicted label categories correspond to the true label categories, abbreviated for brevity.
Figure 7: Difference of confusion matrices between the unified model and individual models. Blue indicates better performance for unified, red vice versa.
More in this topic
Content Moderation
Content moderation studies how to detect, interpret, and respond to harmful or policy-violating material across digital platforms.
See all 87 papers