Hate speech has become increasingly prevalent on social media, raising concerns about its detection, interpretation, and moderation. Persian, a relatively low-resource language, lacks sufficient annotated data and computational resources for fine-grained hate speech analysis. In this study, we introduce a fine-grained, multi-hop benchmark for explainable Persian hate speech analysis, covering content triage, target identification, hate tactic and moral foundation analysis, textual evidence extraction, harmful narrative inference, and severity assessment. By evaluating state-of-the-art large language models (LLMs), we find that performance generally declines as tasks require more fine-grained semantic and inferential reasoning. Proprietary models generally outperform open-weight models, while reasoning-oriented models show no consistent advantage. Incorporating intermediate reasoning steps substantially improves severity prediction compared with one-shot assessment. Our analysis further reveals that Incitement to Violence or Harm is most strongly associated with higher severity, while political-affiliation targets are particularly associated with subversion-based moral framing. Overall, this work provides insights into LLM-based hate speech analysis and supports more accountable moderation of Persian-language social media. The benchmark is publicly available at https://github.com/kalhorghazal/From-Triage-to-Impact.
Figures & tables
Figure 1: Pipeline illustrating multi-hop reasoning steps, human verification, and LLM evaluation.
Hop
Definition
Output Schema
Initial Content Triage
Determines whether the content is harmful and identifies its primary category.
Labels: Offensive Language; Hate Speech; Dangerous Speech; Not Harmful.
Target Group Identification
Identifies the explicit or implicit target of the speech and the basis of identity targeted by the content.
Targeted Group (multi-label); Basis of Identity (multi-label). Labels provided in Table 2 .
Hate Tactic Identification
Analyzes the rhetorical strategy used to express or legitimize hostility toward the target.
Multi-label: Slurs/Derogatory Language; Dehumanization; Harmful Stereotyping; Incitement to Violence or Harm; Calls for Discrimination or Exclusion; Conspiracy Theories.
Moral Foundation Analysis
Identifies moral foundations invoked or violated to justify hostility, grounded in Moral Foundations Theory.
Extracts explicit textual evidence supporting the identified targets, tactics, and moral foundations.
Target Span; Tactic Span; Moral Span.
Harmful Narrative Inference
Infers the implicit ideological or toxic narrative promoted by the content.
Concise free-text narrative description.
Table 1: Definitions of the reasoning hops and their corresponding output schemas.
Category
Standard Values
Targeted Group
Iranian Officials & Politicians; Iranian Security & Military Forces; Domestic Political Opponents; Clerics & Religious Leaders; Human Rights, Media & Academic Activists; Migrants & Ethnic/Religious Minorities; Foreign Adversaries; Iranian Women; Iranian Men; Artists, Actors & Celebrities; Students, Children & Adolescents; Companies, Organizations, & Their Employees
Basis of Identity
Ethnicity/Nationality; Religion/Belief; Social Class; Gender; Political Affiliation; Migration Status; Occupation/Profession; Organizational/Institutional Affiliation; Group Membership; Social/Family Role; Personal Identity; Mental/Behavioral Traits; Educational/Academic Background; Humanity/Personhood
Table 2: Categories for targeted groups and identity bases used in the study.
Metric
Grok 4.6
GPT-5.2
o3
Sonnet 4.6
DeepSeek R1
DeepSeek V3
Qwen3 Think
Qwen3 Inst
Hop 1: Initial Content Triage
Macro Precision
82.7
82.6
79.0
79.5
77.0
76.7
67.8
69.6
Macro Recall
83.0
80.4
77.7
79.2
75.0
68.6
66.7
52.0
Macro F1
82.8
79.7
77.8
79.2
75.0
68.5
66.4
48.0
Hop 2: Targeted Group
Micro F1
71.1
66.0
68.1
66.7
60.1
58.3
42.7
43.0
Table 3: LLM performance across Hops 1–6 of the multi-hop Persian hate-speech reasoning benchmark. All metrics are reported on a 0–100 scale, with task-appropriate evaluation metrics used for each hop. Higher values indicate better performance.
Metric
Grok 4.6
GPT-5.2
o3
Sonnet 4.6
DeepSeek R1
DeepSeek V3
Qwen3 Think
Qwen3 Inst
Severity Level
Macro Precision
51.5
51.6
45.5
52.0
45.1
49.3
42.9
37.0
Macro Recall
53.1
51.7
47.7
40.5
34.4
43.0
31.9
33.5
Macro F1
50.7
50.8
44.6
43.5
32.1
43.9
27.9
29.9
Weighted Cohen’s κ
.829
.819
.760
.797
.558
.736
.525
.525
Severity Level (Ablation)
Table 4: LLM performance on hate speech severity (multi-hop and one-shot) and potential impact. All metrics are reported on a 0–100 scale, except for weighted Cohen’s κ .
Figure 2: Conditional probability distribution of identity bases given the presence of each moral foundation.
Figure 3: Conditional probability distribution of severity levels given the presence of each hate speech tactic.
Figure 4: Recall of hate speech detection across targeted groups for each LLM.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Data Source
Platform
No. in Benchmark
Kebriaei et al.
X
146
Sabri et al.
X
25
Elahimanesh et al.
X
46
Davardoust et al.
Instagram
81
Nazarizadeh et al.
Instagram
50
Appendix
Table 5: Statistics of datasets used to construct our benchmark, including source, platform, and instance counts.
Hop / Annotation Dimension
IAA Results
Hop 1: Initial Content Triage
Cohen’s κ : .743
Observed Agreement: 81.0
Hop 2: Targeted Group
Micro F1: 82.0
Macro F1: 69.3
Hop 2: Basis of Identity
Micro F1: 66.3
Macro F1: 51.2
Appendix
Table 6: Inter-annotator agreement results across annotation hops.
Hop / Annotation Dimension
Human–LLM Alignment Results
Hop 1: Initial Content Triage
Macro Precision: 89.0
Macro Recall: 90.3
Macro F1: 88.8
Hop 2: Targeted Group
Micro F1: 94.0
Macro F1: 94.3
Hop 2: Basis of Identity
Micro F1: 88.2
Appendix
Table 7: Human–LLM alignment results across annotation hops.
Figure 5: Prompt template for detecting hate speech keywords from a given text, along with its English translation.
Figure 6: Prompt template for detecting offensive language keywords from a given text, along with its English translation.
Figure 7: Prompt template for detecting dangerous speech keywords from a given text, along with its English translation.
Figure 8: Prompt template for single-step severity classification of Persian hate speech in the ablation study, along with its English translation.
Figure 9: Prompt template for multi-hop analytical annotation of Persian hate speech.
Figure 10: English translation of the prompt template for multi-hop analytical annotation of Persian hate speech.
Hate Speech Tactic
χ2
df
padj
Cramer’s V
Slurs/Derogatory Language
118.79
3
<.001
.363
Dehumanization
63.48
3
<.001
.266
Harmful Stereotyping
71.53
3
<.001
.282
Calls for Discrimination or Exclusion
94.83
3
<.001
.325
Incitement to Violence or Harm
645.77
3
<.001
.847
Conspiracy Theories
6.96
3
.073
.088
Appendix
Table 8: Associations between hate speech tactics and severity levels. p -values are adjusted using the Benjamini–Hochberg procedure; Cramer’s V reports effect sizes.
Moral Foundation
Identity Basis
pBH
Cramér’s V
Betrayal
Political Affiliation
<.001
.271
Harm
Personal Identity
<.001
.281
Harm
Organizational/Institutional Affiliation
<.001
.273
Degradation
Religion/Belief
<.001
.209
Subversion
Organizational/Institutional Affiliation
<.001
.199
Degradation
Organizational/Institutional Affiliation
<.001
.194
Appendix
Table 9: Significant associations between moral foundations and identity bases. Associations are tested using χ2 tests of independence based on instance-level presence/absence indicators. p -values are corrected using the Benjamini–Hochberg procedure. Cramér’s V is reported as the effect size.
We introduce ParsHate, a manually annotated dataset of 10,000 Persian tweets spanning 2013-2022, representing the first decade-long benchmark for hate speech detection in Persian. The dataset contains 31% hateful content and supports both hate detection and multi-label fine-grained target identification across seven structured target categories. ParsHate also distinguishes explicit and implicit hate, marks explicit and implicit targets, and provides span-level rationales. Data collection combines random and score-stratified temporal sampling to reduce keyword-driven bias while preserving natural label distributions. Applying SOTA models for Persian hate-speech detection on ParsHate shows moderate performance (79% F1), especially with samples from earlier years, and low performance with target identification (25.5% macro-F1). This emphasizes the diverse sampling of hate speech in ParsHate and its challenging nature that requires more advanced methods for better performance. Dataset is made publicly available.
It is challenging to detect hate speech in Low Resource Languages (LRLs) because of the absence of annotated data, the informality of its language structure, and the lack of standardized grammar. A good example of such a challenge is Roman Urdu which is broadly used by South Asians on social media and has a high variation while lacking contextually consistent spellings. The objective of this paper is to conduct a comprehensive assessment of Large Language Models (LLMs) for Hate Speech Detection (HSD) in Roman Urdu script and fine-tune these models using the Parameter-Efficient Fine-Tuning (PEFT) method called Low-Rank Adaptation (LoRA). To evaluate zero-shot inference, we benchmarked it against PEFT on different transformer models, including Mistral, LLaMA, Falcon, and multilingual BERT. Experiments are conducted on the PURUTT (Parallel Urdu and Roman Urdu Corpus for Toxic Comments and Transliteration) dataset with over 72,000 annotated comments. The results suggest that zero shot models perform moderately (F1 = 0.56), but updating a small fraction of the model trainable parameters improves the classification performance significantly (F1 > 0.93). Our results have shown that PEFT delivers outstanding performance alongside excellent computational efficiency, making it highly suitable for low-resource language processing tasks.
Toneema Zubair, Muhammad Junaid Asif, Faisal Kamiran +2
Department of Computer Sciences Information Technology University Lahore (54000), Pakistan · Artificial Intelligence Technology Centre National Centre for Physics Islamabad (44000), Pakistan
This paper presents a comprehensive evaluation framework for assessing the cultural competence of large language models (LLMs) in Persian. Existing Persian cultural benchmarks rely predominantly on multiple-choice formats and English-centric metrics that fail to capture Persian's morphological complexity and semantic nuance. Our framework introduces a Persian-specific short-answer evaluation that combines rule-based morphological normalization with a hybrid syntactic and semantic similarity module, enabling robust soft-match scoring beyond exact string overlap. Through systematic evaluation of 15 state-of-the-art open- and closed-source models across three culturally grounded Persian datasets, we demonstrate that our hybrid evaluation improves scoring consistency by +10 compared to exact-match baselines by capturing meaning that surface-level methods cannot detect. Our human evaluation further confirms that the proposed semantic similarity metric achieves higher agreement with human judgments than LLM-based judges. We publicly release our evaluation framework, providing the first standardized benchmark for measuring cultural understanding in Persian and establishing a reproducible foundation for cross-cultural LLM evaluation research.