DariMis: Harm-Aware Modeling for Dari Misinformation Detection on YouTube

Jawid Ahmad Baktash; Mosa Ebrahimi; Mohammad Zarif Joya; Mursal Dawodi

cs.CLMar 24, 2026

DariMis: Harm-Aware Modeling for Dari Misinformation Detection on YouTube

Authors: Jawid Ahmad Baktash, Mosa Ebrahimi, Mohammad Zarif Joya, Mursal Dawodi

Organizations: Technical University of Munich, Germany · Syiah Kuala University · Mainz University of Applied Sciences, Germany

Abstract

Dari, the primary language of Afghanistan, is spoken by tens of millions of people yet remains largely absent from the misinformation detection literature. We address this gap with DariMis, the first manually annotated dataset of 9,224 Dari-language YouTube videos, labeled across two dimensions: Information Type (Misinformation, Partly True, True) and Harm Level (Low, Medium, High). A central empirical finding is that these dimensions are structurally coupled, not independent: 55.9 percent of Misinformation carries at least Medium harm potential, compared with only 1.0 percent of True content. This enables Information Type classifiers to function as implicit harm-triage filters in content moderation pipelines. We further propose a pair-input encoding strategy that represents the video title and description as separate BERT segment inputs, explicitly modeling the semantic relationship between headline claims and body content, a key signal of misleading information. An ablation study against single-field concatenation shows that pair-input encoding yields a 7.0 percentage point gain in Misinformation recall (60.1 percent to 67.1 percent), the safety-critical minority class, despite modest overall macro F1 differences (0.09 percentage points). We benchmark a Dari/Farsi-specialized model (ParsBERT) against XLM-RoBERTa-base; ParsBERT achieves the best test performance with accuracy of 76.60 percent and macro F1 of 72.77 percent. Bootstrap 95 percent confidence intervals are reported for all metrics, and we discuss both the practical significance and statistical limitations of the results.

Figures & tables

Statistic	Value	Notes
Total collected	10,587	Raw API harvest
Final annotated	9,224	After cleaning
Partly True	5,535 (60.0%)	Majority class
Misinformation	2,082 (22.6%)	Second class
True	1,607 (17.4%)	Minority class
Missing descr.	3,304 (31.2%)	Title-only

TABLE I: DariMis-Dataset Statistics

Panel of Figure 2 — Fig. 1: Class distributions in DariMis (a) Information Type and (b) Harm Level. Partly True dominates at 60%; Low harm accounts for 74.1% of videos.

Panel of Figure 3 — Fig. 2: Harm Level within each Information Type: (left) absolute counts; (right) row-normalised proportions. Misinformation has 55.9% of instances at Medium or High harm versus only 1.0% for True content.

Info. Type	Low	Med.	High	High%	$\geq$ Med.%
Misinformation	919	786	377	18.1%	55.9%
Partly True	4,329	1,136	70	1.3%	21.8%
True	1,591	16	0	0.0%	1.0%
Total	6,839	1,938	447	4.9%	25.9%

TABLE II: Cross-Tabulation: Information Type

\times

Harm Level

Fig. 3: Heatmaps of the Information Type

\times

Harm Level cross-tabulation: (a) raw counts; (b) row-normalised percentages. The concentration of Misinformation in the Medium and High harm columns, and the near-exclusive Low-harm profile of True content, confirm the structural accuracy–harm coupling.

Fig. 4: Overview of the proposed pair-input modeling framework for DariMis. The video title and description are routed into separate BERT segment inputs (Seg A and Seg B), enabling cross-segment self-attention to capture headline–body semantic inconsistencies— a primary signal of misleading content. The predicted Information Type implicitly encodes harm level (55.9% of Misinformation carries $\geq$ Medium harm), enabling downstream moderation triage without a dedicated harm classifier.

Split	Total	Misinfo.	Partly T.	True
Train (70%)	6,498	$\approx$ 1,457	$\approx$ 3,875	$\approx$ 1,166
Val. (15%)	1,392	$\approx$ 313	$\approx$ 839	$\approx$ 240
Test (15%)	1,393	313	839	241

TABLE III: Stratified Dataset Splits

Input Strategy	Acc.	Mac. F1	Mis. Rec.	Mis. F1
Single (concat)	77.46	72.68	0.601	0.680
Pair (ours)	76.60	72.77	0.671	0.692
$\Delta$ pair vs. single		$+$ 0.09	$+$ 7.0 pp	$+$ 1.2 pp

TABLE IV: Ablation: Input Encoding Strategy (ParsBERT)

Model	Acc.	Mac. F1	95% CI (F1)
ParsBERT (pair)	76.60	72.77	[70.05, 75.32]
ParsBERT (single)	77.46	72.68	[69.98, 75.29]
XLM-RoBERTa-base	74.66	70.83	[68.11, 73.57]

TABLE V: Overall Test-Set Performance with Bootstrap 95% CI

Fig. 5: Test-set performance of both models on all four metrics. ParsBERT with pair-input encoding achieves the highest macro F1 and best Misinformation recall.

Class	Prec.	Rec.	F1	Support
Misinformation	0.714	0.671	0.692	313
Partly True	0.803	0.833	0.818	839
True	0.693	0.656	0.674	241
Macro avg.	0.737	0.720	0.728	1,393
Weighted avg.	0.761	0.766	0.763	1,393

TABLE VI: Per-Class Results — ParsBERT Pair Input (Test Set)

Fig. 6: Per-class F1 for both models. XLM-RoBERTa-base shows near-collapse on Misinformation (F1 = 0.26) and True (F1 = 0.03), while ParsBERT maintains consistent performance across all three classes.

Fig. 7: Confusion matrix for ParsBERT pair input (test set). Diagonal entries are correct predictions. Partly True acts as a semantic attractor: 56.8% of all errors involve it as either the true or predicted class.

True Label	Predicted As	Count	% Errors
Misinformation	Partly True	95	29.1%
Partly True	Misinformation	78	23.9%
True	Partly True	77	23.6%
Partly True	True	62	19.0%
Misinformation	True	8	2.5%
True	Misinformation	6	1.8%

TABLE VII: Error Breakdown — ParsBERT Pair Input (Test Set)

Fake News Detection

Fake news detection identifies false or misleading claims by analyzing content, context, evidence, and how information spreads.

See all 98 papers

DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects
Apr 7, 2026Jason Lucas, Matt Murtagh, Ali Al-Lawati +3Fake News Detection Dialect Bias in Language Models
The Pennsylvania State University, USA · Trinity College Dublin, Ireland · MIT Lincoln Laboratory, USA
Misinformation Span Detection in Videos via Audio Transcripts
Apr 23, 2026Breno Matos, Rennan C. Lima, Savvas Zannettou +2Automated Fact-Checking
Max Planck Institute for Informatics · Federal University of Minas Gerais · Kunumi Institute +1
MALicious INTent Dataset and Inoculating LLMs for Enhanced Disinformation Detection
Mar 15, 2026Arkadiusz Modzelewski, Witold Sosnowski, Eleni Papadopulos +4Fake News Detection Intent Classification
University of Padua, Italy · Polish-Japanese Academy of Information Technology, Poland · NASK National Research Institute, Poland +1

Apr 7, 2026J/K move · Enter open · S save

DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects

Jason Lucas, Matt Murtagh, Ali Al-Lawati, Uchendu Uchendu+2

The Pennsylvania State University, USA · Trinity College Dublin, Ireland · MIT Lincoln Laboratory, USA

Fake News Detection Dialect Bias in Language Models Automated Fact-Checking Language Model Robustness

Open paper AskPDF arXiv

Harmful content detectors, particularly disinformation classifiers, are predominantly developed and evaluated on Standard American English (SAE), leaving their robustness to dialectal variation unexplored. We present DIA-HARM, the first benchmark for evaluating disinformation detection robustness across 50 English dialects spanning U.S., British, African, Caribbean, and Asia-Pacific varieties. Using Multi-VALUE's linguistically grounded transformations, we introduce D-CUBE (Dialectal Disinformation Detection Corpus), a core corpus component of DIA-HARM comprising 195K samples derived from established disinformation benchmarks. Our evaluation of 16 detection models reveals systematic vulnerabilities: human-written dialectal content degrades detection by 1.4-3.6% F1, while AI-generated content remains stable. Fine-tuned transformers substantially outperform zero-shot LLMs (96.6% vs. 78.3% best-case F1), with some models exhibiting catastrophic failures exceeding 33% degradation on mixed content. Cross-dialectal transfer analysis across 2,450 dialect pairs shows that multilingual models (mDeBERTa: 97.2% average F1) generalize effectively, while monolingual models like RoBERTa and XLM-RoBERTa fail on dialectal inputs. These findings demonstrate that current disinformation detectors may systematically disadvantage hundreds of millions of non-SAE speakers worldwide. We release the DIA-HARM benchmark, including the D-CUBE corpus (https://github.com/jsl5710/dia-harm), and evaluation tools (https://jsl5710.github.io/dia-harm).

Abstract

Fake News Detection

Explore similar work

DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects

Misinformation Span Detection in Videos via Audio Transcripts

MALicious INTent Dataset and Inoculating LLMs for Enhanced Disinformation Detection