An Explainable System for Offensive Content Detection in Nastaliq Urdu: Corpus Curation, Transformer Benchmarking, and Word-Level Attribution

Muhammad Shahid Iqbal Malik

cs.CLOct 11, 2026

An Explainable System for Offensive Content Detection in Nastaliq Urdu: Corpus Curation, Transformer Benchmarking, and Word-Level Attribution

Authors: Muhammad Shahid Iqbal Malik

Organizations: School of Computing Sciences, Pak-Austria Fachhochschule, Institute of Applied Sciences and Technology, Haripur 22620, Pakistan

Abstract

Automated moderation systems are deployed at scale, yet they remain opaque to the users and moderators affected by their decisions, and their coverage is almost entirely confined to English. For Urdu written in the Nastaliq script, no such system exists, and the annotated resources it would be built on are drawn largely from Twitter/X, where character limits constrain how users write. Interpretability has received little attention, leaving it unclear what such models actually learn. This study addresses these gaps in three steps. First, we curate and publicly release a corpus of 7604 Nastaliq Urdu Facebook comments, extending an earlier corpus by correcting text-level inconsistencies and re-examining the original labels. Second, we fine-tune four Urdu-capable transformers under a common grid-search protocol and compare them against twelve feature-classifier baselines on identical data partitions. The best configuration, Multilingual Representations for Indian Languages (MuRIL), reaches 87.05% accuracy and 87.06% macro F1, improving on the strongest baseline by 5.52 percentage points, a margin that exceeds the 95% confidence interval on test accuracy; the two leading transformers differ by less than the test set can resolve, and performance is stable across five optimizers. Third, we compute word-level SHAP attributions adapted to right-to-left rendering. Attribution mass concentrates on high-frequency slurs but extends to syntactically neutral function words and to political vocabulary characteristic of the source pages, indicating that aggregate performance may rest partly on lexical and topical regularities rather than a generalizable representation of offensiveness.

Figures & tables

Term	Definition	Term	Definition
OE	Offensive Expressions	NLP	Natural Language Processing
DL	Deep Learning	ML	Machine Learning
XAI	Explainable Artificial Intelligence	AI	Artificial Intelligence
SHAP	Shapley Additive Explanations	LR	Logistic Regression
Bi-LSTM	Bidirectional Long Short-Term Memory	OED	Offensive Expressions Detection
SVM	Support Vector Machine	LSTM	Long Short-Term Memory

Table 1: Glossary of key terms

Year [Ref]	Platform	Feature Engineering	Classification	Algorithms
2017 ( Davidson et al., 2017 )	English [Twitter]	TF-IDF, Part of speech, Reading ease scores	Hate speech and offensive content	SVM, LR
2019 ( Sigurbergsson & Derczynski, 2019 )	English & Danish [Reddit and Facebook]	Linguistic, sentiment, reading ease, pre-trained embeddings	Offensive vs not-offensive	LR, Bi-LSTM
2020 ( Çöltekin, 2020 )	Turkish [Twitter]	Word n-grams	Offensive vs not-offensive	SVM
2020 ( Pitenis et al., 2020 )	Greek [Twitter]	TF-IDF unigram and bigrams	Offensive vs not-offensive	SVM, NB, LSTM
2020 ( Mubarak et al., 2020 )	Arabic [Twitter]	Word embeddings	Offensive vs not-offensive	DT, RF, SVM, LR
2020 ( Akhter et al., 2020 )	Urdu & Roman Urdu [YouTube]	Word n-grams, char n-grams	Offensive vs not-offensive	NB, KNN, RF, SVM

Table 2: Summary of offensive/abusive content detection approaches in high and low resource languages

Fig. 1: Working flow of Explainable OED in Nastaliq Urdu

S#	Facebook pages	Before Pre-processing	After Pre-processing
S#	Facebook pages	# of Posts	# of Posts
1	Dunya News	1000	460
2	Samaa TV	300	167
3	Express News	1200	780
4	Bol News	450	120
5	ARY News	800	333

Table 3: Chosen Facebook pages

Urdu Text Preprocessing
1. Punctuation Removal
Before	After
Quaid-e-Azam used to read speeches after watching them and he was also very handsome ——	Quaid-e-Azam used to read speeches after watching them and he was also very handsome
2. Stop word Removal
Before	After
Now the police will take bribes for this just like they do for drugs	Police bribes drugs

Table 4: Demonstration of pre-processing steps on Urdu corpus with English translation

Dataset	Offensive	Not-Offensive	Total	Train	Validate	Test
Urdu	3802	3802	7604	5475	608	1521

Table 5: Urdu OE corpus

Table 6: Sample list of Urdu stop words

Fig. 2: Word cloud representation of the offensive dataset

Hyperparameters	Grid Search
Sequence length	64, 128
Batch size	16, 32, 64
Learning rate	3e-5, 2e-5, 1e-5, 1.5e-5, 0.99e-5
Weight decay	0.01–0.1 (step size = 0.01)
Warmup ratio	0.06–0.1 (step size = 0.001)
Hidden dropout	0.005–0.1 (step size = 0.01)

Table 7: Range of values for eight hyper-parameters ( M. S. I. Malik, Nazarova, et al., 2023 ; M. S. I. Malik, Younas, et al., 2024 )

Baseline Models	Tuning Parameter	Value
Word2vec	Architecture	Skip-gram
	Window Size	10
	Embedding Size	100
RoBERTa	Max Sequence Length	128
RoBERTa	Output_hidden_states	12 (Base)
MuRIL	Max Sequence Length	128

Table 8: Tuning parameters for the baseline models

Models	B.S	L.R	H.D	W.D	Accuracy	F1-score
Models	B.S	L.R	H.D	W.D	Accuracy	OE	Not	Macro	Weighted
Urdu-BERT	32	2e-5	0.1	0.1	80.87	80.19	81.50	80.85	80.86
	32	1e-5	0.05	0.01	80.93	80.33	81.51	80.92	80.91
	32	1e-5	0.1	0.01	81.13	81.75	80.46	81.11	81.12
	32	1e-5	0.005	0.01	82.18	81.55	82.77	82.16	82.17
Urdu-RoBERTa	32	1e-5	0.1	0.01	83.74	83.42	84.06	83.73	83.74

Table 9: Performance of fine-tuning for BERT, RoBERTa, Twhin-BERT, and MuRIL on the test dataset

Optimizer	Final L.R	H.D	W.D	Accuracy	F1-score
Optimizer	Final L.R	H.D	W.D	Accuracy	OE	Not	Macro	Weighted
AdaBelief	1e-5	0.005	0.1	86.19	86.52	85.89	86.20	86.20
	1.5e-5	0.05	0.01	86.88	86.68	87.02	86.89	86.88
	2e-5	0.01	0.01	86.94	87.09	86.70	86.94	86.95
	1e-5	0.01	0.01	87.03	87.21	86.86	87.05	87.05
AdaBound	1e-5	0.005	0.1	86.20	86.53	85.86	86.19	86.20

Table 10: Generality evaluation of fine-tuned MuRIL model across five state-of-the-art optimizers

Fig. 3: Response of Loss on train, and validate parts by fine-tuned MuRIL [for 15 epochs]

Classifiers	Accuracy	95% CI	F1-score		Precision	Recall
Classifiers	Accuracy	95% CI	Macro	Weighted	Precision	Recall
Word2vec+SVM	74.10	[71.84, 76.24]	73.75	73.75	75.46	74.10
Word2vec+LR	71.27	[68.94, 73.49]	70.96	70.95	72.24	71.28
Word2vec+RF	72.58	[70.28, 74.76]	72.43	72.43	73.09	72.59
Word2vec+CNN	73.31	[71.03, 75.47]	73.08	73.08	74.12	73.31
Urdu-RoBERTa + SVM	75.48	[73.26, 77.58]	75.48	75.48	75.48	75.48

Table 11: Comparison of four fine-tuned transformers with twelve baselines [95% CI denotes the Wilson score interval on test accuracy,

n=1{,}521

]

Fig. 4: Class-wise performance of classifiers and baselines (F1-score)

	Models	X-axis
		R-LR	M-SVM	M-RF	M-CNN	U-BERT	U-RBRT	U-TBERT	MuRIL
Y-axis	R-LR	–	0.201 $\|$ 0.098	0.201 $\|$ 0.088	0.201 $\|$ 0.083	0.201 $\|$ 0.086	0.201 $\|$ 0.059	0.201 $\|$ 0.048	0.201 $\|$ 0.039
	M-SVM	0.235 $\|$ 0.098	–	0.235 $\|$ 0.19	0.235 $\|$ 0.182	0.235 $\|$ 0.097	0.235 $\|$ 0.07	0.235 $\|$ 0.066	0.235 $\|$ 0.071
	M-RF	0.217 $\|$ 0.088	0.217 $\|$ 0.19	–	0.217 $\|$ 0.188	0.217 $\|$ 0.091	0.217 $\|$ 0.07	0.217 $\|$ 0.059	0.217 $\|$ 0.067
	M-CNN	0.213 $\|$ 0.083	0.213 $\|$ 0.182	0.213 $\|$ 0.188	–	0.213 $\|$ 0.076	0.213 $\|$ 0.064	0.213 $\|$ 0.05	0.213 $\|$ 0.058
	U-BERT	0.197 $\|$ 0.086	0.197 $\|$ 0.097	0.197 $\|$ 0.091	0.197 $\|$ 0.076	–	0.197 $\|$ 0.084	0.197 $\|$ 0.076	0.197 $\|$ 0.066

Table 12: Comparison of models using MARS Occlusion matrix

		X-axis
		BERT	RoBERTa	Twhin-BERT	MuRIL
Y-axis	BERT	–	0.01 $\|$ 0.026	0.01 $\|$ 0.025	0.01 $\|$ 0.034
	RoBERTa	0.012 $\|$ 0.026	–	0.012 $\|$ 0.039	0.012 $\|$ 0.05
	Twhin-BERT	0.014 $\|$ 0.025	0.014 $\|$ 0.039	–	0.014 $\|$ 0.058
	MuRIL	0.021 $\|$ 0.034	0.021 $\|$ 0.05	0.021 $\|$ 0.058	–

Table 13: Fine-tuned transformers evaluation using MARS Shine-Through measure

	Transformers	X-axis
		BERT	RoBERTa	Twhin-BERT	MuRIL
Y-axis	BERT	–	0.174 $\|$ 0.058	0.174 $\|$ 0.05	0.174 $\|$ 0.039
	RoBERTa	0.11 $\|$ 0.058	–	0.11 $\|$ 0.034	0.11 $\|$ 0.025
	Twhin-BERT	0.09 $\|$ 0.05	0.09 $\|$ 0.034	–	0.09 $\|$ 0.026
	MuRIL	0.072 $\|$ 0.039	0.072 $\|$ 0.025	0.072 $\|$ 0.026	–

Table 14: Fine-tuned transformers’ evaluation using MARS occlusion measure

Fig. 7: Global SHAP bar plot of the top-5 contributing words across offensive test instances

Table 15: Top-5 contributing features/words from offensive samples

Panel of Figure 23 — Fig. 8: Offensive sample 1, SHAP force plot with base value highlighting important words in shades

Fig. 9: SHAP bar plot containing Shapley values for words in Nastaliq Urdu OE [sample 1]

Panel of Figure 25 — Fig. 10: Offensive sample 2, SHAP force plot with base value highlighting important words in shades

Fig. 11: SHAP bar plot containing Shapley values for words in Nastaliq Urdu OE [sample 2]

Fig. 12: Offensive sample 3, SHAP force plot with base value highlighting important words in shades

Fig. 13: SHAP bar plot containing Shapley values for words in Nastaliq Urdu OE [sample 3]

Panel of Figure 29 — Fig. 14: Non-offensive sample 4, SHAP force plot with base value highlighting important words in shades

Fig. 15: SHAP bar plot containing SHAP values for words in Nastaliq Urdu OE [sample 4]

Hate Speech Detection

Hate Speech Detection identifies hateful or abusive language, its targets and intent, to support safer and more accountable content moderation.

See all 100 papers

MUTEX: Leveraging Multilingual Transformers and Conditional Random Fields for Enhanced Urdu Toxic Span Detection
Mar 5, 2026Inayat Arshad, Fajar Saleem, Ijaz HussainToxicity Detection Low-Resource Language Processing
Department of Computer and Information Sciences, Pakistan Institute of Engineering and Applied Sciences (PIEAS), Lehtrar Road, Nilore, Islamabad, Pakistan
'Ghaib in Translation' aka Unseen Harm: Measuring Cross-Script Safety Inconsistency with 'Missed-in-Urdu' Scores in LLM Hate Speech Detection
Aug 25, 2026Fawzia Zehra, Kara-Isitt, Sonal Khosla +1Multilingual Language Model Evaluation Language Model Safety Evaluation
Fuzzy · Brunel University of London, UK · Hasso Plattner Institute Potsdam, Germany
DunbaaBERT: From Sacrifice to Semantics
May 26, 2026Iffat Maab, Waleed Jamil, Raphael SchmittLanguage Model Pretraining Language Modeling
Research and Development Center for Large Language Models (LLMC), National Institute of Informatics, Tokyo · Independent Researcher, Edinburgh, United Kingdom · School of Computation, Information and Technology, Technical University of Munich, Germany +1

Mar 5, 2026J/K move · Enter open · S save

MUTEX: Leveraging Multilingual Transformers and Conditional Random Fields for Enhanced Urdu Toxic Span Detection

Inayat Arshad, Fajar Saleem, Ijaz Hussain

Department of Computer and Information Sciences, Pakistan Institute of Engineering and Applied Sciences (PIEAS), Lehtrar Road, Nilore, Islamabad, Pakistan

Toxicity Detection Low-Resource Language Processing

Open paper AskPDF arXiv

Urdu toxic span detection remains limited because most existing systems rely on sentence-level classification and fail to identify the specific toxic spans within those text. It is further exacerbated by the multiple factors i.e. lack of token-level annotated resources, linguistic complexity of Urdu, frequent code-switching, informal expressions, and rich morphological variations. In this research, we propose MUTEX: a multilingual transformer combined with conditional random fields (CRF) for Urdu toxic span detection framework that uses manually annotated token-level toxic span dataset to improve performance and interpretability. MUTEX uses XLM RoBERTa with CRF layer to perform sequence labeling and is tested on multi-domain data extracted from social media, online news, and YouTube reviews using token-level F1 to evaluate fine-grained span detection. The results indicate that MUTEX achieves 60% token-level F1 score that is the first supervised baseline for Urdu toxic span detection. Further examination reveals that transformer-based models are more effective at implicitly capturing the contextual toxicity and are able to address the issues of code-switching and morphological variation than other models.

	Models	X-axis
		R-LR	M-SVM	M-RF	M-CNN	U-BERT	U-RBRT	U-TBERT	MuRIL
Y-axis	R-LR	–	0.201 $\|$ 0.098	0.201 $\|$ 0.088	0.201 $\|$ 0.083	0.201 $\|$ 0.086	0.201 $\|$ 0.059	0.201 $\|$ 0.048	0.201 $\|$ 0.039
	M-SVM	0.235 $\|$ 0.098	–	0.235 $\|$ 0.19	0.235 $\|$ 0.182	0.235 $\|$ 0.097	0.235 $\|$ 0.07	0.235 $\|$ 0.066	0.235 $\|$ 0.071
	M-RF	0.217 $\|$ 0.088	0.217 $\|$ 0.19	–	0.217 $\|$ 0.188	0.217 $\|$ 0.091	0.217 $\|$ 0.07	0.217 $\|$ 0.059	0.217 $\|$ 0.067
	M-CNN	0.213 $\|$ 0.083	0.213 $\|$ 0.182	0.213 $\|$ 0.188	–	0.213 $\|$ 0.076	0.213 $\|$ 0.064	0.213 $\|$ 0.05	0.213 $\|$ 0.058
	U-BERT	0.197 $\|$ 0.086	0.197 $\|$ 0.097	0.197 $\|$ 0.091	0.197 $\|$ 0.076	–	0.197 $\|$ 0.084	0.197 $\|$ 0.076	0.197 $\|$ 0.066

Abstract

Hate Speech Detection

Explore similar work

MUTEX: Leveraging Multilingual Transformers and Conditional Random Fields for Enhanced Urdu Toxic Span Detection

'Ghaib in Translation' aka Unseen Harm: Measuring Cross-Script Safety Inconsistency with 'Missed-in-Urdu' Scores in LLM Hate Speech Detection

DunbaaBERT: From Sacrifice to Semantics