MUTEX: Leveraging Multilingual Transformers and Conditional Random Fields for Enhanced Urdu Toxic Span Detection

Inayat Arshad; Fajar Saleem; Ijaz Hussain

cs.CLMar 5, 2026

MUTEX: Leveraging Multilingual Transformers and Conditional Random Fields for Enhanced Urdu Toxic Span Detection

Authors: Inayat Arshad, Fajar Saleem, Ijaz Hussain

Organizations: Department of Computer and Information Sciences, Pakistan Institute of Engineering and Applied Sciences (PIEAS), Lehtrar Road, Nilore, Islamabad, Pakistan

Abstract

Urdu toxic span detection remains limited because most existing systems rely on sentence-level classification and fail to identify the specific toxic spans within those text. It is further exacerbated by the multiple factors i.e. lack of token-level annotated resources, linguistic complexity of Urdu, frequent code-switching, informal expressions, and rich morphological variations. In this research, we propose MUTEX: a multilingual transformer combined with conditional random fields (CRF) for Urdu toxic span detection framework that uses manually annotated token-level toxic span dataset to improve performance and interpretability. MUTEX uses XLM RoBERTa with CRF layer to perform sequence labeling and is tested on multi-domain data extracted from social media, online news, and YouTube reviews using token-level F1 to evaluate fine-grained span detection. The results indicate that MUTEX achieves 60% token-level F1 score that is the first supervised baseline for Urdu toxic span detection. Further examination reveals that transformer-based models are more effective at implicitly capturing the contextual toxicity and are able to address the issues of code-switching and morphological variation than other models.

Figures & tables

Ref.	Approach	Model/Method	Lang.	Strengths	Limitations
[ 25 ]	Span Masking Pretraining	SpanBERT	English	Better span boundaries via span-MLM	English only; high pretraining data demand
[ 1 ]	Document-level Classification	Logist. Reg. + CNN	English	Fast inference; effective for coarse-grained filtering	Lacks interpretability; cannot identify specific toxic segments
[ 2 ]	Binary Toxicity Classification	Logist. Reg. (n-grams)	English	Simple baseline; interpretable features	Cannot distinguish hate speech from offensive language
[ 7 ]	Character-level Span Detection	Transformers	English	Fine-grained detection; benchmark F1: 68.5–70.83%	Character-level evaluation is computationally expensive
[ 11 ]	Explainable Hate Speech	BERT + Attention	English	Provides rationales; human-annotated explanations	Requires expensive human annotations; sentence-level only
[ 12 ]	Sequence Tagging	BiLSTM-CRF	Multi.	Effective for NER; models label dependencies	Requires extensive feature engineering; slower than transformers

Table 1: Comparative Analysis of Toxic Span Detection Approaches

Figure 1: Data collection and annotation pipeline for the Urdu toxic span detection dataset.

Source	Samples	Toxic %	Non-Toxic %
Social Media	5,254	57	43
Urdu Newspapers	4,300	52	48
YouTube Text	4,788	56	44
Total	14,342	54	46

Table 2: Dataset statistics per source domain

Metric	Mean	Min	Max
Post length (tokens)	102	4	980
Dense toxic span length	7.5	3	90
% Dense toxic tokens	0.60	0.05	0.85

Table 3: Exploratory statistics: post lengths and dense toxic spans

Figure 2: Distribution of toxicity categories in the annotated Urdu dataset

Figure 3: Proposed system architecture for Urdu toxic span detection

\mathcal{L}=\sum_{(x,y,d)\in\mathcal{B}}w_{d}\cdot\text{Loss}(f_{\theta}(x),y)

Algorithm 1 Multi-Domain Training for Toxic Span Detection

Figure 4: System output interface for Urdu toxic span detection showing highlighted toxic spans with color-coded severity levels

Model	CRF	Precision	Recall	F1-Score	p-value
BiLSTM	$\times$	$57.0$ 1.2	$55.0$ 1.4	$56.0$ 1.3	$p<0.001^{**}$
mBERT	$\times$	$57.0$ 1.1	$55.0$ 1.3	$56.0$ 1.2	$p<0.001^{**}$
XLM-RoBERTa	$\times$	$60.0$ 0.9	$58.0$ 1.0	$59.0$ 0.9	$p=0.023^{*}$
XLM-RoBERTa+CRF	$\checkmark$	$61.0$ 0.8	$59.0$ 0.9	$60.0$ 0.8	—

Table 4: Token-level toxic span detection performance on the test set of 1,434 samples. Models were selected based on validation F1-score using early stopping. Results averaged over 5 runs with different random seeds (42, 123, 456, 789, 1011). Paired t-test comparing to XLM-RoBERTa+CRF: * p < 0.05, ** p < 0.01.

Figure 5: Performance comparison of baseline and proposed models on Urdu toxic span detection. XLM-RoBERTa+CRF achieves the highest token-level F1-score of 60.0%, demonstrating the effectiveness of combining multilingual contextual embeddings with structured sequence prediction. The CRF layer improves F1 by 1.0 percentage point over XLM-RoBERTa alone by enforcing valid BIO tag sequences.

Category	Samples	F1-Score	Avg Span Length
Hate Speech	2,145	62.3%	8.2 tokens
Personal Insults	3,892	61.5%	6.8 tokens
Offensive Language	5,124	58.7%	7.1 tokens
Profanity	1,652	64.1%	3.5 tokens
Overall	14,342	60.0%	7.5 tokens

Table 5: Token-level F1 performance by toxicity category using XLM-RoBERTa+CRF

Domain	Precision	Recall	F1-Score
Social Media	58.5%	56.8%	57.6%
News	63.2%	61.4%	62.3%
YouTube	59.8%	58.1%	58.9%
Multi-Domain (Overall)	61.0%	59.0%	60.0%

Table 6: Domain-specific performance of XLM-RoBERTa+CRF with multi-domain training

Test Domain	Single-Domain F1	Multi-Domain F1	$\Delta$ F1
Social Media	61.3%	57.6%	-3.7%
News	59.3%	62.3%	+3.0%
YouTube	60.7%	58.9%	-1.8%
Average	60.4%	59.6%	-0.8%

Table 7: Comparison of multi-domain vs. single-domain training strategies

Train $\rightarrow$ Test	Social Media	News	YouTube
Social Media	61.3%	53.8%	55.7%
News	52.4%	59.3%	52.9%
YouTube	54.8%	53.4%	60.7%
Multi-Domain	57.6%	62.3%	58.9%

Table 8: Cross-domain transfer evaluation. Rows indicate training domain, columns indicate test domain. Diagonal values, representing in-domain performance, are bold. All results averaged over 5 runs.

Script Variation
Linguistic Factor	F1-Score	$\Delta$ from Baseline
Pure Urdu (Nastaliq)	61.2%	+1.2%
Roman Urdu	58.0%	-2.0%
Code-Switching
Pure Urdu	60.2%	+0.2%
Code-Switched (Ur-En)	58.6%	-1.4%

Table 9: Domain bias analysis showing F1 impact across linguistic factors

Training Data	F1-Score	Precision	Recall	Std Dev	$\Delta$ from prev	p-value
20% (2,868)	53.5%	52.9%	53.3%	$\pm$ 2.8%	—	—
40% (5,737)	54.2%	55.0%	53.4%	$\pm$ 2.1%	+0.7%	$0.382$
60% (8,605)	57.5%	58.1%	57.0%	$\pm$ 1.6%	+3.3%	$0.003$ **
80% (11,474)	59.3%	59.8%	58.8%	$\pm$ 1.2%	+1.8%	$0.018$ *
100% (14,342)	60.0%	61.0%	59.0%	$\pm$ 0.9%	+0.7%	$0.156$

Table 10: Training analysis compares the performance and training data size with learning curve analysis. Note: p-values of independent samples t-test in comparison to the previous data size. *

p<0.05

, **

p<0.01

Preprocessing Configuration	F1-Score	$\Delta$ from Full	p-value	95% CI
Full Preprocessing (Baseline)	60.0 $\pm$ 0.8%	—	—	59.1, 60.9
w/o Unicode Normalization	58.2 $\pm$ 1.1%	-1.8%	$0.007$ **	-3.1, -0.5
w/o Diacritic Handling	59.0 $\pm$ 0.9%	-1.0%	$0.042$ *	-2.0, -0.04
w/o Roman Urdu Conversion	56.3 $\pm$ 1.3%	-3.7%	$0.001$ **	-5.3, -2.1
w/o URL/Emoji Removal	59.5 $\pm$ 0.7%	-0.5%	$0.223$	-1.30, 0.3
w/o Deduplication	59.8 $\pm$ 0.8%	-0.2%	$0.634$	-1.00, 0.6

Table 11: Ablation study on preprocessing components. Results from 5-fold cross-validation. Paired t-test vs. full preprocessing. * p < 0.05, ** p < 0.01.

Approach	Annotation	F1-Score
XLM-R+CRF(Supervised)	Span-level	60.0%
ARE (Lei et al.) [ 26 ]	Sentence-level	53.8%
Attention Analysis	Sentence-level	52.7%

Table 12: Comparison of supervised vs. weakly-supervised toxic span detection

System	Language	Eval Metric	F1	Dataset
English Benchmarks (SemEval-2021 Task 5):
S-NLP (1st) [ 7 ]	English	Char-level F1	70.8%	10,629
NLRG (2nd) [ 7 ]	English	Char-level F1	70.8%	10,629
IITK (3rd) [ 7 ]	English	Char-level F1	70.1%	10,629
SemEval Baseline [ 7 ]	English	Char-level F1	65.9%	10,629
Our Urdu Systems:

Table 13: Performance comparison with metric-specific contextualization

Figure 6: Comparison of key linguistic complexity factors between Urdu and English, including morphological richness, script variations, code-switching frequency, diacritic ambiguity, and average toxic span length.

Figure 7: Learning curves of token-level F1-score vs. size of training data. Various model architectures on Urdu toxic span detection. The XLM-R+CRF ensemble demonstrates consistent improvement with increased data, achieving 60.0% F1 with full dataset. Diminishing returns after 11,500 samples suggest our dataset size is approaching sufficiency for this task.

Toxicity Detection

Toxicity detection identifies harmful language for moderation and predicts toxic effects of chemicals, drugs, or biological exposures.

See all 46 papers

An Explainable System for Offensive Content Detection in Nastaliq Urdu: Corpus Curation, Transformer Benchmarking, and Word-Level Attribution
Oct 11, 2026Muhammad Shahid Iqbal MalikHate Speech Detection Transformer
School of Computing Sciences, Pak-Austria Fachhochschule, Institute of Applied Sciences and Technology, Haripur 22620, Pakistan
DunbaaBERT: From Sacrifice to Semantics
May 26, 2026Iffat Maab, Waleed Jamil, Raphael SchmittLanguage Model Pretraining Language Modeling
Research and Development Center for Large Language Models (LLMC), National Institute of Informatics, Tokyo · Independent Researcher, Edinburgh, United Kingdom · School of Computation, Information and Technology, Technical University of Munich, Germany +1
ToxiTrace: Gradient-Aligned Training for Explainable Chinese Toxicity Detection
Apr 14, 2026Boyang Li, Hongzhe Shou, Yuanyuan Liang +2Toxicity Detection Explainable Text Classification
School of Data Science and Engineering, East China Normal University · School of International Chinese Studies, East China Normal University

Oct 11, 2026J/K move · Enter open · S save

An Explainable System for Offensive Content Detection in Nastaliq Urdu: Corpus Curation, Transformer Benchmarking, and Word-Level Attribution

Muhammad Shahid Iqbal Malik

School of Computing Sciences, Pak-Austria Fachhochschule, Institute of Applied Sciences and Technology, Haripur 22620, Pakistan

Hate Speech Detection Transformer Low-Resource Language Processing Feature Attribution

Open paper AskPDF arXiv

Automated moderation systems are deployed at scale, yet they remain opaque to the users and moderators affected by their decisions, and their coverage is almost entirely confined to English. For Urdu written in the Nastaliq script, no such system exists, and the annotated resources it would be built on are drawn largely from Twitter/X, where character limits constrain how users write. Interpretability has received little attention, leaving it unclear what such models actually learn. This study addresses these gaps in three steps. First, we curate and publicly release a corpus of 7604 Nastaliq Urdu Facebook comments, extending an earlier corpus by correcting text-level inconsistencies and re-examining the original labels. Second, we fine-tune four Urdu-capable transformers under a common grid-search protocol and compare them against twelve feature-classifier baselines on identical data partitions. The best configuration, Multilingual Representations for Indian Languages (MuRIL), reaches 87.05% accuracy and 87.06% macro F1, improving on the strongest baseline by 5.52 percentage points, a margin that exceeds the 95% confidence interval on test accuracy; the two leading transformers differ by less than the test set can resolve, and performance is stable across five optimizers. Third, we compute word-level SHAP attributions adapted to right-to-left rendering. Attribution mass concentrates on high-frequency slurs but extends to syntactically neutral function words and to political vocabulary characteristic of the source pages, indicating that aggregate performance may rest partly on lexical and topical regularities rather than a generalizable representation of offensiveness.

Abstract

Toxicity Detection

Explore similar work

An Explainable System for Offensive Content Detection in Nastaliq Urdu: Corpus Curation, Transformer Benchmarking, and Word-Level Attribution

DunbaaBERT: From Sacrifice to Semantics

ToxiTrace: Gradient-Aligned Training for Explainable Chinese Toxicity Detection