A Fusion of context-aware based BanglaBERT and Two-Layer Stacked LSTM Framework for Multi-Label Cyberbullying Detection

Mirza Raquib; Asif Pervez Polok; Kedar Nath Biswas; Rahat Uddin Azad; Saydul Akbar Murad; Nick Rahimi

cs.CLFeb 25, 2026

A Fusion of context-aware based BanglaBERT and Two-Layer Stacked LSTM Framework for Multi-Label Cyberbullying Detection

Authors: Mirza Raquib, Asif Pervez Polok, Kedar Nath Biswas, Rahat Uddin Azad, Saydul Akbar Murad, Nick Rahimi

Organizations: Department of Computer and Communication Engineering, International Islamic University Chittagong, Chattogram, Bangladesh · Department of Information and Communication Engineering, Noakhali Science and Technology University, Noakhali, Bangladesh · mPower Social Enterprise · Department of Software Engineering, Daffodil International University, Bangladesh ·

Abstract

Cyberbullying has become a serious and growing concern in todays virtual world. When left unnoticed, it can have adverse consequences for social and mental health. Researchers have explored various types of cyberbullying, but most approaches use single-label classification, assuming that each comment contains only one type of abuse. In reality, a single comment may include overlapping forms such as threats, hate speech, and harassment. Therefore, multilabel detection is both realistic and essential. However, multilabel cyberbullying detection has received limited attention, especially in low-resource languages like Bangla, where robust pre-trained models are scarce. Developing a generalized model with moderate accuracy remains challenging. Transformers offer strong contextual understanding but may miss sequential dependencies, while LSTM models capture temporal flow but lack semantic depth. To address these limitations, we propose a fusion architecture that combines BanglaBERT-Large with a two-layer stacked LSTM. We analyze their behavior to jointly model context and sequence. The model is fine-tuned and evaluated on a publicly available multilabel Bangla cyberbullying dataset covering cyberbully, sexual harassment, threat, and spam. We apply different sampling strategies to address class imbalance. Evaluation uses multiple metrics, including accuracy, precision, recall, F1-score, Hamming loss, Cohens kappa, and AUC-ROC. We employ 5-fold cross-validation to assess the generalization of the architecture.

Figures & tables

Study

Dataset

ML Models

DL Models

Hybrid Models

Metrics

Limitations

English

Bangla

Other

Multi-label

No. of Classes

Accuracy

Precision

Recall

Hamm. Loss

Kappa

MCC

ROC

Cross Val.

XAI

[ 18 ]

✗

✓

✗

MNB, LR, RF, LinearSVC

ANN, RNN

✗

✓

✗

Relatively small dataset and overfitting issues

[ 19 ]

✓

✗

2DCNN

✗

✓

✗

✓

✗

Relatively small dataset (1k–3k images), limiting generalization

[ 20 ]

✗

✓

✗

LR, SVM, XGB, RF, DT, KNN

LSTM, CNN, Bi-LSTM

✗

✓

✗

Uses custom Word2Vec embeddings, lacking transformer-based contextual representations

[ 21 ]

✓

✗

LR, RF, XGB, DT, NB, SVM, ET

✗

✓

✗

✓

✗

TF–IDF-based static features which lack deep contextual and semantic representation.

[ 22 ]

✗

✓

✗

DT, RF, SVM, MLP

✗

Hybrid ML Approach

✓

✗

✓

✗

Shallow machine learning models

TABLE I: Comparison of related studies with the proposed BanglaBERT-LSTM hybrid model.

Fig. 1: Workflow of the Proposed Cyberbullying Detection

Dataset Split	Dataset Type	Class
		Bully	Sexual	Religious	Threat	Spam
Train	Imbalance	5497	1447	1140	1116	752
	Oversample	16472	6189	7418	6841	5986
	Downsample	2685	1047	1163	1142	835
Validation	-	643	105	134	145	77
Test	-	643	105	134	145	77

TABLE II: Dataset Statistics and Class Distribution

Fig. 2: Model Architecture of Proposed BERT-CNN-BiLSTM

	$\displaystyle L\leftarrow-\frac{1}{B}\sum_{i=1}^{B}\sum_{k=1}^{K}\Big[$	$\displaystyle y_{i,k}\log(\hat{y}_{i,k})$
		$\displaystyle+(1-y_{i,k})\log(1-\hat{y}_{i,k})\Big]$

Algorithm 3 Training Procedure of the BERT–LSTM Hybrid Classifier with BCE Loss

Model	Accuracy	Ham.	Precision	Recall	F1	MCC	Kappa	AUC
KNN	81.90	0.1810	61.41	69.80	65.34	37.78	32.78	75.50
Naive Bayes	87.26	0.1274	81.11	62.41	70.54	43.77	34.94	84.59
Logistic Regression	88.13	0.1187	81.81	66.10	73.12	49.43	44.37	86.67
XGBoost	88.50	0.1150	80.20	70.28	74.91	53.34	50.18	84.18
Random Forest	88.95	0.1105	81.46	70.92	75.83	54.96	52.56	86.66
GRU	91.32	0.0868	83.26	80.72	81.97	69.08	69.00	91.88

TABLE III: Model Comparison on Cyberbullying Detection using Baseline Models

Model	Train Acc.	Val. Acc.	Test Acc.	Hamming Loss	Precision	Recall	F1	MCC	Kappa	AUC
mBERT	97.80	92.56	92.56	0.0744	85.84	83.29	84.55	74.30	74.22	93.36
RoBERTa	97.08	92.80	93.46	0.0654	87.19	85.86	86.52	77.39	77.37	94.13
BERT-BiGRU	98.64	93.48	93.68	0.0628	87.39	86.83	87.11	78.64	78.63	94.80
DeBERTa	97.48	93.07	93.72	0.0632	85.91	88.67	87.27	78.18	78.09	93.95
BERT-BiLSTM	98.40	93.88	93.84	0.0616	86.80	88.19	87.49	78.53	78.48	93.82
BERT	98.82	93.37	93.92	0.0608	87.02	88.27	87.64	78.93	78.90	93.80

TABLE IV: Model Comparison on Cyberbullying Detection using Transformer Models

Learning Rate	Train Acc.	Val. Acc.	Test Acc.	Hamming Loss	Precision	Recall	F1	MCC	Kappa	AUC
$1\times 10^{-6}$	99.45	92.95	91.74	0.0826	79.99	88.27	83.93	73.14	72.79	93.93
$1\times 10^{-4}$	99.45	92.95	93.39	0.0661	87.52	85.06	86.27	76.39	76.24	93.18

TABLE V: Learning Rate Comparison for the BERT-LSTM Models

Optimizer	Train Acc.	Val. Acc.	Test Acc.	Hamming Loss	Precision	Recall	F1	MCC	Kappa	AUC
RAdam	99.50	93.48	93.97	0.0603	86.70	89.00	87.83	79.20	79.16	94.82
RMSprop	99.58	93.58	94.07	0.0593	87.04	89.00	88.01	79.57	79.51	95.09
Adafactor	99.57	93.44	94.15	0.0585	86.96	89.48	88.20	79.86	79.79	94.93

TABLE VI: Optimizer Comparison for the BERT-LSTM Model

Technique	Train Acc.	Val. Acc.	Test Acc.	Hamming Loss	Precision	Recall	F1	MCC	Kappa	AUC
Original(Imbalanced)	98.64	93.54	94.17	0.0583	88.35	87.71	88.03	79.40	79.33	92.66
Undersampling(Train)	98.02	90.17	90.42	0.0958	74.72	91.89	82.42	69.20	67.19	92.09
Oversampling(Train)	99.53	93.41	94.31	0.0569	88.23	88.51	88.37	80.31	80.25	95.08

TABLE VII: Performance Analysis of the Proposed BERT-LSTM Model Under Different Sampling Strategies

Fold	Accuracy	Hamming Loss	Precision	Recall	F1	MCC	Kappa	AUC
1	93.62	0.0638	83.71	85.31	84.46	78.13	77.99	94.91
2	93.45	0.0655	83.41	83.67	83.41	77.31	77.20	93.63
3	93.61	0.0639	85.48	83.06	84.07	77.65	77.49	93.63
4	93.34	0.0666	83.81	83.99	83.84	77.49	77.45	93.04
5	93.09	0.0691	83.24	83.37	83.23	76.63	76.58	93.79
Average	93.42	0.0658	83.93	83.88	83.70	77.44	77.34	93.80

TABLE VIII: Five-Fold Cross-Validation Performance of the Proposed BERT-LSTM Model

Ref.	Architecture	Accuracy	Hamming Loss	Precision	Recall	F1	MCC	Kappa	AUC
[ 37 ]	Stacked LSTM	93.64	–	94.31	92.05	91.32	–	–	98.02
Proposed	BERT + LSTM	94.31	0.0569	88.23	88.51	88.37	80.31	80.25	95.08

TABLE IX: Comparison of the Proposed BERT-LSTM Model with Existing Work on the Same Dataset

Panel of Figure 13 — Fig. 3: Multi-label Classification Predictions Showing Class Probabilities and Word-level Importance Scores for Each Cyberbullying Category. The Highlighted Words in the Bangla Text Indicate Their Contribution to the Model’s Prediction for Each Label.

Bengali NLP

See all 23 papers

Mar 13, 2026

HMS-BERT: Hybrid Multi-Task Self-Training for Multilingual and Multi-Label Cyberbullying Detection

Cyberbullying on social media is inherently multilingual and multi-faceted, where abusive behaviors often overlap across multiple categories. Existing methods are commonly limited by monolingual assumptions or single-task formulations, which restrict their effectiveness in realistic multilingual and multi-label scenarios. In this paper, we propose HMS-BERT, a hybrid multi-task self-training framework for multilingual and multi-label cyberbullying detection. Built upon a pretrained multilingual BERT backbone, HMS-BERT integrates contextual representations with handcrafted linguistic features and jointly optimizes a fine-grained multi-label abuse classification task and a three-class main classification task. To address labeled data scarcity in low-resource languages, an iterative self-training strategy with confidence-based pseudo-labeling is introduced to facilitate cross-lingual knowledge transfer. Experiments on four public datasets demonstrate that HMS-BERT achieves strong performance, attaining a macro F1-score of up to 0.9847 on the multi-label task and an accuracy of 0.6775 on the main classification task. Ablation studies further verify the effectiveness of the proposed components.

Zixin Feng, Xinying Cui, Yifan Sun +5

School of AI and Advanced Computing, XJTLU Entrepreneur College (Taicang), Xi’an Jiaotong-Liverpool University, Suzhou 215123, China

Multi-Label Classification Multi-Task Learning Self-Training for Language Models

Ask PDF

Apr 29, 2026

Comparative Analysis of AutoML and BiLSTM Models for Cyberbullying Detection on Indonesian Instagram Comments

This study compares machine learning and deep learning approaches for cyberbullying detection in Indonesian-language Instagram comments. Using a balanced dataset of 650 comments labeled as Bullying and Non-Bullying, the study evaluates Naive Bayes, Logistic Regression, and Support Vector Machine with TF-IDF features, as well as BiLSTM and BiLSTM with Bahdanau Attention. A preprocessing pipeline tailored to informal Indonesian text is applied, including slang normalization, stopword removal, and stemming. The results show that Logistic Regression performs best among the machine learning models, while BiLSTM with Attention achieves the strongest overall deep learning performance. The findings highlight the value of domain-specific preprocessing and show that although deep learning captures contextual patterns more effectively, machine learning remains a competitive option for resource-constrained deployments.

Raihana Adelia Putri, Aisyah Musfirah, Anggi Puspita Ningrum +3

Institut Teknologi Sumatera

Long Short-Term Memory Networks Attention Mechanisms

Ask PDF

Jul 13, 2026

Beyond Benchmarks: Exposing the Hidden Crisis in Bangla Hate Speech Detection

The spread of hate speech (HS) across different social media platforms (SMPs) poses a major concern for online safety and ethical moderation. Automatic detection of HS remains a challenging task, especially in under-resourced languages like Bangla, due to cultural context, implicit expressions, and informal linguistic patterns. This study aimed to expose the crisis of Bangla HS detection systems by diagnosing how and why benchmark-trained models fail to identify implicit, context-dependent HS. Six architectures (FastText + CNN, FastText + LSTM, FastText + BiLSTM, BanglaBERT, BanglaBERT + CNN, and BanglaBERT + BiLSTM) were trained on benchmark datasets (about 75,000 posts) and a merged multi-source dataset (about 120,000 posts), then externally validated on an annotated dataset (about 200 posts) collected from Facebook, Twitter, and YouTube, labeled as HS and non-HS, where HS was further categorized as explicit and implicit. BanglaBERT achieved an F1-score of 91.4% on benchmark datasets but declined to 75.3% on the external set and 63.4% for implicit HS involving sarcasm and emojis. The accuracy of FastText + CNN dropped from 78.0% to 51.2% under similar conditions. Emoji-aware preprocessing improved implicit HS detection by up to 12%, whereas emoji removal caused a notable decline in performance (F1: 0.75 to 0.63). Frequent misclassifications in politically charged or satirical comments revealed over-policing risks. This study not only exposes the generalization crisis due to implicit, culturally embedded, and emoji-laden expressions but also underscores the need for developing adaptive, emoji-aware, and culturally grounded frameworks that ensure ethical moderation while preserving freedom of expression. Findings of this study provide insights for researchers, SMPs, and policymakers to design more context-sensitive HS detection systems for low-resource languages.

Faria Afrin Tisha, Fariya Tabassum, Hafsa Binte Kibria +2

Department of Electrical and Computer Engineering, Rajshahi University of Engineering and Technology, Rajshahi, 6204, Bangladesh. · Department of Computer Science, University of York, Deramore Lane, Heslington, York YO10 5GH, UK.

Hate Speech Detection Low-Resource Language Processing

Ask PDF

Abstract

Bengali NLP

Explore similar work

HMS-BERT: Hybrid Multi-Task Self-Training for Multilingual and Multi-Label Cyberbullying Detection

Comparative Analysis of AutoML and BiLSTM Models for Cyberbullying Detection on Indonesian Instagram Comments

Beyond Benchmarks: Exposing the Hidden Crisis in Bangla Hate Speech Detection