A Fusion of context-aware based BanglaBERT and Two-Layer Stacked LSTM Framework for Multi-Label Cyberbullying Detection

Mirza Raquib; Asif Pervez Polok; Kedar Nath Biswas; Rahat Uddin Azad; Saydul Akbar Murad; Nick Rahimi

cs.CLFeb 25, 2026

A Fusion of context-aware based BanglaBERT and Two-Layer Stacked LSTM Framework for Multi-Label Cyberbullying Detection

Authors: Mirza Raquib, Asif Pervez Polok, Kedar Nath Biswas, Rahat Uddin Azad, Saydul Akbar Murad, Nick Rahimi

Organizations: Department of Computer and Communication Engineering, International Islamic University Chittagong, Chattogram, Bangladesh · Department of Information and Communication Engineering, Noakhali Science and Technology University, Noakhali, Bangladesh · mPower Social Enterprise · Department of Software Engineering, Daffodil International University, Bangladesh ·

Abstract

Cyberbullying has become a serious and growing concern in todays virtual world. When left unnoticed, it can have adverse consequences for social and mental health. Researchers have explored various types of cyberbullying, but most approaches use single-label classification, assuming that each comment contains only one type of abuse. In reality, a single comment may include overlapping forms such as threats, hate speech, and harassment. Therefore, multilabel detection is both realistic and essential. However, multilabel cyberbullying detection has received limited attention, especially in low-resource languages like Bangla, where robust pre-trained models are scarce. Developing a generalized model with moderate accuracy remains challenging. Transformers offer strong contextual understanding but may miss sequential dependencies, while LSTM models capture temporal flow but lack semantic depth. To address these limitations, we propose a fusion architecture that combines BanglaBERT-Large with a two-layer stacked LSTM. We analyze their behavior to jointly model context and sequence. The model is fine-tuned and evaluated on a publicly available multilabel Bangla cyberbullying dataset covering cyberbully, sexual harassment, threat, and spam. We apply different sampling strategies to address class imbalance. Evaluation uses multiple metrics, including accuracy, precision, recall, F1-score, Hamming loss, Cohens kappa, and AUC-ROC. We employ 5-fold cross-validation to assess the generalization of the architecture.

Figures & tables

Study

Dataset

ML Models

DL Models

Hybrid Models

Metrics

Limitations

English

Bangla

Other

Multi-label

No. of Classes

Accuracy

Precision

Recall

Hamm. Loss

Kappa

MCC

ROC

Cross Val.

XAI

[ 18 ]

✗

✓

✗

MNB, LR, RF, LinearSVC

ANN, RNN

✗

✓

✗

Relatively small dataset and overfitting issues

[ 19 ]

✓

✗

2DCNN

✗

✓

✗

✓

✗

Relatively small dataset (1k–3k images), limiting generalization

[ 20 ]

✗

✓

✗

LR, SVM, XGB, RF, DT, KNN

LSTM, CNN, Bi-LSTM

✗

✓

✗

Uses custom Word2Vec embeddings, lacking transformer-based contextual representations

[ 21 ]

✓

✗

LR, RF, XGB, DT, NB, SVM, ET

✗

✓

✗

✓

✗

TF–IDF-based static features which lack deep contextual and semantic representation.

[ 22 ]

✗

✓

✗

DT, RF, SVM, MLP

✗

Hybrid ML Approach

✓

✗

✓

✗

Shallow machine learning models

TABLE I: Comparison of related studies with the proposed BanglaBERT-LSTM hybrid model.

Fig. 1: Workflow of the Proposed Cyberbullying Detection

Dataset Split	Dataset Type	Class
		Bully	Sexual	Religious	Threat	Spam
Train	Imbalance	5497	1447	1140	1116	752
	Oversample	16472	6189	7418	6841	5986
	Downsample	2685	1047	1163	1142	835
Validation	-	643	105	134	145	77
Test	-	643	105	134	145	77

TABLE II: Dataset Statistics and Class Distribution

Fig. 2: Model Architecture of Proposed BERT-CNN-BiLSTM

	$\displaystyle L\leftarrow-\frac{1}{B}\sum_{i=1}^{B}\sum_{k=1}^{K}\Big[$	$\displaystyle y_{i,k}\log(\hat{y}_{i,k})$
		$\displaystyle+(1-y_{i,k})\log(1-\hat{y}_{i,k})\Big]$

Algorithm 3 Training Procedure of the BERT–LSTM Hybrid Classifier with BCE Loss

Model	Accuracy	Ham.	Precision	Recall	F1	MCC	Kappa	AUC
KNN	81.90	0.1810	61.41	69.80	65.34	37.78	32.78	75.50
Naive Bayes	87.26	0.1274	81.11	62.41	70.54	43.77	34.94	84.59
Logistic Regression	88.13	0.1187	81.81	66.10	73.12	49.43	44.37	86.67
XGBoost	88.50	0.1150	80.20	70.28	74.91	53.34	50.18	84.18
Random Forest	88.95	0.1105	81.46	70.92	75.83	54.96	52.56	86.66
GRU	91.32	0.0868	83.26	80.72	81.97	69.08	69.00	91.88

TABLE III: Model Comparison on Cyberbullying Detection using Baseline Models

Model	Train Acc.	Val. Acc.	Test Acc.	Hamming Loss	Precision	Recall	F1	MCC	Kappa	AUC
mBERT	97.80	92.56	92.56	0.0744	85.84	83.29	84.55	74.30	74.22	93.36
RoBERTa	97.08	92.80	93.46	0.0654	87.19	85.86	86.52	77.39	77.37	94.13
BERT-BiGRU	98.64	93.48	93.68	0.0628	87.39	86.83	87.11	78.64	78.63	94.80
DeBERTa	97.48	93.07	93.72	0.0632	85.91	88.67	87.27	78.18	78.09	93.95
BERT-BiLSTM	98.40	93.88	93.84	0.0616	86.80	88.19	87.49	78.53	78.48	93.82
BERT	98.82	93.37	93.92	0.0608	87.02	88.27	87.64	78.93	78.90	93.80

TABLE IV: Model Comparison on Cyberbullying Detection using Transformer Models

Learning Rate	Train Acc.	Val. Acc.	Test Acc.	Hamming Loss	Precision	Recall	F1	MCC	Kappa	AUC
$1\times 10^{-6}$	99.45	92.95	91.74	0.0826	79.99	88.27	83.93	73.14	72.79	93.93
$1\times 10^{-4}$	99.45	92.95	93.39	0.0661	87.52	85.06	86.27	76.39	76.24	93.18

TABLE V: Learning Rate Comparison for the BERT-LSTM Models

Optimizer	Train Acc.	Val. Acc.	Test Acc.	Hamming Loss	Precision	Recall	F1	MCC	Kappa	AUC
RAdam	99.50	93.48	93.97	0.0603	86.70	89.00	87.83	79.20	79.16	94.82
RMSprop	99.58	93.58	94.07	0.0593	87.04	89.00	88.01	79.57	79.51	95.09
Adafactor	99.57	93.44	94.15	0.0585	86.96	89.48	88.20	79.86	79.79	94.93

TABLE VI: Optimizer Comparison for the BERT-LSTM Model

Technique	Train Acc.	Val. Acc.	Test Acc.	Hamming Loss	Precision	Recall	F1	MCC	Kappa	AUC
Original(Imbalanced)	98.64	93.54	94.17	0.0583	88.35	87.71	88.03	79.40	79.33	92.66
Undersampling(Train)	98.02	90.17	90.42	0.0958	74.72	91.89	82.42	69.20	67.19	92.09
Oversampling(Train)	99.53	93.41	94.31	0.0569	88.23	88.51	88.37	80.31	80.25	95.08

TABLE VII: Performance Analysis of the Proposed BERT-LSTM Model Under Different Sampling Strategies

Fold	Accuracy	Hamming Loss	Precision	Recall	F1	MCC	Kappa	AUC
1	93.62	0.0638	83.71	85.31	84.46	78.13	77.99	94.91
2	93.45	0.0655	83.41	83.67	83.41	77.31	77.20	93.63
3	93.61	0.0639	85.48	83.06	84.07	77.65	77.49	93.63
4	93.34	0.0666	83.81	83.99	83.84	77.49	77.45	93.04
5	93.09	0.0691	83.24	83.37	83.23	76.63	76.58	93.79
Average	93.42	0.0658	83.93	83.88	83.70	77.44	77.34	93.80

TABLE VIII: Five-Fold Cross-Validation Performance of the Proposed BERT-LSTM Model

Ref.	Architecture	Accuracy	Hamming Loss	Precision	Recall	F1	MCC	Kappa	AUC
[ 37 ]	Stacked LSTM	93.64	–	94.31	92.05	91.32	–	–	98.02
Proposed	BERT + LSTM	94.31	0.0569	88.23	88.51	88.37	80.31	80.25	95.08

TABLE IX: Comparison of the Proposed BERT-LSTM Model with Existing Work on the Same Dataset

Panel of Figure 13 — Fig. 3: Multi-label Classification Predictions Showing Class Probabilities and Word-level Importance Scores for Each Cyberbullying Category. The Highlighted Words in the Bangla Text Indicate Their Contribution to the Model’s Prediction for Each Label.

Bengali NLP

See all 23 papers

HMS-BERT: Hybrid Multi-Task Self-Training for Multilingual and Multi-Label Cyberbullying Detection
Mar 13, 2026Zixin Feng, Xinying Cui, Yifan Sun +5Multi-Label Classification Multi-Task Learning
School of AI and Advanced Computing, XJTLU Entrepreneur College (Taicang), Xi’an Jiaotong-Liverpool University, Suzhou 215123, China
Comparative Analysis of AutoML and BiLSTM Models for Cyberbullying Detection on Indonesian Instagram Comments
Apr 29, 2026Raihana Adelia Putri, Aisyah Musfirah, Anggi Puspita Ningrum +3Long Short-Term Memory Networks Attention Mechanisms
Institut Teknologi Sumatera
Beyond Benchmarks: Exposing the Hidden Crisis in Bangla Hate Speech Detection
Jul 13, 2026Faria Afrin Tisha, Fariya Tabassum, Hafsa Binte Kibria +2Hate Speech Detection Low-Resource Language Processing
Department of Electrical and Computer Engineering, Rajshahi University of Engineering and Technology, Rajshahi, 6204, Bangladesh. · Department of Computer Science, University of York, Deramore Lane, Heslington, York YO10 5GH, UK.

Mar 13, 2026J/K move · Enter open · S save

HMS-BERT: Hybrid Multi-Task Self-Training for Multilingual and Multi-Label Cyberbullying Detection

Zixin Feng, Xinying Cui, Yifan Sun, Zheng Wei+4

School of AI and Advanced Computing, XJTLU Entrepreneur College (Taicang), Xi’an Jiaotong-Liverpool University, Suzhou 215123, China

Multi-Label Classification Multi-Task Learning Self-Training for Language Models

Open paper AskPDF arXiv

Cyberbullying on social media is inherently multilingual and multi-faceted, where abusive behaviors often overlap across multiple categories. Existing methods are commonly limited by monolingual assumptions or single-task formulations, which restrict their effectiveness in realistic multilingual and multi-label scenarios. In this paper, we propose HMS-BERT, a hybrid multi-task self-training framework for multilingual and multi-label cyberbullying detection. Built upon a pretrained multilingual BERT backbone, HMS-BERT integrates contextual representations with handcrafted linguistic features and jointly optimizes a fine-grained multi-label abuse classification task and a three-class main classification task. To address labeled data scarcity in low-resource languages, an iterative self-training strategy with confidence-based pseudo-labeling is introduced to facilitate cross-lingual knowledge transfer. Experiments on four public datasets demonstrate that HMS-BERT achieves strong performance, attaining a macro F1-score of up to 0.9847 on the multi-label task and an accuracy of 0.6775 on the main classification task. Ablation studies further verify the effectiveness of the proposed components.

Abstract

Bengali NLP

Explore similar work

HMS-BERT: Hybrid Multi-Task Self-Training for Multilingual and Multi-Label Cyberbullying Detection

Comparative Analysis of AutoML and BiLSTM Models for Cyberbullying Detection on Indonesian Instagram Comments

Beyond Benchmarks: Exposing the Hidden Crisis in Bangla Hate Speech Detection