Multi-Head Attention based interaction-aware architecture for Bangla Handwritten Character Recognition: Introducing a Primary Dataset

Mirza Raquib; Asif Pervez Polok; Kedar Nath Biswas; Farida Siddiqi Prity; Saydul Akbar Murad; Nick Rahimi

cs.CVApr 8, 2026

Multi-Head Attention based interaction-aware architecture for Bangla Handwritten Character Recognition: Introducing a Primary Dataset

Authors: Mirza Raquib, Asif Pervez Polok, Kedar Nath Biswas, Farida Siddiqi Prity, Saydul Akbar Murad, Nick Rahimi

Organizations: Department of Computer and Communication Engineering, International Islamic University Chittagong, Bangladesh. · mPower Social Enterprise. · Department of Information and Communication Engineering, Noakhali Science and Technology University, Bangladesh. · Department of CSE, Netrokona University, Netrokona, Bangladesh. · School of Computing Sciences and Computer Engineering, University of Southern Mississippi, USA.

Abstract

Character recognition is the fundamental part of an optical character recognition (OCR) system. Word recognition, sentence transcription, document digitization, and language processing are some of the higher-order activities that can be done accurately through character recognition. Nonetheless, recognizing handwritten Bangla characters is not an easy task because they are written in different styles with inconsistent stroke patterns and a high degree of visual character resemblance. The datasets available are usually limited in intra-class and inequitable in class distribution. We have constructed a new balanced dataset of Bangla written characters to overcome those problems. This consists of 78 classes and each class has approximately 650 samples. It contains the basic characters, composite (Juktobarno) characters and numerals. The samples were a diverse group comprising a large age range and socioeconomic groups. Elementary and high school students, university students, and professionals are the contributing factors. The sample also has right and left-handed writers. We have further proposed an interaction-aware hybrid deep learning architecture that integrates EfficientNetB3, Vision Transformer, and Conformer modules in parallel. A multi-head cross-attention fusion mechanism enables effective feature interaction across these components. The proposed model achieves 98.84% accuracy on the constructed dataset and 96.49% on the external CHBCR benchmark, demonstrating strong generalization capability. Grad-CAM visualizations further provide interpretability by highlighting discriminative regions. The dataset and source code of this research is publicly available at: https://huggingface.co/MIRZARAQUIB/Bangla_Handwritten_Character_Recognition.

Figures & tables

Study	Dataset Name	Dataset Type			Architecture Type	Identified Gap	Resolution in Proposed Framework
		Basic	Digits	Compound
[ 19 ]	Ekush (Primary)	✓	✓	✓	EkushNet (22-Layer DCNN)	Fixed CNN architecture without adaptive feature fusion	Interaction-aware hybrid architecture enabling adaptive feature fusion
[ 20 ]	BanglaLekha-Isolated, ISI, CMATERdb, IUBMCdb	✓	✓	✓	CNN with residual unit and Spatial Attention	Single-stream convolutional modeling	Employes parallel CNN–Transformer streams
[ 21 ]	BanglaLekha, Ekush, CMATERdb	✓	✓	✓	MSBNet (Tri-scale Lightweight CNN)	No hybrid CNN–Transformer modeling	Integrates hybrid CNN–Transformer
[ 2 ]	Primary	✓	✓	✓	BNVGLENET (VGG + LeNet)	No global self-attention modeling	Utilizes multi-head cross-attention modeling
[ 22 ]	BanglaLekha, ISI, CMATERdb, Ekush	✓	✓	✗	Lightweight CNN	Less time consuming architecture but some pretrained models shows better performance	Improves generalization through balanced primary dataset with diverse writer variability

TABLE I: Comparative gap analysis and how the proposed framework addresses key architectural and dataset limitations

Fig. 1: Proposed workflow for handwritten character recognition.

Phase	Character Type	Participants	Characters Taken	Total Samples
Phase 1	Sorobarno+Benjonbarno	150	50	7,500
Phase 2	Sorobarno+Digits	500	21	10,500
Phase 3	Benjonbarno	500	39	19,500
Phase 4	Juktobarno	650	18	11,700
Extension	Digits	150	10	1,500
Total	-	1,700	-	50,700

TABLE II: Summary of Dataset Collection

Character Category	Number of Characters	Total Samples
Sorobarno + Benjonbarno	50	7,500
Sorobarno	11	5,500
Digits	10	6,500
Benjonbarno	39	19,500
Juktobarno	18	11,700
Total Samples		50,700

TABLE III: Distribution of Characters in the Dataset

Gender Distribution		Hand Preference
Category	Participants (%)	Category	Participants (%)
Male	820 (48.24%)	Left Hand	190 (11.18%)
Female	880 (51.76%)	Right Hand	1510 (88.82%)
Total: 1700		Total: 1700

TABLE IV: Gender and Hand Preference Distribution of Participants

Age Group	Participants (%)	Age Group	Participants (%)
10–15	200 (11.76%)	26–30	250 (14.71%)
16–20	400 (23.53%)	31–35	200 (11.76%)
21–25	500 (29.41%)	36+	150 (8.82%)
Total Participants: 1700

TABLE V: Age Group Distribution of Dataset Contributors

Fig. 3: Model Architecture of the Proposed Bangla Handwritten Character recognition Framework

\mathcal{L}=-\sum_{i=1}^{C}y_{i}\log(\hat{y}_{i})

Algorithm 2 Training Procedure of the Multi-Head Cross-Attention Framework

Model	Train (%)	Val (%)	Test (%)	Error (%)	Precision (%)	Recall (%)	F1 (%)	MCC (%)	Kappa (%)	AUC (%)
MobileNetV2	98.87	95.94	96.02	3.98	96.04	96.02	96.02	95.98	95.98	99.96
Xception	99.02	96.71	96.80	3.20	96.81	96.80	96.80	96.76	96.76	99.97
ConvNeXt-Small	99.18	97.08	97.15	2.85	97.17	97.15	97.15	97.11	97.11	99.98
Conformer	99.12	96.28	96.36	3.64	96.35	96.36	96.35	96.31	96.31	99.98
EfficientNet	99.09	96.75	96.69	3.31	96.70	96.69	96.70	96.64	96.64	99.97
ViT	99.03	97.24	97.12	2.88	97.11	97.12	97.11	97.08	97.08	99.98

TABLE VI: Performance Comparison of Backbone Models

Model	Train (%)	Val (%)	Test (%)	Error (%)	Precision (%)	Recall (%)	F1 (%)	MCC (%)	Kappa (%)	AUC (%)
Proposed + Prewitt	98.86	96.19	96.25	3.75	96.26	96.25	96.25	96.20	96.20	99.96
Proposed + Canny	98.98	96.78	96.82	3.18	96.83	96.82	96.82	96.78	96.78	99.97
Proposed + Gaussian	99.04	97.15	97.19	2.81	97.20	97.19	97.19	97.15	97.15	99.97
Proposed + CLAHE	99.21	97.56	97.63	2.37	97.64	97.63	97.63	97.59	97.59	99.98
Proposed + Gaussian + Canny	99.40	97.91	97.98	2.02	97.99	97.98	97.98	97.95	97.95	99.98
Proposed + CLAHE + Canny	99.56	98.24	98.30	1.70	98.31	98.30	98.30	98.27	98.27	99.98

TABLE VII: Performance Comparison of Preprocessing and Feature Extraction Variants

Individual Branch Baselines
Configuration	Streams	Accuracy (%)	Precision (%)	Recall (%)	F1 (%)	MCC (%)	Kappa (%)	AUC (%)	Error (%)
EfficientNet	Single	96.69	96.70	96.69	96.70	96.64	96.64	99.97	3.31
ViT	Single	97.12	97.11	97.12	97.11	97.08	97.08	99.98	2.88
Conformer	Single	96.36	96.35	96.36	96.35	96.31	96.31	99.98	3.64
ConvNeXt-Small	Single	97.15	97.17	97.15	97.15	97.11	97.11	99.98	2.85
Stream Combinations

TABLE VIII: Extended Ablation Study of Stream Design and Fusion Mechanisms

Fold	Accuracy (%)	Precision (%)	Recall (%)	F1 (%)	MCC (%)	Kappa (%)	AUC (%)
Fold 1	98.69	98.71	98.69	98.69	98.65	98.65	99.97
Fold 2	98.76	98.79	98.76	98.76	98.73	98.73	99.98
Fold 3	98.88	98.90	98.88	98.88	98.85	98.85	99.98
Fold 4	98.91	98.93	98.91	98.91	98.88	98.88	99.98
Fold 5	98.84	98.86	98.84	98.84	98.81	98.81	99.98
Average	98.82	98.84	98.82	98.82	98.78	98.78	99.98

TABLE IX: Five-Fold Cross-Validation Performance of the Proposed Model

Work	Dataset	Architecture	Dataset Size	Class	Train Acc	Val Acc	Test Acc	Precision	Recall	F1
VashaNet-V1	Primary	26-Layer DCNN	5,750	50	98.58	94.60	94.60	95.00	95.00	95.00
VashaNet-V2	Mixed	19-Layer DCNN	22,000	50	95.82	95.20	95.20	93.53	93.29	93.29
Proposed	Primary	Hybrid Fusion	50,700	78	99.71	98.50	98.84	98.87	98.84	98.84

TABLE X: Performance Comparison of Proposed Model with Previous Works

Fig. 4: Per-class performance results.

Dataset	Train (%)	Val (%)	Test (%)	Error (%)	Precision (%)	Recall (%)	F1 (%)	MCC (%)	Kappa (%)	AUC (%)
CHBCR	99.62	96.65	96.49	3.51	96.52	96.49	96.49	96.45	96.44	99.91
Introduced	99.71	98.50	98.84	1.16	98.87	98.84	98.84	98.83	98.83	99.98

TABLE XI: Comparision of Proposed Architecture on External CHBCR with our Introduced Dataset

Fig. 6: Grad-CAM Analysis of the proposed framework.

Explore similar work

Confidence-Guided Diffusion Augmentation for Enhanced Bangla Compound Character Recognition
May 11, 2026Md. Sultan Al RayhanSynthetic Data Augmentation Optical Character Recognition
Department of Computer Science and Engineering East West University, Dhaka, Bangladesh
Open Vocabulary Word Recognition From Transcribed Bangla Texts
Oct 1, 2026Faias Satter, Sk. Md. Masudul AhsanObject Detection Optical Character Recognition
Department of Computer Science and Engineering Khulna University of Engineering & Technology Khulna-9203, Bangladesh
BornoViT: A Novel Efficient Vision Transformer for Bengali Handwritten Basic Characters Classification
Feb 28, 2026Rafi Hassan Chowdhury, Naimul Haque, Kaniz FatihaOptical Character Recognition Efficient ViTs
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur 1704, Bangladesh

May 11, 2026J/K move · Enter open · S save

Confidence-Guided Diffusion Augmentation for Enhanced Bangla Compound Character Recognition

Md. Sultan Al Rayhan

Department of Computer Science and Engineering East West University, Dhaka, Bangladesh

Synthetic Data Augmentation Optical Character Recognition Conditional Diffusion Models

Open paper AskPDF arXiv

Recognition of handwritten Bangla compound characters remains a challenging problem due to complex character structures, large intra-class variation, and limited availability of high-quality annotated data. Existing Bangla handwritten character recognition systems often struggle to generalize across diverse writing styles, particularly for compound characters containing intricate ligatures and diacritical variations. In this work, we propose a confidence-guided diffusion augmentation framework for low-resolution Bangla compound character recognition. Our framework combines class-conditional diffusion modeling with classifier guidance to synthesize high-quality handwritten compound character samples. To further improve generation quality, we introduce Squeeze-and-Excitation enhanced residual blocks within the diffusion model's U-Net backbone. We additionally propose a confidence-based filtering mechanism where pre-trained classifiers act as quality gates to retain only highly class-consistent synthetic samples. The filtered synthetic images are fused with the original training data and used to retrain multiple classification architectures. Experiments conducted on the AIBangla compound character dataset demonstrate consistent performance improvements across ResNet50, DenseNet121, VGG16, and Vision Transformer architectures. Our best-performing model achieves 89.2% classification accuracy, surpassing the previously published AIBangla benchmark by a substantial margin. The results demonstrate that quality-aware diffusion augmentation can effectively enhance handwritten character recognition performance in low-resource script domains.

Abstract

Explore similar work

Confidence-Guided Diffusion Augmentation for Enhanced Bangla Compound Character Recognition

Open Vocabulary Word Recognition From Transcribed Bangla Texts

BornoViT: A Novel Efficient Vision Transformer for Bengali Handwritten Basic Characters Classification