IsharaBART: An efficient BART-Based Framework for Low-Resource Bangla Sign Language Translation
Organizations: Department of Electrical and Electronic Engineering, University of Chittagong, Chittagong-4331, Bangladesh.
Abstract
Millions of Bangla Sign Language (BdSL) users lack reliable automatic translation tools. Existing systems are limited by data scarcity, unidirectional design, and computationally expensive architectures that are not suitable for low-resource deployment. We introduce IsharaBART, a bidirectional generative framework for gloss-sentence translation in BdSL. IsharaBART uses Bidirectional and Auto-Regressive Transformers (BART)-based conditional language modeling with denoising learning objectives. We developed a parallel dataset of 1,000 professionally annotated gloss-sentence pairs, balanced across tenses and structures. We expanded the dataset to 4,831 pairs through four augmentation techniques: rule-based tense transformations, masked token substitutions, back-translation, and one-shot prompting. A compact denoising sequence-to-sequence model was pre-trained from scratch on monolingual Bangla text from six different corpora, then fine-tuned on the annotated and augmented datasets. We compare IsharaBART against four larger baseline models under identical conditions, where it achieves comparable or superior bidirectional performance. On the gloss-to-sentence (G2S) task, it attains Bilingual Evaluation Understudy (BLEU-4) score up to 51.7, and on the sentence-to-gloss (S2G) task, it achieves 61.40, outperforming multilingual models despite having fewer than 10% of the parameters. Ablation studies demonstrate that pre-training provides 62.5-90.4% faster convergence and 11.5-118% higher BLEU-4 than scratch. Low-rank adaptation (LoRA) reduces trainable parameters by 99.3%, but with 28.0-52.2% loss of BLEU-4 compared to full-parameter fine-tuning.
Figures & tables
| Study | Year | Approach | Dataset/Language | Key Achievement | Limitations |
| Phase I: Hand-Centric & Multi-Modal Foundations (2015-2019) | |||||
| Kang et al. [ 15 ] | 2015 | CNN + Depth Maps | 31 ASL alphabets/digits | 99.99% (same signer), 83.58-85.49% (cross-subject) | Isolated gestures only; no temporal continuity |
| Oberweger et al. [ 16 ] | 2015 | CNN for 3D Joints | Depth-based hand pose | 10-20mm mean joint error | Requires depth calibration; fails with occlusions |
| Huang et al. [ 17 ] | 2015 | 3D CNN Multi-Modal | RGB-Depth-Skeleton | 94.2% accuracy; first multi-modal fusion | Static feature fusion; limited to short sequences |
| Rastgoo et al. [ 18 ] | 2018 | RBM Fusion | Massey/NYU/ASL datasets | 11-28% improvement over prior work | Superseded by LSTM-based methods |
| Wei et al. [ 19 ] | 2019 | 3D ResNet + BiLSTM | CSL | 95.2% precision; sentence-level translation | Rule-based grammar; limited scalability |
| Model | Provider | Details |
| llama-3.3-70b-versatile | Groq API | 70B parameters; temperature=0.85 for diversity; max_tokens=150; top_p=1 |
| Prompt Template | ||
| I will provide you an example of a BdSL Gloss and its corresponding Bangla Sentence. Example Input: Sentence: {Example Sentence} Gloss: {Example Gloss} Task: Generate a completely NEW Gloss and Sentence pair. 1. Structure: Follow the same grammatical style as the example. 2. Content: The meaning MUST be different (change the subject, object, or verb). 3. Format: Output ONLY the result in this exact format: Sentence: [New Sentence] Gloss: [New Gloss] | ||
| Technique | Step-by-Step Process and Example |
|---|---|
| Rule-Based Transformation | Original: Sentence: \bengalifont তুমি তাড়াতাড়ি কাজটি করো । Gloss: \bengalifont তুমি তাড়াতাড়ি কাজ এটি কর Step 1: Identify verb tense/mood ( \bengalifont করো - imperative present) Step 2: Apply tense transformation rule (imperative future) Step 3: Transform verb: \bengalifont করো \bengalifont করবে Step 4: Update gloss accordingly: \bengalifont কর \bengalifont করা হবে Augmented: Sentence: \bengalifont তুমি তাড়াতাড়ি কাজটি করবে । Gloss: \bengalifont তুমি তাড়াতাড়ি কাজ এটি করা হবে |
| Masked Token Substitution | Original: Sentence: \bengalifont সে পরিশ্রমে ভাল রেজাল্ট করেছে। Gloss: \bengalifont সে কাজ পরিশ্রম অনেক ভাল রেজাল্ট পাওয়া Step 1: Randomly select a content word ( \bengalifont রেজাল্ট ) Step 2: Mask the token: \bengalifont সে পরিশ্রমে ভাল [MASK] \bengalifont করেছে। Step 3: Use BanglaBERT to predict: [MASK] \bengalifont ফল Step 4: Substitute in sentence and update gloss Augmented: Sentence: \bengalifont সে পরিশ্রমে ভাল ফল করেছে। Gloss: \bengalifont সে কাজ পরিশ্রম ভাল ফল পাওয়া |
| Back-Translation | Original: Sentence: \bengalifont তার সাথে আমার প্রায়ই দেখা হয়। Step 1: Translate to English using NLLB200: “I see him often.” Step 2: Translate back to Bangla using IndicTrans2: \bengalifont আমি প্রায়ই তাকে দেখি। Step 3: Generate gloss for paraphrased sentence Augmented: Sentence: \bengalifont আমি প্রায়ই তাকে দেখি। (paraphrased) Gloss: \bengalifont তার সাথে আমার হঠাৎ হঠাৎ দেখা হয় |
| One-Shot Prompt | Seed Example: Sentence: \bengalifont তুমি তাড়াতাড়ি কাজটি করো । Gloss: \bengalifont তুমি তাড়াতাড়ি কাজ এটি কর Step 1: Provide seed pair to LLM (Llama-3.3-70B) Step 2: Prompt LLM Step 3: LLM generates by changing subject/object/verb Generated: Sentence: \bengalifont সে ধীরে ধীরে খাবারটি খায় । Gloss: \bengalifont সে ধীরে ধীরে খাবার এটি খা |
| Metric | Evaluator 1 | Evaluator 2 | Overall | Agreement |
|---|---|---|---|---|
| Adequacy | ||||
| Mean SD | 98.83 8.09 | 99.58 5.95 | 99.21 7.11 | *** |
| Median | 100 | 100 | 100 | MAE = 0.75 |
| High Quality ( ) | 298 (99.3%) | 298 (99.3%) | 298 (99.3%) | — |
| Fluency | ||||
| Mean SD | 2.98 0.14 | 2.99 0.08 | 2.99 0.11 | |
| Property | BART |
|---|---|
| Model family | Seq2Seq denoising LM |
| Architecture | Encoder + decoder |
| Encoder layers | 4 |
| Decoder layers | 4 |
| Hidden dimension | 512 |
| Attention heads | 8 / 8 (enc / dec) |
| Model | Params | R-1 | R-2 | R-L | B-1 | B-2 | B-3 | B-4 | COMET |
|---|---|---|---|---|---|---|---|---|---|
| G2S – Annotated Dataset (1,000 pairs) | |||||||||
| mBART-50 | 611M | 77.86 | 63.36 | 76.63 | 65.15 | 54.64 | 46.39 | 39.93 | 0.92 |
| mT5-Small | 300M | 77.76 | 62.37 | 76.20 | 63.78 | 52.78 | 44.76 | 38.76 | 0.90 |
| BanglaT5 | 247M | 82.94 | 70.65 | 81.71 | 72.31 | 62.84 | 55.16 | 48.91 | 0.95 |
| BanglaGPT | 124M | 61.27 | 45.16 | 60.76 | 41.88 | 31.69 | 23.83 | 17.82 | 0.78 |
| IsharaBART (ours) | 56M | 79.86 | 65.22 | 78.32 | 72.73 | 59.29 | 50.05 | 43.30 | 0.93 |
| Input (Gloss) | Ground Truth (Bangla) | Predicted (Bangla) |
| Excellent Performance | ||
| \bengalifont তুমি তুমি আস্তে আস্তে বড় হবে, ভালো মানুষ কাজ কাজ চেষ্টা করবে (YOU YOU SLOWLY SLOWLY GROW WILL, GOOD PERSON WORK WORK TRY WILL) | \bengalifont তুমি বড় হয়ে ভালো মানুষ হওয়ার চেষ্টা করবে। (You will try to become a good person when you grow up.) | \bengalifont তুমি বড় হয়ে ভালো মানুষ হওয়ার চেষ্টা করবে। (You will try to become a good person when you grow up.) |
| \bengalifont যখন তুমি আস্তে আস্তে বড় হবে, তখন তুমি কাজ কি হবে (WHEN YOU SLOWLY SLOWLY GROW WILL, THEN YOU WORK WHAT WILL) | \bengalifont যখন তুমি বড় হবে, তখন তুমি কি করবে? (When you grow up, what will you do?) | \bengalifont যখন তুমি বড় হবে, তখন তুমি কি করবে? (When you grow up, what will you do?) |
| \bengalifont শিক্ষক ছাত্র নতুন বই পড়া বলা না (TEACHER STUDENT NEW BOOK READ TELL NOT) | \bengalifont শিক্ষক ছাত্রদের নতুন বই পড়তে বলবেন না। (The teacher did not tell the students to read the new book.) | \bengalifont শিক্ষক ছাত্রদের নতুন বই পড়তে বলবেন না। (The teacher did not tell the students to read the new book.) |
| \bengalifont আমার হাত একটি চিঠি আছে (MY HAND ONE LETTER HAVE) | \bengalifont আমার কাছে একটি চিঠি আছে। (I have a letter.) | \bengalifont আমার কাছে একটি চিঠি আছে। (I have a letter.) |
| \bengalifont ছাত্র খেলা জন্য মাঠ যাচ্ছিল (STUDENT PLAY FOR FIELD GOING-PAST) | \bengalifont ছাত্ররা খেলার জন্য মাঠে যাচ্ছিল। (The students were going to the field to play.) | \bengalifont ছাত্ররা খেলার জন্য মাঠে যাচ্ছিল। (The students were going to the field to play.) |
| Model | Params | R-1 | R-2 | R-L | B-1 | B-2 | B-3 | B-4 | COMET |
|---|---|---|---|---|---|---|---|---|---|
| S2G – Annotated Dataset (1,000 pairs) | |||||||||
| mBART-50 | 611M | 78.84 | 63.35 | 76.74 | 70.55 | 56.89 | 45.96 | 37.03 | 0.92 |
| mT5-Small | 300M | 74.88 | 60.78 | 73.60 | 65.35 | 52.81 | 43.23 | 35.96 | 0.87 |
| BanglaT5 | 247M | 80.06 | 65.41 | 78.54 | 73.60 | 60.20 | 49.09 | 39.29 | 0.93 |
| BanglaGPT | 124M | 58.60 | 43.38 | 58.01 | 45.97 | 33.23 | 23.54 | 16.19 | 0.77 |
| IsharaBART (ours) | 56M | 77.16 | 63.01 | 75.78 | 69.08 | 56.71 | 47.18 | 39.39 | 0.94 |
| Input (Bangla) | Ground Truth (Gloss) | Predicted (Gloss) |
| Excellent Performance | ||
| \bengalifont সে কাজে হাত দেয়ার আগেই কাজ শেষহয়ে গিয়েছিল। (He/she had finished the work before even starting it.) | \bengalifont সে কাজ হাত শুরু আগেই কাজ শেষ ছিল (HE/SHE WORK HAND START BEFORE WORK FINISHED WAS) | \bengalifont সে কাজ হাত শুরু আগেই কাজ শেষ ছিল (HE/SHE WORK HAND START BEFORE WORK FINISHED WAS) |
| \bengalifont শিক্ষক ছাত্রদের নতুন বই পড়তে বলবেন না। (The teacher will not ask the students to read the new book.) | \bengalifont শিক্ষক ছাত্র নতুন বই পড়া বলা না (TEACHER STUDENT NEW BOOK READ TELL NOT) | \bengalifont শিক্ষক ছাত্র নতুন বই পড়া হবে না (TEACHER STUDENT NEW BOOK READ WILL NOT) |
| \bengalifont ডাক্তার কোনো নতুন ওষুধ দেবেন না। (The doctor will not prescribe any new medicine.) | \bengalifont ডাক্তার কোন নতুন ওষুধ লিখে দেওয়া না (DOCTOR ANY NEW MEDICINE PRESCRIBE GIVE NOT) | \bengalifont ডাক্তার কোন নতুন ওষুধ লিখে দেওয়া না (DOCTOR ANY NEW MEDICINE PRESCRIBE GIVE NOT) |
| \bengalifont নাফিস তার অভ্যাস বদলাতে চেয়েছিল। (Nafis wanted to change his habit.) | \bengalifont নাফিস তার আগে অভ্যাস বদলানো চাওয়া ছিল (NAFIS HIS/HER BEFORE HABIT CHANGE WANT WAS) | \bengalifont নাফিস তার আগে অভ্যাস বদলানো চাওয়া ছিল (NAFIS HIS/HER BEFORE HABIT CHANGE WANT WAS) |
| \bengalifont যখন সে কাঁদছিল, আমি তার হাত ধরেছিলাম। (When he/she was crying, I was holding his/her hand.) | \bengalifont যখন সে কান্না ছিল তখন আমি তার হাত ধরা ছিল (WHEN HE/SHE CRY WAS THEN I HIS/HER HAND HOLD WAS) | \bengalifont যখন সে কান্না ছিল তখন আমি তার হাত ধরা ছিল (WHEN HE/SHE CRY WAS THEN I HIS/HER HAND HOLD WAS) |
| G2S | S2G | |||||||
| Metric | Eval. 1 | Eval. 2 | Overall | Agreement | Eval. 1 | Eval. 2 | Overall | Agreement |
| Adequacy | ||||||||
| Mean SD | ||||||||
| Median | 100 | 100 | 100 | MAE = 11.5 | 100 | 100 | 100 | MAE = 12.5 |
| High quality ( ) | 42 (84%) | 46 (92%) | 42 (84%) | — | 38 (76%) | 38 (76%) | 38 (76%) | — |
| Fluency | ||||||||
| Dataset | Translation Direction | Fine-Tuning Strategy | BLEU-4 | COMET | ROUGE-L |
|---|---|---|---|---|---|
| Annotated | G2S | Full fine-tuning | 43.30 | 0.9279 | 78.32 |
| LoRA | 31.17 | 0.8973 | 72.27 | ||
| Change | ( ) | ( ) | ( ) | ||
| Annotated | S2G | Full fine-tuning | 39.39 | 0.9283 | 75.78 |
| LoRA | 22.15 | 0.9221 | 68.97 | ||
| Change | ( ) | ( ) | ( ) |
| Dataset | Language | Size | Domain | G2S | S2G |
|---|---|---|---|---|---|
| PHOENIX-14T [ 7 , 68 ] | German (DGS) | 8,257 | Weather | 18.75 | 20.68 |
| How2Sign [ 69 ] | English (ASL) | 35,000 | Instructional | 8.0 | — |
| iSign [ 70 ] | English (ISL) | 118,000 | Open | 17.6 | — |
| IsharaKhobor [ 71 ] | Bangla (BdSL) | 5,642 | News | — | — |
| Bangla-SGP [ 72 ] | Bangla (BdSL) | 4,000 | Open | 27.3 | — |
| Ours | Bangla (BdSL) | 4,831 | Open | 51.7 | 61.4 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Training Steps | 30,000 |
| Batch Size (per device) | 64 |
| Gradient Accumulation | 4 steps |
| Effective Batch Size | 256 |
| Learning Rate | |
| LR Schedule | Polynomial decay |
| Hyperparameter | IsharaBART | mBART-50 | mT5-Small | BanglaT5 | BanglaGPT |
|---|---|---|---|---|---|
| Epochs | 20 | 20 | 20 | 20 | 20 |
| Batch Size (per device) | 16 | 16 | 16 | 16 | 16 |
| Gradient Accumulation | 2 | 2 | 2 | 2 | 4 |
| Effective Batch Size | 32 | 32 | 32 | 32 | 64 |
| Learning Rate | |||||
| LR Schedule | Linear decay | Linear decay | Linear decay | Linear decay | Linear decay |
| Hyperparameter | Value |
|---|---|
| Task type | Seq2Seq language modeling |
| LoRA rank ( ) | 8 |
| LoRA scaling factor ( ) | 16 |
| LoRA dropout | 0.1 |
| Target modules | q_proj, k_proj, v_proj, out_proj |
Sign Language Translation
Sign language translation maps continuous signing into written or spoken-language text using visual, pose, and linguistic cues.
See all 55 papers