Sign Language Recognition (SLR) presents complex challenges due to variable signing speeds and spatial intricacies. Specifically, Indian Sign Language (ISL) requires precise modeling of its inherently bimanual nature and is bottlenecked by the scarcity of large-vocabulary benchmarks and the inability of standard models to handle irregular signing velocities. To address this, a structurally-aware hybrid framework for efficient, gloss ISL recognition is proposed. To minimize computational redundancy, an entropy-driven SVD sampler that dynamically extracts informative keyframes is used. Because this subsampling creates irregular physical time gaps, the proposed architecture employs a Continuous Dual Positional Encoding paired with a distance-aware Elastic Transformer to preserve natural signing rhythm. Spatially, a Kinematically-Conditioned Adaptive GCN explicitly models bimanual geometry using unified bone-joint features. To validate our framework, a 120-class, 9,453-video native ISL dataset is created. Augmented by Physical Node Dropout and ArcFace optimization to mitigate subject-specific memorization, our framework achieves a competitive 89.8% accuracy, establishing a strong baseline for complex sign language modeling.
Figures & tables
Datasets
Words
Videos
Avg. Samples
Nandy et al. (2010)
22
600
27.3
Kishore and Kumar (2012)
80
800
10
INCLUDE Sridhar et al. (2020)
263
4287
16.3
CISLR Joshi et al. (2022)
4765
7050
1.5
ISL-CSLTR Elakkiya and Natarajan (2021)
186
700
3.8
Our
120
9453
78.8
Table 1: Comparison of existing Indian Sign Language (ISL) datasets based on vocabulary size, total videos, and average videos per word.
Figure 1: Overall architecture of the proposed hybrid framework. The architecture is a hybrid: a Kinematically-Conditioned Part-GCN handles spatial skeleton structure, while the distance-aware Elastic Transformer performs temporal reasoning over the resulting pose tokens.
Category
Words
Class Weight
Avg. samples
Medical
27
[0.64, 1.26]
61.30
Geographical
23
[0.90, 2.66]
86.90
Technical
18
[0.90, 2.66]
87.11
Action Verbs
20
[0.86, 2.52]
86.25
Daily Life
32
[0.85, 2.47]
78.30
Full Dataset
120
[0.81, 2.41]
78.80
Table 2: Dataset distribution across semantic categories including word counts, weight ranges, and signer averages.
Figure 2: The dynamic preprocessing pipeline of our framework. Raw sequences undergo Motion-Guided Region of Interest (ROI) extraction, followed by entropy-driven SVD temporal subsampling, and finally RTMW skeletal extraction to form the unified input representation.
Category
Train Acc
Train F1
Test Acc
Test F1
Medical
99.2%
99.2%
98.2%
98.2%
Geographical
97.0%
96.8%
91.0%
90.1%
Technical
98.0%
98.0%
90.7%
90.5%
Action Verbs
97.5%
97.5%
88.3%
88.3%
Daily Life
96.2%
96.3%
89.1%
89.2%
Full
93.3%
93.5%
89.8%
90.0%
Table 3: Category-level performance breakdown of the proposed model.
Model
Dataset
Train Acc
Test Acc
MobileNetV2 BiLSTM Sridhar et al. (2020)
INCLUDE50
–
94.5%
DCDW-LSTM Dey and Biswas (2023)
INCLUDE50
–
91.3%
BERT Transformer Prathap et al. (2023)
INCLUDE50
–
89.5%
C3D-BiLSTM MHAttention Dey et al. (2024)
INCLUDE50
–
92.46%
Meta-Learning + LSTM RNN Katti et al. (2023)
INCLUDE50
–
84.52%
MediaPipe CNN BiLSTM Ghanimi et al. (2024)
INCLUDE50
–
97.44%
Table 4: Performance of the proposed framework on established public sign language benchmarks. Training accuracies are reported for our model only to demonstrate the absence of underfitting; they were not available in the original baseline publications (denoted –).
Architecture
Train Acc
Test Acc
CTR-GCN Chen et al. (2021)
69.9%
64.8%
ST-GCN Yan et al. (2018)
85.1%
83.0%
Proposed (Ours)
93.3%
89.8%
Table 5: Comparison of our proposed model against established skeleton baselines on the 120-class ISL dataset. All models utilize the proposed SVD preprocessing pipeline.
Approach
Pre-trained Prior
Train Acc (%)
Test Acc (%)
SlowFast_R50
Kinetics-400
63.40
30.62
MViTv2 Li et al. (2022)
Kinetics-400
33.84
33.63
VideoMAE
Kinetics-400
99.00
42.60
VideoMAE Tong et al. (2022)
SSv2
99.00
74.62
VideoMAE (Staged Fine-Tuning)
SSv2
100.00
81.20
VideoMAE (LoRA Only)
SSv2
80.00
60.00
Table 6: Performance comparison of alternative vision architectures. These models were evaluated exclusively on the Geographical word category subset due to their extreme computational overhead.
Modification
Train Acc
Train F1
Test Acc
Test F1
Proposed Full Framework (Ours)
93.3%
93.5%
89.8%
90.0%
Loss Functions
w/o Weighted Loss
93.1%
93.3%
89.4%
89.5%
w/o ArcFace
94.2%
94.4%
88.0%
88.2%
w/o ArcFace + w/o Weighted Loss
93.9%
94.2%
87.4%
87.8%
Architecture Modules
Table 7: Ablation study of the proposed framework on the 120-class ISL dataset.
Metric
Measurement
Trainable Parameters
6.13 Million
Theoretical Compute
252.57 MFLOPs
Model Weights (float32)
23.39 MB
Peak VRAM ( B=1 )
34.91 MB
Inference Latency
2.87 ± 0.19 ms
Throughput (RTX 4090)
348.7 samples/sec
Table 8: Computational footprint of the proposed architecture.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Hardware
GPU Model
RTX 4090 (24GB)
CPU Model
i9-13900K
Preprocessing & Split
Motion Threshold ( τL )
25
ROI Padding Factor ( λ )
1.2
Appendix
Table 9: Comprehensive summary of model implementation configuration and hyperparameters.
Skeleton-based isolated sign language recognition (ISLR) demands fine-grained understanding of articulated motion across multiple spatial scales, from subtle finger movements to global body dynamics. Existing approaches typically rely on deep feed-forward architectures, which increase model capacity but lack mechanisms for recurrent refinement and structured representation. We propose LA-Sign, a looped transformer framework with geometry-aware alignment for ISLR. Instead of stacking deeper layers, LA-Sign derives its depth from recurrence, repeatedly revisiting latent representations to progressively refine motion understanding under shared parameters. To further regularise this refinement process, we present a geometry-aware contrastive objective that projects skeletal and textual features into an adaptive hyperbolic space, encouraging multi-scale semantic organisation. We study three looping designs and multiple geometric manifolds, demonstrating that encoder-decoder looping combined with adaptive Poincare alignment yields the strongest performance. Extensive experiments on WLASL and MSASL benchmarks show that LA-Sign achieves state-of-the-art results while using fewer unique layers, highlighting the effectiveness of recurrent latent refinement and geometry-aware representation learning for sign language recognition.
Muxin Pu, Mei Kuan Lim, Chun Yong Chong +1
School of Information Technology, Monash University · S-Lab, Nanyang Technological University
How do you build a sign language recognizer that works on a phone? That question drove this work. We built EfficientSign, a lightweight model which takes EfficientNet-B0 and focuses on two attention modules (Squeeze-and-Excitation for channel focus, and a spatial attention layer that focuses on the hand gestures). We tested it against five other approaches on 12,637 images of Indian Sign Language alphabets, all 26 classes, using 5-fold cross-validation. EfficientSign achieves the accuracy of 99.94% (+/-0.05%), which matches the performance of ResNet18's 99.97% accuracy, but with 62% fewer parameters (4.2M vs 11.2M). We also experimented with feeding deep features (1,280-dimensional vectors pulled from EfficientNet-B0's pooling layer) into classical classifiers. SVM achieved the accuracy of 99.63%, Logistic Regression achieved the accuracy of 99.03% and KNN achieved accuracy of 96.33%. All of these blow past the 92% that SURF-based methods managed on a similar dataset back in 2015. Our results show that attention-enhanced learning model provides an efficient and deployable solution for ISL recognition without requiring a massive model or hand-tuned feature pipelines anymore.
Automated sign language recognition for underrepresented languages remains a largely unsolved problem. Central African Sign Language (CASL) exemplifies this challenge: the CASL-W60 benchmark contains 60 isolated signs, and the best published result is 69.93%. We present TransSLR, a lightweight Temporal Transformer Encoder trained from scratch on normalized 64-frame pose sequences. By operating on geometric keypoint representations rather than raw RGB, TransSLR achieves 80.39% top-1 accuracy and 91.07% top-5 accuracy on a signer-independent evaluation split. This result is 10.46 percentage points higher than the published 69.93% result. TransSLR has 8.67 million trainable parameters. We also evaluate zero-shot transfer from a high-resource sign-language model, which achieves 0.00% exact-match accuracy under our manual gloss-matching protocol. Our experiments compare pose-based, RGB-based, and multimodal approaches to sign-language recognition for a low-resource language.
Lucia Yen Wanchi, Samuel Johnny, Victor Tolulope Olufemi +2