Sign Language Recognition (SLR) presents complex challenges due to variable signing speeds and spatial intricacies. Specifically, Indian Sign Language (ISL) requires precise modeling of its inherently bimanual nature and is bottlenecked by the scarcity of large-vocabulary benchmarks and the inability of standard models to handle irregular signing velocities. To address this, a structurally-aware hybrid framework for efficient, gloss ISL recognition is proposed. To minimize computational redundancy, an entropy-driven SVD sampler that dynamically extracts informative keyframes is used. Because this subsampling creates irregular physical time gaps, the proposed architecture employs a Continuous Dual Positional Encoding paired with a distance-aware Elastic Transformer to preserve natural signing rhythm. Spatially, a Kinematically-Conditioned Adaptive GCN explicitly models bimanual geometry using unified bone-joint features. To validate our framework, a 120-class, 9,453-video native ISL dataset is created. Augmented by Physical Node Dropout and ArcFace optimization to mitigate subject-specific memorization, our framework achieves a competitive 89.8% accuracy, establishing a strong baseline for complex sign language modeling.
Figures & tables
Datasets
Words
Videos
Avg. Samples
Nandy et al. (2010)
22
600
27.3
Kishore and Kumar (2012)
80
800
10
INCLUDE Sridhar et al. (2020)
263
4287
16.3
CISLR Joshi et al. (2022)
4765
7050
1.5
ISL-CSLTR Elakkiya and Natarajan (2021)
186
700
3.8
Our
120
9453
78.8
Table 1: Comparison of existing Indian Sign Language (ISL) datasets based on vocabulary size, total videos, and average videos per word.
Figure 1: Overall architecture of the proposed hybrid framework. The architecture is a hybrid: a Kinematically-Conditioned Part-GCN handles spatial skeleton structure, while the distance-aware Elastic Transformer performs temporal reasoning over the resulting pose tokens.
Category
Words
Class Weight
Avg. samples
Medical
27
[0.64, 1.26]
61.30
Geographical
23
[0.90, 2.66]
86.90
Technical
18
[0.90, 2.66]
87.11
Action Verbs
20
[0.86, 2.52]
86.25
Daily Life
32
[0.85, 2.47]
78.30
Full Dataset
120
[0.81, 2.41]
78.80
Table 2: Dataset distribution across semantic categories including word counts, weight ranges, and signer averages.
Figure 2: The dynamic preprocessing pipeline of our framework. Raw sequences undergo Motion-Guided Region of Interest (ROI) extraction, followed by entropy-driven SVD temporal subsampling, and finally RTMW skeletal extraction to form the unified input representation.
Category
Train Acc
Train F1
Test Acc
Test F1
Medical
99.2%
99.2%
98.2%
98.2%
Geographical
97.0%
96.8%
91.0%
90.1%
Technical
98.0%
98.0%
90.7%
90.5%
Action Verbs
97.5%
97.5%
88.3%
88.3%
Daily Life
96.2%
96.3%
89.1%
89.2%
Full
93.3%
93.5%
89.8%
90.0%
Table 3: Category-level performance breakdown of the proposed model.
Model
Dataset
Train Acc
Test Acc
MobileNetV2 BiLSTM Sridhar et al. (2020)
INCLUDE50
–
94.5%
DCDW-LSTM Dey and Biswas (2023)
INCLUDE50
–
91.3%
BERT Transformer Prathap et al. (2023)
INCLUDE50
–
89.5%
C3D-BiLSTM MHAttention Dey et al. (2024)
INCLUDE50
–
92.46%
Meta-Learning + LSTM RNN Katti et al. (2023)
INCLUDE50
–
84.52%
MediaPipe CNN BiLSTM Ghanimi et al. (2024)
INCLUDE50
–
97.44%
Table 4: Performance of the proposed framework on established public sign language benchmarks. Training accuracies are reported for our model only to demonstrate the absence of underfitting; they were not available in the original baseline publications (denoted –).
Architecture
Train Acc
Test Acc
CTR-GCN Chen et al. (2021)
69.9%
64.8%
ST-GCN Yan et al. (2018)
85.1%
83.0%
Proposed (Ours)
93.3%
89.8%
Table 5: Comparison of our proposed model against established skeleton baselines on the 120-class ISL dataset. All models utilize the proposed SVD preprocessing pipeline.
Approach
Pre-trained Prior
Train Acc (%)
Test Acc (%)
SlowFast_R50
Kinetics-400
63.40
30.62
MViTv2 Li et al. (2022)
Kinetics-400
33.84
33.63
VideoMAE
Kinetics-400
99.00
42.60
VideoMAE Tong et al. (2022)
SSv2
99.00
74.62
VideoMAE (Staged Fine-Tuning)
SSv2
100.00
81.20
VideoMAE (LoRA Only)
SSv2
80.00
60.00
Table 6: Performance comparison of alternative vision architectures. These models were evaluated exclusively on the Geographical word category subset due to their extreme computational overhead.
Modification
Train Acc
Train F1
Test Acc
Test F1
Proposed Full Framework (Ours)
93.3%
93.5%
89.8%
90.0%
Loss Functions
w/o Weighted Loss
93.1%
93.3%
89.4%
89.5%
w/o ArcFace
94.2%
94.4%
88.0%
88.2%
w/o ArcFace + w/o Weighted Loss
93.9%
94.2%
87.4%
87.8%
Architecture Modules
Table 7: Ablation study of the proposed framework on the 120-class ISL dataset.
Metric
Measurement
Trainable Parameters
6.13 Million
Theoretical Compute
252.57 MFLOPs
Model Weights (float32)
23.39 MB
Peak VRAM ( B=1 )
34.91 MB
Inference Latency
2.87 ± 0.19 ms
Throughput (RTX 4090)
348.7 samples/sec
Table 8: Computational footprint of the proposed architecture.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Hardware
GPU Model
RTX 4090 (24GB)
CPU Model
i9-13900K
Preprocessing & Split
Motion Threshold ( τL )
25
ROI Padding Factor ( λ )
1.2
Appendix
Table 9: Comprehensive summary of model implementation configuration and hyperparameters.