eess.SPMar 26, 2026

Development of ML model for triboelectric nanogenerator based sign language detection system

Authors: Meshv Patel, Bikash Baro, Sayan Bayan, Mohendra Roy

Organizations: Department of ICT, Pandit Deendayal Energy University, Gandhinagar, India · Department of Physics, Rajiv Gandhi University, Rono Hills, Itanagar, Arunachal Pradesh

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Sign language recognition (SLR) is vital for bridging communication gaps between deaf and hearing communities. Vision-based approaches suffer from occlusion, computational costs, and physical constraints. This work presents a comparison of machine learning (ML) and deep learning models for a custom triboelectric nanogenerator (TENG)-based sensor glove. Utilizing multivariate time-series data from five flex sensors, the study benchmarks traditional ML algorithms, feedforward neural networks, LSTM-based temporal models, and a multi-sensor MFCC CNN-LSTM architecture across 11 sign classes (digits 1-5, letters A-F). The proposed MFCC CNN-LSTM architecture processes frequency-domain features from each sensor through independent convolutional branches before fusion. It achieves 93.33% accuracy and 95.56% precision, a 23-point improvement over the best ML algorithm (Random Forest: 70.38%). Ablation studies reveal 50-timestep windows offer a tradeoff between temporal context and training data volume, yielding 84.13% accuracy compared to 58.06% with 100-timestep windows. MFCC feature extraction maps temporal variations to execution-speed-invariant spectral representations, and data augmentation methods (time warping, noise injection) are essential for generalization. Results demonstrate that frequency-domain feature representations combined with parallel multi-sensor processing architectures offer enhancement over classical algorithms and time-domain deep learning for wearable sensor-based gesture recognition. This aids assistive technology development.

Figures & tables

Fig. 1 : Schematic representation and real image of the fabricated STENG or sensor
Fig. 2 : Photograph of sensors attached to commercial nitrile gloves with variation of numeric and alphabetic sign languages
Fig. 3 : Generated output voltage for sign language F
ModelAccuracyPrecisionRecallF1-Score
MFCC CNN-LSTM93.33%95.56%93.33%92.89%
LSTM (Window 50)84.13%83.40%84.13%82.62%
Comprehensive LSTM82.54%83.96%82.54%82.72%
Random Forest70.38%70.53%70.38%70.35%
TABLE I : Comparative Performance of All Approaches
ReferenceSensor TypeGesturesBest AccuracyMethod
Ji et al. [ 16 ]IMU (16-node)20 dynamic98.85%Attention-BiLSTM
Zhu et al. [ 17 ]Fiber optic12∼\sim 95%ML-assisted
This workSTENG (ZnO)11 static93.33%MFCC CNN-LSTM
TABLE II : Comparison with Prior Glove-Based Sign Language Recognition Works
Fig. 4 : Comparative performance of all models across different approaches
Fig. 5 : MFCC CNN-LSTM training performance across 78 epochs
Fig. 6 : Confusion matrix of MFCC CNN-LSTM on the test set
WindowAccuracyPrecisionRecallF1-ScoreEpochsChunks
Size      
5084.13%83.40%84.13%82.62%121415
7566.67%62.67%66.67%60.20%61274
10058.06%54.95%58.06%51.47%88205
TABLE III : Performance Across Different Window Sizes
Fig. 7 : Training accuracy and loss across epochs for different window sizes
ModelAccuracyPrecisionRecallF1-Score
Random Forest70.38%70.53%70.38%70.35%
K-Nearest Neighbors68.64%69.16%68.64%68.81%
Gradient Boosting58.12%58.61%58.12%57.99%
Decision Tree57.60%57.85%57.60%57.69%
Support Vector Machine56.88%60.10%56.88%57.20%
TABLE IV : Traditional ML Performance (Top 5 Models)
Fig. 8 : Performance comparison of traditional ML models
Fig. 9 : Comprehensive LSTM training performance across 107 epochs
More in this topic

Sign Language Recognition

Sign language recognition uses video, images, or sensor data to identify signs, fingerspelling, and continuous signing across people and settings.

See all 56 papers

Explore similar work

CardsList
Aug 3, 2026

TransSLR: A Lightweight Transformer for Sign Language Recognition

Automated sign language recognition for underrepresented languages remains a largely unsolved problem. Central African Sign Language (CASL) exemplifies this challenge: the CASL-W60 benchmark contains 60 isolated signs, and the best published result is 69.93%. We present TransSLR, a lightweight Temporal Transformer Encoder trained from scratch on normalized 64-frame pose sequences. By operating on geometric keypoint representations rather than raw RGB, TransSLR achieves 80.39% top-1 accuracy and 91.07% top-5 accuracy on a signer-independent evaluation split. This result is 10.46 percentage points higher than the published 69.93% result. TransSLR has 8.67 million trainable parameters. We also evaluate zero-shot transfer from a high-resource sign-language model, which achieves 0.00% exact-match accuracy under our manual gloss-matching protocol. Our experiments compare pose-based, RGB-based, and multimodal approaches to sign-language recognition for a low-resource language.

Lucia Yen Wanchi, Samuel Johnny, Victor Tolulope Olufemi +2

Carnegie Mellon University Africa

Sign Language Recognition
AskPDF
Aug 4, 2026

A Low-Cost Hybrid Reservoir Computing Model for Isolated Sign Language Video Recognition

Sign language recognition (SLR) enhances communication between hearing and hearing-impaired individuals. Although deep learning (DL) has achieved promising performance in SLR, its high computational cost limits deployment on edge devices. To address this challenge, we propose a lightweight reservoir computing (RC)-based approach for SLR. In the proposed method, MediaPipe extracts body and hand keypoints to capture the spatial and temporal dynamics of gestures. These keypoints are then processed by a hybrid reservoir computing (HRC) architecture that combines deep reservoir computing (DRC) and bidirectional reservoir computing (BRC), transforming the input into a high-dimensional dynamic representation. A ridge regression model maps the final HRC state to class labels. This HRC-based SLR method achieved Top-1, Top-5, and Top-10 accuracies of 61.12%, 86.05%, and 92.56%, respectively, on the Word-Level American Sign Language 100 (WLASL100) video dataset, demonstrating competitive performance compared to deep learning-based approaches. Additionally, due to the lightweight nature of RC, the training time was drastically reduced to only a few seconds compared with DL-based methods such as Bi-GRU.This method offers low computational cost, showing its potential for deployment on edge devices.

Nitin Kumar Singh, Arie Rachmad Syulistyo, Yuichiro Tanaka +1

Woosong University Kazakhstan, Turkistan City, Kazakhstan · Data Science and Artificial Intelligence Laboratory (DSAIL), Woosong University Kazakhstan, Turkistan City, Kazakhstan · Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu, Kitakyushu, 808-0196, Japan +2

Reservoir ComputingSign Language Recognition
AskPDF
Oct 10, 2026

A Hybrid Kinematic GCN and Elastic Transformer for Efficient Indian Sign Language Recognition

Sign Language Recognition (SLR) presents complex challenges due to variable signing speeds and spatial intricacies. Specifically, Indian Sign Language (ISL) requires precise modeling of its inherently bimanual nature and is bottlenecked by the scarcity of large-vocabulary benchmarks and the inability of standard models to handle irregular signing velocities. To address this, a structurally-aware hybrid framework for efficient, gloss ISL recognition is proposed. To minimize computational redundancy, an entropy-driven SVD sampler that dynamically extracts informative keyframes is used. Because this subsampling creates irregular physical time gaps, the proposed architecture employs a Continuous Dual Positional Encoding paired with a distance-aware Elastic Transformer to preserve natural signing rhythm. Spatially, a Kinematically-Conditioned Adaptive GCN explicitly models bimanual geometry using unified bone-joint features. To validate our framework, a 120-class, 9,453-video native ISL dataset is created. Augmented by Physical Node Dropout and ArcFace optimization to mitigate subject-specific memorization, our framework achieves a competitive 89.8% accuracy, establishing a strong baseline for complex sign language modeling.

Pushp Garg, Himangshu Sarma, Mrinmoy Ghorai

Human Computer Interaction Lab, IIIT SriCity, India

Sign Language RecognitionTransformer
AskPDF