MambaHand: Local Identity and Behavioral Sequence Learning with Mamba for Dynamic Hand Gesture Authentication
Authors: Zenan Lin, Junduan Huang, Xilai Wang, Wenxiong Kang
Organizations: School of Automation Science and Engineering, South China University of Technology, Guangzhou, China · School of Artificial Intelligence, South China Normal University, Foshan, China
Dynamic hand gesture authentication (DHGA) is an emerging biometric technique that contains both physiological and behavioral identity characteristics. However, existing methods primarily rely on convolutional networks or manually designed motion representations, making it difficult to jointly capture fine-grained local identity differences and long-range behavioral dependencies during hand gesture execution. To address these challenges, we propose a two-stream identity representation learning framework, MambaHand Net, to combine CNNs with Mamba for DHGA. The proposed framework consists of a Local Identity Stream (LIS) and a Behavioral Sequential Stream (BSS). The LIS extracts frame-wise local identity cues with a convolutional network, while the BSS preserves spatial tokens from intermediate visual features and constructs first-order and second-order temporal difference sequences to model multi-order behavioral variations. Moreover, the bidirectional Mamba is incorporated into the BSS to model semantic motion across different spatial locations, enabling efficient long-range modeling of hand gesture execution habits. Finally, attention-guided spatial token aggregation and a two-stream fusion strategy are used to integrate local identity cues and multi-order behavioral patterns into a discriminative identity representation. Our method achieves state-of-the-art (SOTA) average performance on the SCUT-DHGA benchmark dataset, demonstrating that combining CNNs with Mamba is an effective new solution for DHGA.
Figures & tables
Figure 1: Overall architecture of the proposed MambaHand Net. The framework consists of a LIS and a BSS built upon a shared encoder. The LIS extracts frame-level local identity cues, while the BSS preserves spatial tokens from intermediate features and constructs first-order and second-order temporal difference sequences for behavioral modeling. Two Bi-Mamba-based branches are employed to capture multi-order behavioral dynamics, followed by ABTA, branch fusion, and two-stream feature fusion for final identity representation learning.
Experiments
Training Gestures
Testing Gesture
Enrollment
Verification
MG i
All gestures
Gi
Gi
UMG i
All gestures except Gi
Gi
Gi
Table 1: Experimental protocols used in this work on the SCUT-DHGA dataset. “ Gi ” denotes the i -th hand gesture.
Type
Networks
EER(%)
Parameters (M)
FLOPs (G)/Video
UMG
MG
Average
Single
Cross
Single
Cross
3D CNN
DHGA-net [ 16 ]
1.470
7.833
1.281
7.163
4.437
4.5
31.2
R(2+1)D [ 22 ]
1.096
9.078
1.096
10.081
5.338
33.6
108.4
ResNet3D-18 [ 8 ]
0.611
7.326
0.459
6.696
3.773
33.5
117.6
ResNet3D-101 [ 24 ]
2.989
13.719
1.622
13.078
7.852
86.4
323.4
Table 2: Comparison of our approach with other networks.
Convolutional Neural Networks (CNNs) capture local features efficiently but struggle with global context due to their limited receptive field. On the other hand, transformers effectively capture global dependencies through self-attention but suffer from high redundancy and computational costs. Thus, to leverage the advantages of both CNNs and transformers, we propose a unified model (UniCon-Former) that aims to provide robust and efficient performance on dynamic hand gesture recognition. The unified approach helps the model to learn both local and global features. At the beginning of each transformer stage, the convolution projections help in decreasing the dimension of the input vectors of the transformer block. This creates a pyramidal structure at each transformer stage. These features enable the UniCon-Former to reduce resource usage than vanilla transformers, making it flexible for learning multi-scale and high-resolution features, which is required in hand gesture recognition. We have performed experiments with NVGesture and Briareo datasets and achieved state-of-the-art results with fewer parameters and MACs.
Mallika Garg, Debashis Ghosh, Pyari Mohan Pradhan
Indian Institute of Technology Kharagpur, INDIA · Indian Institute of Technology Roorkee, INDIA
This paper proposes a method for dynamic hand gesture recognition based on the composition of two models: the MediaPipe Hand Landmarker, responsible for extracting 21 skeletal keypoints of the hand, and a convolutional neural network (CNN) trained to classify gestures from a spatiotemporal matrix representation of dimensions 90 by 21 of those keypoints. The method is applied to the recognition of LIBRAS (Brazilian Sign Language) gestures for device control in a home automation system, covering 11 classes of static and dynamic gestures. For real-time inference, a sliding window with temporal frame triplication is used, enabling continuous recognition without recurrent networks. Tests achieved 95% accuracy under low-light conditions and 92% under normal lighting. The results indicate that the approach is effective, although systematic experiments with greater user diversity are needed for a more thorough evaluation of generalization.
Jasmine Moreira
CPGEI Universidade Tecnológica Federal do Paraná Curitiba, Paraná, Brazil
In human computer interaction, real-time detection and classification of dynamic hand gestures is challenging as: 1) the system must run in a real-time video stream and there is no noticeable lag in response after performing a gesture; 2) there is a large difference in how people perform gestures, making recognition more difficult. In this paper, an online hand gesture recognition system is proposed, which is able to localize gestures in real-time video stream and recognize what these gestures are. To improve the robustness of the system, the sliding window approach is used to refine results from multiple windows. All of the models in my project are trained on Jester database, achieving 98+% accuracy for detector and 90+% accuracy for classifier. For the overall performance of the system, the best group can respond within three seconds and reach 37.5% Levenshtein accuracy on the homemade dataset. The project codes used in this work are publicly available.
Yinghao Qin, Tijana Timotijevic
School of Electronic Engineering and Computer Science Queen Mary, University of London London, UK