cs.CVNov 15, 2026

DoF-Query Transformer for Adaptive Fusion of Egocentric Vision and Forearm HD-sEMG in Continuous Hand Kinematics

Authors: Jiahang Liu, Wenbo Zhang, Junsong Wang, Zhenzhi Ying, Siqi Hou, Naohiko Sugita, Liming Shu

Organizations: School of Mechanical Engineering, Dalian University of Technology, Dalian, Liaoning, China. · Department of Mechanical Engineering, The University of Tokyo, Tokyo, Japan.

Abstract

Continuous and accurate estimation of human hand kinematics is important not only for intuitive human-machine interaction, but also for capturing fine-grained human manipulation demonstrations that can provide action supervision for embodied-intelligence datasets and dexterous robot learning. However, individual sensing modalities may be insufficient for reliable estimation under varying operating conditions. We present a finger-unencumbered multimodal framework for continuous regression of 20 hand joint angles from egocentric RGB images and 128-channel forearm high-density surface electromyography (HD-sEMG). The framework combines a Vision Transformer-based visual encoder, a Transformer-based EMG encoder, and a degree of freedom (DoF) query Transformer fusion decoder. To account for the varying informativeness of visual and EMG cues across joint DoFs and input conditions, the decoder assigns a separate learnable query to each DoF, enabling DoF-specific and context-adaptive fusion of the two modalities. The framework was evaluated on synchronized multimodal data from 10 participants. Averaged across participants, the proposed fusion model achieved a mean coefficient of determination (R2R^2) of 0.880.88 and a mean absolute error (MAE) of 2.80∘2.80^\circ, compared with 0.820.82 and 3.21∘3.21^\circ for the vision-only model and 0.680.68 and 5.13∘5.13^\circ for the EMG-only model. It also outperformed the evaluated representative unimodal baselines and alternative fusion designs. DoF-wise analyses showed larger fusion gains for visually ambiguous joints and viewing conditions, while synthetic visual occlusion demonstrated greater robustness than vision-only estimation. These results highlight the complementary value of egocentric vision and forearm HD-sEMG for capturing fine-grained human hand kinematics, with potential applications in human demonstration acquisition and dexterous robot learning.

Explore similar work

May 7, 2026

EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation

Hand pose estimation from egocentric video is challenging under adverse visual conditions, such as occlusion. Wrist-worn surface electromyography (sEMG) measures muscle activity without requiring visual access to the hand, offering a wearable complement to egocentric cameras. Evaluating their combined use requires synchronized physiological and visual recordings with continuous pose annotations, which existing EMG-based hand pose datasets do not jointly provide. We introduce EgoEMG, a multimodal dataset pairing bilateral wrist sEMG with egocentric video and motion-capture-derived hand poses, supplemented by external multi-view RGB-D recordings. EgoEMG contains 25.9 hours of recordings from 68 participants, covering 60 bare-hand gesture classes and interactions with 10 everyday objects. We establish EMG-only, vision-only, and fusion benchmarks with shared 22-dimensional pose targets and evaluations on unseen users, gestures, and objects. For EMG-only estimation, EMGFormer achieves the lowest overall MAE among the compared methods on EgoEMG and competitive results on EMG2Pose. Our residual fusion model combines ViT and EMGFormer features. Adding EMG reduces error for both single-frame and temporal vision models, with MAE decreasing from 7.39∘7.39^\circ to 7.06∘7.06^\circ for ViT-L. The largest absolute reduction occurs when evaluating interactions with an unseen object. Generalization to unseen users remains challenging. Project page: https://zhenqis123.github.io/egoemg-site/.
Oct 3, 2026

EVFormer: An Egocentric Vision-EMG Bidirectional Attention Model for Bimanual Hand Pose Estimation

Egocentric bimanual hand pose estimation is important for virtual interaction, wearable control, and rehabilitation, but visual observations are often degraded by self-occlusion, hand-hand contact, and object manipulation. We propose EVFormer, a multimodal framework that combines the current RGB frame with the preceding 200 ms of bilateral wrist surface electromyography (sEMG) to estimate 44 finger and wrist joint angles. EVFormer separately encodes visual spatial features and sEMG temporal features, enables cross-modal information exchange through sequential bidirectional cross-attention, and integrates the two modalities using feature-wise gated fusion. We evaluate EVFormer in a single-participant feasibility study using one synchronized public EgoEMG recording with chronologically separated training, validation, and test splits. On 296 test samples, EVFormer achieves a mean absolute error of 11.482 degrees, compared with 13.228-13.610 degrees for vision-only, sEMG-only, late-fusion, and training-mean baselines. This corresponds to relative error reductions of 13.20% compared with the vision-only model and 14.23% compared with late fusion. EVFormer also achieves the lowest error in four of the five evaluated gesture classes. These results provide preliminary evidence that feature-level interaction between egocentric vision and sEMG can improve bimanual hand pose estimation. Further evaluation across participants, recording sessions, sensor placements, and real-world interaction conditions is required to establish the generalizability of the approach.
Jul 24, 2026

Multimodal Surface EMG Hand Gesture Recognition Using Query-Based Transformers for Prosthetic Control

Hand gesture recognition via surface electromyography (sEMG) is fundamental to prosthetic control. In this field, deep learning approaches have become the gold standard. However, current architectures struggle to scale; model performance typically decreases as the number of hand movements increases. Performance degradation is tied to the increased statistical complexity of decoding expanded gesture sets and compounded by the limitations of state-of-the-art methods, which primarily rely on low-latency unimodal convolutional architectures. Convolutions operate locally, limiting model's ability to capture long-range sequential patterns. Unimodal setups cannot leverage complementary information from coordinated signals characterizing movement execution, such as inertial and eye-tracking data. These limitations motivate architectures that integrate local and global features across multimodal physiological sequences. To bridge this gap, this study introduces EMG-CrossFormer, an end-to-end hybrid convolutional-transformer for seamless multimodal integration. EMG-CrossFormer combines representations from an arbitrary number of unimodal encoders through cascaded cross-attention fusion layers, and decodes the fused representations using learnable gesture queries. EMG-CrossFormer was evaluated on four NinaPro datasets (DB2, DB3, DB7, and DB10) and benchmarked against six state-of-the-art models using an increasing number of modalities. Using only sEMG, EMG-CrossFormer achieved mean accuracies of 72.33%, 52.48%, 79.16%, and 73.49% on DB2, DB3, DB7, and DB10, respectively. Incorporating inertial signals improved performance to 90.66%, 80.40%, 92.79%, and 92.06%. These results show that joint local-global feature modeling improves sEMG-only decoding and that multimodal fusion substantially amplifies this benefit, underscoring the value of both design principles for complex hand gesture recognition.