cs.CVNov 15, 2026

DoF-Query Transformer for Adaptive Fusion of Egocentric Vision and Forearm HD-sEMG in Continuous Hand Kinematics

Authors: Jiahang Liu, Wenbo Zhang, Junsong Wang, Zhenzhi Ying, Siqi Hou, Naohiko Sugita, Liming Shu

Organizations: School of Mechanical Engineering, Dalian University of Technology, Dalian, Liaoning, China. · Department of Mechanical Engineering, The University of Tokyo, Tokyo, Japan.

Abstract

Continuous and accurate estimation of human hand kinematics is important not only for intuitive human-machine interaction, but also for capturing fine-grained human manipulation demonstrations that can provide action supervision for embodied-intelligence datasets and dexterous robot learning. However, individual sensing modalities may be insufficient for reliable estimation under varying operating conditions. We present a finger-unencumbered multimodal framework for continuous regression of 20 hand joint angles from egocentric RGB images and 128-channel forearm high-density surface electromyography (HD-sEMG). The framework combines a Vision Transformer-based visual encoder, a Transformer-based EMG encoder, and a degree of freedom (DoF) query Transformer fusion decoder. To account for the varying informativeness of visual and EMG cues across joint DoFs and input conditions, the decoder assigns a separate learnable query to each DoF, enabling DoF-specific and context-adaptive fusion of the two modalities. The framework was evaluated on synchronized multimodal data from 10 participants. Averaged across participants, the proposed fusion model achieved a mean coefficient of determination (R2R^2) of 0.880.88 and a mean absolute error (MAE) of 2.80∘2.80^\circ, compared with 0.820.82 and 3.21∘3.21^\circ for the vision-only model and 0.680.68 and 5.13∘5.13^\circ for the EMG-only model. It also outperformed the evaluated representative unimodal baselines and alternative fusion designs. DoF-wise analyses showed larger fusion gains for visually ambiguous joints and viewing conditions, while synthetic visual occlusion demonstrated greater robustness than vision-only estimation. These results highlight the complementary value of egocentric vision and forearm HD-sEMG for capturing fine-grained human hand kinematics, with potential applications in human demonstration acquisition and dexterous robot learning.

Explore similar work

CardsList
  1. Multimodal Surface EMG Hand Gesture Recognition Using Query-Based Transformers for Prosthetic Control

    Jul 24, 2026Federico Del Pup, Elisa Tentori, Manfredo AtzoriEMG-Based Gesture RecognitionHybrid CNN-Transformer Architectures