Organizations: Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea · Department of Computer Science, RPTU Kaisersalutern-Landau, Kaiserslautern, Germany · Embedded Intelligence, German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany
Human operators are still frequently exposed to hazardous environments such as disaster zones and industrial facilities, where intuitive and reliable teleoperation of mobile robots and Unmanned Aerial Vehicles (UAVs) is essential. In this context, hands-free teleoperation enhances operator mobility and situational awareness, thereby improving safety in hazardous environments. While vision-based gesture recognition has been explored as one method for hands-free teleoperation, its performance often deteriorates under occlusions, lighting variations, and cluttered backgrounds, limiting its applicability in real-world operations. To overcome these limitations, we propose a multimodal gesture recognition framework that integrates inertial data (accelerometer, gyroscope, and orientation) from Apple Watches on both wrists with capacitive sensing signals from custom gloves. We design a late fusion strategy based on the log-likelihood ratio (LLR), which not only enhances recognition performance but also provides interpretability by quantifying modality-specific contributions. To support this research, we introduce a new dataset of 20 distinct gestures inspired by aircraft marshalling signals, comprising synchronized RGB video, IMU, and capacitive sensor data. Experimental results demonstrate that our framework achieves performance comparable to a state-of-the-art vision-based baseline while significantly reducing computational cost, model size, and training time, making it well suited for real-time robot control. We therefore underscore the potential of sensor-based multimodal fusion as a robust and interpretable solution for gesture-driven mobile robot and drone teleoperation.
Figures & tables
Fig. 1 : An example of drone control using the proposed sensor-based gesture recognition.
Fig. 2 : Hardware setup with textile sensing gloves and Apple Watch devices on both hands is shown in Fig. 2(a) . Purple outlines mark capacitive sensors (bottom) and IMU sensors (top), while orange outlines mark Apple Watch devices on the wrist used for IMU sensing and quaternion orientation.
Fig. 3 : Framework architecture for sensor-based gesture recognition on our dataset. Each sensor modality is processed individually through convolutional and temporal feature extraction layers, and the resulting modality-specific features are fused using either log-likelihood ratio (LLR) fusion Eq. 2 or self-attention fusion for classification.
Gesture Label
ID
Brake
0
Brake Fire Left
1
Brake Fire Right
2
Come Close
3
Cut Engine Left
4
Cut Engine Right
5
TABLE I : Gesture Label Mapping
Fig. 4 : Example gesture classes are illustrated, including ’brake,’ ’brake fire right,’ and ’move away.’ Each gesture is represented by two sequential image frames, displaying a non-mirrored, first-person perspective.
Sensor Type
Feature Names
Capacitive Sensors (CAPA)
CAPA_0, CAPA_1, CAPA_2, CAPA_3
Accelerometer (ACC)
ACC_X, ACC_Y, ACC_Z
Gyroscope (GYRO)
GYRO_X, GYRO_Y, GYRO_Z
TABLE II : Glove Sensor Channels
Sensor Type
Feature Names
Accelerometer (ACC)
ACC_X, ACC_Y, ACC_Z
Gyroscope (GYRO)
GYRO_X, GYRO_ GYRO_Z
Quaternion Orientation (QUAT)
QUAT_W, QUAT_X, QUAT_Y, QUAT_Z
TABLE III : Apple Watch Sensor Channels
Model
F1 Score (%)
Precision (%)
Recall (%)
(LOSO Split)
LLR Fusion (sensor)
95.40 ± 5.91
95.99 ± 4.84
95.52 ± 5.85
Self-Attention Fusion (sensor)
94.42 ± 5.20
95.05 ± 4.44
94.45 ± 5.09
PoseConv3D (video)
94.70 ± 2.33
95.03 ± 1.93
94.73 ± 2.35
(LOPO Split)
LLR Fusion (sensor)
93.59 ± 5.20
94.45 ± 4.70
93.59 ± 5.31
TABLE IV : Model Performance Comparison
Fig. 5 : Computational resource comparison between sensor-based (orange) and vision-based (blue) models. From left to right, the plots compare GFLOPs per inference, model size, and training time. Training time is reported as the average across all dataset split variations.
Fig. 6 : Randomly sampled LLR contribution per modality, when correctly predicted Move Away gesture
Fig. 7 : Randomly sampled attention weights for a case when correctly predicted Move Away gesture
Sensor Modalities
F1 Score (%)
LOSO
LOPO
(All 4 sensors)
CAP + ACC + GYRO + QUAT
95.40 ± 5.91
93.59 ± 5.20
(3 sensors)
CAP + ACC + GYRO
95.80 ± 5.61
91.78 ± 4.11
CAP + ACC + QUAT
95.24 ± 5.40
92.11 ± 4.52
TABLE V : Performance Comparison in Modality Combinations in Terms of F1 Score (LOSO vs. LOPO)
Gesture recognition is a cornerstone of Human-Computer Interaction (HCI) for smart eyewear, enabling natural and device-free control in augmented reality environments. Traditional vision-based approaches face significant challenges regarding power consumption, computational latency, and user privacy. This paper proposes a lightweight, privacy-preserving gesture recognition system based on the fusion of low-resolution Time-of-Flight (ToF) and Infrared (IR) thermal sensors. We used an 8 times 8 multizone ToF sensor (VL53L8CH) and an 8 times 8 IR array (AMG8833) to capture complementary depth and thermal cues. A compact Convolutional Neural Network (CNN) with a specialized grouped-convolution architecture is designed to fuse these modalities efficiently on a microcontroller (MCU). Experimental results on a custom dataset of 7 static gestures, validated via k-fold cross-validation, demonstrate that the proposed fusion strategy significantly outperforms single-sensor baselines with an accuracy of 92.3% and a macro F1-score of 0.93. Finally, on-device benchmarks on STM32F4 and STM32H7 MCUs confirm the system's suitability for resource-constrained wearables, requiring only 6,343 parameters and achieving millisecond-level inference latency with a total system power of 50 mW.
Pietro Bartoli, Christian Veronesi, Tommaso Bondini +2
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB) Politecnico di Milano, Milan, Italy
Robust human-robot interaction in complex environments requires accurate gesture perception, semantic scene understanding, and reliable task planning under limited onboard computing resources. This paper presents a cloud-edge multimodal interaction framework that integrates an enhanced YOLO-based gesture detector with coordinated large language model (LLM) and vision-language model (VLM) agents. The proposed detector, incorporates the Convolutional Block Attention Module (CBAM) into the neck and replaces the baseline bounding-box regression objective with Distance-IoU (DIoU) loss. These modifications improve feature discrimination and localization for small or partially occluded gestures in complex backgrounds. The cloud layer performs gesture detection, scene understanding, multimodal fusion, and action planning, whereas the TonyPi robot locally handles data acquisition, communication, action execution, and feedback. Experiments on a public gesture dataset and a custom dataset show that YOLO-DC achieves precision values of 98.9% and 95.0%, with [email protected] values of 90.7% and 92.7%, respectively. System-level evaluation yields success rates of 95%, 88%, and 82% for single-action, composite-action, and vision-dependent tasks. A 30 participant evaluation yields an overall mean satisfaction score of 3.69 out of 5. These results demonstrate the feasibility of combining refined gesture detection with multimodal agents for resource-constrained robotic interaction.
While gesture recognition using vision or robot skins is an active research area in Human-Robot Collaboration (HRC), this paper explores deep learning methods relying solely on a robot's built-in joint sensors, eliminating the need for external sensors. We evaluated various convolutional neural network (CNN) architectures and collected a dataset to study the impact of data representation and model architecture on the recognition accuracy. Our results show that spectrogram-based representations significantly improve accuracy, while model architecture plays a smaller role. We also tested generalization to new robot poses, where spectrogram-based models performed better. Implemented on a Franka Emika Research robot, two of our methods, STFT2DCNN and STT3DCNN, achieved over 95% accuracy in contact detection and gesture classification. These findings demonstrate the feasibility of external-sensor-free tactile recognition and promote further research toward cost-effective, scalable solutions for HRC.
Deqing Song, Weimin Yang, Maryam Rezayati +1
Dep. of Informatics, University of Zurich Switzerland · Institute of Mechatronics systems, Zurich University of Applied Science, Switzerland