Why Personalization Matters: Cross-Subject Challenges in EMG-IMU-based HRI Activity Recognition
Authors: Ruan Rithelle Chagas de Faria Carminati, Giovanni Braglia, Luigi Biagiotti, Ronnier Frates Rohrich, Andre Schneider de Oliveira, Mikael Nedel Hartmann, André Eugenio Lazzaretti
Organizations: Universidade Tecnológica Federal do Paraná, Curitiba, Brazil · University of Modena and Reggio Emilia, Emilia-Romagna, Italy · Istituto Italiano di Tecnologia, Genoa, Italy
This paper investigates wearable-based recognition of human activities and gestures to support Human-Robot Interaction (HRI) in object-handover and assembly-like scenarios. Electromyography (EMG) and Inertial Measurement Unit (IMU) signals were collected using a Myo armband, culminating in a novel dataset introduced as MAGIC-HRI (Multimodal Activity, Gesture and Intention Collection) with a large taxonomy of 53 movement classes, including Brazilian Sign Language (LIBRAS) numbers (0-9), hand gestures, object/tool handover actions (pick up/give/hold), tool-manipulation tasks, and generic assembly/idle motions, collected from 11 participants with 10 samples per class (530 samples per participant). Signals are segmented by detecting muscle activation via an EMG energy envelope, then processed using sliding windows; time- and frequency-domain features are extracted. Multiple classical classifiers are tuned via cross-validated grid search, with Random Forest as the strongest baseline. A Leave-One-Subject-Out (LOSO) protocol reveals a large generalization gap, indicating substantial subject dependence. A personalized adaptation experiment suggests that injecting a small number of samples from a new user can markedly improve recognition. Overall, the study contributes a broad, HRI-driven multimodal dataset, a rigorous evaluation emphasizing generalization, and practical evidence that personalization is likely required for robust deployment in practical HRI.
Figures & tables
Fig. 1: General EMG/IMU-based industrial HRI in our context.
Ref.
Input data
LOSO
Dataset
Classes
Subjects
[ 4 ]
Vision, EMG/IMU
✕
✕
18
1
[ 19 ]
Vision, Sound, EMG/IMU
✕
✕
10
10
[ 20 ]
EMG/IMU
✕
✕
9
6
[ 18 ]
Vision, Force/Torque, EMG/IMU
✕
✕
8
6
[ 5 ]
Vision, Force/Torque, EMG/IMU
✕
✕
3
7
[ 17 ]
Vision
✕
✕
7
-
TABLE I: Comparison of our proposed method with related works in terms of input data and classification scope.
Fig. 2: Overview of the wearable-based activity and gesture recognition pipeline evaluated in this work.
Classifier
Accuracy
F1-Score
Random Forest
74.73%
74.95%
Support Vector Machine
70.61%
69.91%
1D Convolutional Neural Network
70.67%
62.39%
Multi-Layer Perceptron
63.83%
63.35%
Linear Discriminant Analysis
51.33%
50.57%
Decision Tree
49.07%
48.71%
TABLE II: Accuracy performance of various classifiers across MAGIC-HRI class categories.
Overlap
F1-Score
F1-Score LOSO
50% Overlap
81.17%
20.29%
75% Overlap
91.15%
20.19%
90% Overlap
99.20%
19.87%
TABLE III: Model performance metrics across different overlap configurations.
Fig. 3: Model performance as a function of feature reduction based on SHAP values.
Fig. 4: Impact of sample injection volume on LOSO F1-Score.
With each sensing modality exhibiting inherent strengths and limitations, multi-modal approaches for wearable Human Activity Recognition (HAR) are becoming increasingly relevant -- particularly for recognizing Activities of Daily Living (ADLs), where individual modalities often produce ambiguous signals for similar or complex activities. This work introduces HARMES, a multi-modal wearable dataset combining three wrist-recorded modalities: motion sensing via an Inertial Measurement Unit (IMU), atmospheric environmental sensors (humidity, temperature, and pressure), and audio. Collected from 20 participants performing household activities in their own homes, HARMES totals over 80 hours of recorded data, with approximately three hours of labeled activity data per participant across 15 ADL classes. To the best of our knowledge, HARMES is the first dataset to combine this particular sensor trio, and it is nearly six times larger than the previously largest wrist-inertial-acoustic HAR dataset. In an extensive benchmark, we evaluate cross-subject generalization and conduct an ablation study revealing that modality contributions are activity-dependent and can provide complementary value, particularly for activities that are ambiguous from motion data alone. HARMES is freely available at Zenodo, alongside example code for loading the dataset and training models on GitHub.
Robin Burchard, Pascal-André Brückner, Marius Bock +2
University of Siegen, Siegen, Germany · University of Bonn and & Lamarr Institute for Machine Learning and Artificial Intelligence, Bonn, Germany
Reliable estimation of neuromuscular activation is a key enabler for adaptive and personalized control in wearable robotics. However, surface electromyography (EMG) remains difficult to deploy robustly outside laboratory settings due to electrode sensitivity, signal non-stationarity, and strong subject dependence. In this work, we propose an adaptive IMU-to-EMG learning framework that reconstructs continuous muscle activation envelopes from wearable inertial measurements across heterogeneous movement conditions. The approach combines a Transformer encoder with Gaussian Error Gated Linear Units (GEGLU-Transformer) to enhance cross-subject generalization and enable rapid subject-specific personalization. Under a strict leave-one-subject-out (LOSO) protocol on a multi-condition lower-limb biomechanics dataset, the proposed architecture achieves r = 0.706 +/- 0.139 and R^2 = 0.474 +/- 0.208 without subject-specific adaptation. With only 0.5% adaptation data, performance increases to r = 0.761 +/- 0.030 and R^2 = 0.559 +/- 0.047, demonstrating rapid adaptation and early performance saturation. These results support attention-based architectures combined with lightweight adaptation as a practical and scalable alternative to direct EMG sensing for real-world wearable robotic applications.
AR smart glasses need continuous behavioral context to offer proactive assistance, yet their most practical always-on sensor, the head-mounted Inertial Measurement Unit (IMU), detects only motion primitives such as walking or standing. We push beyond motion primitives to behavioral-level recognition, defining five categories that balance AR application need with sensor observability. To this end, we construct a 160K-sample Ego4D dataset with a four-tier quality assurance framework spanning 8 activity scenarios, and propose HiT-HAR, a 703K-parameter hierarchical model that outperforms prior head-mounted IMU models on five-class action and eight-class scenario recognition. We further map the observability frontier of head-mounted IMU through per-class separability analysis, identifying which behavioral categories are reliably observable (Locomotion), which benefit from temporal context (Object Transfer, Task Operation), and where scenario-dependent signal overlap poses remaining challenges. Our results indicate that architectural choices exploiting temporal context and scenario structure outperform simply scaling model size. The code and dataset are publicly available at https://github.com/Harvard-AI-and-Robotics-Lab/HiT-HAR.