An Effective, Reliable, and Robust Framework for Human Activity Recognition Using Wearable Sensors
Authors: Nafees Ahmad, Ho-fung Leung, Muhammad Adil Abid, Sadia Shakil
Organizations: The Chinese University of Hong Kong, China · Independent Researcher, China · Malmö University, Sweden · National University of Sciences and Technology (NUST), 44000, Islamabad, Pakistan
Human Activity Recognition (HAR) through wearable sensors greatly improves the quality of human life through its multiple applications. For HAR, multi-sensor channel information is vital for optimal performance. Current work states that applying an attention neural network to prioritize discriminatory sensor channels helps the model classify activity more precisely. However, obtaining discriminatory information from multisensory channels is not always trivial, such as when collecting data from older hospitalized patients. In this context, existing HAR methods struggle to classify activities, particularly activities with similar natures. Moreover, HAR models predominantly suffer from overfitting due to the small size of available datasets, which leads to poor performance. Data augmentation (DA) is a viable solution to this problem. However, available DA methods have various drawbacks, including the possibility of being domain-dependent, resulting in distorted models for test sequences. To address these HAR problems, we propose a novel framework, ALAE-TAE-CutMix+, which focuses on two aspects. First, it enhances the latent information across each sensor channel and learns to exploit the relation among multiple latent features and the ongoing activity. Consequently, the discriminatory feature representations of each activity is enriched. Second, a new augmentation strategy is introduced to address the shortcomings of existing multi-sensor channel data augmentation. We then extend the framework to create a further enhanced version, namely ALAE-CIE-TAE-CutMix+, which learns to capture the interactions between the features of each pair of sensor channels. We find that although the first framework performs slightly better than the latter, the latter is nonetheless more reliable and robust. Both frameworks significantly outperform SOTA approaches on the four HAR datasets from diverse domains.
Figures & tables
Figure 1. Overview of the proposed ALAE-TAE-CutMix + framework for human activity recognition.
Figure 2. Overview of the attention module.
Figure 3. Illustration of our temporal CutMix component generated virtual example: (a) a segment of walking activity ( xa ) from the training partition with its one-hot encoded label ( ya ), (b) a training segment of lying activity corresponding to its one-hot encoded label ( yb ), (c) a newly generated virtual sample with the target label and lambda’s ( λ∼BetaDistribution(α,α) ) value indicates the mixing ratio of input segments. The sub-segment SA from sensors channels 4−8 with their time-stamps 14−24 is cropped from xa and pasted on the corresponding location in xb . Each colour in the sequences Xa , Xb , and Xv , represents a sensor channel, while the range of numbers 0 to 24 on the horizontal axis indicates their time-stamps. (Best viewed in color.)
Figure 4. Overview of Cross-Channel Interaction Encoder and showing how it is integrated into ALAE-TAE-CutMix + framework.
Figure 5. Pie charts illustrating the four benchmark datasets investigated in our work. The activities listed within these benchmark datasets and their percentage distribution are presented. (Best viewed in color.)
Dataset
# Subjects
# Sensor Channels
# Activities
# Training Segments
Hospital
12
6
7
4607
GOTOV
29
9
16
291723
Skoda
1
60
10
15548
Opportunity
4
79
18
54246
Table 1. Statistical overview of the datasets.
HAR Study
Hospital
GOTOV
Skoda
Opportunity
Baselines
LSTM Learner Baseline ( Guan and Plötz, 2017 )
62.7
61.1
90.4
65.9
DeepConvLSTM ( Ordóñez and Roggen, 2016 )
62.8
66.9
91.2
67.2
b-LSTM-S ( Hammerla et al., 2016 )
63.6
63.9
92.1
68.4
Att. Model ( Murahari and Plötz, 2018 )
64.1
70.7
91.3
70.7
Attend and Discriminate ( Abedin et al., 2021 )
66.6
76.2
92.8
74.6
ALAE-TAE-CutMix +
70.5
79.4
94.8
75.1
Table 2. An F-score based performance comparison between the proposed frameworks and the baseline studies.
Figure 6. Confusion matrices to show class-wise recognition performance of ALAE-TAE-CutMix + framework achieved on the hold-out test splits across the four benchmark datasets.
Figure 7. Confusion matrices to show class-wise recognition performance of ALAE-CIE-TAE-CutMix + framework achieved on the hold-out test splits across the four benchmark datasets.
Figure 8. Activity-wise recognition performance comparison among both proposed models and the runner-up Attend and Discriminate model ( Abedin et al., 2021 ) on (a) Hospital and (b) GOTOV dataset.
HAR Study
Hospital
GOTOV
Skoda
Opportunity
Attend and Discriminate ( Abedin et al., 2021 )
64.8 ± 0.8
75.3 ± 0.9
92.2 ± 0.2
70.6 ± 2.8
ALAE-TAE-CutMix +
67.7 ± 1.1
77.8 ± 0.7
94.5 ± 0.1
71.3 ± 0.4
ALAE-CIE-TAE-CutMix +
68.7 ± 0.7
78.4 ± 0.2
94.6 ± 0.1
71.7 ± 1.0
Table 3. Performance comparison between the proposed ALAE-CIE-TAE-CutMix + , ALAE-TAE-CutMix + and the current best performing model based on average results and the standard deviations of all experiments.
HAR Model
F-score
LSTM Learner Baseline
62.7
Ours (CutMix)
66.2
Ours (ALAE)
66.2
Ours (CutMix + )
66.7
Ours (CutMix + TAE)
66.7
Ours (ALAE + TAE)
66.8
Table 4. The results of ablation study on the Hospital dataset.
HAR Model
F-score
LSTM Learner Baseline
61.1
Ours (CutMix)
64.8
Ours (ALAE)
73.7
Ours (CutMix + )
63.3
Ours (CutMix + TAE)
64.9
Ours (ALAE + TAE)
74.4
Table 5. The results of ablation study on the GOTOV dataset.
Data Augmentation
Hospital
GOTOV
ALAE-TAE
ALAE-CIE-TAE
ALAE-TAE
ALAE-CIE-TAE
No Augmentation
68.2
66.7
72.4
70.6
Scaling
67.1
66.8
72.7
75.5
Jittering
67
67.2
71.3
75.7
Magnitude Warping
68.5
67.9
71.0
74.7
CutMix with No Label Mixing
68.2
68.3
76.1
78.3
Table 6. The F-score based comparison between proposed CutMix and its variants with different data augmentation methods on Hospital and GOTOV dataset.
Figure 9. The performance analysis of the proposed simpler framework on limited training data across different datasets.
Figure 10. The performance analysis of the proposed CIE-enhanced framework on limited training data across different datasets.
Figure 11. Impact of different segment sizes on the performance of the proposed simpler framework.
Figure 12. Impact of different segment sizes on the performance of the proposed CIE-enhanced framework.
Human Activity Recognition (HAR) has shown remarkable effectiveness in various applications, such as smart healthcare and intelligent manufacturing. However, a major challenge faced by HAR is the distribution shift across different sensor data domains, which often leads to decreased performance when deployed for real-world applications. To address this issue, this paper introduces GenHAR, a novel framework designed to mitigate the domain gap by learning domain-invariant sensor representations. GenHAR aims to enhance the generalization capabilities of HAR on target domains purely with data from the source domain. The key novelty of GenHAR lies in two aspects. Firstly, GenHAR tokenizes sensor data and learns correlations among frequency sensor channel dimensions to improve the robustness of HAR models. Secondly, GenHAR improves the efficiency via selective masking and an efficient attention mechanism. We conduct a systematic analysis of GenHAR by comparing it with state-of-the-art HAR methods on real-world human activity datasets. Results show that GenHAR outperforms state-of-the-art methods by 9.97% in accuracy, and reduces Floating Point Operations by 6.4 times. Moreover, we deploy GenHAR at a leading logistics company in 4 cities, and have detected 2.15 billion real-time activities. We release our code at: https://github.com/Sensor-FoundationModel/GenHAR.
Zhiqing Hong, Zelong Li, Xiubin Fan +5
HKUST (GZ) Guangzhou, China · JD Logistics Beijing, China · Beijing Normal–Hong Kong Baptist University Zhuhai, China +2
Sensor-based human activity recognition (HAR) has achieved significant progressed in fully supervised learning settings. However, these supervised learning models rely on large amount of labeled data, which require labor-intensive collection and meticulous annotation. To address these challenges, this paper proposes a Joint Embedding Predictive Architecture framework tailored for sensor-based HAR, designed to learn robust and generalizable representations from unlabeled datasets. The proposed framework features an encoder designed to explicitly model both the fine-grained local temporal representations within individual window and the long-term temporal sequence of adjacent windows. Furthermore, we introduce an improved Variance-Invariance-Covariance Regularization (VICReg) objective function that incorporates computationally lightweight norm term to stabilize the JEPA pre-training phase. This term balances variance, invariance and covariance constraints to prevent representation collapse. The proposed HAR-JEPA framework is evaluated using two benchmark continuously performed activity datasets. The results show that high-quality representations are successfully learned by the proposed framework. Furthermore, the representations learned by HAR-JEPA demonstrates superior generalization on minority, high variance transitional activities such as sit-to-stand and sit-to-lie where supervised learning tend to overfit due to limited support.
Mohd Halim Mohd Noor, Abdulrahman M. A. Baraka
School of Computer Sciences, Universiti Sains Malaysia, 11800 Pulau Pinang, Malaysia · Artificial Intelligence Research Center, Ajman University, Ajman, United Arab Emirates · Al-Quds Open University, Ramallah, Palestine
Sensor-based Human Activity Recognition (HAR) has attracted increasing attention in medical and healthcare monitoring, particularly with the growth of Internet of Medical Things (IoMT). However, in real-world wearable sensing scenarios, IoMT signals are often corrupted by missing measurements, sensor failures, and environmental noise, which significantly degrade the performance of conventional deep learning models that assume clean and complete inputs. To address this challenge, we propose a Manifold-Consistent Spatio-Temporal Network (MCSTN) for robust HAR under imperfect sensing conditions. The proposed framework introduces a dual-level corruption modeling mechanism that simulates realistic sensor imperfections through both physical-level corruption and diffusion-driven continuous corruption. By enforcing representation consistency across multiple corrupted views, the model learns stable and corruption-invariant semantic representations. Furthermore, we design a dual-stream spatio-temporal architecture that explicitly decouples temporal dynamics modeling and spatial correlation learning. The temporal stream captures long-term activity dynamics, while the spatial stream models inter-sensor relationships, enabling more effective spatio-temporal representation learning. Extensive experiments on three widely used HAR benchmark datasets, PAMAP2, Opportunity, and WISDM, demonstrate that the proposed MCSTN achieves competitive performance compared with existing state-of-the-art methods, particularly under imperfect sensing conditions. These results validate the effectiveness and robustness of the proposed framework for real-world wearable IoMT sensing applications.
Jiangtao Fan, Anish Jindal, Amir Atapour-Abarghouei
Department of Computer Science, Durham University, Durham, DH1 3LE, UK