An Effective, Reliable, and Robust Framework for Human Activity Recognition Using Wearable Sensors
Authors: Nafees Ahmad, Ho-fung Leung, Muhammad Adil Abid, Sadia Shakil
Organizations: The Chinese University of Hong Kong, China · Independent Researcher, China · Malmö University, Sweden · National University of Sciences and Technology (NUST), 44000, Islamabad, Pakistan
Human Activity Recognition (HAR) through wearable sensors greatly improves the quality of human life through its multiple applications. For HAR, multi-sensor channel information is vital for optimal performance. Current work states that applying an attention neural network to prioritize discriminatory sensor channels helps the model classify activity more precisely. However, obtaining discriminatory information from multisensory channels is not always trivial, such as when collecting data from older hospitalized patients. In this context, existing HAR methods struggle to classify activities, particularly activities with similar natures. Moreover, HAR models predominantly suffer from overfitting due to the small size of available datasets, which leads to poor performance. Data augmentation (DA) is a viable solution to this problem. However, available DA methods have various drawbacks, including the possibility of being domain-dependent, resulting in distorted models for test sequences. To address these HAR problems, we propose a novel framework, ALAE-TAE-CutMix+, which focuses on two aspects. First, it enhances the latent information across each sensor channel and learns to exploit the relation among multiple latent features and the ongoing activity. Consequently, the discriminatory feature representations of each activity is enriched. Second, a new augmentation strategy is introduced to address the shortcomings of existing multi-sensor channel data augmentation. We then extend the framework to create a further enhanced version, namely ALAE-CIE-TAE-CutMix+, which learns to capture the interactions between the features of each pair of sensor channels. We find that although the first framework performs slightly better than the latter, the latter is nonetheless more reliable and robust. Both frameworks significantly outperform SOTA approaches on the four HAR datasets from diverse domains.
Figures & tables
Figure 1. Overview of the proposed ALAE-TAE-CutMix + framework for human activity recognition.
Figure 2. Overview of the attention module.
Figure 3. Illustration of our temporal CutMix component generated virtual example: (a) a segment of walking activity ( xa ) from the training partition with its one-hot encoded label ( ya ), (b) a training segment of lying activity corresponding to its one-hot encoded label ( yb ), (c) a newly generated virtual sample with the target label and lambda’s ( λ∼BetaDistribution(α,α) ) value indicates the mixing ratio of input segments. The sub-segment SA from sensors channels 4−8 with their time-stamps 14−24 is cropped from xa and pasted on the corresponding location in xb . Each colour in the sequences Xa , Xb , and Xv , represents a sensor channel, while the range of numbers 0 to 24 on the horizontal axis indicates their time-stamps. (Best viewed in color.)
Figure 4. Overview of Cross-Channel Interaction Encoder and showing how it is integrated into ALAE-TAE-CutMix + framework.
Figure 5. Pie charts illustrating the four benchmark datasets investigated in our work. The activities listed within these benchmark datasets and their percentage distribution are presented. (Best viewed in color.)
Dataset
# Subjects
# Sensor Channels
# Activities
# Training Segments
Hospital
12
6
7
4607
GOTOV
29
9
16
291723
Skoda
1
60
10
15548
Opportunity
4
79
18
54246
Table 1. Statistical overview of the datasets.
HAR Study
Hospital
GOTOV
Skoda
Opportunity
Baselines
LSTM Learner Baseline ( Guan and Plötz, 2017 )
62.7
61.1
90.4
65.9
DeepConvLSTM ( Ordóñez and Roggen, 2016 )
62.8
66.9
91.2
67.2
b-LSTM-S ( Hammerla et al., 2016 )
63.6
63.9
92.1
68.4
Att. Model ( Murahari and Plötz, 2018 )
64.1
70.7
91.3
70.7
Attend and Discriminate ( Abedin et al., 2021 )
66.6
76.2
92.8
74.6
ALAE-TAE-CutMix +
70.5
79.4
94.8
75.1
Table 2. An F-score based performance comparison between the proposed frameworks and the baseline studies.
Figure 6. Confusion matrices to show class-wise recognition performance of ALAE-TAE-CutMix + framework achieved on the hold-out test splits across the four benchmark datasets.
Figure 7. Confusion matrices to show class-wise recognition performance of ALAE-CIE-TAE-CutMix + framework achieved on the hold-out test splits across the four benchmark datasets.
Figure 8. Activity-wise recognition performance comparison among both proposed models and the runner-up Attend and Discriminate model ( Abedin et al., 2021 ) on (a) Hospital and (b) GOTOV dataset.
HAR Study
Hospital
GOTOV
Skoda
Opportunity
Attend and Discriminate ( Abedin et al., 2021 )
64.8 ± 0.8
75.3 ± 0.9
92.2 ± 0.2
70.6 ± 2.8
ALAE-TAE-CutMix +
67.7 ± 1.1
77.8 ± 0.7
94.5 ± 0.1
71.3 ± 0.4
ALAE-CIE-TAE-CutMix +
68.7 ± 0.7
78.4 ± 0.2
94.6 ± 0.1
71.7 ± 1.0
Table 3. Performance comparison between the proposed ALAE-CIE-TAE-CutMix + , ALAE-TAE-CutMix + and the current best performing model based on average results and the standard deviations of all experiments.
HAR Model
F-score
LSTM Learner Baseline
62.7
Ours (CutMix)
66.2
Ours (ALAE)
66.2
Ours (CutMix + )
66.7
Ours (CutMix + TAE)
66.7
Ours (ALAE + TAE)
66.8
Table 4. The results of ablation study on the Hospital dataset.
HAR Model
F-score
LSTM Learner Baseline
61.1
Ours (CutMix)
64.8
Ours (ALAE)
73.7
Ours (CutMix + )
63.3
Ours (CutMix + TAE)
64.9
Ours (ALAE + TAE)
74.4
Table 5. The results of ablation study on the GOTOV dataset.
Data Augmentation
Hospital
GOTOV
ALAE-TAE
ALAE-CIE-TAE
ALAE-TAE
ALAE-CIE-TAE
No Augmentation
68.2
66.7
72.4
70.6
Scaling
67.1
66.8
72.7
75.5
Jittering
67
67.2
71.3
75.7
Magnitude Warping
68.5
67.9
71.0
74.7
CutMix with No Label Mixing
68.2
68.3
76.1
78.3
Table 6. The F-score based comparison between proposed CutMix and its variants with different data augmentation methods on Hospital and GOTOV dataset.
Figure 9. The performance analysis of the proposed simpler framework on limited training data across different datasets.
Figure 10. The performance analysis of the proposed CIE-enhanced framework on limited training data across different datasets.
Figure 11. Impact of different segment sizes on the performance of the proposed simpler framework.
Figure 12. Impact of different segment sizes on the performance of the proposed CIE-enhanced framework.
School of Computer Sciences, Universiti Sains Malaysia, 11800 Pulau Pinang, Malaysia · Artificial Intelligence Research Center, Ajman University, Ajman, United Arab Emirates · Al-Quds Open University, Ramallah, Palestine