Organizations: Korea University Seoul, Republic of Korea · Massachusetts Institute of Technology Cambridge, Massachusetts, USA · Dept. of Artificial Intelligence Korea University Seoul, Republic of Korea
Human Activity Recognition (HAR) with self-administered wearables, such as at-home rehabilitation and exercise monitoring, often requires reattaching inertial measurement units (IMUs) across sessions. In multi-IMU settings, this can induce independent orientation offsets across body locations, a deployment shift that conventional scalar HAR models do not structurally handle. Existing remedies rely on rotation augmentation, whose robustness depends on sampled transformations, or calibration and orientationnormalization pipelines requiring additional reference-frame assumptions or explicit procedures. We present Truly Rotation-Invariant HAR (TRI-HAR), a rotation-invariant framework that makes robustness to independent per-location IMU orientation offsets a structural model property. TRI-HAR reshapes accelerometer and gyroscope streams into triaxial vectors, applies a shared SO(3)-equivariant backbone and invariant projection to each IMU location, and fuses the resulting invariant features for activity classification. Across four multi-IMU benchmarks, TRI-HAR preserves macro-F1 under fixed independent per-location SO(3) rotations and outperforms rotation-augmented baselines under this target shift without requiring rotational augmentation.
Figures & tables
Figure 1. Body-worn IMUs may be reattached with independent orientation offsets across uses. TRI-HAR performs per-location equivariant encoding and invariant projection before fusion for stable activity predictions. Teaser diagram for TRI-HAR. The left side shows a person wearing IMUs at the chest, wrist, thigh, and ankle during an initial placement. The middle shows the same activity during inference after sensors have been reattached, with each IMU coordinate frame rotated differently and labeled with independent rotations R1 through R4. The right side shows a simplified TRI-HAR block with equivariant encoding, invariant projection, and location-wise fusion, producing the same activity label before and after reattachment.
Figure 2. Architecture of TRI-HAR. Windowed input signals are vectorized into triaxial streams and grouped by physical IMU location. A shared location encoder Eθ is applied to each group X(l) to produce an invariant feature hl ; the features h1,…,hL are then concatenated in a fixed location order and passed to an MLP classifier. Dimensions inside Eθ are shown for one location group, and the encoder weights are shared across locations. Diagram of the TRI-HAR architecture. A sensor input window is first vectorized from flattened accelerometer and gyroscope channels into triaxial streams, then grouped by IMU location. Each location group is passed through the same shared encoder, labeled Eθ, which contains VN-EdgeConv layers, a VN invariant layer, and pooling, producing one invariant feature hl per location. The invariant features from all locations are concatenated and passed through an MLP to produce activity-class predictions.
Dataset
Freq.
WL
#Subj.
#Cls.
#Locs.
PAMAP2 ( Reiss and Stricker, 2012 )
33
5.12
8
12
3
DSADS ( Altun et al., 2010 )
25
5.00
8
19
5
Opportunity ( Chavarriaga et al., 2013 )
30
1.00
4
17
7
RealDISP ( Baños et al., 2012 )
50
2.40
17
33
9
Table 1. Characteristics of the datasets used in the evaluation
Model
Train Aug.
PAMAP2 ( Reiss and Stricker, 2012 )
DSADS ( Altun et al., 2010 )
Opportunity ( Chavarriaga et al., 2013 )
RealDISP ( Baños et al., 2012 )
I
SO(3)loc-fix
I
SO(3)loc-fix
I
SO(3)loc-fix
I
SO(3)loc-fix
MC-CNN ( Yang et al., 2015 )
none
80.45 ± 8.18
31.52 ± 6.21
82.80 ± 8.09
30.63 ± 5.80
41.18 ± 6.62
11.82 ± 3.90
87.48 ± 10.39
48.95 ± 9.81
loc-sample
—
79.14 ± 6.75
—
81.75 ± 6.43
—
33.56 ± 8.10
—
87.19 ± 8.35
DeepConvLSTM ( Ordóñez and Roggen, 2016 )
none
71.13 ± 10.39
21.84 ± 10.05
76.05 ± 11.27
25.85 ± 6.74
39.44 ± 7.99
6.16 ± 1.85
86.99 ± 10.21
53.37 ± 16.25
loc-sample
—
76.65 ± 8.80
—
77.97 ± 7.94
—
30.01 ± 9.06
—
87.65 ± 7.65
MLP-HAR ( Zhou et al., 2024 )
none
80.08 ± 6.73
28.52 ± 9.27
83.23 ± 7.80
22.55 ± 6.46
42.01 ± 3.93
5.45 ± 1.32
88.49 ± 6.82
39.70 ± 4.80
Table 2. Benchmark comparison under fixed independent per-location test rotations. Values are macro-F1 (%, mean ± standard deviation) over cross-subject folds. Columns report performance on the original test data I and under SO(3)loc-fix ; entries under I are omitted for loc-sample rows for compactness. Bold denotes the best value within each dataset/test column.
Model
none
loc-sample
Δaug
MC-CNN
67.95
74.84
+6.89
DeepConvLSTM
67.68
72.90
+5.22
MLP-HAR
77.55
74.67
−2.88
TinyHAR
72.55
78.49
+5.94
SA-HAR
49.89
51.28
+1.39
TRI-HAR
85.29
—
Table 3. Macro-F1 (%) on RealDISP Mutual-displacement fold. Δaug is augmentation-induced change in baseline score. Bold marks the highest score.
Setting
Model
FP32 (MiB)
CPU (ms)
GPU (ms)
3-IMU
MLP-HAR
0.67
0.86/1.08
0.69/1.09
MC-CNN
23.14
1.79/2.02
0.53/0.94
TRI-HAR
13.40
67.38/67.61
2.58/2.97
5-IMU
MLP-HAR
1.84
0.97/1.21
1.14/1.48
MC-CNN
28.46
2.61/2.84
0.88/1.25
TRI-HAR
21.41
85.56/85.78
3.01/3.36
Table 4. Host-side live p99 latency for the 3-IMU and 5-IMU settings. CPU and GPU entries report model/window-to-label in ms. FP32 denotes parameter memory.
Inertial Measurement Unit (IMU)-based Human Activity Recognition (HAR) aims to interpret and classify user behaviors from temporal motion signals. Recently, deep learning frameworks have advanced this task by learning and extracting discriminative spatiotemporal representations, significantly improving recognition performance. However, IMU-based HAR still faces several critical challenges, particularly limited training samples and static knowledge utilization, both of which severely hinder its large-scale deployment. In this paper, we introduce MoRA, the first Retrieval-Augmented Module specifically designed for motion series. It can be flexibly integrated into any existing HAR model, enhancing recognition performance while maintaining inference efficiency. To address issues such as information redundancy in retrieval results and rigid fusion strategies, we propose an uncertainty-adaptive fusion unit within MoRA. This unit leverages previous physical knowledge from IMU signals to dynamically adjust the fusion strategy between original outputs and retrieved information, enabling more robust recognition. Extensive experiments on ten real-world datasets demonstrate that MoRA significantly improves the performance of existing IMU-based HAR models, consistently delivering stable and effective gains. The source code of MoRA is available at: https://github.com/liavonpenn/mora.
Peng Liao, Shangsong Liang, Lin Chen +1
School of Computer Science and Engineering, Sun Yat-sen University · Engineering Research Centre of Applied Technology on Machine Translation and Artificial Intelligence, Macao Polytechnic University
Human Activity Recognition (HAR) using inertial measurement units (IMUs) enables a wide range of applications, yet the field still lacks a unified model that can generalize across diverse subjects, devices, and activities. Training such a model is difficult due to two key challenges: sensing heterogeneity -- differences in sampling rates, channel configurations, and sensor placements -- and poor generalization to unseen activities and label vocabularies. We introduce HALO (Heterogeneity-Aware Language-aligned Open-set model), a domain-specific IMU foundation model that addresses both challenges through a two-stage training framework. Stage 1 pretrains the IMU encoder with heterogeneity-aware self-supervised learning, including adaptive-pooling tokenization, channel-independent feature extraction, and contextualized sensor conditioning that injects natural-language sensor descriptions into each channel embedding. Stage 2 aligns this IMU encoder with text embeddings via synonym-aware soft contrastive learning, enabling open-set recognition via cosine-similarity retrieval without per-dataset classifiers. Trained on 10 public HAR datasets and evaluated on 7 held-out datasets, HALO outperforms five state-of-the-art baselines on all 8 aggregate metrics, and still leads on 3 of 4 settings under baseline-matched inputs. Despite using only ~35M trainable parameters -- 10x fewer than the latest foundation model MOMENT (341.2M) -- HALO improves zero-shot open-set accuracy, measured over all 87 training labels, by 13.7 percentage points. On two further datasets with severe distribution shift, every model including HALO collapses zero-shot. A video demonstration of HALO's performance in real world is available at https://youtu.be/rooVKragtFU
Zihan Ding, Liyu Zhang, Xiaomin Ouyang
Hong Kong University of Science and Technology Hong Kong, China
AR smart glasses need continuous behavioral context to offer proactive assistance, yet their most practical always-on sensor, the head-mounted Inertial Measurement Unit (IMU), detects only motion primitives such as walking or standing. We push beyond motion primitives to behavioral-level recognition, defining five categories that balance AR application need with sensor observability. To this end, we construct a 160K-sample Ego4D dataset with a four-tier quality assurance framework spanning 8 activity scenarios, and propose HiT-HAR, a 703K-parameter hierarchical model that outperforms prior head-mounted IMU models on five-class action and eight-class scenario recognition. We further map the observability frontier of head-mounted IMU through per-class separability analysis, identifying which behavioral categories are reliably observable (Locomotion), which benefit from temporal context (Object Transfer, Task Operation), and where scenario-dependent signal overlap poses remaining challenges. Our results indicate that architectural choices exploiting temporal context and scenario structure outperform simply scaling model size. The code and dataset are publicly available at https://github.com/Harvard-AI-and-Robotics-Lab/HiT-HAR.