Human Activity Recognition (HAR) using inertial measurement units (IMUs) enables a wide range of applications, yet the field still lacks a unified model that can generalize across diverse subjects, devices, and activities. Training such a model is difficult due to two key challenges: sensing heterogeneity -- differences in sampling rates, channel configurations, and sensor placements -- and poor generalization to unseen activities and label vocabularies. We introduce HALO (Heterogeneity-Aware Language-aligned Open-set model), a domain-specific IMU foundation model that addresses both challenges through a two-stage training framework. Stage 1 pretrains the IMU encoder with heterogeneity-aware self-supervised learning, including adaptive-pooling tokenization, channel-independent feature extraction, and contextualized sensor conditioning that injects natural-language sensor descriptions into each channel embedding. Stage 2 aligns this IMU encoder with text embeddings via synonym-aware soft contrastive learning, enabling open-set recognition via cosine-similarity retrieval without per-dataset classifiers. Trained on 10 public HAR datasets and evaluated on 7 held-out datasets, HALO outperforms five state-of-the-art baselines on all 8 aggregate metrics, and still leads on 3 of 4 settings under baseline-matched inputs. Despite using only ~35M trainable parameters -- 10x fewer than the latest foundation model MOMENT (341.2M) -- HALO improves zero-shot open-set accuracy, measured over all 87 training labels, by 13.7 percentage points. On two further datasets with severe distribution shift, every model including HALO collapses zero-shot. A video demonstration of HALO's performance in real world is available at https://youtu.be/rooVKragtFU
Figures & tables
Figure 1. HALO is a domain-specific IMU foundation model that can tackle various sensing heterogeneity for open-set human activity recognition.
Figure 2. Dataset heterogeneity across 14 HAR datasets in our benchmark. Training datasets (blue circles) and test datasets (red triangles) span 20–100 Hz in sampling rates, 3–45 channels, and 4–19 activity labels.
Figure 3. System overview of HALO. The sensing heterogeneity is addressed through heterogeneity-aware pretraining of the IMU encoder (Stage 1), followed by synonym-aware soft contrastive learning to align sensor features with text semantics (Stage 2), enabling generalizable and open-set activity recognition. Zero-shot inference is achieved by swapping label banks at runtime without retraining.
Figure 4. Heterogeneity-aware Pretraining. Each row shows a challenge (left) and the corresponding design in HALO (right): adaptive-pooling tokenization for variable sampling rates; channel-independent feature extraction for variable channels; and contextualized sensor conditioning for variable sensor placements.
Figure 5. IMU-to-text alignment and zero-shot inference. Top: During training, the IMU encoder is aligned with a frozen text encoder in a shared embedding space via synonym-aware contrastive learning, so that semantically related activities are clustered together. Bottom: At inference, the IMU segment is encoded and matched against an open label library via cosine similarity-enabling zero-shot recognition without retraining or dataset-specific classifiers.
Dataset
Hz
#Subj
#Act
#Ch
Description
Cov.
Diff.
UCI-HAR ( Anguita et al., 2013 )
50
30
6
9
Smartphone IMU
HHAR ( Stisen et al., 2015 )
∼ 50
9
6
6
Phone + watch
UniMiB ( Micucci et al., 2017 )
50
30
17
3
Phone acc, ADLs
MHEALTH ( Banos et al., 2014 )
50
10
12
6+
Body IMU suite
PAMAP2 ( Reiss and Stricker, 2012 )
100
9
18
27
Multi-IMU
WISDM ( Kwapisz et al., 2011 )
20
51
18
6
Phone + watch
Table 1. Dataset overview. Top: 10 training datasets. Bottom: 7 held-out test datasets. † Main test datasets. ‡ Severe OOD datasets; reported separately. Cov. : Fraction of test activities that have an equivalent in the training set. Diff. : Difficulty level determined by distribution shift and label novelty.
Model
Paradigm
Dim
Train.
Infer.
LiMU-BERT ( Xu et al., 2021 )
Masked recon.
72
72.7K
72.7K
MOMENT ( Goswami et al., 2024 )
General TS
6144
341.2M
341.2M
CrossHAR ( Chang et al., 2024 )
Self-sup HAR
72
531.4K
531.4K
LanHAR ( Li et al., 2024 )
Text-aligned
768
13.0M ∗
122.9M
LLaSA ( Imran et al., 2025 )
LLM generative
—
∼ 6.7B
∼ 6.7B
HALO (ours)
Text-aligned
384
∼ 35M
∼ 58M
Table 2. Baseline overview. Train. : parameters updated during pretraining and alignment. Infer. : Total parameters used at inference, including frozen modules. HALO’s inference total includes a single frozen MiniLM (22.7M) shared for both contrastive alignment and channel conditioning; all text embeddings are precomputable. ∗ LanHAR freezes SciBERT (109.9M) during stage-2 and unfreezes it for fine-tuning.
ZS-O
ZS-C
1%
10%
Model
Acc
F1
Acc
F1
Acc
F1
Acc
F1
HALO (ours)
42.0
21.0
53.1
37.2
76.5
70.0
85.7
81.7
MOMENT
28.3
7.2
44.7
33.2
73.8
69.8
81.3
76.8
CrossHAR
21.2
5.5
38.2
31.7
61.1
55.2
78.6
75.2
LiMU-BERT
21.7
7.5
33.1
23.1
35.7
22.7
51.3
43.6
LanHAR
15.8
6.2
28.4
21.7
43.3
35.2
56.2
51.5
Table 3. Overall performance in average accuracy (%) and macro F1 (%) across 5 main test datasets. Bold : best result for each metric. ZS-O/ZS-C: zero-shot open-set/closed-set settings. HALO and LanHAR are evaluated at each dataset’s native sampling rate; other baselines use data resampled to 20 Hz. † LLaSA: published 7B model, zero-shot only (no supervised)
Accuracy (%)
Macro F1 (%)
Dataset
ZS-O
ZS-C
1%
10%
ZS-O
ZS-C
1%
10%
MotionSense
49.3
64.1
88.6
95.0
30.3
49.3
87.1
94.6
RealWorld
48.0
48.0
75.1
85.6
33.3
37.9
74.0
86.4
MobiAct
42.2
50.0
65.3
72.9
16.3
12.8
35.2
51.7
Shoaib
49.2
54.1
81.6
95.9
17.6
51.5
81.2
95.7
Opportunity
21.1
49.3
72.0
79.3
7.4
34.2
72.6
80.0
Table 4. Per-dataset results of HALO on the five main test datasets. ZS-O/ZS-C: zero-shot open/closed-set. All datasets are held out from training.
HARTH
VTT-ConIoT
ZS-O
ZS-C
1%
10%
ZS-O
ZS-C
1%
10%
HALO
2.0
2.5
64.7
78.3
1.3
2.2
1.9
16.9
MOMENT
0.4
1.9
66.7
65.4
1.6
5.2
18.4
34.8
CrossHAR
0.3
0.9
42.4
71.9
0.7
5.0
10.6
30.9
LiMU-BERT
0.0
20.2
54.4
19.1
3.4
7.1
4.3
15.0
LanHAR
0.9
1.1
3.4
70.0
8.3
6.9
7.7
7.7
Table 5. Accuracy on severe out-of-domain datasets (%). ZS-O/ZS-C: zero-shot open/closed-set. All models collapse on HARTH zero-shot (sensor distribution shift); VTT-ConIoT is capped at ∼ 50% by novel activities. † LLaSA: zero-shot only, subsampled.
Configuration
ZS-O
ZS-C
1%
10%
Full model
44.9
51.9
74.7
85.4
− Channel-text fusion
25.3 (–19.6)
43.8 (–8.1)
72.9 (–1.8)
82.5 (–2.9)
− Signal augmentation
45.3 (+0.4)
53.0 (+1.1)
70.9 (–3.8)
83.5 (–1.9)
− Text augmentation
42.7 (–2.2)
49.9 (–2.0)
67.7 (–7.0)
84.2 (–1.2)
Table 6. Ablation study of design components. Each row removes one component while keeping all others fixed. Results report average accuracy (%) across five main test datasets, with performance deltas relative to the full model shown in parentheses.
Configuration
ZS-O
ZS-C
1%
10%
HALO (native Hz, rich desc.)
42.0
53.1
76.5
85.7
HALO (20 Hz, generic desc.)
30.6
49.4
71.7
84.7
Best baseline (MOMENT)
28.3
44.7
73.8
81.3
Table 7. Analysis of native sampling rate and evaluation fairness with average accuracy (%) across five main test datasets. In Row 2, HALO is evaluated using the same 20Hz, 6 - channel inputs with generic textual descriptions as four of the five baselines (LanHAR uses 50Hz data). Even under these conservative and standardized inputs, HALO outperforms baselines in three of the four evaluation settings.
Figure 6. Analysis of embedding space. UMAP projection of learned IMU and text embeddings at epoch 100. Circles: per-activity IMU centroids; diamonds: SBERT text centroids; lines connect matched IMU–text pairs.
Figure 7. Patch size sensitivity. Zero-shot closed-set accuracy (%) across candidate patch durations. The dashed vertical line indicates the selected patch size of 1.0 s. Original datasets are evaluated at 20 Hz, while Shoaib and Opportunity are evaluated at their native sampling rates.
Model
Params
ZS-O
ZS-C
1%
10%
Tiny
6M
32.8
52.1
72.6
86.4
Small
35M
46.0
53.4
77.5
85.3
Medium
63M
37.8
47.7
72.6
84.5
Table 8. Scaling of model size. Average accuracy (%) on 3 main test datasets at 20 Hz. The Small model (35M) achieves the best performance, while the Medium model regresses on all zero-shot metrics despite having twice as many parameters.
Model
Params
Avg (ms)
p95 (ms)
Peak Mem. (MB)
CPU Wt/ms
CrossHAR
110K
0.29
0.30
1.20
224.0K
LiMU-BERT
110K
0.31
0.36
0.27
188.0K
LLaSA
105K
0.33
0.46
0.02
187.2K
LanHAR
8.8M
0.77
0.89
0.05
71.8K
MOMENT
24.0M
37.69
37.98
25.05
1.8K
HALO (ours)
34.0M
7.16
7.37
32.81
2414.4K
Table 9. Inference overhead on an iPhone 16 Pro. All models are benchmarked with same setting: identical 6-channel IMU inputs using Core ML, with 10 warm-up iterations followed by 100 timed iterations. Params denotes the profiled model size. CPU Wt/ms reports cycle-weighted CPU activity normalized by benchmark duration.
Figure 8. Deployment setting of smartphone-based IMU HAR in the real-world case study.
Figure 9. Performance continuous IMU-based HAR on the smartphone.
Zero-shot learning (ZSL) for inertial measurement unit (IMU)-based human activity recognition (HAR) faces a central challenge: bridging the gap between sensor embeddings and semantic class representations. We systematically evaluate seven configurations combining three inference methods with two training pipelines on the PAMAP2 dataset, using 14 seen and 4 unseen activity classes with subjects 108 and 109 held out for testing. We find that the modality gap is a training-time phenomenon governed by the encoder objective. A temporal convolutional network (TCN) trained with cross-entropy over label-name Sentence- BERT prototypes yields sensor embeddings with a mean cosine similarity of 0.30 to the corresponding text prototypes, while replacing the label-name prototype targets with discriminative activity descriptions raises this to 0.69. This alignment improvement transfers consistently across all three inference methods. The strongest result combines contrastive training with inverted softmax correction, achieving 73.2% accuracy and 0.583 macro F1 on unseen classes, compared to 58.3% accuracy and 0.34 macro F1 for the label-name baseline. A secondary finding is that richer text descriptions reduce inter-prototype separability in Sentence-BERT space, because shared biomechanical vocabulary causes the language model to compress the prototype cloud. This effect does not negate the benefits of contrastive alignment provided prototype descriptions retain sufficient discriminative vocabulary. We also demonstrate that overall accuracy is a misleading primary metric when test-set class distributions are imbalanced, and recommend macro-averaged F1 as the standard reporting metric for ZSL-HAR benchmarks.
Inertial Measurement Unit (IMU)-based Human Activity Recognition (HAR) aims to interpret and classify user behaviors from temporal motion signals. Recently, deep learning frameworks have advanced this task by learning and extracting discriminative spatiotemporal representations, significantly improving recognition performance. However, IMU-based HAR still faces several critical challenges, particularly limited training samples and static knowledge utilization, both of which severely hinder its large-scale deployment. In this paper, we introduce MoRA, the first Retrieval-Augmented Module specifically designed for motion series. It can be flexibly integrated into any existing HAR model, enhancing recognition performance while maintaining inference efficiency. To address issues such as information redundancy in retrieval results and rigid fusion strategies, we propose an uncertainty-adaptive fusion unit within MoRA. This unit leverages previous physical knowledge from IMU signals to dynamically adjust the fusion strategy between original outputs and retrieved information, enabling more robust recognition. Extensive experiments on ten real-world datasets demonstrate that MoRA significantly improves the performance of existing IMU-based HAR models, consistently delivering stable and effective gains. The source code of MoRA is available at: https://github.com/liavonpenn/mora.
Peng Liao, Shangsong Liang, Lin Chen +1
School of Computer Science and Engineering, Sun Yat-sen University · Engineering Research Centre of Applied Technology on Machine Translation and Artificial Intelligence, Macao Polytechnic University
Human Activity Recognition (HAR) with self-administered wearables, such as at-home rehabilitation and exercise monitoring, often requires reattaching inertial measurement units (IMUs) across sessions. In multi-IMU settings, this can induce independent orientation offsets across body locations, a deployment shift that conventional scalar HAR models do not structurally handle. Existing remedies rely on rotation augmentation, whose robustness depends on sampled transformations, or calibration and orientationnormalization pipelines requiring additional reference-frame assumptions or explicit procedures. We present Truly Rotation-Invariant HAR (TRI-HAR), a rotation-invariant framework that makes robustness to independent per-location IMU orientation offsets a structural model property. TRI-HAR reshapes accelerometer and gyroscope streams into triaxial vectors, applies a shared SO(3)-equivariant backbone and invariant projection to each IMU location, and fuses the resulting invariant features for activity classification. Across four multi-IMU benchmarks, TRI-HAR preserves macro-F1 under fixed independent per-location SO(3) rotations and outperforms rotation-augmented baselines under this target shift without requiring rotational augmentation.
Seungyeol Baek, Yoonbyung Chai, Yonghyeon Lee +2
Korea University Seoul, Republic of Korea · Massachusetts Institute of Technology Cambridge, Massachusetts, USA · Dept. of Artificial Intelligence Korea University Seoul, Republic of Korea