OntoAR-HAR: Ontology-Guided Agentic RAG with Personalized Memory for Human Activity Recognition
Authors: Xiangqing Lu, Tao Zhu, Liang Yu, Shuangjian Li, Lulu Ban, Furong Duan, Mingxing Nie, Liming Chen, +1 more
Organizations: School of Computer Science, University of South China, Hengyang, China · School of Computer Science and Technology, Dalian University of Technology, Dalian, China · Hunan University, Changsha, China · School of Computing, Ulster University, Belfast, United Kingdom
LLMs increasingly support sensor-based human activity recognition (HAR) by linking motion signals with activity semantics and physical knowledge. Retrieval-augmented generation (RAG) grounds decisions in sensor examples and domain features, enabling training-free target-user inference with less dependence on raw numerical arrays. However, existing LLM/RAG-based HAR typically searches single-level activity spaces using static population knowledge. Consequently, they compare many unrelated candidates using the same static population evidence and overlook personalized motion patterns. We propose OntoAR-HAR, an ontology-guided agentic RAG framework combining static population and personalized knowledge. Its tree-structured activity ontology decomposes classification into local decisions among semantically related activities. The static store supplies node-constrained examples, while offline-discovered executable signal features provide numerical evidence for local distinctions. During inference, the personalized store collects selected user observations, which the LLM integrates with static knowledge and signal features while traversing the ontology to an activity label. Cross-subject and heterogeneous-sensing experiments show complementary gains from hierarchical retrieval and personalized memory across multiple LLM backbones, achieving state-of-the-art performance. By explicitly modeling hierarchical relations across activity granularities, OntoAR-HAR also addresses the long-overlooked semantic inclusion problem and avoids treating related labels as independent categories.
Figures & tables
Fig. 1: Offline construction. The activity hierarchy is manually grouped and refined using retrieval-only confusions on balanced training samples. Source embeddings form the static population store; an LLM iteratively proposes signal features whose executable implementations are evaluated on confused examples.
Feature family
Representative statistics or derived quantities
Time-domain statistics
Mean, standard deviation, variance, maximum, minimum, median, root mean square, peak, range, interquartile range, skewness, kurtosis, and signal magnitude area.
Frequency-domain statistics
Low-, mid-, and high-frequency band powers and their ratios; dominant frequency and power; secondary peak frequency; spectral centroid, entropy, skewness, and kurtosis; weighted mean frequency; and spectral energy.
Short-time Fourier transform
Maximum, mean, and standard deviation in each frequency band, plus statistics of short-time spectral entropy and spectral centroid.
Change and impact
Mean, root mean square, and standard deviation of first differences; zero-crossing rate; impact rate of change; and peaks.
Periodicity and dependence
First autocorrelation peak, first minimum, and first zero crossing; fourth-order autoregressive coefficients; and recurrence rate.
Multiscale complexity
Wavelet-packet energy and permutation entropy.
TABLE I: Representative Signal-Feature Families Considered during Offline Construction. The Rule-Based Generator Implements Supported Candidates; the Validated Feature Set Is Specific to Each Decision Node.
Fig. 2: Online inference. At each ontology node, retrieval is restricted to the current subtree. Static population examples, personalized examples, and executable-feature reports form the decision context; the selected child determines the next local problem.
Method
Metric
Easy
Medium
Hard
Avg.
Opp.
UCI-HAR
Shoaib
PAMAP2
USC-HAD
Mhealth
WISDM
DSADS
# Classes
4
6
7
12
12
12
18
19
–
# Channels
30
6
30
18
6
15
6
30
–
Cross-modal alignment (full-shot)
ImageBind
Acc
82.8
90.9
98.5
94.3
52.1
82.9
77.2
90.0
83.6
F1
84.6
90.7
98.5
94.6
52.2
83.2
76.8
89.3
83.7
TABLE II: Cross-Subject Results (%). Acc and Macro-F1 Are Shown in Separate Rows. The Dataset Descriptors Give the Number of Activity Classes and Sensor Channels; OntoAR-HAR Reports the Mean with Standard Deviation as a Subscript over Ten Random Samples. Best Results Are Bold, and Second-Best Results Are Underlined.
Hierarchical F1 (%)
Dataset
Single
Hier.
Δ
Dataset
Single
Hier.
Δ
DSADS
95.26
96.05
+0.79
Shoaib
99.29
99.52
+0.23
Mhealth
89.31
91.04
+1.73
UCI-HAR
96.67
97.50
+0.83
Opp.
89.50
92.75
+3.25
USC-HAD
78.54
81.11
+2.57
PAMAP2
89.03
89.58
+0.55
WISDM
81.77
82.99
+1.22
Avg.
89.92
91.32
+1.40
TABLE III: Semantic-Inclusion Evaluation. The Upper Block Reports Dataset-Level Hierarchical F1; the Lower Block Gives Overall Metrics. Percentage Changes Are in Points.
Fig. 3: Controlled component study with Gemini 2.5 Flash. Each vertical bar starts from single-level retrieval; the upper segments show the incremental gains from hierarchy and personalized memory.
Method
UCI-HAR
PAMAP2
USC-HAD
Mhealth
DSADS
Average
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
ImageBind
14.1
5.0
14.6
8.1
11.8
7.3
7.9
1.7
2.1
1.2
10.1
4.7
IMU2CLIP
17.8
11.4
1.9
1.1
12.8
9.3
7.9
1.3
4.6
1.1
9.0
4.8
IMUGPT
1.1
0.3
8.9
1.5
6.0
6.9
9.8
2.5
7.5
2.0
6.7
2.6
HARGPT
15.0
12.7
11.1
2.1
9.5
3.6
10.4
7.4
5.8
3.4
10.4
5.8
ZARA
16.7
4.9
17.1
8.6
16.7
4.8
20.0
6.7
16.7
6.8
17.4
6.4
TABLE IV: Heterogeneous-Dataset Results (%) over Six Harmonized Activity Categories without Body-Location Filtering. Acc and Macro-F1 Are Shown Side by Side. Best Results Are Bold, and Second-Best Results Are Underlined.
Fig. 4: Example decision path. Static population retrieval, personalized retrieval, and node-specific feature reports are kept separate in the context, allowing the final decision to be traced through the ontology.
Backbone
Opp.
UCI
Shoaib
PAMAP2
USC
Mhealth
WISDM
DSADS
Avg.
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Acc
F1
Gemini 2.0 Flash
83.00
82.71
92.50
92.33
96.19
96.19
85.42
85.94
71.67
71.60
85.83
85.51
78.12
77.30
90.53
90.44
85.41
85.25
Gemini 2.5 Flash
92.50
92.51
95.42
95.38
100.00
100.00
85.83
86.01
74.58
74.60
90.42
89.33
79.51
78.54
93.68
93.05
88.99
88.68
Gemini 3.0 Flash
90.50
90.62
94.17
94.09
100.00
100.00
89.17
89.49
82.50
82.01
89.58
89.45
80.90
80.27
94.74
94.79
90.20
90.09
DeepSeek R1
80.50
80.28
94.12
94.17
96.67
96.66
85.00
85.22
71.25
71.50
78.33
77.80
79.51
78.66
92.63
91.98
84.75
84.53
Grok 4
85.50
85.41
92.50
92.44
99.52
99.52
87.50
87.90
72.50
72.41
85.00
84.58
78.82
77.83
93.15
92.78
86.81
86.61
TABLE V: Detailed Performance of Different LLM Backbones in the Controlled Comparison (%). Accuracy and Macro-F1 Are Shown Side by Side. Best Results Are Bold, and Second-Best Results Are Underlined.
Dataset (C/W)
Ret. (s)
LLM (s)
Tokens (K)
Cost
DSADS (30/125)
0.057
1.39
1.91/0.35/2.25
0.001442
Mhealth (15/100)
0.036
1.67
1.29/0.31/1.60
0.001170
Opportunity (30/60)
0.077
2.59
2.71/0.50/3.20
0.002055
PAMAP2 (18/200)
0.054
2.52
2.31/0.45/2.76
0.001822
Shoaib (30/100)
0.092
1.48
1.61/0.47/2.08
0.001655
UCI-HAR (6/128)
0.019
4.96
1.03/0.18/1.22
0.000768
TABLE VI: Per-Query Efficiency with Gemini 2.5 Flash. C/W Denotes Channels/Window; Ret. Is Local Retrieval Time; Tokens (K) Are Prompt/Completion/Total, with the Total Computed before Rounding; Cost Is in USD.
Fig. 5: Retrieval-only accuracy and the change obtained after LLM reasoning with the same retrieved evidence. Positive changes occur on all eight datasets.