Patient Activity Recognition (PAR) in clinical settings uses activity data to improve safety and quality of care. Although significant progress has been made, current models mainly identify which activity is occurring. They often spatially compose sub-sparse visual cues using global and local attention mechanisms, yet only learn logically implicit patterns due to their neural-pipeline. Advancing clinical safety requires methods that can infer why a set of visual cues implies a risk, and how these can be compositionally reasoned through explicit logic beyond mere classification. To address this, we proposed Logi-PAR, the first Logic-Infused Patient Activity Recognition Framework that integrates contextual fact fusion as a multi-view primitive extractor and injects neural-guided differentiable rules. Our method automatically learns rules from visual cues, optimizing them end-to-end while enabling the implicit emergence patterns to be explicitly labelled during training. To the best of our knowledge, Logi-PAR is the first framework to recognize patient activity by applying learnable logic rules to symbolic mappings. It produces auditable why explanations as rule traces and supports counterfactual interventions (e.g., risk would decrease by 65% if assistance were present). Extensive evaluation on clinical benchmarks (VAST and OmniFall) demonstrates state-of-the-art performance, significantly outperforming Vision-Language Models and transformer baselines. The code is available via: https://github.com/zararkhan985/Logi-PAR.git}
Figures & tables
Figure 1: Overcoming Black-Box Correlations to Logical Inference in Clinical Risk Detection . Standard attention mechanisms frequently miss decisive but sparse risk cues, mistaking high-risk exits for resting states due to background bias. Logi-PAR overcomes this by enforcing a hierarchical inference structure: it first extracts reliable Atomic Facts via fine-grained attention and then applies learnable logic rules. This explicitly models the causal structure of an Unattended Bed Exit , ensuring correct classification and generating human-verifiable explanations.
Figure 2: Overview of the proposed Logi-PAR framework. The framework processes multi-view images through a shared perception backbone to extract a Probabilistic Fact Graph. These facts are then fed into the Differentiable Causal-Logic Layer, where the Gated Soft-Logic Composition (Eq. 2) dynamically assembles atomic facts into complex clinical risk states, enabling both accurate classification and explanations.
VAST
OmniFall
Models
Category
Backbone
Acc
mR@k
m@P
F1
AUC
F@R ↓
CGS
NPR
mAP
CF@val
mS@Acc
F1
SlowFast Feichtenhofer et al. (2019)
Vis-CNN
ResNet-50
79.5
76.2
74.1
75.8
0.82
0.28
52.1
48.3
81.5
8.4
68.2
72.1
ActionCLIP Wang et al. (2021)
Vis-VLM
ViT-B/16
82.3
80.1
78.5
79.4
0.85
0.22
55.4
51.2
84.2
10.5
71.5
75.8
ClinActNet Majid et al. (2025a)
Multimodal
Transformer
84.1
81.5
80.2
82.0
0.86
0.19
58.2
53.1
86.1
11.2
73.1
78.4
VideoMAE-V2 Wang et al. (2023)
Vis-T
ViT-g
88.4
85.2
82.1
83.6
0.89
0.12
62.1
58.4
89.2
12.5
75.2
81.0
UniFormerV2 Li et al. (2023)
Vis-T
UniFormer-L
89.1
86.8
84.5
85.6
0.91
0.10
64.5
60.1
90.1
14.2
78.4
83.5
Table 1: Comparison with state-of-the-art methods on VAST and OmniFall benchmarks. We report Accuracy (Acc), Mean Recall@k (mR@k), Mean Precision (m@P), F1-score, AUC, and False Positive Rate at Recall (F@R) for VAST. For OmniFall, we report CGS, NPR, mAP, CF@val, mS@Acc, and F1. (-) indicates missing values in original reports. Bold indicates best performance, underline indicates second best.
Model Variant
Label
VAST
OmniFall
F1
m@P
F@R ↓
CGS
F1
w/o Fact Fusion Mechanism ( ϕ )
A
68.5
61.5
11.7
62.1
75.8
w/o Differentiable Logic
B
85.0
78.5
12.5
65.0
81.2
w/o Causal Constraints
C
88.0
89.1
18.1
82.3
87.5
Logi-PAR (Full)
D
91.8
92.4
4.2
89.4
91.0
Table 2: Ablation study of Logi-PAR components, we validate the necessity of each component with label. F@R (lower is better). Best results are in bold .
Figure 3: Differentiable Logic rules impact during training . The (Blue Line) visualizes how Logi-PAR maintains high accuracy, while the (Red Line) sparsity regularization ( λ2 ) forces the model to "prune" unnecessary logic gates, drastically reducing the number of active rules.
Figure 4: Visualization of practical deployment on VAST sample (Video ID: P04_Exit_03). int 3-Step PAR. Heatmaps compare the baseline global attention (top), which erroneously fixates on the pillow, against Logi-PAR’s fact-specific attention (bottom). The backbone visualization confirms that Logi-PAR effectively distributes attention across views to resolve occlusion from multi-view, thereby providing the logic module ( ψ ) with a complete set of atomic facts for reliable PAR inference.
Large language models (LLMs) offer a new approach to wearable-sensor-based human activity recognition. Retrieval-augmented generation (RAG) further introduces similar motion examples and discriminative sensor knowledge, enabling decisions grounded in observations relevant to the current query. More evidence, however, does not necessarily mean better-organized evidence. When online prompts and LLM agents primarily orchestrate inference, decisions about which activities to compare and when to invoke discriminative knowledge remain concentrated at inference time. As the activity vocabulary grows, the model must repeatedly process more candidates and associated knowledge, whereas pruning candidates too early risks excluding the true activity. We propose confusion-hierarchy evidence-aware reasoning (CHEAR), which moves part of this knowledge organization offline. CHEAR constructs and validates an activity hierarchy from recurrent cross-subject retrieval errors and builds corresponding activity-pair knowledge. At inference time, the hierarchy orders candidate comparisons; the LLM retains candidates progressively, and local discriminative knowledge is accessed only for the remaining activities. Across eight datasets, CHEAR achieves 86.79% mean cross-subject accuracy, a 75.91% candidate reduction rate, and 96.44% true-activity retention. Per-sample token use is 85.15% lower on average than in a variant without the hierarchical structure. Mean accuracy across eight cross-dataset transfer directions is 88.81%. These results show that organizing candidates and discriminative knowledge around observed activity confusions reduces online LLM input while maintaining recognition performance.
Wearable human activity recognition (HAR) has made steady progress, yet much of this progress remains grounded in fixed-window, closed-set classification benchmarks. This formulation is poorly matched to everyday behavior, where activities are open-ended, unscripted, personalized, variable in duration, and often compositional. To address this mismatch, we introduce ActivityNarrated, an open-ended narrative paradigm for language-grounded wearable activity understanding. We formulate this setting as dense sensor signal captioning with a comprehensive benchmark protocol that measures temporal localization, caption quality, sensor-language alignment, conventional closed-set classification as a downstream diagnostic, and additional robustness measures. We further present ActNarrator, a 3-stage architecture that discretizes continuous IMU signals into reusable motion tokens and uses an external frozen small language model to generate open-vocabulary activity captions. Experiments show that our method provides high quality dense sensor captioning with superior adaptivity and robustness, enabling various downstream tasks by turning sensor-based human activity understanding into sensor-grounded text-level reasoning. This includes downstream classification where ActNarrator outperforms state-of-the-art HAR models by 3.8 - 31.6 % in Macro-F1. This paradigm also enables novel activity understanding capabilities such as complex question-answering over long time horizons.
Skeleton-based human activity recognition has achieved strong empirical performance, yet most existing models remain black boxes and difficult to interpret. In this work, we introduce a neurosymbolic formulation of skeleton-based HAR that reframes action recognition as concept-driven first-order logical reasoning over motion primitives. Our framework bridges representation learning and symbolic inference by grounding first-order logic predicates in learnable spatial and temporal motion concepts. Specifically, we employ a standard spatio-temporal skeleton encoder to extract latent motion representations, which are then mapped to interpretable concept predicates via a spatio-temporal concept decoder that explicitly separates pose-centric and dynamics-centric abstractions. These concept predicates are composed through differentiable first-order logic layers, enabling the model to learn human-readable logical rules that govern action semantics. To impose semantic structure on the learned concepts, we align skeleton representations with LLM-derived descriptions of atomic motion primitives, establishing a shared conceptual space for perception and reasoning. Extensive experiments on NTU RGB+D 60/120 and NW-UCLA demonstrate that our approach achieves competitive recognition performance while providing explicit, interpretable explanations grounded in logical structure. Our results highlight neurosymbolic reasoning as an effective paradigm for interpretable spatio-temporal action understanding. Code: https://github.com/Mr-TalhaIlyas/REASON
Talha Ilyas, Deval Mehta, Zongyuan Ge
Department of ECSE, Faculty of Engineering, Monash University, Australia · Department of DSAI, Faculty of Information Technology, Monash University, Australia · AIM for Health Lab, Faculty of Information Technology, Monash University, Australia