Recognition of Daily Activities through Multi-Modal Deep Learning: A Video, Pose, and Object-Aware Approach for Ambient Assisted Living
Authors: Kooshan Hashemifard, Pau Climent-Pérez, Francisco Florez-Revuelta
Organizations: Research Group on Ambient Intelligence for Active and Healthy Ageing, Department of Computing Technology, University of Alicante, Ctra. San Vicente del Raspeig, s/n, 03690, San Vicente del Raspeig, Spain. · Alicante Institute for Health and Biomedical Research (ISABIAL), Avda. Pintor Baeza, 12, 03010, Alicante, Spain. · ValgrAI - Valencian Graduate School and Research Network of Artificial Intelligence, Camí de Vera s/n, 46022, Valencia, Spain.
Recognition of daily activities is a critical element for effective Ambient Assisted Living (AAL) systems, particularly to monitor the well-being and support the independence of older adults in indoor environments. However, developing robust activity recognition systems faces significant challenges, including intra-class variability, inter-class similarity, environmental variability, camera perspectives, and scene complexity. This paper presents a multi-modal approach for the recognition of activities of daily living tailored for older adults within AAL settings. The proposed system integrates visual information processed by a 3D Convolutional Neural Network (CNN) with 3D human pose data analyzed by a Graph Convolutional Network. Contextual information, derived from an object detection module, is fused with the 3D CNN features using a cross-attention mechanism to enhance recognition accuracy. This method is evaluated using the Toyota SmartHome dataset, which consists of real-world indoor activities. The results indicate that the proposed system achieves competitive classification accuracy for a range of daily activities, highlighting its potential as an essential component for advanced AAL monitoring solutions. This advancement supports the broader goal of developing intelligent systems that promote safety and autonomy among older adults.
Figures & tables
Figure 1: Architecture of the Proposed Method
Figure 2: Sequential skeleton rotation stages adapted from ( Climent-Perez and Florez-Revuelta, 2021 ) . ( a ) Original frame; ( b ) left: base pose, center: Y -axis rotated pose to face forward, right: fully rotated pose ( Y and Z axes).
Figure 3: Pose processing module with two tasks, calculating temporal attention weights and pose estimation.
Figure 4: Cross-Attention mechanism between scene context and extracted visual features.
Methods
Pose
RGB
CS
CV1
CV2
Pose Only
2s-AGCN ( Shi et al., 2019b )
✓
×
60.9
21.6
32.3
ST-GCN ( Yan et al., 2018 )
✓
×
53.8
12.5
51.0
UNIK ( Yang et al., 2021 )
✓
×
63.1
22.9
61.4
MS-G3D Net ( Liu et al., 2020 )
✓
×
66.7
17.5
59.4
ViA ( Yang et al., 2024 )
✓
×
64.0
35.6
65.4
Table 1: Comparison of mean per-class accuracy (%) on Toyota SmartHome dataset across different evaluation protocols. ∘ indicates that the modality has been used only in training.
Configuration
CS
CV1
CV2
Single-Modal Baselines
Video-only (I3D)
53.4
34.9
45.1
Pose-only (GCNs)
52.1
42.6
51.2
Bi-Modal Baselines
Video + Pose
68.4
42.7
62.1
Video + Object
66.8
39.5
60.3
Table 2: Comprehensive ablation study results showing mean per-class accuracy (%) on Toyota SmartHome dataset across different evaluation protocols.
Number of Heads
CS
CV1
CV2
2 heads
69.2
43.5
64.8
4 heads
69.4
43.2
65.1
8 heads
70.1
44.2
65.4
16 heads
69.8
43.9
65.6
Table 3: Performance comparison across different numbers of attention heads showing accuracy results on CS, CV1, and CV2 datasets.
Figure 5: Sample frames from the intervals corresponding to temporal attention weights αt values for activity class "Eating Snacks".
School of Computer Science, University of Nottingham Ningbo China, Ningbo, China · School of Mathematical Sciences, University of Nottingham Ningbo China, Ningbo, China