A Multi-Modal CNN-LSTM Framework with Multi-Head Attention and Focal Loss for Real-Time Elderly Fall Detection

Lijie Zhou; Luran Wang

cs.LGMar 20, 2026

A Multi-Modal CNN-LSTM Framework with Multi-Head Attention and Focal Loss for Real-Time Elderly Fall Detection

Authors: Lijie Zhou, Luran Wang

Organizations: School of Computer Science, University of Nottingham Ningbo China, Ningbo, China · School of Mathematical Sciences, University of Nottingham Ningbo China, Ningbo, China

Abstract

The increasing global aging population has intensified the demand for reliable health monitoring systems, particularly those capable of detecting critical events such as falls among elderly individuals. Traditional fall detection approaches relying on single-modality acceleration data suffer from high false alarm rates, while conventional machine learning methods require extensive hand-crafted feature engineering. This paper proposes a novel multi-modal deep learning framework, MultiModalFallDetector, designed for real-time elderly fall detection using wearable sensors. Our approach integrates multiple innovations: a multi-scale CNN-based feature extractor capturing motion dynamics at varying temporal resolutions; fusion of tri-axial accelerometer, gyroscope, and four-channel physiological signals; incorporation of a multi-head self-attention mechanism for dynamic temporal weighting; adoption of Focal Loss to mitigate severe class imbalance; introduction of an auxiliary activity classification task for regularization; and implementation of transfer learning from UCI HAR to SisFall dataset. Extensive experiments on the SisFall dataset, which includes real-world simulated fall trials from elderly participants (aged 60-85), demonstrate that our framework achieves an F1-score of 98. 7, Recall of 98. 9, and AUC-ROC of 99. 4, significantly outperforming baseline methods including traditional machine learning and standard deep learning approaches. The model maintains sub- 50ms inference latency on edge devices, confirming its suitability for real-time deployment in geriatric care settings.

Figures & tables

Fig. 1: Overview of the proposed MultiModalFallDetector architecture. The five-stage pipeline consists of input preprocessing and segmentation, modality-specific feature extraction via multi-scale CNNs, temporal dynamics modeling via bidirectional LSTM, contextual attention weighting via multi-head self-attention, and dual-head prediction for fall detection and activity classification.

Dataset	Subjects	Activities	Sampling Rate
SisFall	23 elderly + 14 young	15 falls + 19 ADLs	200 Hz
UCI HAR	30 young	6 activities	50 Hz

TABLE I: Dataset characteristics for training and evaluation

Fig. 2: Visual comparison of data augmentation techniques applied to accelerometer signals. The original signal is shown in blue, while augmented versions demonstrate the effects of jitter (noise addition), scaling (amplitude variation), and rotation (orientation change).

Platform	Latency (ms)	Throughput (samples/s)
NVIDIA RTX 3060	2.3	435
Raspberry Pi 4	45	22
Jetson Nano	12	83

TABLE II: Inference latency measurements across different hardware platforms

Method	Precision	Recall	F1-Score	AUC-ROC
Threshold (2.0g)	0.72	0.89	0.80	0.83
SVM + Handcrafted	0.85	0.82	0.83	0.90
Random Forest	0.83	0.84	0.83	0.89
CNN-only	0.88	0.86	0.87	0.93
LSTM-only	0.86	0.88	0.87	0.92
CNN-LSTM (no attention)	0.91	0.90	0.90	0.95

TABLE III: Performance comparison of proposed method against baseline approaches on the SisFall test set

Fig. 3: ROC curve comparison of the proposed MultiModalFallDetector against baseline methods. The proposed model achieves an AUC-ROC of 0.97, demonstrating superior discriminative performance across all classification thresholds.

Variant	Recall	F1-Score	AUC
Full model	0.989	0.987	0.994
w/o Multi-scale CNN (kernel=3 only)	0.92	0.91	0.95
w/o Attention	0.93	0.91	0.95
w/o Focal Loss (BCE)	0.88	0.89	0.94
w/o Auxiliary task	0.94	0.93	0.96
w/o Transfer learning	0.91	0.90	0.95

TABLE IV: Ablation study results on the SisFall test set

Fig. 4: Confusion matrix of the proposed MultiModalFallDetector on the SisFall test set. Diagonal elements represent correct classifications, with the model achieving high accuracy across all activity classes while maintaining excellent fall detection performance.

Fig. 5: Attention weight visualization for fall and walking sequences. In the fall sequence (top), attention weights concentrate sharply around the impact moment (timestep $\sim$ 50). In the walking sequence (bottom), attention is distributed more uniformly across the periodic gait pattern.

Fig. 6: Transfer learning efficacy comparison. The model pre-trained on UCI HAR (blue) demonstrates faster convergence and higher final performance compared to training from scratch (red) on the SisFall dataset.

Explore similar work

You Don't Need Attention: Gated Convolutional Modeling for Watch-Based Fall Detection
May 19, 2026Sana Alamgeer, Ronish Kumar, Awatif Yasmin +2Fall Detection Wearable Sensing
1Texas State University, San Marcos, USA
Real-time fall detection based on vision for low-power edge platforms
Jul 14, 2026Wenjun Xia, Zhicheng Peng, Haopeng Li +1Fall Detection Physics-Informed ML
Jiangsu University, Zhenjiang, China
Explainable Fall Detection for Elderly Monitoring via Temporally Stable SHAP in Skeleton-Based Human Activity Recognition
Apr 14, 2026Mohammad Saleh, Azadeh TabatabaeiSHAP Feature Attribution Human Activity Recognition
Department of Computer Engineering, University of Science and Culture, Tehran, Iran

May 19, 2026J/K move · Enter open · S save

You Don't Need Attention: Gated Convolutional Modeling for Watch-Based Fall Detection

Sana Alamgeer, Ronish Kumar, Awatif Yasmin, Muhammad Irshad+1

1Texas State University, San Marcos, USA

Fall Detection Wearable Sensing Convolutional Neural Networks

Open paper AskPDF arXiv

Existing deep learning approaches for wearable fall detection systems rely on self-attention mechanisms that impose quadratic computational overhead, distributing weights across all time steps. This global weight distribution impairs the precise localization of the brief impact signatures that characterize falls within short, fixed-length windows. To overcome this challenge, we propose Gated-CNN, a lightweight dual-stream architecture that processes accelerometer and gyroscope streams through independent one-dimensional convolutional feature extractors, followed by (i) a sigmoid gating module that selectively suppresses uninformative background activations while amplifying fall-discriminative features, (ii) a global average pooling layer that compresses each stream into a compact fixed-length descriptor, and (iii) a shared classification head that fuses both descriptors for binary fall prediction. For offline evaluation, we evaluate the model across five wrist-mounted inertial measurement unit (IMU) datasets, achieving average F1-scores of 93%, 93%, 90%, 91%, and 90% on SmartFallMM, WEDA-Fall, FallAllD, UMAFall, and UP-Fall, outperforming Transformer baselines. For real-time evaluation, we deployed the model on a Google Pixel Watch 3 and tested across 12 participants. The model achieves an average F1-score of 97% and an accuracy of 98% with zero missed falls, showing that sigmoid gating offers a more structurally aligned and computationally efficient alternative to attention for commodity smartwatch-based fall detection.

Abstract

Explore similar work

You Don't Need Attention: Gated Convolutional Modeling for Watch-Based Fall Detection

Real-time fall detection based on vision for low-power edge platforms

Explainable Fall Detection for Elderly Monitoring via Temporally Stable SHAP in Skeleton-Based Human Activity Recognition