From Stress to Affect: Multimodal Deep Learning for Physiological Emotion Recognition Across Wearable Sensor Modalities

Desta Haileselassie Hagos; Saurav Keshari Aryal; Legand L. Burge

cs.LGSep 17, 2026

From Stress to Affect: Multimodal Deep Learning for Physiological Emotion Recognition Across Wearable Sensor Modalities

Authors: Desta Haileselassie Hagos, Saurav Keshari Aryal, Legand L. Burge

Organizations: Department of Electrical Engineering and Computer Science, Howard University, Washington DC, USA

Abstract

Physiological emotion recognition using wearable sensors has important applications in mental health monitoring, affective computing, and human-computer interaction. However, existing studies typically evaluate a single model, sensing configuration, or dataset, limiting our understanding of how these factors influence recognition performance. We present a comparative study of temporal deep learning architectures for physiological emotion recognition using two multimodal wearable datasets: WESAD and EmoWear. Bidirectional long short-term memory (LSTM), temporal convolutional network (TCN), and Transformer models are evaluated under wrist-only, chest-only, and multimodal sensing configurations using participant-independent leave-one-subject-out cross-validation (LOSO-CV). We also investigate soft-voting ensembles, sensor ablation, sampling frequency, and gradient-based saliency. The Transformer achieved the highest multimodal accuracy on WESAD (99.02% +/- 0.51%), whereas the LSTM achieved the best multimodal accuracy on EmoWear for both arousal (91.80% +/- 1.06%) and valence (89.96% +/- 0.36%). These results show that relative architecture performance depends on dataset characteristics rather than one architecture being uniformly superior. Multimodal sensing consistently outperformed wrist-only and chest-only configurations across both datasets. Sampling-frequency analysis showed that 4 Hz provides a practical operating point, with performance comparable to higher frequencies at substantially lower training cost. These findings provide guidance for selecting architectures, sensing modalities, and sampling frequencies for wearable physiological emotion recognition.

Figures & tables

Property	WESAD	EmoWear
Participants	15 (12M, 3F)	48 (27M, 21F)
Affective construct	Discrete	Dimensional
Emotion labels	Baseline, Stress, Amusement	Arousal, Valence
Number of classes	3	2 (binary per dimension)
Emotion elicitation	Controlled lab (TSST + video)	Naturalistic video clips
Wrist device	Empatica E4	Empatica E4

TABLE I: Comparison of the WESAD and EmoWear datasets.

Model	Class	Precision	Recall	F1-score	Accuracy
Logistic Regression	Baseline	65.61	72.86	69.05	59.68 $\pm$ 16.86
	Stress	55.82	51.06	53.33
	Amusement	40.87	32.82	36.41
	Macro Avg	54.10	52.25	52.93 $\pm$ 20.47
	Weighted Avg	58.52	59.59	58.85 $\pm$ 19.90
Random Forest	Baseline	73.95	77.82	75.84	67.50 $\pm$ 10.89

TABLE II: Multimodal classification performance of summary-statistics baselines on the WESAD dataset.

Model	Class	Precision	Recall	F1-score	Accuracy
LSTM	Baseline	95.92	96.66	96.29	95.81 $\pm$ 0.48
	Stress	97.03	98.37	97.70
	Amusement	93.14	88.54	90.78
	Macro Avg	95.36	94.52	94.92 $\pm$ 0.53
	Weighted Avg	95.79	95.81	95.79 $\pm$ 0.46
Transformer	Baseline	93.53	94.46	94.00	93.33 $\pm$ 1.68

TABLE III: Unimodal classification performance of LSTM, Transformer, and TCN on WESAD wrist signals.

Model	Class	Precision	Recall	F1-score	Accuracy
LSTM	Baseline	96.62	99.29	97.93	97.70 $\pm$ 0.55
	Stress	99.37	99.00	99.18
	Amusement	98.29	90.34	94.15
	Macro Avg	98.09	96.21	97.09 $\pm$ 0.72
	Weighted Avg	97.73	97.70	97.67 $\pm$ 0.56
Transformer	Baseline	99.29	98.94	99.11	99.02 $\pm$ 0.51

TABLE IV: Multimodal classification performance on the WESAD dataset using fused wrist and chest signals.

Class	Precision	Recall	F1-score	Accuracy
Baseline	98.87	99.36	99.12	98.91 $\pm$ 0.13
Stress	99.50	99.75	99.62
Amusement	97.94	95.96	96.94
Macro Avg	98.77	98.36	98.56 $\pm$ 0.17
Weighted Avg	98.90	98.91	98.90 $\pm$ 0.13

TABLE V: Ensemble fusion results combining LSTM, TCN, and Transformer predictions on the multimodal WESAD dataset.

Input	Model	Precision	Recall	F1-score	Accuracy
Wrist-Only
LSTM	Baseline	96.41	93.47	94.92	94.38 $\pm$ 0.88
	Stress	97.63	97.99	97.81
	Amusement	83.33	90.81	86.91
	Macro Avg	92.46	94.09	93.21 $\pm$ 0.97
	Weighted Avg	94.58	94.38	94.44 $\pm$ 0.85

TABLE VI: Ablation study results for wrist-only and chest-only inputs on the WESAD dataset.

Fig. 1: Saliency map for a representative stress sample from the WESAD dataset using the multimodal LSTM model. Higher saliency values indicate greater contributions of the corresponding physiological signals at each time step to the predicted stress class.

Model	Setting	F1-score (%)	Accuracy (%)	Study
Decision Tree	All modalities	58.05 $\pm$ 1.61	63.56 $\pm$ 1.73	Schmidt et al. [ 13 ]
Random Forest	All modalities	64.08 $\pm$ 1.68	74.97 $\pm$ 1.11	Schmidt et al. [ 13 ]
AdaBoost	All modalities	68.85 $\pm$ 0.89	79.57 $\pm$ 0.93	Schmidt et al. [ 13 ]
LDA	All modalities	71.56	75.80	Schmidt et al. [ 13 ]
kNN	All modalities	48.70	56.14	Schmidt et al. [ 13 ]
Random Forest	All modalities	65.73	67.56	Garg et al. [ 14 ]

TABLE VII: Comparison with previously reported results on the WESAD dataset under the three-class LOSO-CV protocol. Results from previous studies are reported as originally published.

Model	Setting	F1-score (%)	Accuracy (%)	Study
Decision Tree	Wrist-only	43.62 $\pm$ 1.33	53.98 $\pm$ 1.79	Schmidt et al. [ 13 ]
Random Forest	Wrist-only	62.86 $\pm$ 0.65	74.85 $\pm$ 0.20	Schmidt et al. [ 13 ]
AdaBoost	Wrist-only	64.12 $\pm$ 0.98	75.21 $\pm$ 0.77	Schmidt et al. [ 13 ]
LDA	Wrist-only	63.24	70.74	Schmidt et al. [ 13 ]
kNN	Wrist-only	37.20	45.54	Schmidt et al. [ 13 ]
Decision Tree	Chest-only	53.06 $\pm$ 0.50	57.68 $\pm$ 0.40	Schmidt et al. [ 13 ]

TABLE VIII: Performance comparison of unimodal models on the WESAD dataset under the LOSO-CV protocol. Results from previous studies are reported as originally published.

Model	Class	Precision	Recall	F1-score	Accuracy
Wrist-Only
Logistic Regression	Low Arousal	58.68	55.54	57.07	54.26 $\pm$ 0.12
	High Arousal	49.50	52.70	51.05
	Macro Avg	54.09	54.12	54.06 $\pm$ 0.12
	Weighted Avg	54.53	54.26	54.35 $\pm$ 0.12
Random Forest	Low Arousal	81.50	72.55	76.77	75.96 $\pm$ 0.10

TABLE IX: Arousal classification performance of summary-statistics baselines on the EmoWear dataset.

Model	Class	Precision	Recall	F1-score	Accuracy
Wrist-Only
Logistic Regression	Low Valence	52.63	62.14	56.99	52.21 $\pm$ 0.08
	High Valence	51.57	41.88	46.23
	Macro Avg	52.10	52.01	51.61 $\pm$ 0.13
	Weighted Avg	52.11	52.21	51.71 $\pm$ 0.12
Random Forest	Low Valence	78.65	69.36	73.72	74.80 $\pm$ 0.11

TABLE X: Valence classification performance of summary-statistics baselines on the EmoWear dataset.

Model	Class	Precision	Recall	F1-score	Accuracy
Arousal
LSTM	Low Arousal	85.95	83.53	84.72	83.51 $\pm$ 0.77
	High Arousal	80.74	83.48	82.09
	Macro Avg	83.34	83.51	83.40 $\pm$ 0.80
	Weighted Avg	83.59	83.51	83.53 $\pm$ 0.78
Transformer	Low Arousal	76.62	65.17	70.44	70.05 $\pm$ 0.25

TABLE XI: Wrist-only classification performance of LSTM, Transformer, and TCN on the EmoWear dataset.

Model	Class	Precision	Recall	F1-score	Accuracy
Arousal
LSTM	Low Arousal	75.56	70.21	72.79	71.27 $\pm$ 0.20
	High Arousal	66.82	72.55	69.57
	Macro Avg	71.19	71.38	71.18 $\pm$ 0.17
	Weighted Avg	71.61	71.27	71.33 $\pm$ 0.20
Transformer	Low Arousal	73.42	53.17	61.67	63.83 $\pm$ 0.38

TABLE XII: Chest-only classification performance of LSTM, Transformer, and TCN on the EmoWear dataset.

Model	Class	Precision	Recall	F1-score	Accuracy
Arousal
LSTM	Low Arousal	93.40	91.48	92.43	91.80 $\pm$ 1.06
	High Arousal	89.95	92.18	91.05
	Macro Avg	91.67	91.83	91.74 $\pm$ 1.06
	Weighted Avg	91.84	91.80	91.81 $\pm$ 1.06
Transformer	Low Arousal	85.77	74.08	79.49	79.08 $\pm$ 0.36

TABLE XIII: Multimodal classification performance on the EmoWear dataset using fused wrist and chest signals.

Task	Class	Precision	Recall	F1-score	Accuracy
Wrist-Only
Arousal	Low Arousal	86.84	80.21	83.39	82.51 $\pm$ 0.03
	High Arousal	78.09	85.30	81.54
	Macro Avg	82.47	82.76	82.46 $\pm$ 0.04
	Weighted Avg	82.88	82.51	82.55 $\pm$ 0.04
Valence	Low Valence	79.86	80.40	80.13	79.68 $\pm$ 1.24

TABLE XIV: Soft-voting ensemble results on the EmoWear dataset across sensor configurations and emotional dimensions.

Setting	Model	Arousal (%)	Valence (%)
Wrist-Only	LSTM	$83.51\pm 0.77$	$81.96\pm\text{1.62}$
	Transformer	$70.05\pm 0.25$	$64.79\pm 0.70$
	TCN	$\text{72.22}\pm\text{0.09}$	$\text{67.73}\pm\text{0.27}$
	Ensemble	$82.51\pm 0.03$	$79.68\pm 1.24$
Chest-Only	LSTM	$71.27\pm 0.20$	$67.47\pm 0.23$
	Transformer	$63.83\pm 0.38$	$60.77\pm 0.35$

TABLE XV: Ablation study summary for EmoWear across sensor configurations, models, and emotional dimensions.

Hz	Timesteps	Stride	Arousal		Valence		Train Time (min)
Hz	Timesteps	Stride	Accuracy (%)	Macro F1 (%)	Accuracy (%)	Macro F1 (%)	Arousal	Valence
1	3	1	82.25 $\pm$ 0.37	82.22 $\pm$ 0.35	80.18 $\pm$ 0.72	80.18 $\pm$ 0.72	7.87 $\pm$ 0.63	5.75 $\pm$ 0.04
2	5	1	88.54 $\pm$ 0.07	88.49 $\pm$ 0.08	87.13 $\pm$ 0.18	87.12 $\pm$ 0.18	20.98 $\pm$ 0.92	17.11 $\pm$ 0.02
4	10	2	90.16 $\pm$ 0.98	90.11 $\pm$ 0.98	90.48 $\pm$ 0.61	90.48 $\pm$ 0.61	32.98 $\pm$ 2.52	30.41 $\pm$ 0.06
8	20	4	91.47 $\pm$ 0.83	91.41 $\pm$ 0.83	89.64 $\pm$ 0.57	89.64 $\pm$ 0.57	57.01 $\pm$ 3.77	54.31 $\pm$ 0.38
16	40	8	91.06 $\pm$ 0.98	91.01 $\pm$ 0.98	90.21 $\pm$ 0.41	90.21 $\pm$ 0.41	99.02 $\pm$ 0.41	107.26 $\pm$ 8.00

TABLE XVI: Sampling-frequency sensitivity analysis for the multimodal LSTM on EmoWear. Results are reported as mean

\pm

std across three seeds. Training time is reported in minutes.

Fig. 2: Sampling-frequency sensitivity analysis for the multimodal LSTM on EmoWear. Accuracy and macro F1-score are shown as a function of sampling frequency for arousal (left) and valence (right). Error bars indicate

\pm

1 standard deviation across three seeds.

Fig. 3: Accuracy vs. training time for the multimodal LSTM on EmoWear across sampling frequencies. Accuracy (blue, left axis) and training time in minutes (green, right axis) are shown for arousal (left) and valence (right). Error bars indicate

\pm

1 standard deviation across three seeds.

Fig. 4: Representative saliency map for a correctly classified high-arousal sample from the EmoWear dataset using the multimodal LSTM model. Higher saliency values indicate greater contributions of the corresponding physiological signals at each time step to the predicted class.

Fig. 5: Representative saliency map for a correctly classified high-valence sample from the EmoWear dataset using the multimodal LSTM model. Higher saliency values indicate greater contributions of the corresponding physiological signals at each time step to the predicted class.

Model	WESAD	EmoWear
Model	Accuracy (%)	Arousal (%)	Valence (%)
LSTM	97.70 $\pm$ 0.55	91.80 $\pm$ 1.06	89.96 $\pm$ 0.36
Transformer	99.02 $\pm$ 0.51	79.08 $\pm$ 0.36	77.04 $\pm$ 0.55
TCN	96.76 $\pm$ 0.59	78.69 $\pm$ 0.65	76.59 $\pm$ 0.03
Ensemble	98.91 $\pm$ 0.13	88.65 $\pm$ 0.59	89.03 $\pm$ 0.57

TABLE XVII: Multimodal accuracy (%) of each architecture on WESAD and EmoWear. EmoWear results are shown for arousal and valence separately.

Dataset	Setting	Best Deep Learning (%)	Best Baseline (%)
WESAD	Multimodal	99.02 (Transformer)	67.50 (Random Forest)
EmoWear	Wrist Arousal	83.51 (LSTM)	75.96 (Random Forest)
	Wrist Valence	81.96 (LSTM)	74.80 (Random Forest)
	Chest Arousal	71.27 (LSTM)	69.06 (Random Forest)
	Chest Valence	67.47 (LSTM)	67.30 (Random Forest)
	Multimodal Arousal	91.80 (LSTM)	79.05 (Random Forest)

TABLE XVIII: Accuracy gap between the best deep learning model and the best classical machine learning baseline across datasets and sensor configurations (%).

Emotion Recognition

Emotion recognition uses language, voice, face, body movement, and physiological signals to infer emotional categories or dimensions such as valence and arousal.

See all 124 papers

Jun 12, 2026

Deep Temporal Modeling and Ensemble Fusion for Multimodal Emotion Recognition from Physiological Signals

Physiological stress and emotion recognition are important for health monitoring and affective computing. In this work, we present a comprehensive evaluation of deep learning models such as Long Short-Term Memory (LSTM), Temporal Convolutional Networks (TCN), and Transformer on the WESAD dataset for multimodal affect recognition using wrist and chest sensor signals. We perform ablation studies to assess the individual contributions of each modality by training models on wrist-only and chest-only inputs. In addition, we implement a late-fusion ensemble strategy that combines predictions from all three architectures trained on multimodal input. We also employ early fusion at the sensor level by concatenating wrist and chest signals before feeding them into each model. Our results show that Transformer models consistently achieve the highest accuracy in multimodal settings, while TCN models perform best in the wrist-only configuration. The ensemble method yields the highest overall accuracy (98.91 +/- 0.13%) and macro-F1 score (98.56 +/- 0.17%). These findings demonstrate the effectiveness of sensor fusion and ensemble-based fusion in developing robust systems for physiological emotion recognition.

Desta Haileselassie Hagos, Saurav Keshari Aryal, Patrick Ymele-Leki +2

Howard University Washington, DC, USA

Temporal Convolutional Networks Deep Ensembles Multimodal Emotion Recognition Emotion Recognition Multimodal Sensor Fusion

Ask PDF

Apr 30, 2026

Introducing WARM-VR: Benchmark Dataset for Multimodal Wearable Affect Recognition in Virtual Reality

With the growing integration of human-computer interaction into everyday life, advances in machine learning have enabled systems to better perceive and respond to users' emotional states. Most existing affect recognition datasets focus on static environments, limiting their applicability to immersive multimedia contexts such as Virtual Reality (VR). In this paper, we introduce WARM-VR, a novel publicly available multimodal dataset designed to support affect recognition in immersive, multisensory environments using wearable sensing instrumentation. Data were collected from 31 participants aged 19-37 using wearable sensors: a wristband measuring Blood Volume Pulse (BVP), EDA, skin Temperature, three-axis Acceleration, and a chest strap recording ECG signals. Participants engaged in immersive VR experiences designed to elicit relaxation through a calming beach environment following stress induction via an arithmetic task. These sessions incorporated synchronized multimedia stimuli: visual, auditory, and olfactory. Affective states were assessed subjectively through validated self-report questionnaires and objectively through the analysis of physiological measurements. Statistical analysis of the questionnaires confirmed that VR relaxation significantly reduced negative affect, particularly with olfactory enhancement. Furthermore, we established a benchmark on the dataset using widely recognized machine learning algorithms. The best performance for binary classification from BVP data of valence, was obtained with a CNN and a CNN-Bi-GRU model, both achieving an average F1-score of 0.63 and an AUC of 0.69. For arousal, a lightweight Transformer architecture provided the most balanced results (F1-0 0.54 and F1-1 0.63), outperforming recurrent hybrids. In the relaxation task, a CNN-Bi-GRU model reached the highest overall performance (average F1-score 0.64, AUC 0.69).

Karim Alghoul, Faisal Mohd, Fedwa Laamarti +2

School of Electrical Engineering and Computer Science, University of Ottawa, Canada · Computer Vision Department, Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

Wearable Sensing Multimodal Emotion Recognition Emotion Recognition Affective Computing

Ask PDF

May 14, 2026

Decision-Level Fusion for Robust Wearable Affect Recognition

Automatic recognition of affective state from wearable physiology has clear societal impact for public health, preventive care, and stress-aware interventions, but real deployments require robustness to non-stationary dynamics, artefacts, and missing sensors. We study this problem on WESAD, using baseline, stress, and amusement conditions, where common fixed-basis spectral features such as FFT bandpower and Welch PSD can oversmooth short-lived discriminative patterns. We propose a non-stationary pipeline that combines Fourier-Bessel Series Expansion (FBSE) with EWT data-driven spectral segmentation to extract mode-wise transient descriptors. For multimodal integration, we adopt decision-level aggregation over per-modality predictors and weight each modality by predictive uncertainty and modality reliability. Results on WESAD, using 15 subjects and ECG, EDA, BVP, EMG, and ACC signals across three classes, indicate that decision-level aggregation is approximately 84 percent of the time at least as good as feature-level aggregation, and approximately 48 percent of the time strictly better, suggesting improved robustness under heterogeneous and partially reliable sensing.

Lokesh Singh, Athina Georgara, Jayati Deshmukh +2

University of Southampton · Southampton, United Kingdom

Multimodal Robustness Frequency-Domain Feature Learning Multimodal Fusion Affective Computing Multimodal Sensor Fusion

Ask PDF

Abstract

Emotion Recognition

Explore similar work

Deep Temporal Modeling and Ensemble Fusion for Multimodal Emotion Recognition from Physiological Signals

Introducing WARM-VR: Benchmark Dataset for Multimodal Wearable Affect Recognition in Virtual Reality

Decision-Level Fusion for Robust Wearable Affect Recognition