MUPAS: Multimodal Unified Perception with Asymmetric Attention for Stress Assessment
Organizations: Honda Research Institute Japan Wako City, Japan · Department of Electronic Engineering Hellenic Mediterranean University Chania, Greece · Research Institute of Electrical Communication Tohoku University Sendai, Japan · University of Canberra Canberra, Australia ·
Abstract
Stress is a leading contributor to long-term physical and mental health decline. The automatic assessment of stress is a promising approach for continuous monitoring and mitigating human biases. However, the development of automated systems remains challenging due to the heterogeneity of physiological and behavioral signals. This paper presents MUPAS, a multimodal framework for automatic stress assessment that combines facial video and ECG signals within a unified perception architecture. Both modalities are mapped into a shared token representation without modality-specific components, and processed through asymmetric attention comprising segment-local cross-attention and global self-attention across segment states. Experiments on a newly collected 58-subject stress dataset, evaluated under a stratified subject-level protocol, show that facial video achieves 69.44% test accuracy, while ECG-only performance remains close to chance under the proposed representation. Unified Video-ECG fusion achieves a test accuracy of 69.58% at approximately half the number of parameters of feature-level fusion, with negligible additional computational cost. Feature-level fusion does not exceed the video-only baseline, while unified input-level integration provides a parameter-efficient mechanism for jointly processing the two modalities.
Figures & tables
| Split | Difficulty Group | |||
| Q1 – Hard | Q2 – Med-Hard | Q3 – Med-Easy | Q4 – Easy | |
| Training (38) | P017, P018, P022, P026, P034, P035, P042, P045, P050, P056 | P001, P002, P003, P007, P012, P021, P033, P040, P048 | P004, P014, P016, P032, P036, P046, P047, P052, P053, P054 | P005, P010, P020, P028, P029, P037, P039, P041, P057 |
| Validation (8) | P038, P055 | P009, P023 | P006, P013 | P019, P030 |
| Testing (12) | P008, P025, P044 | P011, P024, P043 | P015, P031, P058 | P027, P051, P059 |
| Input | Computational Cost | Validation | Testing | ||||
| Modality | Stride | Params (M) | GFLOPs | Accuracy | Precision | F1 | Accuracy |
| Video | 5 | 6.39 | 70.46 | 68.80 | 69.39 | 67.39 | 60.56 |
| Video | 10 | 6.05 | 35.67 | 66.39 | 66.20 | 66.14 | 63.75 |
| Video | 15 | 5.93 | 24.07 | 67.77 | 68.33 | 67.89 | 69.44 |
| Video | 20 | 5.87 | 18.27 | 70.42 | 71.86 | 68.45 | 65.97 |
| Video | 30 | 5.82 | 12.48 | 68.60 | 69.91 | 68.72 | 64.86 |
| Input | Computational Cost | Validation | Testing | ||||
| Modality | Stride | Params (M) | GFLOPs | Accuracy | Precision | F1 | Accuracy |
| ECG | 1 | 5.84 | 1.12 | 50.00 | 27.71 | 35.66 | 51.81 |
| ECG | 2 | 5.84 | 1.12 | 54.32 | 68.85 | 45.81 | 52.50 |
| ECG | 5 | 5.84 | 1.12 | 55.93 | 66.83 | 49.70 | 52.50 |
| ECG | 10 | 5.84 | 1.12 | 55.67 | 70.89 | 48.21 | 52.64 |
| ECG | 15 | 5.84 | 1.12 | 53.23 | 61.90 | 45.07 | 53.33 |
| Input | Computational Cost | Validation | Testing | |||||
| Modality | Stride | Fusion | Params (M) | GFLOPs | Accuracy | Precision | F1 | Accuracy |
| Video-ECG | 5-1 | feature-level | 12.11 | 71.37 | 69.36 | 69.27 | 68.67 | 68.61 |
| Video-ECG | 10-1 | feature-level | 11.76 | 36.58 | 70.48 | 70.74 | 70.57 | 67.22 |
| Video-ECG | 15-1 | feature-level | 11.65 | 24.99 | 71.27 | 71.02 | 70.98 | 68.33 |
| Video-ECG | 20-1 | feature-level | 11.59 | 19.19 | 67.74 | 67.54 | 67.39 | 68.78 |
| Video-ECG | 30-1 | feature-level | 11.53 | 13.39 | 68.57 | 68.41 | 68.45 | 67.22 |
| Study | Modality | Protocol | Accuracy |
| [ 37 ] | EDA | Hold-out | 70.97 |
| [ 36 ] | Video | Hold-out | 69.44 |
| MUPA 2 S | Video–ECG | Hold-out | 69.58 |
| [ 34 ] | Video | LOSO | |
| MUPA 2 S | Video–ECG | LOSO |
Multimodal Fusion
Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.
See all 805 papers