Vision Transformers have shown tremendous success in numerous computer vision applications; however, they have not been exploited for stress assessment using physiological signals such as Electrocardiogram (ECG). In order to get the maximum benefit from the vision transformer for multilevel stress assessment, in this paper, we transform the raw ECG data into 2D spectrograms using short time Fourier transform (STFT). These spectrograms are divided into patches for feeding to the transformer encoder. We also perform experiments with 1D CNN and ResNet-18 (CNN model). We perform leave-onesubject-out cross validation (LOSOCV) experiments on WESAD and Ryerson Multimedia Lab (RML) dataset. One of the biggest challenges of LOSOCV based experiments is to tackle the problem of intersubject variability. In this research, we address the issue of intersubject variability and show our success using 2D spectrograms and the attention mechanism of transformer. Experiments show that vision transformer handles the effect of intersubject variability much better than CNN-based models and beats all previous state-of-the-art methods by a considerable margin. Moreover, our method is end-to-end, does not require handcrafted features, and can learn robust representations. The proposed method achieved 71.01% and 76.7% accuracies with RML dataset and WESAD dataset respectively for three class classification and 88.3% for binary classification on WESAD.
Figures & tables
Fig. 1: Overview of the proposed method. We transform an ECG signal into spectrogram which is then split into flattened patches, add position embeddings and then feed to the transformer encoder. Stress recognition is performed by adding an extra learnable “classification token” to the sequence.
Fig. 2: Spectrograms representing low, medium, and high stress levels from the RML dataset. Each spectrogram displays the time–frequency distribution of the ECG signal.
Fig. 3: Visualization of features in spectrogram
Fig. 4: Figure shows one encoder block of a Vision Transformer. This block is repeated L times in the full encoder as indicated by LX at the top.
Fine Tuned Parameters
Values
Learning Rate
0.001
Momentum
0.9
Weight Decay
0.005
BatchSize
16
TABLE I: Parameters for fine tuning the vision transformer
Methods
Accuracy
F1 Score
P.Schmidt et al. [ 19 ]
66.29
56.03
P.Garg et al. [ 18 ]
67.56
65.73
Z.Ahmad et al. [ 6 ]
72.7
73.1
Proposed Method
76.7
76.8
TABLE IX: Performance comparison of the proposed method On the WESAD dataset with other methods for three class classification using LOSOCV splitting.
Methods
Accuracy
F1 Score
B.Behinaein et al. [ 30 ]
80.4
69.7
P.Garg et al. [ 18 ]
84.17
83.34
P.Schmidt et al. [ 19 ]
85.44
81.31
Bota et al. [ 33 ]
85.7
-
Proposed Method
88.3
88.4
TABLE X: Performance comparison of the proposed method on the WESAD dataset with other methods for binary classification using LOSOCV splitting.
Fig. 5: Visualization of attention maps for three stress levels. (a) Low stress, (b) Medium stress, and (c) High stress, each showing the spectrogram with attention maps extracted from the 1st, 5th, and 10th encoder layers.
Fig. 6: Visualization of Class Activation Maps (CAMs) for three stress levels: (a) low, (b) medium, and (c) high. Each spectrogram is shown with CAMs extracted from the 2nd, 8th, and 16th convolutional layers of ResNet-18.
Fig. 7: t-SNE visualization of extracted features for Subject 4 of the RML dataset. (a) Vision Transformer (b) ResNet-18
Walkky LLC, Boston, MA, USA · Northwell Cardiovascular Institute, Lenox Hill Hospital, New York, NY, USA · Weiss Consulting Groups, Boca Raton, FL, USA +2