Multimodal BVP and Respiration for Automated Pain Recognition with Hankel-FCN
Authors: Valentina Becchetti, Muhammad Umar Khan, Stefanos Gkikas, Alessandro Giuseppi, Raul Fernandez Rojas
Organizations: Department of Computer, Control & Management Engineering, Sapienza University of Rome, Rome, Italy · BioSIS (Biosensing & Intelligent Systems) Lab, Centre for Intelligent Computing & Systems, University of Canberra, Canberra, ACT, Australia · Honda Research Institute Japan, Wako City, Japan
Automated pain recognition is a fundamental challenge for the advancement of clinical care and empathetic human-computer interaction, due to the subjective and dynamic nature of pain. In this study, we propose a deep learning framework for objective pain assessment based on multimodal physiological signals from the AI4Pain dataset. The approach focuses on Blood Volume Pulse (BVP) and Respiration (RESP), capturing complementary aspects of autonomic responses to nociceptive stimulation. To explicitly model their temporal dynamics, the signals are transformed into structured trajectory representations using Hankel matrix embeddings, enabling the learning of dynamical patterns beyond conventional time-series representations. These embeddings are processed by modality-specific Fully Convolutional Networks, and their outputs are integrated through a late fusion strategy for three-class classification (No Pain, Low Pain, High Pain). Experimental results demonstrate that the proposed method achieves an accuracy of 67.09% and an F1-score of 66.41%, providing competitive performance compared to existing approaches, while relying on no handcrafted feature engineering. Multimodal fusion significantly outperformed either modality alone, and the Hankel embedding showed a trend towards further improvement. Further validation is required to establish generalisability to broader populations and clinical settings.
Figures & tables
Fig. 1: Filtered and normalized BVP and RESP signals.
Component
Hyperparameter
Search Space
General
Learning Rate
[10−4,5×10−3]
Weight Decay
[10−8,10−3]
Hankel Embedding
Depth
{10,20,30,40,50}
Delay
{1,2}
Stride
{1,2}
FCN
Convolution Blocks
{1,2,3,4}
TABLE I: Hyperparameter Search Space for the Proposed Model.
Fig. 2: Representative architecture of the proposed multimodal Hankel-FCN framework. Modality-specific Hankel embeddings and convolutional branches extract temporal features from BVP and RESP signals, which are fused for pain classification (NP, LP, HP).
Modality
Acc (%)
F1 (%)
TPR (NP)
TPR (LP)
TPR (HP)
BVP
58.08 ± 13.90
56.89 ± 14.97
62.95 ± 22.58
56.28 ± 23.48
55.00 ± 18.74
RESP
52.74 ± 8.87
48.03 ± 11.24
64.36 ± 31.23
50.64 ± 28.58
43.21 ± 29.82
BVP, RESP
67.09 ± 12.87
66.41 ± 13.61
68.68 ± 19.50
64.74 ± 19.64
68.08 ± 18.49
TABLE II: Performance of the proposed Hankel-FCN model under unimodal and multimodal input configurations.
Fig. 3: Comparison of the independently optimized multimodal FCN pipelines with and without Hankel embedding across the 65 outer LOSO folds. Bars show the mean, and error bars indicate the standard deviation across held-out participants.
Fig. 4: Mean confusion matrix across the 65 outer LOSO folds for three-class pain classification (NP, LP, HP).
Ref.
Modality
Model
Acc. (%)
[ 19 ] 1,b
EDA, BVP, RESP, SpO 2
SVM (RBF)
52.30
[ 21 ] b
EDA
SVM (RBF)
66.40
[ 22 ] a
fNIRS & Video
Twins-Pain ViT
46.46
[ 23 ] a,b
EDA
GIAFormer
51.62
[ 20 ] a
fNIRS
SVM (RBF)
66.55
[ 21 ] b
BVP
Ensemble
59.60
TABLE III: Comparison with previous studies using the AI4Pain dataset for three-class pain classification using LOSO-CV.
Pain is a complex and pervasive phenomenon affecting a large percentage of the population, and accurate assessment is essential for effective clinical management and intervention. Computational pain recognition systems enable continuous monitoring, support clinical decision-making, and help mitigate pain-related distress and functional decline. This study introduces a unified tokenization framework for heterogeneous 3D modalities in pain recognition that provides a single processing pipeline across behavioral and brain-activity 3D data, without requiring separate architectures for each modality or handcrafted inductive biases. The framework preserves spatial, temporal, and time--frequency structure while mapping diverse inputs into a shared token space. Extensive experiments show that the proposed approach effectively processes facial videos and fNIRS data in both raw-signal and spectrogram-based representations. On the AI4Pain benchmark dataset, the proposed framework achieves state-of-the-art performance while maintaining high computational efficiency and enabling real-time assessment on both GPU and CPU hardware.
Stefanos Gkikas, Christian Arzate Cruz, Valentina Becchetti +3
Honda Research Institute Japan Wako City, Japan · Sapienza University of Rome Rome, Italy · University of Canberra Canberra, Australia
Automatic pain localization, which involves identifying the anatomical origin of pain from peripheral physiological signals without patient self-report, is a clinically critical but largely unaddressed problem, particularly for non-verbal patients. This paper presents a systematic comparison of classical feature engineering and deep sequence learning for subject-independent three-class pain localization using the AI4Pain 2026 Challenge dataset, which comprises four synchronously recorded wearable modalities: electrodermal activity, blood volume pulse, respiration, and peripheral oxygen saturation recorded from 65 participants under controlled TENS-induced pain. A 115-dimensional hand-crafted feature set spanning time-domain, frequency-domain, modality-specific, and cross-modal descriptors is benchmarked against end-to-end deep architectures. Extremely Randomized Trees achieves the highest macro-F1 of 0.539, outperforming the best deep model by 7.4 percentage points, with EDA spectral features emerging as the dominant discriminators. A consistent 26-point gap between pain detection (F1,=,0.815) and localization (F1,=,0.552) across all models points to a fundamental ceiling imposed by the anatomical diffuseness of peripheral autonomic pathways at 10-second resolution.
Ioannis Kyprakis, Stefanos Gkikas, Eric Nichols +2
Hellenic Mediterranean University Heraklion, Greece · Honda Research Institute Japan Wako City, Japan
Pain is a multifaceted and widespread phenomenon with substantial clinical and societal burden, making reliable automated assessment a critical objective. This paper presents a lightweight transformer architecture that fuses multiple fNIRS representations through a unified tokenization mechanism, enabling joint modeling of complementary signal views without requiring modality-specific adaptations or increasing architectural complexity. The proposed token-mixing strategy preserves spatial, temporal, and time-frequency characteristics by projecting heterogeneous inputs onto a shared latent representation, using a structured segmentation scheme to control the granularity of local aggregation and global interaction. The model is evaluated on the AI4Pain dataset using stacked raw waveform and power spectral density representations of fNIRS inputs. Experimental results demonstrate competitive pain recognition performance while remaining computationally compact, making the approach suitable for real-time inference on both GPU and CPU hardware.
Stefanos Gkikas, Christian Arzate Cruz, Yu Fang +6
Honda Research Institute Japan, Wako City, Japan · BioSIS (Biosensing & Intelligent Systems) Lab, Centre for Intelligent Computing and Systems, University of Canberra, Canberra, Australia · Department of Electronic Engineering, Hellenic Mediterranean University, Chania, Greece