Reliability-Aware Weighted Multi-Scale Spatio-Temporal Maps for Heart Rate Monitoring
Authors: Arpan Bairagi, Rakesh Dey, Siladittya Manna, Umapada Pal
Organizations: ⋆CVPR Unit, Indian Statistical Institute, Kolkata, India · † Department of Computational and Data Sciences, Indian Institute of Science, Bengaluru, India
Remote photoplethysmography (rPPG) allows for the contactless estimation of physiological signals from facial videos by analyzing subtle skin color changes. However, rPPG signals are extremely susceptible to illumination changes, motion, shadows, and specular reflections, resulting in low-quality signals in unconstrained environments. To overcome these issues, we present a Reliability-Aware Weighted Multi-Scale Spatio-Temporal (WMST) map that models pixel reliability through the suppression of environmental noises. These noises are modeled using different weighting strategies to focus on more physiologically valid areas. Leveraging the WMST map, we develop an SSL contrastive learning approach based on Swin-Unet, where positive pairs are generated from conventional rPPG signals and temporally expanded WMST maps. Moreover, we introduce a new High-High-High (HHH) wavelet map as a negative example that maintains motion and structural details while filtering out physiological information. Here, our aim is to estimate heart rate (HR), and the experiments on public rPPG benchmarks show that our approach enhances motion and illumination robustness with lower HR estimation error and higher Pearson correlation than existing Self-Supervised Learning (SSL) based rPPG methods.
Figures & tables
Figure 1: Qualitative comparison between the MST map and the proposed WMST map. Darker regions mean lower weights, and brighter regions mean higher weights (Best view by 300% zoom).
Setting
Paired t -test
Wilcoxon signed-rank
WMST vs MST
1.90×10−13
3.18×10−13
Table 1: One-sided statistical significance analysis of SNR on VIPL-HR (WMST > MST).
Figure 2: Distribution of SNR of G channel values across all ROI combinations obtained from all non-empty subsets from the set of six ROIs.
Figure 3: Qualitative comparison of signal quality between the MST map and the proposed WMST map (Best viewed with 300% zoom).
Method
MAE ↓
RMSE ↓
R↑
Contrast-Phys [ 23 ]
17.8
22.8
0.17
SSPD [ 26 ]
6.04
9.56
0.59
Li et al. [ 11 ]
5.19
8.26
0.78
Greip et al. [ 27 ]
7.35
9.70
0.55
RS-rPPG. [ 20 ]
5.98
10.5
0.56
RS+rPPG [ 21 ]
4.40
7.19
0.79
Table 2: Performance comparison on the VIPL-HR dataset with SOTA SSL methods. The best and second best result is denoted by bold and underline, respectively.
Method
MAE ↓
RMSE ↓
R↑
Gideon et al. [ 6 ]
2.3
2.9
0.99
SiNC [ 22 ]
0.59
0.83
0.99
Contrast-phys [ 23 ]
1.00
1.40
0.99
Yue et.al. [ 25 ]
0.58
0.84
0.99
Li et al. [ 11 ]
0.48
0.64
0.99
RS+rPPG [ 21 ]
0.62
0.99
0.99
Table 3: Performance comparison on the UBFC-rPPG dataset with SOTA SSL methods. The best and second best result is denoted by bold and underline, respectively.
Setting
MAE ↓
R↑
RS-rPPG [ 20 ] + WMST map
6.01
0.65
Ours w/o HHH map
4.78
0.80
Ours w/o WMST map
4.55
0.81
Ours with HHH map and WMST map
4.32
0.86
Table 4: Ablation study on the VIPL-HR dataset.
Figure 4: Qualitative comparison of predicted HR predictions between RS+rPPG and the proposed method on the VIPL-HR dataset. We have segmented all of the predictions into four segments for ease of visualization (Best viewed with 300% zoom).
Remote photoplethysmography (rPPG) enables contactless measurement of heart rate and other vital signs by analyzing subtle color variations in facial skin induced by cardiac pulsation. Current rPPG methods are mainly based on either end-to-end modeling from raw videos or intermediate spatial-temporal map (STMap) representations. The former preserves complete spatiotemporal information and can capture subtle heartbeat-related signals, but it also introduces substantial noise from motion artifacts and illumination variations. The latter stacks the temporal color changes of multiple facial regions of interest into compact two-dimensional representations, significantly reducing data volume and computational complexity, although some high-frequency details may be lost. To effectively integrate the mutual strengths, we propose PhysNeXt, a dual-input deep learning framework that jointly exploits video frames and STMap representations. By incorporating a spatio-temporal difference modeling unit, a cross-modal interaction module, and a structured attention-based decoder, PhysNeXt collaboratively enhances the robustness of pulse signal extraction. Experimental results demonstrate that PhysNeXt achieves more stable and fine-grained rPPG signal recovery under challenging conditions, validating the effectiveness of joint modeling of video and STMap representations. The codes will be released.
Junzhe Cao, Bo Zhao, Zhiyi Niu +5
Harbin Institute of Technology, Shenzhen · Great Bay University · Hefei University of Technology +1
Physiological awareness is important for service, social, and assistive robots that interact with humans in everyday environments. Remote photoplethysmography (rPPG) enables non-contact heart-rate (HR) estimation from an RGB camera, making it a promising sensing modality for robot-mounted vision systems. However, illumination variation remains a major barrier to robust deployment. This paper presents an end-to-end spatial-temporal transformer framework for remote HR estimation on a new dataset with varied illumination. Our estimator integrates PRNet-based 3D face alignment, clip-level illumination augmentation, the Residual Temporal Standardization Module, and controlled hybrid temporal-frequency supervision. The training objective combines a Soft-Shifted Pearson waveform loss with a spectral Kullback-Leibler divergence loss, where a tuned weight (β) controls the contribution of frequency-domain heart-rate guidance. Experiments on a static all-level mix protocol covering three illumination levels show that β=5 provides the strongest result among the tested beta settings, achieving a best-run HR mean absolute error (MAE) of 0.79 bpm and an HR correlation of 0.982. Compared with the PhysFormer baseline evaluated on our dataset, our estimator reduces HR MAE by 93.6 %, while increasing HR correlation from 0.088 to 0.982, making it usable when illumination varies.
Zhi Wei Xu, Torbjörn E. M. Nordling
Department of Mechanical Engineering, National Cheng Kung University, Tainan, Taiwan.
Remote photoplethysmography (rPPG) enables non-contact heart rate (HR) monitoring from facial videos, but RGB-only methods are vulnerable to illumination changes, motion artifacts, and skin-tone-dependent optical reflectance. We propose CardiacMamba, a fair and robust RGB-RF fusion framework that integrates optical facial cues and radio-frequency cardiac motion cues through state space modeling. CardiacMamba introduces a Temporal Difference Mamba Module (TDMM) to enhance subtle RF temporal variations, a bidirectional SSM-based interaction mechanism to align heterogeneous RGB-RF dynamics, and a Channel-wise Fast Fourier Transform (CFFT) module for channel-domain spectral refinement. On the EquiPleth dataset, CardiacMamba achieves state-of-the-art performance with 0.96 bpm MAE, 3.06 bpm RMSE, and 0.97 Pearson correlation, while reducing the observed light-dark skin-tone MAE gap to 0.26 bpm and maintaining robustness under RGB degradation and RF-missing conditions