Remote photoplethysmography (rPPG) enables non-contact heart rate (HR) monitoring from facial videos, but RGB-only methods are vulnerable to illumination changes, motion artifacts, and skin-tone-dependent optical reflectance. We propose CardiacMamba, a fair and robust RGB-RF fusion framework that integrates optical facial cues and radio-frequency cardiac motion cues through state space modeling. CardiacMamba introduces a Temporal Difference Mamba Module (TDMM) to enhance subtle RF temporal variations, a bidirectional SSM-based interaction mechanism to align heterogeneous RGB-RF dynamics, and a Channel-wise Fast Fourier Transform (CFFT) module for channel-domain spectral refinement. On the EquiPleth dataset, CardiacMamba achieves state-of-the-art performance with 0.96 bpm MAE, 3.06 bpm RMSE, and 0.97 Pearson correlation, while reducing the observed light-dark skin-tone MAE gap to 0.26 bpm and maintaining robustness under RGB degradation and RF-missing conditions
Figures & tables
Figure 1: The overall architecture of CardiacMamba. It consists of three stages: Dual-level Feature Extraction and Alignment, Bidirectional Feature Interaction, and Bidirectional Feature Fusion.
Figure 2: Channel-wise Fast Fourier Transform (CFFT) for refining RGB and RF representations through channel-domain spectral interaction.
Table 1: Comparison of different methods on the EquiPleth dataset. The best results are marked in bold .
Method
Train
Test
MAE
RMSE
Base
RGB&RF
RGB
21.7
25.7
Base
RGB&RF
RF
21.4
24.3
Base
RGB&RF
RGB&RF
20.3
24.8
Vilesov et al. [ 7 ]
RGB&RF
RGB
6.82
13.32
Vilesov et al. [ 7 ]
RGB&RF
RF
7.25
9.62
Vilesov et al. [ 7 ]
RGB&RF
RGB&RF
1.12
3.42
Table 2: Testing under missing-modality conditions.
Vim
CFFT
SSM
RFAM
TDMM
MAE
RMSE
ρ
×
✓
✓
✓
✓
1.7
4.9
0.91
✓
×
✓
✓
✓
4.92
6.33
0.77
✓
✓
×
✓
✓
1.86
5.51
0.89
✓
✓
✓
×
✓
1.85
5.31
0.9
✓
✓
✓
✓
×
3.82
8.06
0.81
✓
✓
✓
✓
✓
0.96
3.06
0.97
Table 3: Ablation on different modules, including ‘Vim’ (short for Vision Mamba), ‘CFFT’ (short for Channel-wise Fast Fourier Transform), ‘RFAM’ (short for RF Alignment Module), and ‘TDMM’ (short for Temporal Difference Mamba Module).
Model
Condition
MAE
RMSE
ρ
Ours (RGB-Only)
Normal
1.20
4.23
0.95
+ Gaussian Noise
8.54
15.12
0.41
Ours (Full model)
Normal
0.96
3.06
0.97
+ Gaussian Noise
2.15
5.80
0.88
Table 4: Ablation study on model robustness under simulated RGB signal degradation. We added Gaussian noise to the RGB input to test performance. Best results under noisy conditions are in bold .
Figure 4: Visual representations of features (from left to right): human face, feature heat map, radar spectrum diagram, and its feature heat map.
Figure 5: Bland-Altman plots comparing estimated and ground-truth heart rates.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Method
Input
Δ MAE
Δ RMSE
Δρ
ICA [ 18 ]
RGB
4.42
3.15
-0.36
CHROM [ 1 ]
RGB
4.97
4.17
-0.38
BCG [ 19 ]
RGB
0.99
1.25
0.05
PhysNet [ 2 ]
RGB
2.22
4.05
-0.25
FFT-based RF [ 3 ]
RF
1.32
2.06
0.32
Vilesov et al. [ 7 ]
RGB&RF
0.67
1.44
-0.10
Appendix
Table 5: Comparison of the Fairness of various methods on the RGB and RF fusion task, with the differences in performance ( Δ ) between light and dark skin tones in terms of MAE, RMSE, and correlation coefficient ( ρ ).
Physiological awareness is important for service, social, and assistive robots that interact with humans in everyday environments. Remote photoplethysmography (rPPG) enables non-contact heart-rate (HR) estimation from an RGB camera, making it a promising sensing modality for robot-mounted vision systems. However, illumination variation remains a major barrier to robust deployment. This paper presents an end-to-end spatial-temporal transformer framework for remote HR estimation on a new dataset with varied illumination. Our estimator integrates PRNet-based 3D face alignment, clip-level illumination augmentation, the Residual Temporal Standardization Module, and controlled hybrid temporal-frequency supervision. The training objective combines a Soft-Shifted Pearson waveform loss with a spectral Kullback-Leibler divergence loss, where a tuned weight (β) controls the contribution of frequency-domain heart-rate guidance. Experiments on a static all-level mix protocol covering three illumination levels show that β=5 provides the strongest result among the tested beta settings, achieving a best-run HR mean absolute error (MAE) of 0.79 bpm and an HR correlation of 0.982. Compared with the PhysFormer baseline evaluated on our dataset, our estimator reduces HR MAE by 93.6 %, while increasing HR correlation from 0.088 to 0.982, making it usable when illumination varies.
Zhi Wei Xu, Torbjörn E. M. Nordling
Department of Mechanical Engineering, National Cheng Kung University, Tainan, Taiwan.
Remote photoplethysmography (rPPG) enables non-contact heart-rate estimation from facial videos, but its weak physiological signal is easily corrupted by motion, illumination changes, occlusion, skin-appearance variation, and device noise. Existing rPPG methods typically rely on a single model to directly predict heart rate or recover pulse waveforms, while different strong estimators may produce conflicting yet individually plausible candidates for the same video. To resolve these conflicts, we propose PhysAgent, an inference-time multi-agent candidate-verification framework. Unlike direct prediction approaches, PhysAgent neither trains a new base rPPG model nor asks Multimodal Large Language Models (MLLMs) to output heart rate directly. In contrast, it treats outputs from multiple base estimators as physiological hypotheses to be verified and uses a lightweight 4B MLLM, Qwen3-VL-4B, to drive multi-agent reasoning over video conditions, signal reliability, and candidate disagreement. A deterministic physiological verifier checks the fusion proposal, and a reproducible numerical fusion process produces the final heart rate. Experimental results on multiple public rPPG benchmarks show that PhysAgent improves fusion stability and reliability across different datasets and source-domain settings, while avoiding the irreproducibility and physiological inconsistency of direct MLLM prediction or unconstrained ensemble fusion. The code will be released soon.
Yehui Yang, Bo Zhao, Junzhe Cao +4
1Great Bay University · 2Shenzhen University of Advanced Technology · 3Macao Polytechnic University +1
Remote photoplethysmography (rPPG) estimates physiological signals from facial videos by analyzing subtle pulse induced skin color variations. Despite recent progress, existing self-supervised rPPG methods mainly reconstruct masked pixels or low-level visual representations, which can bias the model toward facial appearance rather than latent physiological dy namics. Moreover, most recent Mamba-based approaches scan facial video tokens only in chronological order, limiting their ability to exploit the cyclic structure of pulse signals. To ad dress these limitations, we propose RhythmJEPA, a rhythm structured joint-embedding predictive learning framework for rPPG. Instead of reconstructing RGB frames, RhythmJEPA predicts latent teacher representations from masked facial videos, thereby encouraging physiology-aware representation learning in the embedding space. To explicitly model pulse-related tem poral structure, we introduce a Cyclic Rhythm-State Plan ner (CRSP), which estimates frame-wise latent physiological states and decodes the most plausible cyclic state path via dynamic programming with a constrained transition grammar. Guided by the decoded states, we further design a Dual Order Mamba Encoder (DOM), which combines conventional chronological scanning with state-ordered scanning to capture both local temporal continuity and long-range rhythm-consistent dependencies. Finally, a lightweight Spatial Pulse Mixer (SPM) extracts compact pulse-sensitive facial tokens with a favorable balance between complexity and performance. Experiments on PURE, UBFC-rPPG, and MMPD show competitive performance over representative rPPG methods. The codes are available at https://github.com/deconasser/RhythmJEPA.