Camera-based heart rate (HR) and respiratory rate (RR) estimates vary in reliability from window to window, and a common remedy is to fuse several extraction methods with weights from reference-free signal quality indicators. However, few studies test whether these indicators identify the most accurate method in each window or outperform simpler fusion. We evaluate 11 quality indicators and 8 inter-method similarity measures for fusing 7 HR and 10 RR extractors on four public datasets (75 test subjects). Filters, indicator directions, weighting parameters and a learned ridge baseline are selected on a separate five-subject set before testing. Fusion is compared with equal averaging, a constant typical-rate predictor, the best single method selected in advance and in hindsight, and a per-window oracle with a permutation-based null, using subject-level bootstrap confidence intervals. For HR, quality-weighted fusion reduced mean absolute error by 10.9% [5.2, 18.4] relative to averaging and by 15.6% [8.1, 23.1] relative to the method selected in advance, and was statistically indistinguishable from the best method selected in hindsight. Weighting methods by their agreement did not improve on averaging. For RR, the indicators could not identify the more accurate estimates: sharpened quality weighting increased error (-2.6% [-4.8, -0.8]), fused estimates were uncorrelated with the reference (r <= 0.22), and averaging was less accurate than the typical-rate predictor (-3.1% [-6.1, -0.4]). With a second respiratory reference, apparent RR gains reflected estimates moving toward its typical rate. Quality-guided fusion is worthwhile for video HR; for RR, 30-s windows improved estimates but not fusion.
Figures & tables
Figure 1: Quality and similarity framework. Candidate HR and RR signals are extracted by classical, motion-based and deep learning methods, filtered over the full recording, converted to rates in 10-s windows by MUSIC, scored by quality indicators, inter-method similarity or a learned ridge score, and fused or filtered. Steps in the dashed box are fixed on the five-subject Trainset before any test set is evaluated. Fusion is evaluated against a common reference and against averaging, a constant predictor and a null oracle.
Set
Subj.
Rec.
fps
HR ref.
RR ref.
HR (bpm)
RR (brpm)
Trainset
5
15
30
PPG
belt
80.5 ± 11.7
11.9 ± 4.9
OMS-Test
6
18
30
PPG
belt
73.7 ± 12.9
12.3 ± 5.5
COHFACE
39
160
20
PPG
rate series
70.3 ± 11.3
12.7 ± 3.6
MAHNOB
25
48
61
ECG
belt
72.2 ± 12.3
15.3 ± 5.2
StressID
5
60
62
ECG
belt
79.8 ± 16.8
14.2 ± 5.6
Table 1: Datasets. Windows are 10 s with a 1-s step. Reference rates are mean ± SD over windows.
HR
RR
Indicator
Definition
Dir.
ρ×
Dir.
ρ×
Zero-crossing rate, QZCR
sign changes per sample
–
0.11
–
0.00
Hjorth mobility, QHJ-M
Var(Δx)/Var(x)
↓
0.15
–
−0.01
Hjorth complexity, QHJ-C
mobility of Δx / mobility of x
↓
0.29
–
0.05
Signal-to-noise ratio, QSNR
mean in-band / mean out-of-band power (dB)
↑
−0.25
↑
−0.12
Irrelevant power ratio, QIPR
out-of-band / total magnitude
–
0.04
–
0.01
Table 2: Quality indicators and the directions inferred on the Trainset ( ↓ lower is better, ↑ higher is better, – dropped because the sign was not the same in all five Trainset subjects). ρ× is the cross-method rank correlation with absolute error on the Trainset (raw sign; negative means higher values go with lower error).
Figure 2: Single-method MAE (top rows) and Pearson r (bottom rows) for each method and rate estimator. The Trainset-chosen estimator (MUSIC, bold) is used for everything that follows; red labels mark the Trainset DN1 pair (HR: POS + PF; RR: Mot-S + PF).
Figure 3: HR bias (top) and MAE (bottom) by reference HR for the three deep models and OMIT. All methods over-estimate low and under-estimate high heart rates.
Figure 4: Filtering: MAE averaged over methods as the fraction of removed windows grows, removing windows by AMM score, at random, or by true error (oracle).
Figure 5: Median within-method ρ (horizontal) against the median partial ρ given the predicted rate (vertical), one point per indicator, task and dataset, including the Trainset. Points on the diagonal carry information beyond the rate; points pulled toward the horizontal axis carry it mainly through the rate.
Figure 6: Spearman ρ between indicator and absolute error, within methods (circles) and across methods in the same window (squares), median over subjects. Pool indicators are oriented by their Trainset direction so that ρ>0 means useful; dropped indicators are shown with the lower-is-better orientation. ± marks indicators whose within- and across-method ρ differ in sign.
Row
OMS-Test
COHFACE
MAHNOB
StressID
Mean
Heart rate (bpm)
Baseline (test-chosen) †
5.46 (0.70)
7.29 (0.51)
5.99 (0.62)
13.07 (0.06)
7.95
Baseline-train
6.01 (0.70)
9.11 (0.41)
10.91 (0.07)
13.07 (0.06)
9.78
Equal mean
6.75 (0.64)
7.14 (0.55)
8.66 (0.38)
14.63 (0.07)
9.29
Median
5.97 (0.67)
6.87 (0.56)
8.18 (0.39)
14.12 (0.16)
8.79
Constant
11.73
12.26
11.29
13.13
12.10
Table 3: Fusion and controls, ALL group: MAE with Pearson r in parentheses. Mean is the unweighted mean over datasets. † Uses test labels (diagnostic only). QS / QH : quality-based soft/hard fusion; SS / SH : similarity-based soft/hard fusion; L-Q: learned score; τ∗ : Trainset-tuned temperature. r is undefined for constant predictions.
Comparison
HR
RR
Baseline → Equal mean
−19.5 [ −33.4 , −10.8 ]
5.6 [1.9, 9.1]
Baseline-train → Equal mean
4.5 [ −8.8 , 12.0]
13.5 [10.5, 16.8]
Constant → Equal mean
24.0 [14.7, 31.4]
−3.1 [ −6.1 , −0.4 ]
Equal mean →QS -AMM ( τ=1 )
4.9 [2.9, 7.6]
0.2 [ −0.1 , 0.4]
Equal mean →QS -AMM ( τ∗ )
10.9 [5.2, 18.4]
−2.6 [ −4.8 , −0.8 ]
Equal mean → Trainset- QS -SMM
3.9 [1.7, 6.7]
0.2 [ −0.1 , 0.5]
Table 4: Pooled bootstrap gains (75 subjects, four datasets, equal dataset weight). Positive values favour the second configuration. 95% percentile confidence intervals in brackets.
Figure 7: Pooled bootstrap gains (Table 4 ) with 95% confidence intervals; right of zero favours the second configuration of each pair (A → B). Coloured intervals exclude zero. † Chosen with test labels.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Indicator
Dir.
ρ×
Subject signs
Label-free
HR
QZCR
–
0.11
+ + − + +
↓
QHJ-M
↓
0.15
+ + + + +
↓
QHJ-C
↓
0.29
+ + + + +
↓
QSNR
↑
− 0.25
−−−−−
↑
QIPR
–
0.04
+ + + + −
↓
Appendix
Table S3.1 : Indicator directions inferred on the Trainset.
Group
Parameter
HR
RR
Global
Indicator pool
QHJ-M,QHJ-C,QSNR,QTMCC,QHFD
QSNR,QKURT,QSKEW,QHFD
DN1 method
POS + PF
Mot-S + PF
Constant (median)
78.95
12.52
ALL
SMM QH / QS
QHJ-C,QTMCC (both)
QSNR,QKURT,QHFD / QSNR,QSKEW
SMM SH / SS
SNCC,SPFD / SPFD,SBPC
SPFD,SBPC / SNCC,SPFD
τ∗ ( Q / S )
0.2 / 0.5
0.1 / 0.1
Appendix
Table S3.2 : Choices fixed on the Trainset.
Task
Feature
L-Q
L-Q+rate
Direction
Agrees
RR
QSNR
− 0.120
− 0.127
↑
yes
QKURT
− 0.038
− 0.035
↑
yes
QSKEW
0.025
0.029
↓
yes
QHFD
− 0.002
0.000
↓
no
predicted rate
—
0.043
—
—
HR
QHJ-M
− 0.076
− 0.041
↓
no
Appendix
Table S3.3 : Learned ridge coefficients.
Figure S4.1 : Trainset MAE of quality-weighted soft fusion against the temperature τ , per group. Dotted lines: equal mean ( τ=∞ ); the minimum defines τ∗ .
Task
Group
Methods
Setting
Largest weight
Effective methods
HR
ALL
7
τ=1
0.21
6.25–6.33
τ∗=0.20
0.42–0.43
2.99–3.09
NLM
4
τ=1
0.30–0.31
3.83–3.88
τ∗=0.50
0.36–0.38
3.47–3.61
DLM
3
τ=1
0.40–0.42
2.82–2.88
τ∗=∞
0.33
3.00
Appendix
Table S4.1 : Weight sharpness of quality-weighted soft fusion.
Task
Estimator
Trainset
OMS-Test
COHFACE
MAHNOB
StressID
HR
FFT
8.33
9.05
9.01
11.28
19.06
Welch
9.02
10.24
10.47
13.17
19.96
MUSIC
7.72
8.22
8.24
10.58
17.25
Peak
9.67
9.08
9.28
10.96
16.59
RR
FFT
4.69
5.20
4.91
6.58
6.10
Welch
4.92
5.40
5.06
6.63
6.20
Appendix
Table S5.3 : Mean single-method MAE per rate estimator.
HR
RR
Indicator
raw
partial
raw
partial
QZCR
2 / 0
6 / 0
3 / 3
9 / 0
QHJ-M
1 / 0
9 / 0
2 / 6
10 / 0
QHJ-C
0 / 0
3 / 3
9 / 1
0 / 6
QSNR
3 / 0
1 / 0
6 / 0
9 / 0
QIPR
0 / 0
0 / 0
2 / 0
4 / 0
Appendix
Table S6.3 : Significant within-method correlations after Benjamini–Hochberg correction.
COHFACE
MAHNOB-HCI
Indicator
raw
partial
median partial ρ
raw
partial
median partial ρ
QZCR
3 / 0
9 / 0
0.11
0 / 3
0 / 0
− 0.01
QHJ-M
2 / 0
10 / 0
0.19
0 / 6
0 / 0
0.00
QHJ-C
1 / 1
0 / 6
− 0.08
8 / 0
0 / 0
0.03
QSNR
5 / 0
9 / 0
0.12
1 / 0
0 / 0
0.03
QIPR
2 / 0
4 / 0
0.05
0 / 0
0 / 0
0.01
Appendix
Table S6.4 : RR significance counts by dataset.
Comparison
Gain [95% CI]
Baseline † → Equal mean
− 6.9 [ − 11.1, − 3.0]
Baseline-train → Equal mean
8.6 [5.5, 11.6]
Constant → Equal mean
− 21.8 [ − 27.3, − 16.7]
Equal mean →QS -AMM ( τ=1 )
2.8 [1.9, 3.6]
Equal mean →QS -AMM ( τ∗ )
8.1 [ − 1.0, 15.1]
Equal mean → Trainset- QS -SMM
4.5 [2.4, 6.2]
Appendix
Table S9.2 : Pooled bootstrap gains (%) under the peak RR reference (ALL group, 75 subjects); positive favours the second configuration.
10-s windows
30-s windows
Trainset-chosen RR estimator
MUSIC
Welch
RR indicator pool
QSNR , QKURT , QSKEW , QHFD
QKURT , QSKEW , QPerI
Best single method per test set, mean MAE (brpm)
4.71
4.18
Highest single-method r on a test set
0.19
0.28
Belt references, spectral vs. peak: r
0.35–0.51
0.70–0.82
Belt references, spectral vs. peak: MAE (brpm)
2.7–4.5
1.5–3.1
Appendix
Table S10.1 : RR in 10-s and 30-s windows. Gains are pooled bootstrap gains (%, 95% CI); positive favours the second configuration.
Dataset
ρ×
Subjects ρ>0
p
Trainset
0.10
4 / 5
0.125
OMS-Test
0.12
6 / 6
0.031
COHFACE
0.16
35 / 39
< 0.001
MAHNOB
0.09
17 / 25
0.011
StressID
0.08
4 / 5
0.125
Appendix
Table S10.3 : Cross-method ρ between QSNR and absolute error with 30-s windows.
Remote photoplethysmography (rPPG) enables non-contact heart rate (HR) monitoring from facial videos, but RGB-only methods are vulnerable to illumination changes, motion artifacts, and skin-tone-dependent optical reflectance. We propose CardiacMamba, a fair and robust RGB-RF fusion framework that integrates optical facial cues and radio-frequency cardiac motion cues through state space modeling. CardiacMamba introduces a Temporal Difference Mamba Module (TDMM) to enhance subtle RF temporal variations, a bidirectional SSM-based interaction mechanism to align heterogeneous RGB-RF dynamics, and a Channel-wise Fast Fourier Transform (CFFT) module for channel-domain spectral refinement. On the EquiPleth dataset, CardiacMamba achieves state-of-the-art performance with 0.96 bpm MAE, 3.06 bpm RMSE, and 0.97 Pearson correlation, while reducing the observed light-dark skin-tone MAE gap to 0.26 bpm and maintaining robustness under RGB degradation and RF-missing conditions
Remote photoplethysmography (rPPG) enables non-contact heart-rate estimation from facial videos, but its weak physiological signal is easily corrupted by motion, illumination changes, occlusion, skin-appearance variation, and device noise. Existing rPPG methods typically rely on a single model to directly predict heart rate or recover pulse waveforms, while different strong estimators may produce conflicting yet individually plausible candidates for the same video. To resolve these conflicts, we propose PhysAgent, an inference-time multi-agent candidate-verification framework. Unlike direct prediction approaches, PhysAgent neither trains a new base rPPG model nor asks Multimodal Large Language Models (MLLMs) to output heart rate directly. In contrast, it treats outputs from multiple base estimators as physiological hypotheses to be verified and uses a lightweight 4B MLLM, Qwen3-VL-4B, to drive multi-agent reasoning over video conditions, signal reliability, and candidate disagreement. A deterministic physiological verifier checks the fusion proposal, and a reproducible numerical fusion process produces the final heart rate. Experimental results on multiple public rPPG benchmarks show that PhysAgent improves fusion stability and reliability across different datasets and source-domain settings, while avoiding the irreproducibility and physiological inconsistency of direct MLLM prediction or unconstrained ensemble fusion. The code will be released soon.
Yehui Yang, Bo Zhao, Junzhe Cao +4
1Great Bay University · 2Shenzhen University of Advanced Technology · 3Macao Polytechnic University +1
Wearable electrocardiogram (ECG) and photoplethysmogram (PPG) sensors are complementary but individually fragile: motion artifact, poor contact, and sensor dropout can degrade one or both signals. Fusion strategies that assume both modalities are equally trustworthy can become less reliable than a single clean modality under degradation. We present CardioFusion-AI, a framework whose signal-processing front end, including R-peak and systolic-peak detection, an Orphanidou-type signal-quality index, and beat-by-beat pulse transit time estimation, is validated on 53 real intensive-care recordings (848 windows; heart-rate mean absolute error 1.61 bpm for ECG and 2.78 bpm for PPG) and a real annotated fetal ECG database (R-peak F1 0.89-0.98). We then conduct a controlled synthetic degradation study comparing eight ECG-PPG fusion strategies across six degradation regimes spanning graded corruption and complete modality loss, using five independent training seeds. Attention fusion achieved the lowest descriptive overall error (1.66+/-0.43 bpm). Both adaptive gates reallocated weight toward the healthy modality under complete modality loss, but showed near-zero correlation between gate weight and signal quality under graded degradation (r = 0.10-0.24). Signal-quality conditioning produced a specific improvement under missing-PPG conditions (1.56+/-0.59 bpm), approaching the 1.48 bpm unimodal ceiling. With only five training seeds, no pairwise comparison survives Holm-corrected significance testing; effect sizes and confidence intervals are therefore reported. These results indicate that modality availability and modality quality are functionally distinct problems for adaptive fusion.