Quality and Similarity Indicators for Video-Based Heart Rate and Respiratory Rate Fusion: A Controlled Evaluation
Organizations: Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Oulu, Finland
Abstract
Camera-based heart rate (HR) and respiratory rate (RR) estimates vary in reliability from window to window, and a common remedy is to fuse several extraction methods with weights from reference-free signal quality indicators. However, few studies test whether these indicators identify the most accurate method in each window or outperform simpler fusion. We evaluate 11 quality indicators and 8 inter-method similarity measures for fusing 7 HR and 10 RR extractors on four public datasets (75 test subjects). Filters, indicator directions, weighting parameters and a learned ridge baseline are selected on a separate five-subject set before testing. Fusion is compared with equal averaging, a constant typical-rate predictor, the best single method selected in advance and in hindsight, and a per-window oracle with a permutation-based null, using subject-level bootstrap confidence intervals. For HR, quality-weighted fusion reduced mean absolute error by 10.9% [5.2, 18.4] relative to averaging and by 15.6% [8.1, 23.1] relative to the method selected in advance, and was statistically indistinguishable from the best method selected in hindsight. Weighting methods by their agreement did not improve on averaging. For RR, the indicators could not identify the more accurate estimates: sharpened quality weighting increased error (-2.6% [-4.8, -0.8]), fused estimates were uncorrelated with the reference (r <= 0.22), and averaging was less accurate than the typical-rate predictor (-3.1% [-6.1, -0.4]). With a second respiratory reference, apparent RR gains reflected estimates moving toward its typical rate. Quality-guided fusion is worthwhile for video HR; for RR, 30-s windows improved estimates but not fusion.
Figures & tables
| Set | Subj. | Rec. | fps | HR ref. | RR ref. | HR (bpm) | RR (brpm) |
| Trainset | 5 | 15 | 30 | PPG | belt | 80.5 11.7 | 11.9 4.9 |
| OMS-Test | 6 | 18 | 30 | PPG | belt | 73.7 12.9 | 12.3 5.5 |
| COHFACE | 39 | 160 | 20 | PPG | rate series | 70.3 11.3 | 12.7 3.6 |
| MAHNOB | 25 | 48 | 61 | ECG | belt | 72.2 12.3 | 15.3 5.2 |
| StressID | 5 | 60 | 62 | ECG | belt | 79.8 16.8 | 14.2 5.6 |
| HR | RR | |||||
| Indicator | Definition | Dir. | Dir. | |||
| Zero-crossing rate, | sign changes per sample | – | 0.11 | – | 0.00 | |
| Hjorth mobility, | 0.15 | – | ||||
| Hjorth complexity, | mobility of / mobility of | 0.29 | – | 0.05 | ||
| Signal-to-noise ratio, | mean in-band / mean out-of-band power (dB) | |||||
| Irrelevant power ratio, | out-of-band / total magnitude | – | 0.04 | – | 0.01 | |
| Row | OMS-Test | COHFACE | MAHNOB | StressID | Mean |
| Heart rate (bpm) | |||||
| Baseline (test-chosen) † | 5.46 (0.70) | 7.29 (0.51) | 5.99 (0.62) | 13.07 (0.06) | 7.95 |
| Baseline-train | 6.01 (0.70) | 9.11 (0.41) | 10.91 (0.07) | 13.07 (0.06) | 9.78 |
| Equal mean | 6.75 (0.64) | 7.14 (0.55) | 8.66 (0.38) | 14.63 (0.07) | 9.29 |
| Median | 5.97 (0.67) | 6.87 (0.56) | 8.18 (0.39) | 14.12 (0.16) | 8.79 |
| Constant | 11.73 | 12.26 | 11.29 | 13.13 | 12.10 |
| Comparison | HR | RR |
| Baseline Equal mean | [ , ] | 5.6 [1.9, 9.1] |
| Baseline-train Equal mean | 4.5 [ , 12.0] | 13.5 [10.5, 16.8] |
| Constant Equal mean | 24.0 [14.7, 31.4] | [ , ] |
| Equal mean -AMM ( ) | 4.9 [2.9, 7.6] | 0.2 [ , 0.4] |
| Equal mean -AMM ( ) | 10.9 [5.2, 18.4] | [ , ] |
| Equal mean Trainset- -SMM | 3.9 [1.7, 6.7] | 0.2 [ , 0.5] |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Indicator | Dir. | Subject signs | Label-free | |
| HR | ||||
| – | 0.11 | + + + + | ||
| 0.15 | + + + + + | |||
| 0.29 | + + + + + | |||
| 0.25 | ||||
| – | 0.04 | + + + + | ||
| Group | Parameter | HR | RR |
| Global | Indicator pool | ||
| DN1 method | POS + PF | Mot-S + PF | |
| Constant (median) | 78.95 | 12.52 | |
| ALL | SMM / | (both) | / |
| SMM / | / | / | |
| ( / ) | 0.2 / 0.5 | 0.1 / 0.1 |
| Task | Feature | L-Q | L-Q+rate | Direction | Agrees |
| RR | 0.120 | 0.127 | yes | ||
| 0.038 | 0.035 | yes | |||
| 0.025 | 0.029 | yes | |||
| 0.002 | 0.000 | no | |||
| predicted rate | — | 0.043 | — | — | |
| HR | 0.076 | 0.041 | no |
| Task | Group | Methods | Setting | Largest weight | Effective methods |
| HR | ALL | 7 | 0.21 | 6.25–6.33 | |
| 0.42–0.43 | 2.99–3.09 | ||||
| NLM | 4 | 0.30–0.31 | 3.83–3.88 | ||
| 0.36–0.38 | 3.47–3.61 | ||||
| DLM | 3 | 0.40–0.42 | 2.82–2.88 | ||
| 0.33 | 3.00 |
| Task | Estimator | Trainset | OMS-Test | COHFACE | MAHNOB | StressID |
| HR | FFT | 8.33 | 9.05 | 9.01 | 11.28 | 19.06 |
| Welch | 9.02 | 10.24 | 10.47 | 13.17 | 19.96 | |
| MUSIC | 7.72 | 8.22 | 8.24 | 10.58 | 17.25 | |
| Peak | 9.67 | 9.08 | 9.28 | 10.96 | 16.59 | |
| RR | FFT | 4.69 | 5.20 | 4.91 | 6.58 | 6.10 |
| Welch | 4.92 | 5.40 | 5.06 | 6.63 | 6.20 |
| HR | RR | |||
| Indicator | raw | partial | raw | partial |
| 2 / 0 | 6 / 0 | 3 / 3 | 9 / 0 | |
| 1 / 0 | 9 / 0 | 2 / 6 | 10 / 0 | |
| 0 / 0 | 3 / 3 | 9 / 1 | 0 / 6 | |
| 3 / 0 | 1 / 0 | 6 / 0 | 9 / 0 | |
| 0 / 0 | 0 / 0 | 2 / 0 | 4 / 0 | |
| COHFACE | MAHNOB-HCI | |||||
| Indicator | raw | partial | median partial | raw | partial | median partial |
| 3 / 0 | 9 / 0 | 0.11 | 0 / 3 | 0 / 0 | 0.01 | |
| 2 / 0 | 10 / 0 | 0.19 | 0 / 6 | 0 / 0 | 0.00 | |
| 1 / 1 | 0 / 6 | 0.08 | 8 / 0 | 0 / 0 | 0.03 | |
| 5 / 0 | 9 / 0 | 0.12 | 1 / 0 | 0 / 0 | 0.03 | |
| 2 / 0 | 4 / 0 | 0.05 | 0 / 0 | 0 / 0 | 0.01 | |
| Comparison | Gain [95% CI] |
| Baseline † Equal mean | 6.9 [ 11.1, 3.0] |
| Baseline-train Equal mean | 8.6 [5.5, 11.6] |
| Constant Equal mean | 21.8 [ 27.3, 16.7] |
| Equal mean -AMM ( ) | 2.8 [1.9, 3.6] |
| Equal mean -AMM ( ) | 8.1 [ 1.0, 15.1] |
| Equal mean Trainset- -SMM | 4.5 [2.4, 6.2] |
| 10-s windows | 30-s windows | |
| Trainset-chosen RR estimator | MUSIC | Welch |
| RR indicator pool | , , , | , , |
| Best single method per test set, mean MAE (brpm) | 4.71 | 4.18 |
| Highest single-method on a test set | 0.19 | 0.28 |
| Belt references, spectral vs. peak: | 0.35–0.51 | 0.70–0.82 |
| Belt references, spectral vs. peak: MAE (brpm) | 2.7–4.5 | 1.5–3.1 |
| Dataset | Subjects | ||
| Trainset | 0.10 | 4 / 5 | 0.125 |
| OMS-Test | 0.12 | 6 / 6 | 0.031 |
| COHFACE | 0.16 | 35 / 39 | 0.001 |
| MAHNOB | 0.09 | 17 / 25 | 0.011 |
| StressID | 0.08 | 4 / 5 | 0.125 |
Heart Rate Estimation
Heart rate estimation recovers beats per minute from wearable, contact, or remote physiological signals.
See all 34 papers