Good Practice Guide for quantifying uncertainties for machine learning models applied to photoplethysmography signals
Authors: P. Harris, C. Bench, M. Rinkevičius, V. Marozas, L. Coquelin, A. Thompson, M. Nandi, U. Hackstein, +1 more
Organizations: PTB Physikalisch-Technische Bundesanstalt · NPL National Physical Laboratory · CMI Cesky Metrologicky Institut · IMBiH Institut za mjeriteljstvo Bosne i Hercegovine · IPQ Instituto Português da Qualidade, I.P. · LNE Laboratoire National de métrologie et d'Essais · FC Qompium NV · FVB Forschungsverbund Berlin e.V.
Abstract
This Good Practice Guide presents work done in the QUMPHY project (Uncertainty quantification for machine learning models applied to photoplethysmography signals) that considered both machine learning and uncertainty quantification for problems which used photoplethysmography (PPG) signals from wearable devices as input. It provides high-level guidance on what types of machine learning model might be used and how different models compare when applied to both regression and classification tasks. It provides guidance on the implementation of different methods for uncertainty quantification, covering both model-dependent and model-independent techniques, and on the validation of the results provided by those methods. It also describes six benchmark problems together with pointers to different benchmark datasets for each problem. Software is described that can assist practitioners in implementing the methods described herein and there is a brief consideration of ethical issues. It concludes with a summary and recommendations.
Purpose: Uncertainty Quantification (UQ) has gained traction in an attempt to improve the interpretability and robustness of machine learning predictions. Specifically (medical) biosignals such as electroencephalography (EEG), electrocardiography (ECG), electrooculography (EOG), and electromyography (EMG) could benefit from good UQ, since these suffer from a poor signal-to-noise ratio, and good human interpretability is pivotal for medical applications. To determine how uncertainty estimation can be used for biosignal tasks, we investigate current methods, use cases, applications, evaluations, and uncertainty measures. Methods: In this paper, we systematically review the state of the art of applying Uncertainty Quantification to Machine Learning tasks in the biosignal domain. All works from Web of Science, Scopus, IEEE XPlore and PsycINFO that discuss uncertainty in Machine Learning on one of the aforementioned biosignals is included. Results: We present various methods, shortcomings, uncertainty measures and theoretical frameworks that currently exist in this application domain based on the 53 reviewed papers and related literature. We address misconceptions in the field, provide recommendations for future work, and discuss gaps in the literature in relation to diagnostic implementations as well as control for prostheses or brain-computer interfaces. Conclusion: Overall it can be concluded that promising UQ methods are available, but that research is needed on how people and systems may interact with an uncertainty-model in a (clinical) environment.
Ivo Pascal de Jong, Andreea Ioana Sburlea, Matias Valdenegro-Toro
Uncertainty quantification (UQ) is critical for safety-critical domains like healthcare, yet it is rarely evaluated under realistic out-of-distribution (OOD) conditions. Here, we assessed predictive performance and uncertainty reliability for deep learning-based blood pressure (BP) estimation from photoplethysmography (PPG) signals under both in-distribution (ID) and OOD settings. Using an XResNet1D-50 trained on PulseDB and tested on four external datasets, we compared deep ensembles (DE) and Monte Carlo dropout (MCD) with Gaussian negative log-likelihood (GNLL) and mean squared error (MSE) losses, optionally followed by post-hoc recalibration via conformal prediction (CP), temperature scaling (TS), and isotonic regression (IR). The key findings of our study are as follows: (1) DE provides stronger predictive robustness under domain shift than MCD, an advantage that becomes clear primarily under external shift. (2) Recalibrated GNLL-based methods yield the best uncertainty calibration (e.g., GNLL+DE+CP for systolic blood pressure (SBP), GNLL+DE+TS for diastolic blood pressure (DBP)), while MSE-based uncertainty requires recalibration to become practically useful. (3) Across settings, CP and TS offer the most consistent gains, with IR remaining competitive in several cases. Overall, our results identify DE-based methods as most robust for predictive performance under domain shift, GNLL as strongest for native UQ, and recalibration as essential for making MSE-based uncertainty practical. These findings highlight the need to jointly assess predictive accuracy and calibration on external data for trustworthy cuffless BP estimation
Mohammad Moulaeifard, Ciaran Bench, Philip J. Aston +1
In principle, deep generative models can be used to perform domain adaptation; i.e. align the input feature representations of test data with that of a separate discriminative model's training data. This can help improve the discriminative model's performance on the test data. However, generative models are prone to producing hallucinations and artefacts that may degrade the quality of generated data, and therefore, predictive performance when processed by the discriminative model. While uncertainty quantification can provide a means to assess the quality of adapted data, the standard framework for evaluating the quality of predicted uncertainties may not easily extend to generative models due to the common lack of ground truths (among other reasons). Even with ground truths, this evaluation is agnostic to how the generated outputs are used on the downstream task, limiting the extent to which the uncertainty reliability analysis provides insights about the utility of the uncertainties with respect to the intended use case of the adapted examples. Here, we describe how decision-theoretic uncertainty quantification can address these concerns and provide a convenient framework for evaluating the trustworthiness of generated outputs, in particular, for domain adaptation. We consider a case study in photoplethysmography time series denoising for Atrial Fibrillation classification. This formalises a well-known heuristic method of using a downstream classifier to assess the quality of generated outputs.