EgoHRV: Continuous Heart Rate Variability Estimation from Egocentric Systems for Autonomic Response and Skill Assessment

Berken Utku Demirel; Christian Holz

cs.CVAug 19, 2026

EgoHRV: Continuous Heart Rate Variability Estimation from Egocentric Systems for Autonomic Response and Skill Assessment

Authors: Berken Utku Demirel, Christian Holz

Organizations: Department of Computer Science, ETH Zürich, Zürich, Switzerland

Abstract

Egocentric vision systems capture human behavior from visible cues, but overlook physiological indicators of autonomic states such as stress, engagement, and attention. Heart rate variability (HRV) is a widely used noninvasive marker of autonomic regulation under stress. HRV reflects small timing differences between successive heartbeats and has so far been out of reach for egocentric platforms, where motion and noise in gaze video mask exactly this fine-grained timing. We propose EgoHRV, a method that estimates HRV as well as heart rate (HR) from the gaze cameras that are already integrated into egocentric headsets. Our pipeline combines a 3D backbone with a novel low--high decomposition module that extracts the blood volume pulse (BVP) signal from gaze video. Our cross-domain pretraining aligns the frequency-domain representations of contact-based and camera-derived signals. This alignment gives EgoHRV the temporal precision to recover HRV from the subtle fluctuations in gaze video. EgoHRV achieves state-of-the-art accuracy for HR and HRV estimation from egocentric video, and its uncertainty-aware design improves downstream behavioral modeling. Integrating our HRV estimates and confidence measures into EgoExo4D's proficiency estimator raises accuracy by 17.8%. Beyond skill, continuous HRV estimation also opens egocentric systems to stress- and arousal-aware estimation tasks. Code: https://github.com/eth-siplab/EgoHRV

Figures & tables

Figure 1: EgoHRV continuously estimates heart rate as well as heart rate variability with uncertainty from gaze cameras in egocentric systems to enrich understanding of human behavior, cognitive effort, stress, and skill assessment.

Figure 2: EgoHRV estimates HR and HRV from egocentric gaze videos. Our method introduces three key components: (1) A low–high decomposition module integrated into the 3D backbone to suppress high-frequency noise and preserve waveform shape. (2) Cross-domain pretraining of blood volume pulse signals using large-scale physiological datasets for learning frequency-domain representations to transfer into BVP signals obtained from cameras. (3) PhysFusion , a multimodal fusion head that combines TimeSformer video embeddings with temporally weighted HRV for proficiency estimation in the EgoExo4D benchmark [ 27 ] .

Panel of Figure 3 — Figure 3: Reliability diagrams for uncertainty estimation with and without IMU input.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Specification
Input	$\mathbf{y}\in\mathbb{R}^{B\times 1\times T}$ (from 3D backbone).
Low-pass	Box filter (AvgPool/Conv) with $k{=}11$ , stride $1$ , pad $5$ : $\ \mathbf{y}_{\mathrm{lo}}=\mathbf{y}*h,\ h=\tfrac{1}{11}\mathbf{1}_{11}$ .
High-pass	$\mathbf{y}_{\mathrm{hi}}=\mathbf{y}-\mathbf{y}_{\mathrm{lo}}$ .
Head lo	Conv1D $1{\rightarrow}16$ ( $k{=}9$ , $p{=}4$ ) $\rightarrow$ GELU $\rightarrow$ Conv1D $16{\rightarrow}1$ ( $k{=}1$ ). Output $\tilde{\mathbf{y}}_{\mathrm{lo}}\in\mathbb{R}^{B\times 1\times T}$ .
Head hi	Conv1D $1{\rightarrow}16$ ( $k{=}5$ , $p{=}2$ ) $\rightarrow$ GELU $\rightarrow$ Conv1D $16{\rightarrow}1$ ( $k{=}1$ ). Output $\tilde{\mathbf{y}}_{\mathrm{hi}}\in\mathbb{R}^{B\times 1\times T}$ .
Fusion gain	Learnable scalar $g$ (initialized to $0.3$ ).

Appendix

Table 5: Low-high decomposition block details.

BVP encoder–decoder ( $f_{\text{BVP}}$ )
Component	Specification
Input	$\mathbf{z}\in\mathbb{R}^{B\times 1\times 128}$ (FFT magnitude of BVP segment).
Downsampling	Four sequential blocks: (1) Conv1D $1{\rightarrow}N$ ( $k{=}3$ , $p{=}1$ ) $\rightarrow$ BN $\rightarrow$ ReLU; (2) Conv1D $N{\rightarrow}2N$ ( $k{=}3$ , $p{=}1$ ) $\rightarrow$ BN $\rightarrow$ ReLU; (3–4) additional down paths with concatenated pooled inputs (kernel $3$ , stride $2$ ).
Upsampling	Three symmetric decoder stages with linear interpolation (scale factor 2) followed by concatenation and Conv1D–BN–ReLU blocks ( $k{=}3$ , $p{=}1$ ).
Output (mean head)	Conv1D $N{\rightarrow}1$ ( $k{=}3$ , $p{=}1$ ) $\rightarrow$ Linear (128 $\rightarrow$ 1) $\Rightarrow\mu$ .
Global pooling	AdaptiveAvgPool1D(1) $\rightarrow$ Linear( $N{\rightarrow}128$ ) to form BVP embedding.

Appendix

Table 6: Architectural details of the cross-domain models used for estimating IBIs. All inputs are Fourier magnitude features.

k

: kernel size,

p

: padding.

Component	Specification
Inputs	Per-timestep HR $h_{t}$ and log-variance $\log\sigma_{t}^{2}$ ; tensors $h,\log\sigma^{2}\in\mathbb{R}^{B\times T}$ . Stack to $[B,T,2]$ .
Projection	Linear $(2\rightarrow 128)$ applied at each time step $\Rightarrow$ $[B,T,128]$ ( $d_{\text{phys}}=128$ ).
Temporal encoder	TransformerEncoder, 2 layers; each layer: $d_{\text{model}}{=}128$ , nhead $=4$ , dim_feedforward $=256$ , dropout $=0.1$ , batch_first $=$ True.
Attention pooling	Learned scores via Linear $(128\rightarrow 1)$ ; masked softmax over $T$ ; weighted sum $\Rightarrow$ physiology vector $v_{\text{phys}}\in\mathbb{R}^{B\times 128}$ .
Fusion	Concatenate ViT CLS ( $\in\mathbb{R}^{B\times\text{embed\_dim}}$ ) with $v_{\text{phys}}$ $\Rightarrow$ $[B,\text{embed\_dim}{+}128]$ .
Head	Linear $(\text{embed\_dim}{+}128\rightarrow\text{num\_classes})$ for prediction.

Appendix

Table 7: The proposed PhysFusion block details (video CLS fused with physiological stream).

Device	Time (ms)	Windows/s	Real-time factor	Peak mem. (MB)
RTX 4090	5.75	173.9	$747.8\times$	74
CPU (10 threads)	447.7	2.23	$9.6\times$	–

Appendix

Table 8: Inference efficiency for a 128-frame (4.3 s) input window. The real-time factor is the input duration divided by inference latency.

Figure 4: Spectral similarity between model outputs and ground-truth is evaluated using kernel density estimates. Our method shows the highest overlap with the true contact PPG frequency distribution, indicating better recovery of relevant frequency components.

Figure 5: Qualitative comparison of reconstructed waveforms. (a) compares our method (blue) against the ground-truth contact BVP (purple) and the ablated variant without low–high decomposition (yellow). (b) compares our method (blue) with PulseFormer [ 4 ] (green) for the same segment. Our approach aligns closely with the ground-truth waveform, whereas PulseFormer and the ablated model exhibit phase misalignment.

Figure 6: Reliability diagram of empirical versus nominal coverage. The model exhibits slightly conservative uncertainty at low confidence and near-perfect calibration at high confidence.

Panel of Figure 7 — Figure 7: Uncertainty correlates with error and supports reliability gating.

Explore similar work

Illumination-Robust Camera-Based Heart-Rate Estimation for Physiological Sensing in Robots
Jun 10, 2026Zhi Wei Xu, Torbjörn E. M. NordlingRemote Photoplethysmography Distribution Shift Robustness
Department of Mechanical Engineering, National Cheng Kung University, Tainan, Taiwan.
PhysAgent: A Multi-Agent Framework for Reliable Remote Heart Rate Estimation
Jul 29, 2026Yehui Yang, Bo Zhao, Junzhe Cao +4Remote Photoplethysmography Heart Rate Estimation
1Great Bay University · 2Shenzhen University of Advanced Technology · 3Macao Polytechnic University +1
EyeTAG: Eye Trajectory-Aware Gaze Estimation
Oct 1, 2026Jungmin Lee, Niamat Ullah, Yoseob HanGaze Estimation
Department of Information and Telecommunication Engineering Soongsil University Seoul, Republic of Korea · Department of Electronic Engineering Soongsil University Seoul, Republic of Korea

Jun 10, 2026J/K move · Enter open · S save

Illumination-Robust Camera-Based Heart-Rate Estimation for Physiological Sensing in Robots

Zhi Wei Xu, Torbjörn E. M. Nordling

Department of Mechanical Engineering, National Cheng Kung University, Tainan, Taiwan.

Remote Photoplethysmography Distribution Shift Robustness Transformer Heart Rate Estimation Photoplethysmography Robotic Perception

Open paper AskPDF arXiv

Physiological awareness is important for service, social, and assistive robots that interact with humans in everyday environments. Remote photoplethysmography (rPPG) enables non-contact heart-rate (HR) estimation from an RGB camera, making it a promising sensing modality for robot-mounted vision systems. However, illumination variation remains a major barrier to robust deployment. This paper presents an end-to-end spatial-temporal transformer framework for remote HR estimation on a new dataset with varied illumination. Our estimator integrates PRNet-based 3D face alignment, clip-level illumination augmentation, the Residual Temporal Standardization Module, and controlled hybrid temporal-frequency supervision. The training objective combines a Soft-Shifted Pearson waveform loss with a spectral Kullback-Leibler divergence loss, where a tuned weight (

\mathbfβ

) controls the contribution of frequency-domain heart-rate guidance. Experiments on a static all-level mix protocol covering three illumination levels show that

\mathbfβ=5

provides the strongest result among the tested beta settings, achieving a best-run HR mean absolute error (MAE) of 0.79 bpm and an HR correlation of 0.982. Compared with the PhysFormer baseline evaluated on our dataset, our estimator reduces HR MAE by 93.6 %, while increasing HR correlation from 0.088 to 0.982, making it usable when illumination varies.

Abstract

Explore similar work

Illumination-Robust Camera-Based Heart-Rate Estimation for Physiological Sensing in Robots

PhysAgent: A Multi-Agent Framework for Reliable Remote Heart Rate Estimation

EyeTAG: Eye Trajectory-Aware Gaze Estimation