Egocentric vision systems capture human behavior from visible cues, but overlook physiological indicators of autonomic states such as stress, engagement, and attention. Heart rate variability (HRV) is a widely used noninvasive marker of autonomic regulation under stress. HRV reflects small timing differences between successive heartbeats and has so far been out of reach for egocentric platforms, where motion and noise in gaze video mask exactly this fine-grained timing. We propose EgoHRV, a method that estimates HRV as well as heart rate (HR) from the gaze cameras that are already integrated into egocentric headsets. Our pipeline combines a 3D backbone with a novel low--high decomposition module that extracts the blood volume pulse (BVP) signal from gaze video. Our cross-domain pretraining aligns the frequency-domain representations of contact-based and camera-derived signals. This alignment gives EgoHRV the temporal precision to recover HRV from the subtle fluctuations in gaze video. EgoHRV achieves state-of-the-art accuracy for HR and HRV estimation from egocentric video, and its uncertainty-aware design improves downstream behavioral modeling. Integrating our HRV estimates and confidence measures into EgoExo4D's proficiency estimator raises accuracy by 17.8%. Beyond skill, continuous HRV estimation also opens egocentric systems to stress- and arousal-aware estimation tasks. Code: https://github.com/eth-siplab/EgoHRV
Figures & tables
Figure 1: EgoHRV continuously estimates heart rate as well as heart rate variability with uncertainty from gaze cameras in egocentric systems to enrich understanding of human behavior, cognitive effort, stress, and skill assessment.
Figure 2: EgoHRV estimates HR and HRV from egocentric gaze videos. Our method introduces three key components: (1) A low–high decomposition module integrated into the 3D backbone to suppress high-frequency noise and preserve waveform shape. (2) Cross-domain pretraining of blood volume pulse signals using large-scale physiological datasets for learning frequency-domain representations to transfer into BVP signals obtained from cameras. (3) PhysFusion , a multimodal fusion head that combines TimeSformer video embeddings with temporally weighted HRV for proficiency estimation in the EgoExo4D benchmark [ 27 ] .
Figure 3: Reliability diagrams for uncertainty estimation with and without IMU input.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Specification
Input
y∈RB×1×T (from 3D backbone).
Low-pass
Box filter (AvgPool/Conv) with k=11 , stride 1 , pad 5 : ylo=y∗h,h=111111 .
Three symmetric decoder stages with linear interpolation (scale factor 2) followed by concatenation and Conv1D–BN–ReLU blocks ( k=3 , p=1 ).
Output (mean head)
Conv1D N→1 ( k=3 , p=1 ) → Linear (128 → 1) ⇒μ .
Global pooling
AdaptiveAvgPool1D(1) → Linear( N→128 ) to form BVP embedding.
Appendix
Table 6: Architectural details of the cross-domain models used for estimating IBIs. All inputs are Fourier magnitude features. k : kernel size, p : padding.
Component
Specification
Inputs
Per-timestep HR ht and log-variance logσt2 ; tensors h,logσ2∈RB×T . Stack to [B,T,2] .
Projection
Linear (2→128) applied at each time step ⇒[B,T,128] ( dphys=128 ).
Learned scores via Linear (128→1) ; masked softmax over T ; weighted sum ⇒ physiology vector vphys∈RB×128 .
Fusion
Concatenate ViT CLS ( ∈RB×embed_dim ) with vphys⇒[B,embed_dim+128] .
Head
Linear (embed_dim+128→num_classes) for prediction.
Appendix
Table 7: The proposed PhysFusion block details (video CLS fused with physiological stream).
Device
Time (ms)
Windows/s
Real-time factor
Peak mem. (MB)
RTX 4090
5.75
173.9
747.8×
74
CPU (10 threads)
447.7
2.23
9.6×
–
Appendix
Table 8: Inference efficiency for a 128-frame (4.3 s) input window. The real-time factor is the input duration divided by inference latency.
Figure 4: Spectral similarity between model outputs and ground-truth is evaluated using kernel density estimates. Our method shows the highest overlap with the true contact PPG frequency distribution, indicating better recovery of relevant frequency components.
Figure 5: Qualitative comparison of reconstructed waveforms. (a) compares our method (blue) against the ground-truth contact BVP (purple) and the ablated variant without low–high decomposition (yellow). (b) compares our method (blue) with PulseFormer [ 4 ] (green) for the same segment. Our approach aligns closely with the ground-truth waveform, whereas PulseFormer and the ablated model exhibit phase misalignment.
Figure 6: Reliability diagram of empirical versus nominal coverage. The model exhibits slightly conservative uncertainty at low confidence and near-perfect calibration at high confidence.
Figure 7: Uncertainty correlates with error and supports reliability gating.
Physiological awareness is important for service, social, and assistive robots that interact with humans in everyday environments. Remote photoplethysmography (rPPG) enables non-contact heart-rate (HR) estimation from an RGB camera, making it a promising sensing modality for robot-mounted vision systems. However, illumination variation remains a major barrier to robust deployment. This paper presents an end-to-end spatial-temporal transformer framework for remote HR estimation on a new dataset with varied illumination. Our estimator integrates PRNet-based 3D face alignment, clip-level illumination augmentation, the Residual Temporal Standardization Module, and controlled hybrid temporal-frequency supervision. The training objective combines a Soft-Shifted Pearson waveform loss with a spectral Kullback-Leibler divergence loss, where a tuned weight (β) controls the contribution of frequency-domain heart-rate guidance. Experiments on a static all-level mix protocol covering three illumination levels show that β=5 provides the strongest result among the tested beta settings, achieving a best-run HR mean absolute error (MAE) of 0.79 bpm and an HR correlation of 0.982. Compared with the PhysFormer baseline evaluated on our dataset, our estimator reduces HR MAE by 93.6 %, while increasing HR correlation from 0.088 to 0.982, making it usable when illumination varies.
Zhi Wei Xu, Torbjörn E. M. Nordling
Department of Mechanical Engineering, National Cheng Kung University, Tainan, Taiwan.
Remote photoplethysmography (rPPG) enables non-contact heart-rate estimation from facial videos, but its weak physiological signal is easily corrupted by motion, illumination changes, occlusion, skin-appearance variation, and device noise. Existing rPPG methods typically rely on a single model to directly predict heart rate or recover pulse waveforms, while different strong estimators may produce conflicting yet individually plausible candidates for the same video. To resolve these conflicts, we propose PhysAgent, an inference-time multi-agent candidate-verification framework. Unlike direct prediction approaches, PhysAgent neither trains a new base rPPG model nor asks Multimodal Large Language Models (MLLMs) to output heart rate directly. In contrast, it treats outputs from multiple base estimators as physiological hypotheses to be verified and uses a lightweight 4B MLLM, Qwen3-VL-4B, to drive multi-agent reasoning over video conditions, signal reliability, and candidate disagreement. A deterministic physiological verifier checks the fusion proposal, and a reproducible numerical fusion process produces the final heart rate. Experimental results on multiple public rPPG benchmarks show that PhysAgent improves fusion stability and reliability across different datasets and source-domain settings, while avoiding the irreproducibility and physiological inconsistency of direct MLLM prediction or unconstrained ensemble fusion. The code will be released soon.
Yehui Yang, Bo Zhao, Junzhe Cao +4
1Great Bay University · 2Shenzhen University of Advanced Technology · 3Macao Polytechnic University +1
Gaze estimation under natural head-eye motion underpins applications from driver monitoring to human-computer interaction. Single-frame methods predict each frame independently, so consecutive outputs fluctuate as jitter. Multi-frame methods reduce this, but they learn motion implicitly inside appearance features, so the gaze trajectory is never an explicit variable. We propose EyeTAG (Eye Trajectory-Aware Gaze Estimation), a causal multi-frame framework built around an explicit first-order gaze prior: at each step it differentiates its own recent predictions and feeds the resulting trajectory back as a compact kinematic token. Because differencing is translation-invariant in gaze space, this token carries subject-invariant motion rather than personal gaze offsets. Face and eye streams supply visual evidence, fused by cross-attention and a causal Transformer decoder. EyeTAG reduces the mean angular error by about 1.0∘ on Gaze360 and performs on par with the strongest baseline on EVE (2.56∘ vs. 2.58∘). Within-model ablations, which keep the encoder and the rest of the architecture fixed and vary only the gaze history, show that the differential formulation, rather than temporal context alone, removes the systematic saccade bias that persists even with an absolute gaze-history prior. Our code is available at https://github.com/peter8366/EyeTAG.
Jungmin Lee, Niamat Ullah, Yoseob Han
Department of Information and Telecommunication Engineering Soongsil University Seoul, Republic of Korea · Department of Electronic Engineering Soongsil University Seoul, Republic of Korea