Organizations: The Hong Kong University of Science and Technology (Guangzhou), China · Sichuan University, China · Nanyang Technological University, Singapore · Tsinghua Shenzhen International Graduate School, Tsinghua University, China
Video-based remote physiological measurement (RPM) has broad applications, while Vision-Language Models (VLMs) provide a promising opportunity to enable end-to-end physiological monitoring for future human-aware intelligent systems. However, existing RPM methods rely on task-specific models and cannot be seamlessly integrated with the multimodal understanding and interaction capabilities of VLMs. We revisit RPM with VLMs and provide empirical evidence for two major bottlenecks: generic visual semantic representations fail to preserve fine-grained physiological cues; while discrete token prediction in autoregressive generation is inherently misaligned with continuous numerical measurement. To address these challenges, we propose VITAL (Visual-physiological Token Alignment for Language Models), a visual-physiological token alignment framework for RPM and preserving general vision tasks. VITAL retains the original visual pathway for semantic understanding and introduces an independent physiological pathway to jointly model semantic and physiological information. Moreover, we propose a structured numerical alignment approach to improve the reliability of generative heart rate estimation by enhancing numerical relationship modeling. Extensive experiments show that VITAL improves cross-domain estimation performance on multiple rPPG datasets while preserving the general visual understanding capability of VLMs, demonstrating its effectiveness as a unified visual-physiological model. Code will be released.
Figures & tables
Figure 1: Empirical test of VLM-based RPM. (a) Increasing source-domain training data does not improve the cross-domain generalization of native VLMs, whereas an rPPG-oriented physiological encoder yields consistent gains. (b) Even with physiological representations, generative VLMs exhibit source-prior attraction under HR distribution shift. Models are trained on UBFC-rPPG and evaluated on PURE; darker gray indicates higher source-domain HR density. Red boxes highlight repeated-value banding.
Decoder
Estimation
Pairwise collision rate (%) ↓
MAE ↓
RMSE ↓
r↑
ΔGT≥5
≥10
≥15
≥20
Regression
9.57
13.85
0.900
6.01
3.62
2.20
1.19
Autoregressive
8.44
13.90
0.818
7.12
6.55
5.77
4.02
Table 1: Output-interface analysis on UBFC-rPPG → PURE. Collision rates are the percentages of sample pairs with ∣ΔPred∣≤1 BPM among pairs satisfying the indicated minimum ground-truth HR separation.
Figure 2: Overview of VITAL. (a) Dual-Path Physiological Grounding preserves native semantic representations while introducing PhysTokens as fine-grained physiological evidence, together with a progressive alignment training process. (b) Place-Value-Aware Numerical Supervision weights digit errors according to decimal significance. (c) Cross-Sample Ordinal Supervision preserves relative HR ordering through the first differing digit.
Method
PURE
UBFC-rPPG
BUAA
VIPL-HR
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
Green
15.47
23.31
0.23
22.29
12.77
15.88
0.29
13.72
8.85
13.54
0.44
9.84
13.67
29.38
0.10
22.00
CHROM
9.26
14.12
0.73
13.97
9.60
13.87
0.43
11.56
6.58
11.88
0.55
8.72
18.87
24.28
0.24
20.19
POS
9.58
16.27
0.65
15.55
9.03
11.20
0.40
9.28
7.20
12.54
0.56
8.71
19.18
24.93
0.23
20.33
PhysNet
10.96
15.88
0.54
13.01
8.09
13.26
0.68
11.25
6.04
8.36
0.56
8.31
14.81
18.62
0.17
18.44
PhysFormer
17.51
21.45
0.14
19.96
14.88
18.72
0.26
15.08
8.81
11.82
0.49
11.78
14.00
17.39
0.28
17.15
Table 2: HR estimation results on MSDG protocol. † denotes STMap-based methods. In this and following tables, best results are bolded, and second-best results are underlined. Significant ( p <.05) improvements are marked ’ ∗ ’ using paired t -tests.
Method
PhysDrive
MMPD
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
FactorizePhys
14.29
17.88
0.13
15.96
12.74
15.04
0.17
13.37
Period-LLM
24.04
26.39
0.01
23.61
23.42
28.97
0.02
26.55
PhysLLM
16.63
19.32
0.11
16.90
12.49
15.72
0.17
14.75
VITAL
12.65 ∗
15.08 ∗
0.17 ∗
13.74 ∗
10.63 ∗
13.23 ∗
0.22 ∗
12.90 ∗
Table 3: HR estimation results on challenging datasets.
Figure 3: Prediction distributions on PURE and BUAA. Red boxes highlight repeated-value banding.
Method
MVBench ↑
MAE ↓
Params ↓
Memory ↓
Qwen3-VL-4B (Native)
64.05% ∗
20.98
4.438B
8.512 GiB
Period-LLM
59.38%
17.89
4.438B
8.512 GiB
VITAL
60.53%
10.63 ∗
4.460B
8.748 GiB
Table 4: General video understanding on MVBench, physiological performance on MMPD, and model overhead with Qwen3-VL-4B backbone.
Method
PURE
UBFC-rPPG
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
Qwen3-VL-2B
4.51
10.67
0.85
10.47
2.98
5.63
0.95
5.25
Qwen3-VL-4B
3.97
9.29
0.89
9.15
2.14
3.64
0.98
3.42
Qwen3-VL-8B
3.88 ∗
9.14 ∗
0.91 ∗
9.03 ∗
2.11 ∗
3.12 ∗
0.98 ∗
2.96 ∗
LLaVA-NeXT-Video-7B
4.86
9.57
0.88
9.45
2.77
4.12
0.97
3.48
Table 5: Cross-domain HR estimation of VITAL with different VLM backbones under the MSDG protocol. Models are trained on the other three source domains and evaluated on the indicated unseen target domain.
Method
Balanced Accuracy (%) ↑
HR Estimation
Glasses
Lighting
Recovery State
MAE ↓
r↑
Qwen3-VL-4B (Native)
89.55% ∗
46.94%
50.04%
21.76
0.01
Period-LLM
83.55%
45.10%
50.16%
18.69
0.03
VITAL
84.59%
49.81% ∗
56.54% ∗
10.91 ∗
0.18 ∗
Table 6: Zero-shot evaluation of visual semantics and physiological states on MMPD.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Training Schedule
Warm-up
Stage I
Stage II
PURE
UBFC-rPPG
BUAA
VIPL-HR
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
Stage II
No
No
Yes
14.21
16.98
0.62
15.07
7.45
10.19
0.73
8.58
15.03
18.74
0.72
16.62
18.24
22.06
0.51
19.35
Warm-up → Stage II
Yes
No
Yes
13.49
16.74
0.65
14.16
7.83
10.32
0.74
8.59
14.76
17.09
0.75
15.95
16.43
21.21
0.55
18.89
Stage I → Stage II
No
Yes
Yes
4.35
10.27
0.87
10.03
2.71
4.15
0.97
3.80
4.98
6.81
0.89
4.72
8.64
12.13
0.70
11.88
Warm-up → Stage I → Stage II
Yes
Yes
Yes
3.97
9.29
0.89
9.15
2.14
3.64
0.98
3.42
4.70
6.40
0.91
4.56
8.61
11.95
0.71
11.61
Appendix
Table 7: Ablation of projector warm-up and the two stages of Progressive PhysToken Alignment under the MSDG protocol. Each dataset serves once as the unseen target domain, with the remaining three datasets used as source domains. All schedules use the same total optimization budget and matched numbers of rPPG and general-purpose instruction samples. The best and second-best results are shown in bold and underlined, respectively.
Training Objective
PURE
UBFC-rPPG
BUAA
VIPL-HR
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
Lbase
4.83
10.87
0.85
10.55
2.39
4.12
0.96
4.03
6.38
8.15
0.87
5.29
8.90
13.11
0.67
12.18
Lbase+LNTL
5.23
11.45
0.82
10.72
3.40
5.06
0.92
4.57
6.29
8.03
0.88
5.25
9.02
12.94
0.68
12.05
Lbase+LNTL+Lord
4.35
9.76
0.86
9.63
2.61
4.53
0.95
4.39
6.01
7.83
0.87
5.20
8.72
12.49
0.67
12.17
Lbase+LPV
4.33
9.54
0.89
9.32
2.26
3.97
0.95
3.89
5.42
7.32
0.88
5.19
8.67
12.20
0.67
12.19
Lbase+LPV+Lord
3.97
9.29
0.89
9.15
2.14
3.64
0.98
3.42
4.70
6.40
0.91
4.56
8.61
11.95
0.71
11.61
Appendix
Table 8: Comparison of numerical supervision objectives under the MSDG protocol. Each dataset serves once as the unseen target domain, with the remaining three datasets used as source domains. All variants retain LLM and Laux and differ only in their additional numerical supervision. Loss weights are omitted from the objective labels for brevity. The best and second-best results are shown in bold and underlined, respectively.
Number of PhysTokens
PURE
UBFC-rPPG
BUAA
VIPL-HR
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
MAE ↓
RMSE ↓
r↑
SD ↓
5
8.83
14.09
0.83
13.92
6.95
8.87
0.94
7.66
7.43
9.65
0.82
7.83
12.85
17.35
0.53
14.87
10
4.27
9.86
0.87
9.70
3.18
3.90
0.96
3.73
5.04
6.83
0.86
6.41
9.52
11.87
0.70
12.02
20
3.97
9.29
0.89
9.15
2.14
3.64
0.98
3.42
4.70
6.40
0.91
4.56
8.61
11.95
0.71
11.61
40
4.52
10.03
0.88
9.91
2.33
3.87
0.97
3.53
4.86
6.69
0.87
5.19
9.03
12.23
0.65
12.05
80
4.34
9.76
0.87
9.77
2.59
3.91
0.96
3.62
5.22
6.90
0.85
5.35
9.45
12.55
0.64
12.34
Appendix
Table 9: Effect of PhysToken sequence length under the MSDG protocol. Each dataset serves once as the unseen target domain, with the remaining three datasets used as source domains. The best and second-best results are shown in bold and underlined, respectively.
Target Domain
Valid Digit Positions
Fully Aligned Samples
HR-Gap Retention
Same-Length Retention
Span-Compatible Retention
Final Valid Pairs
Valid Pairs per Batch
PURE
100.00%
100.00%
87.46%
62.14%
100.00%
54.35%
2.72
UBFC-rPPG
100.00%
100.00%
87.11%
82.13%
100.00%
71.54%
3.58
BUAA
100.00%
100.00%
90.33%
61.75%
100.00%
55.78%
2.79
VIPL-HR
100.00%
100.00%
89.98%
66.55%
100.00%
59.88%
2.99
Micro-average
100.00%
100.00%
88.38%
68.58%
100.00%
60.61%
3.03
Appendix
Table 10: Coverage analysis of Structured Numerical Alignment using the Qwen3-VL tokenizer. Valid Digit Positions is computed over all formatted digit positions. HR-Gap Retention is computed relative to all offset-based initial candidate pairs, Same-Length Retention relative to HR-gap-valid pairs, and Span-Compatible Retention relative to same-length pairs. Final Valid Pairs is computed relative to all offset-based initial candidate pairs. Pair statistics are averaged over five dataloader seeds.
Method
δ=0
δ=0.5
δ=1
δ=2
Continuous Regression Probe
0.72%
2.17%
3.62%
6.64%
Autoregressive Generation
3.16%
5.96%
6.55%
10.01%
Autoregressive Generation + SNA
1.85%
3.72%
4.73%
8.49%
Appendix
Table 11: Pairwise collision rates under different prediction-distance thresholds. The minimum ground-truth HR separation is fixed at 10 BPM. Lower values indicate better cross-sample numerical discriminability. Best results are bolded, and second-best results are underlined.
Method
γ=5
γ=10
γ=15
γ=20
Continuous Regression Probe
6.01%
3.62%
2.20%
1.19%
Autoregressive Generation
7.12%
6.55%
5.77%
4.02%
Autoregressive Generation + SNA
6.32%
4.73%
4.31%
3.67%
Appendix
Table 12: Pairwise collision rates under different minimum ground-truth HR separations. The prediction-distance threshold is fixed at 1 BPM. Best results are bolded, and second-best results are underlined.
Remote photoplethysmography (rPPG) enables contactless physiological measurement from facial videos, yet its subtle pulse-related variations are easily affected by illumination variation, head motion, facial blur, and region-of-interest instability. Existing methods mainly suppress interference during feature learning, while whether the learned temporal features remain affected by interference and how to further suppress such interference before rPPG estimation are rarely examined. To address this limitation, we propose PhysVR, a vision-language model guided interference-aware temporal feature refinement framework for rPPG estimation. Specifically, a physiological backbone produces global temporal features and a coarse rPPG prediction, from which signal-derived physiological reliability evidence is constructed from local temporal characteristics. In parallel, a frozen vision-language model processes sampled facial frames under an interference-oriented prompt, and an evidence head extracts visual interference evidence from the VLM output. Temporal cross-attention integrates the physiological and visual evidence with the global temporal features to construct interference-aware temporal context. Guided by this context, a shared temporal correction unit performs general refinement, while four interference-specific experts selectively suppress different interference through adaptive routing. The refined temporal features are then used for final rPPG estimation. Extensive experiments on five public benchmarks demonstrate that PhysVR consistently outperforms representative methods under both intra-dataset and cross-dataset evaluation protocols.
Zixu Li, Jianjun Qian, Hang Shao +3
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · College of Computer Science and Technology, Qingdao University, Qingdao 266071, China
Remote photoplethysmography (rPPG) enables non-contact heart-rate estimation from facial videos, but its weak physiological signal is easily corrupted by motion, illumination changes, occlusion, skin-appearance variation, and device noise. Existing rPPG methods typically rely on a single model to directly predict heart rate or recover pulse waveforms, while different strong estimators may produce conflicting yet individually plausible candidates for the same video. To resolve these conflicts, we propose PhysAgent, an inference-time multi-agent candidate-verification framework. Unlike direct prediction approaches, PhysAgent neither trains a new base rPPG model nor asks Multimodal Large Language Models (MLLMs) to output heart rate directly. In contrast, it treats outputs from multiple base estimators as physiological hypotheses to be verified and uses a lightweight 4B MLLM, Qwen3-VL-4B, to drive multi-agent reasoning over video conditions, signal reliability, and candidate disagreement. A deterministic physiological verifier checks the fusion proposal, and a reproducible numerical fusion process produces the final heart rate. Experimental results on multiple public rPPG benchmarks show that PhysAgent improves fusion stability and reliability across different datasets and source-domain settings, while avoiding the irreproducibility and physiological inconsistency of direct MLLM prediction or unconstrained ensemble fusion. The code will be released soon.
Yehui Yang, Bo Zhao, Junzhe Cao +4
1Great Bay University · 2Shenzhen University of Advanced Technology · 3Macao Polytechnic University +1
Video-based remote physiological measurement (RPM) is highly accessible but remains fragile under varying illumination, skin tones, and motion. Radio frequency (RF) radar is largely invariant to illumination and appearance, providing complementary cardio-respiratory micro-motion cues; however, requiring radar at inference is often impractical due to its limited ubiquity and deployment overhead. We propose RPM-Distill, a physiology-guided cross-modal distillation framework that leverages synchronized radar only during training while retaining video-only inference. Our key observation is that although RGB and RF waveforms differ in sensing physics and time-domain morphology, they share similar latent periodic rhythm in the frequency domain. We thus distill physiology-structured spectral evidence to improve robustness, via losses that (i) anchor the fundamental peak, (ii) match the off-peak background distribution, and (iii) preserve spectral morphology and sharpness. To avoid negative transfer under sample-level teacher quality and alignment uncertainty, a spectral policy network predicts sample-level distillation gates and component weights from the student--teacher spectral relation map, learned with a meta bilevel objective on a small labeled validation split. Through extensive experiments in challenging conditions and cross-dataset settings, RPM-Distill brings 81% MAE and 21% correlation improvement over unimodal baselines. Code is at https://github.com/WJULYW/RPM-Distill.
Jiyao Wang, Qingyong Hu, Duoxun Tang +3
McGill University, Montreal, Canada · Hong Kong University of Science and Technology, Hong Kong, China · Tsinghua University, Shenzhen, China +1