Why Do VLMs Fail at Remote Physiological Measurement? Rethinking with Physiological Grounding and Numerical Alignment

Xiao Yang; Zekun Zhang; Yuxuan Fan; Duoxun Tang; Ziyang Liu; Dengbo He; Kaishun Wu; Jiyao Wang

cs.CVOct 13, 2026

Why Do VLMs Fail at Remote Physiological Measurement? Rethinking with Physiological Grounding and Numerical Alignment

Authors: Xiao Yang, Zekun Zhang, Yuxuan Fan, Duoxun Tang, Ziyang Liu, Dengbo He, Kaishun Wu, Jiyao Wang

Organizations: The Hong Kong University of Science and Technology (Guangzhou), China · Sichuan University, China · Nanyang Technological University, Singapore · Tsinghua Shenzhen International Graduate School, Tsinghua University, China

Abstract

Video-based remote physiological measurement (RPM) has broad applications, while Vision-Language Models (VLMs) provide a promising opportunity to enable end-to-end physiological monitoring for future human-aware intelligent systems. However, existing RPM methods rely on task-specific models and cannot be seamlessly integrated with the multimodal understanding and interaction capabilities of VLMs. We revisit RPM with VLMs and provide empirical evidence for two major bottlenecks: generic visual semantic representations fail to preserve fine-grained physiological cues; while discrete token prediction in autoregressive generation is inherently misaligned with continuous numerical measurement. To address these challenges, we propose VITAL (Visual-physiological Token Alignment for Language Models), a visual-physiological token alignment framework for RPM and preserving general vision tasks. VITAL retains the original visual pathway for semantic understanding and introduces an independent physiological pathway to jointly model semantic and physiological information. Moreover, we propose a structured numerical alignment approach to improve the reliability of generative heart rate estimation by enhancing numerical relationship modeling. Extensive experiments show that VITAL improves cross-domain estimation performance on multiple rPPG datasets while preserving the general visual understanding capability of VLMs, demonstrating its effectiveness as a unified visual-physiological model. Code will be released.

Figures & tables

Figure 1: Empirical test of VLM-based RPM. (a) Increasing source-domain training data does not improve the cross-domain generalization of native VLMs, whereas an rPPG-oriented physiological encoder yields consistent gains. (b) Even with physiological representations, generative VLMs exhibit source-prior attraction under HR distribution shift. Models are trained on UBFC-rPPG and evaluated on PURE; darker gray indicates higher source-domain HR density. Red boxes highlight repeated-value banding.

Decoder	Estimation			Pairwise collision rate (%) $\downarrow$
Decoder	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	$\Delta\mathrm{GT}\geq 5$	$\geq 10$	$\geq 15$	$\geq 20$
Regression	9.57	13.85	0.900	6.01	3.62	2.20	1.19
Autoregressive	8.44	13.90	0.818	7.12	6.55	5.77	4.02

Table 1: Output-interface analysis on UBFC-rPPG

\rightarrow

PURE. Collision rates are the percentages of sample pairs with

|\Delta\mathrm{Pred}|\leq 1

BPM among pairs satisfying the indicated minimum ground-truth HR separation.

Figure 2: Overview of VITAL. (a) Dual-Path Physiological Grounding preserves native semantic representations while introducing PhysTokens as fine-grained physiological evidence, together with a progressive alignment training process. (b) Place-Value-Aware Numerical Supervision weights digit errors according to decimal significance. (c) Cross-Sample Ordinal Supervision preserves relative HR ordering through the first differing digit.

Method	PURE				UBFC-rPPG				BUAA				VIPL-HR
Method	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$
Green	15.47	23.31	0.23	22.29	12.77	15.88	0.29	13.72	8.85	13.54	0.44	9.84	13.67	29.38	0.10	22.00
CHROM	9.26	14.12	0.73	13.97	9.60	13.87	0.43	11.56	6.58	11.88	0.55	8.72	18.87	24.28	0.24	20.19
POS	9.58	16.27	0.65	15.55	9.03	11.20	0.40	9.28	7.20	12.54	0.56	8.71	19.18	24.93	0.23	20.33
PhysNet	10.96	15.88	0.54	13.01	8.09	13.26	0.68	11.25	6.04	8.36	0.56	8.31	14.81	18.62	0.17	18.44
PhysFormer	17.51	21.45	0.14	19.96	14.88	18.72	0.26	15.08	8.81	11.82	0.49	11.78	14.00	17.39	0.28	17.15

Table 2: HR estimation results on MSDG protocol. † denotes STMap-based methods. In this and following tables, best results are bolded, and second-best results are underlined. Significant (

p

<.05) improvements are marked ’ ∗ ’ using paired

t

-tests.

Method	PhysDrive				MMPD
	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$
FactorizePhys	14.29	17.88	0.13	15.96	12.74	15.04	0.17	13.37
Period-LLM	24.04	26.39	0.01	23.61	23.42	28.97	0.02	26.55
PhysLLM	16.63	19.32	0.11	16.90	12.49	15.72	0.17	14.75
VITAL	12.65 ∗	15.08 ∗	0.17 ∗	13.74 ∗	10.63 ∗	13.23 ∗	0.22 ∗	12.90 ∗

Table 3: HR estimation results on challenging datasets.

Figure 3: Prediction distributions on PURE and BUAA. Red boxes highlight repeated-value banding.

Method	MVBench $\uparrow$	MAE $\downarrow$	Params $\downarrow$	Memory $\downarrow$
Qwen3-VL-4B (Native)	64.05% ∗	20.98	4.438B	8.512 GiB
Period-LLM	59.38%	17.89	4.438B	8.512 GiB
VITAL	60.53%	10.63 ∗	4.460B	8.748 GiB

Table 4: General video understanding on MVBench, physiological performance on MMPD, and model overhead with Qwen3-VL-4B backbone.

Method	PURE				UBFC-rPPG
Method	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$
Qwen3-VL-2B	4.51	10.67	0.85	10.47	2.98	5.63	0.95	5.25
Qwen3-VL-4B	3.97	9.29	0.89	9.15	2.14	3.64	0.98	3.42
Qwen3-VL-8B	3.88 ∗	9.14 ∗	0.91 ∗	9.03 ∗	2.11 ∗	3.12 ∗	0.98 ∗	2.96 ∗
LLaVA-NeXT-Video-7B	4.86	9.57	0.88	9.45	2.77	4.12	0.97	3.48

Table 5: Cross-domain HR estimation of VITAL with different VLM backbones under the MSDG protocol. Models are trained on the other three source domains and evaluated on the indicated unseen target domain.

Method	Balanced Accuracy (%) $\uparrow$			HR Estimation
Method	Glasses	Lighting	Recovery State	MAE $\downarrow$	$r$ $\uparrow$
Qwen3-VL-4B (Native)	89.55% ∗	46.94%	50.04%	21.76	0.01
Period-LLM	83.55%	45.10%	50.16%	18.69	0.03
VITAL	84.59%	49.81% ∗	56.54% ∗	10.91 ∗	0.18 ∗

Table 6: Zero-shot evaluation of visual semantics and physiological states on MMPD.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Training Schedule	Warm-up	Stage I	Stage II	PURE				UBFC-rPPG				BUAA				VIPL-HR
Training Schedule	Warm-up	Stage I	Stage II	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$
Stage II	No	No	Yes	14.21	16.98	0.62	15.07	7.45	10.19	0.73	8.58	15.03	18.74	0.72	16.62	18.24	22.06	0.51	19.35
Warm-up $\rightarrow$ Stage II	Yes	No	Yes	13.49	16.74	0.65	14.16	7.83	10.32	0.74	8.59	14.76	17.09	0.75	15.95	16.43	21.21	0.55	18.89
Stage I $\rightarrow$ Stage II	No	Yes	Yes	4.35	10.27	0.87	10.03	2.71	4.15	0.97	3.80	4.98	6.81	0.89	4.72	8.64	12.13	0.70	11.88
Warm-up $\rightarrow$ Stage I $\rightarrow$ Stage II	Yes	Yes	Yes	3.97	9.29	0.89	9.15	2.14	3.64	0.98	3.42	4.70	6.40	0.91	4.56	8.61	11.95	0.71	11.61

Appendix

Table 7: Ablation of projector warm-up and the two stages of Progressive PhysToken Alignment under the MSDG protocol. Each dataset serves once as the unseen target domain, with the remaining three datasets used as source domains. All schedules use the same total optimization budget and matched numbers of rPPG and general-purpose instruction samples. The best and second-best results are shown in bold and underlined, respectively.

Training Objective	PURE				UBFC-rPPG				BUAA				VIPL-HR
Training Objective	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$
$\mathcal{L}_{\mathrm{base}}$	4.83	10.87	0.85	10.55	2.39	4.12	0.96	4.03	6.38	8.15	0.87	5.29	8.90	13.11	0.67	12.18
$\mathcal{L}_{\mathrm{base}}+\mathcal{L}_{\mathrm{NTL}}$	5.23	11.45	0.82	10.72	3.40	5.06	0.92	4.57	6.29	8.03	0.88	5.25	9.02	12.94	0.68	12.05
$\mathcal{L}_{\mathrm{base}}+\mathcal{L}_{\mathrm{NTL}}+\mathcal{L}_{\mathrm{ord}}$	4.35	9.76	0.86	9.63	2.61	4.53	0.95	4.39	6.01	7.83	0.87	5.20	8.72	12.49	0.67	12.17
$\mathcal{L}_{\mathrm{base}}+\mathcal{L}_{\mathrm{PV}}$	4.33	9.54	0.89	9.32	2.26	3.97	0.95	3.89	5.42	7.32	0.88	5.19	8.67	12.20	0.67	12.19
$\mathcal{L}_{\mathrm{base}}+\mathcal{L}_{\mathrm{PV}}+\mathcal{L}_{\mathrm{ord}}$	3.97	9.29	0.89	9.15	2.14	3.64	0.98	3.42	4.70	6.40	0.91	4.56	8.61	11.95	0.71	11.61

Appendix

Table 8: Comparison of numerical supervision objectives under the MSDG protocol. Each dataset serves once as the unseen target domain, with the remaining three datasets used as source domains. All variants retain

\mathcal{L}_{\mathrm{LM}}

and

\mathcal{L}_{\mathrm{aux}}

and differ only in their additional numerical supervision. Loss weights are omitted from the objective labels for brevity. The best and second-best results are shown in bold and underlined, respectively.

Number of PhysTokens	PURE				UBFC-rPPG				BUAA				VIPL-HR
Number of PhysTokens	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	$r\uparrow$	SD $\downarrow$
5	8.83	14.09	0.83	13.92	6.95	8.87	0.94	7.66	7.43	9.65	0.82	7.83	12.85	17.35	0.53	14.87
10	4.27	9.86	0.87	9.70	3.18	3.90	0.96	3.73	5.04	6.83	0.86	6.41	9.52	11.87	0.70	12.02
20	3.97	9.29	0.89	9.15	2.14	3.64	0.98	3.42	4.70	6.40	0.91	4.56	8.61	11.95	0.71	11.61
40	4.52	10.03	0.88	9.91	2.33	3.87	0.97	3.53	4.86	6.69	0.87	5.19	9.03	12.23	0.65	12.05
80	4.34	9.76	0.87	9.77	2.59	3.91	0.96	3.62	5.22	6.90	0.85	5.35	9.45	12.55	0.64	12.34

Appendix

Table 9: Effect of PhysToken sequence length under the MSDG protocol. Each dataset serves once as the unseen target domain, with the remaining three datasets used as source domains. The best and second-best results are shown in bold and underlined, respectively.

Target Domain	Valid Digit Positions	Fully Aligned Samples	HR-Gap Retention	Same-Length Retention	Span-Compatible Retention	Final Valid Pairs	Valid Pairs per Batch
PURE	100.00%	100.00%	87.46%	62.14%	100.00%	54.35%	2.72
UBFC-rPPG	100.00%	100.00%	87.11%	82.13%	100.00%	71.54%	3.58
BUAA	100.00%	100.00%	90.33%	61.75%	100.00%	55.78%	2.79
VIPL-HR	100.00%	100.00%	89.98%	66.55%	100.00%	59.88%	2.99
Micro-average	100.00%	100.00%	88.38%	68.58%	100.00%	60.61%	3.03

Appendix

Table 10: Coverage analysis of Structured Numerical Alignment using the Qwen3-VL tokenizer. Valid Digit Positions is computed over all formatted digit positions. HR-Gap Retention is computed relative to all offset-based initial candidate pairs, Same-Length Retention relative to HR-gap-valid pairs, and Span-Compatible Retention relative to same-length pairs. Final Valid Pairs is computed relative to all offset-based initial candidate pairs. Pair statistics are averaged over five dataloader seeds.

Method	$\delta=0$	$\delta=0.5$	$\delta=1$	$\delta=2$
Continuous Regression Probe	0.72%	2.17%	3.62%	6.64%
Autoregressive Generation	3.16%	5.96%	6.55%	10.01%
Autoregressive Generation + SNA	1.85%	3.72%	4.73%	8.49%

Appendix

Table 11: Pairwise collision rates under different prediction-distance thresholds. The minimum ground-truth HR separation is fixed at 10 BPM. Lower values indicate better cross-sample numerical discriminability. Best results are bolded, and second-best results are underlined.

Method	$\gamma=5$	$\gamma=10$	$\gamma=15$	$\gamma=20$
Continuous Regression Probe	6.01%	3.62%	2.20%	1.19%
Autoregressive Generation	7.12%	6.55%	5.77%	4.02%
Autoregressive Generation + SNA	6.32%	4.73%	4.31%	3.67%

Appendix

Table 12: Pairwise collision rates under different minimum ground-truth HR separations. The prediction-distance threshold is fixed at 1 BPM. Best results are bolded, and second-best results are underlined.

Remote Photoplethysmography

Remote photoplethysmography (rPPG) estimates pulse and other physiological signals from video without contact sensors.

See all 46 papers

PhysVR: Vision-Language Model Guided Interference-aware Temporal Feature Refinement for Remote Physiological Measurement
Aug 30, 2026Zixu Li, Jianjun Qian, Hang Shao +3Remote Photoplethysmography Photoplethysmography
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · College of Computer Science and Technology, Qingdao University, Qingdao 266071, China
PhysAgent: A Multi-Agent Framework for Reliable Remote Heart Rate Estimation
Jul 29, 2026Yehui Yang, Bo Zhao, Junzhe Cao +4Remote Photoplethysmography Heart Rate Estimation
1Great Bay University · 2Shenzhen University of Advanced Technology · 3Macao Polytechnic University +1
RPM-Distill: Physiology-guided Adaptive Cross-modal Distillation for Robust Remote Physiological Measurement
Jun 26, 2026Jiyao Wang, Qingyong Hu, Duoxun Tang +3Remote Photoplethysmography Adaptive Knowledge Distillation
McGill University, Montreal, Canada · Hong Kong University of Science and Technology, Hong Kong, China · Tsinghua University, Shenzhen, China +1

Aug 30, 2026J/K move · Enter open · S save

PhysVR: Vision-Language Model Guided Interference-aware Temporal Feature Refinement for Remote Physiological Measurement

Zixu Li, Jianjun Qian, Hang Shao, Daoheng Li+2

PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · College of Computer Science and Technology, Qingdao University, Qingdao 266071, China

Remote Photoplethysmography Photoplethysmography

Open paper AskPDF arXiv

Remote photoplethysmography (rPPG) enables contactless physiological measurement from facial videos, yet its subtle pulse-related variations are easily affected by illumination variation, head motion, facial blur, and region-of-interest instability. Existing methods mainly suppress interference during feature learning, while whether the learned temporal features remain affected by interference and how to further suppress such interference before rPPG estimation are rarely examined. To address this limitation, we propose PhysVR, a vision-language model guided interference-aware temporal feature refinement framework for rPPG estimation. Specifically, a physiological backbone produces global temporal features and a coarse rPPG prediction, from which signal-derived physiological reliability evidence is constructed from local temporal characteristics. In parallel, a frozen vision-language model processes sampled facial frames under an interference-oriented prompt, and an evidence head extracts visual interference evidence from the VLM output. Temporal cross-attention integrates the physiological and visual evidence with the global temporal features to construct interference-aware temporal context. Guided by this context, a shared temporal correction unit performs general refinement, while four interference-specific experts selectively suppress different interference through adaptive routing. The refined temporal features are then used for final rPPG estimation. Extensive experiments on five public benchmarks demonstrate that PhysVR consistently outperforms representative methods under both intra-dataset and cross-dataset evaluation protocols.

Abstract

Remote Photoplethysmography

Explore similar work

PhysVR: Vision-Language Model Guided Interference-aware Temporal Feature Refinement for Remote Physiological Measurement

PhysAgent: A Multi-Agent Framework for Reliable Remote Heart Rate Estimation

RPM-Distill: Physiology-guided Adaptive Cross-modal Distillation for Robust Remote Physiological Measurement