Tracking Is Not Sensing: Artifact-Aware Thermal Video Respiration Estimation in Head-Free Marmosets

Penghao Wang; Sen Yang; Fenghua Xie; He Wang; Xindong Song

cs.CVOct 24, 2026

Tracking Is Not Sensing: Artifact-Aware Thermal Video Respiration Estimation in Head-Free Marmosets

Authors: Penghao Wang, Sen Yang, Fenghua Xie, He Wang, Xindong Song

Organizations: Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology · Tsinghua Laboratory of Brain and Intelligence, Tsinghua University

Abstract

Continuous physiological monitoring is essential for linking behavior, internal state, and neural activity in neuroscience and translational research. In common marmosets, a key nonhuman primate model for social neuroscience, respiratory rate provides a basic readout of arousal, stress, and physiological regulation. Contact sensors can provide accurate measurements, but they add preparation burden and may interfere with head-free behavioral protocols. Thermal video offers a noncontact alternative by capturing breathing-related temperature changes around the nose and mouth. However, in head-free marmosets, these cues are weak, localized, and easily corrupted by head motion, pose changes, and fur. Here we show that this setting exposes a broader challenge in video-based physiological sensing: visually reliable tracking does not necessarily imply physiologically valid measurement. Using synchronized thermal video and contact respiration recordings, we show that marmoset respiration is recoverable from thermal video but strongly ROI-sensitive. Under head-free motion, visually plausible nasal tracking can still produce traces inconsistent with contact respiration, demonstrating that tracking is not sensing. We therefore estimate respiratory rate from learned temporal signals extracted from thermal clips and introduce Selective Temporal Attenuation Regularization (STAR). STAR selectively attenuates high-motion thermal discontinuities through local temporal imputation during training, regularizing models toward respiration-consistent dynamics while leaving inference unchanged on raw video. Across multiple deep-learning backbones, STAR improves head-free estimation, reaching 2.23 bpm MAE compared with 20.76 bpm for tracked ROI analysis.

Figures & tables

Figure 1: Tracking is not sensing in head-free thermal respiration monitoring. Left to right: synchronized thermal video and contact respiration provide paired observations; respiration related thermal cues are local and ROI sensitive; under head-free motion, visually successful CSRT tracking can still produce temperature traces that do not match respiration; robust estimation therefore requires recovering respiratory dynamics from thermal clips rather than relying on ROI averaged tracking alone.

ROI setting	MAE $\downarrow$	MAPE $\downarrow$	$\rho$ $\uparrow$
Manual ROI	2.46	3.57%	0.87
Oracle ROI	0.45	0.66%	0.99

Table 1: Fixed-head ROI frequency analysis. The oracle ROI is a post hoc diagnostic upper bound and is not deployable.

Panel of Figure 2 — Figure 2: Manual ROI versus post-hoc oracle ROI under fixed-head conditions.

Method	Windows	MAE $\downarrow$	MAPE $\downarrow$	$\rho$ $\uparrow$
KCF ROI	150	21.68	21.80	0.06
CSRT ROI	150	22.17	22.54	0.08
CSRT ROI	141	20.76	21.55	0.10

Table 2: Head-free tracking results. High ROI availability and visually correct nasal coverage do not translate into accurate respiratory-rate estimation.

Figure 3: Overview of STAR. A spatiotemporal network predicts a respiratory waveform from thermal clips. During training, high motion frames are identified from adjacent thermal differences and locally imputed using temporal neighbors with probability $p_{s}$ , attenuating motion-induced thermal discontinuity while inference uses raw clips.

Method	MAE $\downarrow$	MAPE $\downarrow$	$\rho$ $\uparrow$
Manual ROI	2.46	3.57	0.87
TS-CAN	2.16	3.10	0.83
PhysNet	0.93	1.40	0.95
PhysFormer	0.82	1.21	0.97

Table 3: Head-fixed comparison between traditional ROI methods and spatiotemporal waveform prediction.

Panel of Figure 4 — Figure 4: Grad-CAM visualization under head-fixed conditions.

Method	MAE $\downarrow$	MAPE $\downarrow$	$\rho$ $\uparrow$
Tracked ROI reference
CSRT ROI	20.76	21.55	0.10
Individual dependent
PhysFormer	3.31	3.98	0.91
PhysFormer + STAR	2.71	3.28	0.94
PhysNet	2.69	3.54	0.92

Table 4: Head-free respiratory rate estimation under individual-dependent and individual-independent evaluation.

Variant	MAE $\downarrow$	MAPE $\downarrow$	$\rho$ $\uparrow$
Operation
Raw	1.13	1.32	0.98
Random filter	1.19	1.36	0.97
Low-motion	2.09	2.45	0.93
Filter	2.00	2.39	0.94
Random imput.	1.26	1.53	0.95

Table 5: STAR ablation on B6 under individual-dependent evaluation.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Contact respiration reference hardware. The air bladder sensor converts abdominal respiratory motion into a pressure signal, while the elastic tubular bandage maintains stable contact during acquisition.

Panel of Figure 6 — Figure 6: Synchronized marmoset thermal respiration acquisition setup.

Panel of Figure 7 — Figure 7: Examples of respiratory signal screening.

Size / ND	MAE $\downarrow$	MAPE $\downarrow$	$\rho$ $\uparrow$
0.6	0.59	0.83%	0.99
0.8	0.71	1.00%	0.98
1.0	0.79	1.12%	0.98
1.5	1.75	2.51%	0.92
2.0	1.80	2.65%	0.92
2.5	2.20	3.16%	0.89

Appendix

Table 6: ND normalized ROI scale sensitivity under fixed-head conditions.

Config	Value
Backbones	TS-CAN, PhysNet, PhysFormer
Input modality	Cropped thermal facial clip
Training objective	Backbone specific objective following rPPG-Toolbox
Clip length	2 s / 50 frames
Sampling rate	25 fps
Input size	$1\times 50\times 128\times 128$

Appendix

Table 7: Default training and evaluation configuration for the marmoset experiments.

Panel of Figure 8 — Figure 8: Supplementary agreement analysis for STAR with PhysNet.

Figure 11: Motion intensity stratified success rates on head-free windows.

Figure 12: Motion duration stratified success rates on head-free windows.

Panel of Figure 13 — Figure 13: Additional KCF and CSRT tracking comparisons under head-free motion.

Panel of Figure 14 — Figure 14: Additional examples of tracked ROI respiratory rate estimation under head-free motion. Visual ROI continuity is not sufficient to guarantee an accurate respiratory trace.

Thermal Imaging

Thermal imaging measures infrared radiation to estimate temperature and reveal people, objects, and material changes in low-visibility conditions.

See all 77 papers

Thermal Imaging for Contactless Cardiorespiratory and Sudomotor Response Monitoring
Feb 12, 2026Constantino Álvarez Casado, Mohammad Rahman, Sasan Sharifipour +4Thermal Imaging Heart Rate Estimation
Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland · Candour Ltd, Oulu, Finland
A Skin-Tone-Aware Dual-Representation Remote Photoplethysmography Framework for Contactless Respiratory Rate Estimation
Jun 19, 2026Trishna Saikia, Anup Kumar Gupta, Puneet Gupta +1Remote Photoplethysmography Photoplethysmography
Video-based detection of cessation of breathing in pre-term infants using machine learning
Jul 6, 2026Dineo Serame, Lionel Tarassenko, Mauricio VillarroelHealthcare Multimodal Sensor Fusion
Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, UK

Feb 12, 2026J/K move · Enter open · S save

Thermal Imaging for Contactless Cardiorespiratory and Sudomotor Response Monitoring

Constantino Álvarez Casado, Mohammad Rahman, Sasan Sharifipour, Nhi Nguyen+3

Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland · Candour Ltd, Oulu, Finland

Thermal Imaging Heart Rate Estimation Driver Monitoring

Open paper AskPDF arXiv

Human-machine interfaces in industrial automation need sensing modules that monitor operator actions and physiological state. This is important in factories, vehicles, machinery cabins, and human-robot collaboration, where workload, stress, fatigue, or reduced attention can affect safety. RGB monitoring is limited by low light, shadows, and privacy concerns, while thermal infrared imaging captures skin temperature dynamics without visible illumination. This paper studies thermal video as a contactless computer vision modality for estimating electrodermal activity (EDA), heart rate (HR), and breathing rate (BR), with the goal of supporting adaptive human-machine interfaces and operator-state awareness. We propose a signal-processing pipeline that tracks facial regions, aggregates thermal signals, and separates slow sudomotor trends from faster cardiorespiratory components. HR is estimated using orthogonal matrix image transformation (OMIT) across multiple facial regions, while BR is estimated from nasal and cheek thermal signals using spectral peak detection. We characterize 288 ROI-method configurations against contact references with lag-tolerant metrics using 31 sessions from the public SIMULATOR STUDY 1 (SIM1) driver monitoring dataset. The best fixed EDA configuration reaches a mean absolute correlation of

0.40 \pm 0.23

against palm EDA, with individual sessions reaching

0.89

. BR estimation achieves

3.1 \pm 1.1

bpm mean absolute error, while HR estimation yields

13.8 \pm 7.5

bpm MAE, limited by the

7.5

Hz thermal camera frame rate. The results show that thermal video provides useful respiratory and sudomotor cues, while revealing limitations caused by ROI selection, polarity changes, latency, and subject variability. These findings provide baseline design guidance for thermal computer vision as an auxiliary sensing layer in adaptive industrial HMI systems.

Abstract

Thermal Imaging

Explore similar work

Thermal Imaging for Contactless Cardiorespiratory and Sudomotor Response Monitoring

A Skin-Tone-Aware Dual-Representation Remote Photoplethysmography Framework for Contactless Respiratory Rate Estimation

Video-based detection of cessation of breathing in pre-term infants using machine learning