Towards Privacy-Preserving Thermal Human Perception from Dataset to Deployment

Kailai Sun; Yue Cui; Xinwei Wang; Zijie Gao; Baoshen Guo; Zhiqing Hong; Irfan Qaisar; Xu Yang

cs.CVOct 10, 2026

Towards Privacy-Preserving Thermal Human Perception from Dataset to Deployment

Authors: Kailai Sun, Yue Cui, Xinwei Wang, Zijie Gao, Baoshen Guo, Zhiqing Hong, Irfan Qaisar, Xu Yang,

Organizations: Urban Mobility Lab, Massachusetts Institute of Technology, Cambridge, MA 02139, United States · Singapore-MIT Alliance for Research and Technology Centre (SMART), Massachusetts Institute of Technology, Singapore 138602 · Chongqing General Hospital, Chongqing University, Chongqing 401147, China · Hong Kong Xsmart Century Technology Co., Ltd., Hong Kong, China ·

Abstract

Indoor human perception is the foundation for smart homes and healthcare. However, RGB cameras raise privacy concerns in private indoor environments, while low-resolution thermal cameras provide limited spatial and semantic information. Large-scale indoor thermal datasets and real-world deployments remain limited. Here, we introduce Indoor RGB-T, a large-scale paired RGB--thermal dataset for indoor human perception. The dataset covers 20 indoor rooms and contains 90,576 80x62 thermal frames, 87,016 temporally matched RGB--thermal pairs, and 115,557 annotated human bounding boxes. We establish benchmarks for robust thermal occupant-state detection with 0.75 mAP and occupancy estimation with an accuracy of 0.847 in unseen rooms. We further evaluate RGB-to-thermal and thermal-to-RGB translation, achieving a temperature MAE of 0.654 C and an R2 of 0.923. Finally, we deploy the thermal perception model on a commercial sleep-monitoring system in the real world. Field testing with 100 participants and deployment of approximately 1,000 devices further demonstrate its practical feasibility. We identify practical challenges that motivate future research directions, including environmental thermal interference, sensing range and multi-person occlusion. The code, trained models and partial dataset are available at https://github.com/kailaisun/RGBT and https://huggingface.co/skl24/RGBT.

Figures & tables

Dataset	Collection Environment	Thermal Resolution	Task	BBox	State	Count	Thermal Interference	RGB–Thermal Samples
KAIST [ 21 ]	Urban traffic	$320\times 256$	Pedestrian detection	✓	–	–	NR	95k
LLVIP [ 22 ]	Urban traffic	$1280\times 720$	Pedestrian detection	✓	–	–	NR	15k
M 3 FD [ 23 ]	Urban traffic	$640\times 512$	Object detection	✓	–	–	NR	4k
MFNet [ 24 ]	Urban traffic	$640\times 480$	Semantic segmentation	–	–	–	NR	1.5k
DarkAct [ 37 ]	Indoor/outdoor	NR	Human action recognition	–	✓	–	NR	12k clips
RGBDT500 [ 26 ]	Indoor/outdoor	$1920\times 1080$	Object tracking	✓	–	–	NR	203.7k

TABLE I: Comparison of representative RGB–thermal datasets.

Fig. 1: RGB–thermal data acquisition device. The Meridian Innovation MI0802M7G thermal camera and a $640\times 480$ visible-light camera are mounted in the same enclosure with a fixed relative pose.

Fig. 2: Representative examples from the dataset, showing four modalities: temperature field, pseudo-color thermal image, RGB image, and RGB–thermal overlay. Bounding boxes are annotated in the native thermal image coordinate system and projected to the RGB image for reference.

Fig. 3: Room-level occupancy composition of the dataset. Each bar represents one room and shows the distribution of annotated frames with all occupants on the bed or at least one occupant off the bed.

Room	Frames				Density		Interference		Cams
Room	Thermal	Annot.	RGB	Pairs	Boxes	Occ./fr.	Without	With	Cams
room01	6,933	6,930	4,867	6,907	8,991	1.30	–	–	2
room02	6,746	6,746	4,615	6,746	8,838	1.31	–	–	2
room03	4,278	3,938	1,515	3,936	5,263	1.34	–	–	1
room04	1,056	1,056	1,053	1,053	1,233	1.17	516	540	1
room05	5,152	5,135	5,091	5,096	6,246	1.22	1,357	3,795	1

TABLE II: Dataset statistics and acquisition conditions. Rooms are numbered room01 – room20 ; the mapping to recording sessions and venues is provided in the dataset room index.

Fig. 4: Geometry statistics of occupant bounding boxes. The distributions are computed over 110,781 boxes from 83,049 frames. (a) Box area distribution. (b) Height-to-width ratio distribution. (c) Occupant number distribution per frame.

Fig. 5: Occupant-state distribution over 110,781 boxes from 83,049 frames. (a) Overall distribution of the four occupant states. (b) Room-level state composition normalized by the number of annotated boxes in each room.

Method	$\mathrm{mAP}_{50}$	$\mathrm{mAP}_{50:95}$	Precision	Recall	Macro F1	Latency (ms)	FPS
Faster R-CNN [ 40 ]	0.723	0.412	0.751	0.649	0.640	13.036	76.7
RTMDet [ 41 ]	0.742	0.452	0.751	0.666	0.699	12.049	83.0
DINO [ 42 ]	0.681	0.438	0.686	0.613	0.630	24.378	41.0
YOLO11 [ 43 ]	0.705	0.427	0.704	0.681	0.692	5.339	187.3
RT-DETR [ 44 ]	0.692	0.393	0.733	0.633	0.668	11.659	85.8
YOLO26 [ 45 ]	0.751	0.480	0.778	0.680	0.723	5.828	171.6

TABLE III: Thermal occupancy detection performance on the room-disjoint test set.

Method	State	$\mathrm{mAP}_{50}$	$\mathrm{mAP}_{50:95}$	Precision	Recall	F1-score
Faster R-CNN [ 40 ]	Lying	0.897	0.482	0.676	0.905	0.774
	Sitting	0.834	0.486	0.714	0.839	0.771
	Other	0.387	0.264	0.828	0.174	0.288
	Off-bed	0.773	0.415	0.786	0.679	0.728
RTMDet [ 41 ]	Lying	0.860	0.487	0.872	0.715	0.786
	Sitting	0.794	0.518	0.721	0.758	0.739

TABLE IV: Class-wise thermal occupant state recognition performance on the room-disjoint test set.

Fig. 6: Qualitative examples of thermal occupancy detection with state recognition for four occupant states: lying, sitting, other, and off-bed.

Method	Acc	BAcc	Macro F1	1Occ. P	1Occ. R	1Occ. F1	2Occ. P	2Occ. R	2Occ. F1	ROC-AUC	Latency
ConvNeXt-T [ 46 ]	0.841	0.829	0.829	0.873	0.877	0.875	0.786	0.780	0.783	0.909	5.03
ConvNeXt-S [ 46 ]	0.842	0.824	0.828	0.862	0.893	0.877	0.804	0.754	0.778	0.908	11.46
Swin-T [ 47 ]	0.830	0.823	0.819	0.880	0.846	0.863	0.752	0.801	0.775	0.909	6.54
ResNet18 [ 48 ]	0.817	0.808	0.805	0.866	0.839	0.853	0.738	0.777	0.757	0.892	2.79
ViT-B/16 [ 49 ]	0.820	0.805	0.806	0.855	0.861	0.858	0.758	0.750	0.754	0.895	3.40
DINOv2 ViT-S [ 50 ]	0.702	0.679	0.679	0.764	0.764	0.764	0.594	0.595	0.595	0.745	3.84

TABLE V: Occupancy estimation performance on the room-disjoint test set. 1-Occ. and 2-Occ. denote the one-occupant and two-occupant classes, respectively. Latency is reported in milliseconds.

Method	MAE ( ∘ C) $\downarrow$	RMSE ( ∘ C) $\downarrow$	$R^{2}$ $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Pix2Pix [ 51 ]	0.753	1.000	0.896	25.519	0.572	0.068
ThermalGAN [ 52 ]	0.726	0.957	0.905	25.853	0.582	0.063
Palette DDPM [ 53 ]	2.236	2.892	0.134	18.752	0.466	0.166
U-Net [ 54 ]	0.654	0.860	0.923	26.754	0.616	0.176

TABLE VI: RGB-to-thermal translation performance on the test set. Lower MAE, RMSE, and LPIPS are better, whereas higher

R^{2}

, PSNR, and SSIM are better.

Fig. 7: Qualitative comparison of RGB-to-thermal translation results across different rooms.

Method	PSNR $\uparrow$	SSIM $\uparrow$	MS-SSIM $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$\Delta E\downarrow$	LPIPS $\downarrow$	FID $\downarrow$
Pix2Pix [ 51 ]	18.590	0.700	0.785	0.062	0.118	6.700	0.183	28.860
NAFNet [ 55 ]	16.030	0.437	0.512	0.110	0.163	10.960	0.540	65.590
ControlNet-SDXL [ 56 ]	10.520	0.398	0.186	0.237	0.302	22.830	0.701	68.790
ControlNet [ 57 ]	10.220	0.333	0.207	0.252	0.324	24.220	0.738	78.280
BBDM [ 58 ]	19.470	0.797	0.801	0.069	0.133	7.320	0.161	20.390

TABLE VII: Thermal-to-RGB synthesis performance. Higher PSNR, SSIM, and MS-SSIM are better, whereas lower MAE, RMSE,

\Delta E

, LPIPS, and FID are better.

Fig. 8: Overview of the real-world thermal sleep monitoring system, including thermal awareness, on-device inference, structured output transmission, cloud-side integration, and mobile sleep reporting.

Thermal Imaging

Thermal imaging measures infrared radiation to estimate temperature and reveal people, objects, and material changes in low-visibility conditions.

See all 77 papers

Thermal-Only Crowd Counting with Deployment-Time Privacy Protection
May 16, 2026Yifei Qian, Zhongliang Guo, Chun Tong Lei +4Thermal Imaging Crowd Counting
School of Computer Science, University of Nottingham, NG7 2RD Nottingham, U.K. · School of Computer Science, University of St Andrews, KY16 9AJ St Andrews, U.K. · Department of Data Science, City University of Hong Kong, Hong Kong SAR, China +1
TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translation
Feb 23, 2026Dong-Guw Lee, Tai Hyoung Rhee, Hyunsoo Jang +3Image-to-Image Translation Thermal Imaging
KENTECH
TAP3D: Thermal-Assisted 3D Human Point Clouds
Oct 8, 2026Xie Zhang, Chengxiao Li, Xuan Liu +13D Human Reconstruction Point Cloud Generation
The University of Hong Kong

May 16, 2026J/K move · Enter open · S save

Thermal-Only Crowd Counting with Deployment-Time Privacy Protection

Yifei Qian, Zhongliang Guo, Chun Tong Lei, Bowen Deng+3

School of Computer Science, University of Nottingham, NG7 2RD Nottingham, U.K. · School of Computer Science, University of St Andrews, KY16 9AJ St Andrews, U.K. · Department of Data Science, City University of Hong Kong, Hong Kong SAR, China+1

Thermal Imaging Crowd Counting Privacy-Preserving ML

Open paper AskPDF arXiv

While RGB-Thermal crowd counting has shown promise, the paradigm faces critical limitations: RGB data raises privacy concerns in public surveillance, and multi-modal misalignment degrades fusion performance. We propose the first thermal-only framework specifically designed for privacy-conscious crowd counting, eliminating RGB dependency at inference time and substantially reducing the privacy exposure associated with continuous RGB capture in public surveillance deployments. To mitigate thermal ambiguity, we leverage depth-to-RGB diffusion models as a cross-modal bridge, extracting discriminative features that enhance thermal representations. Critically, we demonstrate that single-step LCM denoising yields features most faithful to the structural content of the depth conditioning signal, while multi-step approaches progressively decouple features from the conditioning input and accumulate errors that degrade counting accuracy. Experiments on RGBT-CC and DroneRGBT datasets show our method achieves competitive performance against state-of-the-art RGB-T fusion methods, while requiring only thermal input during inference, eliminating the need for continuous RGB capture that constitutes the primary privacy concern in real-world surveillance deployment. The code will be made publicly available.

Abstract

Thermal Imaging

Explore similar work

Thermal-Only Crowd Counting with Deployment-Time Privacy Protection

TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translation

TAP3D: Thermal-Assisted 3D Human Point Clouds