Experimental study on surveillance video-based indoor occupancy measurement with occupant-centric control

Irfan Qaisar; Kailai Sun; Qingshan Jia; Qianchuan Zhao

eess.SYMar 27, 2026

Experimental study on surveillance video-based indoor occupancy measurement with occupant-centric control

Authors: Irfan Qaisar, Kailai Sun, Qingshan Jia, Qianchuan Zhao

Organizations: Center for Intelligent and Networked Systems, Department of Automation, BNRist, Tsinghua University, Beijing, 100084, China · Urban Mobility Lab, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · Singapore-MIT Alliance for Research and Technology Centre, Massachusetts Institute of Technology, 138602, Singapore

Abstract

Accurate occupancy information is essential for closed-loop occupant-centric control (OCC) in smart buildings. However, existing vision-based occupancy measurement methods often struggle to provide stable and accurate measurements in real indoor environments, and their implications for downstream HVAC control remain insufficiently studied. To achieve Net Zero emissions by 2050, this paper presents an experimental study of large language models (LLMs)-enhanced vision-based indoor occupancy measurement and its impact on OCC-enabled HVAC operation. Detection-only, tracking-based, and LLM-based refinement pipelines are compared under identical conditions using real surveillance data collected from a research laboratory in China, with frame-level manual ground-truth annotations. Results show that tracking-based methods improve temporal stability over detection-only measurement, while LLM-based refinement further improves occupancy measurement performance and reduces false unoccupied prediction. The best-performing pipeline, YOLOv8+DeepSeek, achieves an accuracy of 0.8824 and an F1-score of 0.9320. This pipeline is then integrated into an HVAC supervisory model predictive control framework in OpenStudio-EnergyPlus. Experimental results demonstrate that the proposed framework can support more efficient OCC operation, achieving a substantial HVAC energy-saving potential of 17.94%. These findings provide an effective methodology and practical foundation for future research in AI-enhanced smart building operations.

Figures & tables

List of Abbreviations
CO 2	Carbon Dioxide
DL	Deep Learning
DeepSORT	Deep Simple Online and Realtime Tracking
HVAC	Heating, Ventilation, and Air Conditioning
IEQ	Indoor Environmental Quality
LLM	Large Language Model

Table 1

Figure 1: Overview of the proposed occupancy measurement and control integration framework.

Figure 2: Floor plan and representative views of the experimental laboratory environment at FIT, Tsinghua University.

		Counting (Regression)		Occupied/Unoccupied (Classification)
Model	$\mathbf{n_{\text{frames}}}$	MAE	RMSE	Accuracy	Precision	Recall	F1	FN	FP
YOLOv8-only	20722	1.0984	1.5827	0.8125	0.8796	0.9051	0.8922	1685	2201
YOLOv8 +DeepSORT	20722	1.1653	1.6348	0.7964	0.8728	0.8926	0.8826	1907	2311
YOLOv8 +ByteTrack	20722	1.0502	1.5419	0.8193	0.8789	0.9154	0.8962	1503	2241
YOLOv8 +VLM (Ollama/LLaVA)	20722	0.9648	1.4216	0.8525	0.9021	0.9287	0.9152	1266	1790
YOLOv8 +LLM (DeepSeek)	20722	0.8923	1.3124	0.8824	0.9245	0.9396	0.9320	1073	1363

Table 1: Global evaluation metrics across models.

Model	Actual: Unoccupied		Actual: Occupied
Model	Pred: Unocc. (TN)	Pred: Occ. (FP)	Pred: Unocc. (FN)	Pred: Occ. (TP)
YOLOv8-only	760	2201	1685	16076
YOLOv8 +DeepSORT	650	2311	1907	15854
YOLOv8 +ByteTrack	720	2241	1503	16258
YOLOv8 +VLM (Ollama/LLaVA)	1171	1790	1266	16495
YOLOv8 +LLM (DeepSeek)	1598	1363	1073	16688

Table 2: Confusion matrices for occupied/unoccupied classification (Occupied = positive class). Totals satisfy

(\mathrm{TN}+\mathrm{FP})=2961

and

(\mathrm{FN}+\mathrm{TP})=17761

Model	ID-switches	Fragmentation
YOLOv8-only	–	–
YOLOv8 + DeepSORT	2143	1687
YOLOv8 + ByteTrack	1326	1048
YOLOv8 +VLLM (Ollama/LLaVA)	–	–
YOLOv8 + LLM (DeepSeek)	–	–

Table 3: Tracking stability metrics for identity-based occupancy measurement pipelines.

Item	Value
Weather file	Beijing, China (EnergyPlus EPW)
Simulation time step	5 minutes
Simulation horizon	Annual (extrapolated from measured days)
Comfort model	Fanger PMV/PPD
Occupancy aggregation	Frame-level $\rightarrow$ 5-minute intervals
Occupied state	Occupied if count $\geq 1$

Table 4: Simulation and MPC supervisory control parameters.

Figure 3: Monthly HVAC cooling and heating energy consumption under baseline and OCC operation.

Figure 4: Monthly average predicted percentage dissatisfied (PPD) under baseline and OCC operation.

Scenario	Avg. PPD (%)	Annual Energy Consumption (kWh)			Savings vs. Baseline (%)
Scenario	Avg. PPD (%)	Cooling	Heating	Total HVAC	Cooling	Heating	Total HVAC
Baseline	10.4699	1475.1	14127.8	15602.9	0.00	0.00	0.00
MPC (YOLOv8-only)	12.0762	1403.1	12421.5	13824.6	4.88	12.08	11.40
MPC (DeepSORT)	13.3376	1403.3	12047.9	13451.2	4.87	14.72	13.79
MPC (ByteTrack)	10.7403	1408.1	13089.0	14497.1	4.54	7.35	7.09
MPC (Ollama/LLaVA)	9.6414	1346.4	11811.5	13157.9	8.72	16.40	15.67

Table 5: Annual average predicted percentage dissatisfied (PPD), HVAC energy consumption, and savings relative to the baseline.

Explore similar work

Large Language Models for HVAC Operations in Building Energy Systems: A Critical Review of Methods, Applications, and Deployment Readiness
Sep 4, 2026Alexander Neubauer, Tianzhen Hong, Han Li +4Language Model-Based Control Heating, Ventilation, and Air Conditioning Control
FLM-Occ: Feed-forward Likelihood Maximization for Efficient Indoor Occupancy Prediction
Jun 19, 2026Guangcheng Chen, Lihuang Fang, Huaqi Tao +33D Occupancy Prediction
Southern University of Science and Technology · Shenzhen Key Laboratory of Robotics and Computer Vision · Guangdong Laboratory of Artificial Intelligence and Digital Economy (Shenzhen)
FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction
Apr 30, 2026Zeyu Jiang, Changqing Zhou, Xingxing Zuo +1Semantic Occupancy Mapping 3D Occupancy Prediction
The Hong Kong University of Science and Technology (Guangzhou) · MBZUAI

Sep 4, 2026J/K move · Enter open · S save

Large Language Models for HVAC Operations in Building Energy Systems: A Critical Review of Methods, Applications, and Deployment Readiness

Alexander Neubauer, Tianzhen Hong, Han Li, Mengbo Yu+3

Language Model-Based Control Heating, Ventilation, and Air Conditioning Control Building Energy Management LLM Evaluation

Open paper AskPDF arXiv

Building automation systems generate rich sensor data yet remain insight-poor because heterogeneous point naming, missing metadata, and fragmented documentation obstruct their operational use. This systematic review analyses and codes 66 peer-reviewed studies on large language models (LLMs) for HVAC operations published between 2023 and March 2026. Each study is classified across five application families and three LLM method families and assessed for evidence realism, deployment readiness, and the responsibility boundary between the LLM and physical HVAC decisions. The corpus is concentrated in building energy modelling (BEM, 32 of 66 papers), while load forecasting remains too sparse for subfield-level conclusions. Only four studies reach pilot-level evidence, and none reports sustained operational deployment. No study was classified as ready-now for industry adoption; three were near-term and 63 research-only. Nevertheless, several bounded, human-in-the-loop uses merit near-term trials, including point-name normalisation, document-grounded operator support, BEM workflow assistance, and advisory interfaces around physics-based controllers. Conventional machine learning (ML), model predictive control (MPC), reinforcement learning (RL) and ontology-based tools remain more adopted for high-frequency control, short-horizon numerical forecasting, and well-posed ontology mapping, while autonomous agentic operation and unvalidated occupant proxies remain research-stage. Current evidence therefore supports LLMs primarily as semantic and workflow layers rather than autonomous HVAC controllers. Future work should prioritise field-validated benchmarks, orchestration evaluation under operational constraints, and LLM-MPC/RL architectures with bounded latency and verifiable safety properties.

Abstract

Explore similar work

Large Language Models for HVAC Operations in Building Energy Systems: A Critical Review of Methods, Applications, and Deployment Readiness

FLM-Occ: Feed-forward Likelihood Maximization for Efficient Indoor Occupancy Prediction

FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction