Large Language Models for HVAC Operations in Building Energy Systems: A Critical Review of Methods, Applications, and Deployment Readiness

Alexander Neubauer; Tianzhen Hong; Han Li; Mengbo Yu; Amin Darbandi; Yannick Fürst; Martin Kriegel

cs.AISep 4, 2026

Large Language Models for HVAC Operations in Building Energy Systems: A Critical Review of Methods, Applications, and Deployment Readiness

Authors: Alexander Neubauer, Tianzhen Hong, Han Li, Mengbo Yu, Amin Darbandi, Yannick Fürst, Martin Kriegel

Abstract

Building automation systems generate rich sensor data yet remain insight-poor because heterogeneous point naming, missing metadata, and fragmented documentation obstruct their operational use. This systematic review analyses and codes 66 peer-reviewed studies on large language models (LLMs) for HVAC operations published between 2023 and March 2026. Each study is classified across five application families and three LLM method families and assessed for evidence realism, deployment readiness, and the responsibility boundary between the LLM and physical HVAC decisions. The corpus is concentrated in building energy modelling (BEM, 32 of 66 papers), while load forecasting remains too sparse for subfield-level conclusions. Only four studies reach pilot-level evidence, and none reports sustained operational deployment. No study was classified as ready-now for industry adoption; three were near-term and 63 research-only. Nevertheless, several bounded, human-in-the-loop uses merit near-term trials, including point-name normalisation, document-grounded operator support, BEM workflow assistance, and advisory interfaces around physics-based controllers. Conventional machine learning (ML), model predictive control (MPC), reinforcement learning (RL) and ontology-based tools remain more adopted for high-frequency control, short-horizon numerical forecasting, and well-posed ontology mapping, while autonomous agentic operation and unvalidated occupant proxies remain research-stage. Current evidence therefore supports LLMs primarily as semantic and workflow layers rather than autonomous HVAC controllers. Future work should prioritise field-validated benchmarks, orchestration evaluation under operational constraints, and LLM-MPC/RL architectures with bounded latency and verifiable safety properties.

Figures & tables

Figure 1: Conceptual role of the LLM semantic and workflow layer in HVAC operations. Fragmented building information sources (left) are interpreted, aligned, and routed by the LLM layer (centre) toward five HVAC application categories (right). Verification and deployment guardrails separate the LLM semantic zone from bounded physical actuation. The timeline (bottom) reflects the observed corpus shift from prompt-centred approaches (M1) toward context-grounded orchestration (M2) and the emerging priority of governed harness engineering.

Category	Mechanisms	Typical HVAC inputs	Primary risk	Representative HVAC use
M1 Prompting / Supervised	ICL, prompt constraints, fine-tuning, PEFT	Instructions, labelled sensor data	Prompt brittleness; dataset shift	FDD classification; BEM code generation
M2 Context-grounded orchestration	Retrieval over manuals, BIM documents, ontologies, memory, and sensor logs; tool-calling interfaces; agentic loops and multi-step debugging workflows	Building documents; ontologies; memory; tool outputs	Retrieval errors; latency; error propagation across chained steps	Diagnostic retrieval; multi-step BEM workflows; model context protocol (MCP)-enabled simulation access
M3 Multimodal input	Vision–language encoding of non-textual inputs	Floor plans; P&ID diagrams; spatial heatmaps; sensor plots; video	Integration complexity; governance	Multimodal setpoint supervision; video-derived modelling guidance

Table 1: Comparison of LLM adaptation categories and their HVAC deployment characteristics.

Figure 2: Review workflow, from search and screening to taxonomy coding and cross-cutting synthesis.

Figure 3: PRISMA flow diagram: records identified via Scopus, screened, assessed for eligibility, and included in the review.

Dimension	Code	Operational definition
Evidence type	Conceptual	No empirical evaluation.
	Simulation-only	Evaluation based exclusively on a simulated building, system, environment, or synthetic dataset.
	Lab study	Evaluation in a structured experimental or user-evaluation setting on purpose-collected data, including hardware or testbed experiments and participant or expert evaluations. The LLM is not deployed as part of routine building operation.
	Retrospective field data	Offline analysis of historical data collected from an operational building. The LLM is not deployed in the live workflow.
	Pilot deployment	Short-term deployment in a real building with measured operational or user outcomes.
	Operational deployment	Sustained or repeated use under routine operational conditions.

Table 2: Operational definitions for evidence and deployment coding.

Criterion	Definition	Threshold for “met”
Data realism	Evaluation uses empirical data or real-world source material rather than exclusively simulated or synthetically generated inputs	Retrospective field, lab, pilot, or operational evidence
Experimental or field realism	Evaluation extends beyond simulation-only or retrospective analysis to a structured experimental setting, pilot installation, or operational deployment	Lab, pilot, or operational evidence
Closed-loop validation	LLM output directly influences a physical or simulated control action and the response is measured	Closed-loop evaluation reported
Reproducibility	Code, dataset, or model weights are publicly available, or results are replicated across multiple runs	Public artefact or multi-run results
Safety / governance reporting	Paper explicitly discusses responsibility boundary, safety constraints, data governance, or failure modes	At least one of these dimensions addressed

Table 3: Study-quality appraisal rubric applied to all 66 corpus papers.

Figure 4: Application-by-method taxonomy (A1–A5 by M1–M3). Cell values show paper counts from the reviewed corpus. Violet border: dominant cell (A3/M2, n=15). Red border: gap cells.

Figure 5: Parallel-categories view of the reviewed corpus across application category, LLM method category, evidence type, responsibility boundary, and publication year (

N=66

). The figure complements the A

\times

M heatmap by showing how the main coding dimensions co-occur across papers. Flow colour indicates application category, allowing each A1–A5 ribbon group to be traced through method, evidence, responsibility boundary, and year. To read the figure, follow a ribbon left to right: each ribbon is a group of papers sharing the same category on every axis, and ribbon width is proportional to the number of papers in that group.

Figure 6: LLM model usage across the reviewed corpus, broken down by publication year. GPT-4/GPT-4o dominates; open-weight models appear primarily in fine-tuning studies.

Figure 7: Papers per year 2023–2026 (March), stacked by application category A1–A5 (top) and method category M1–M3 (bottom). The field shows clear acceleration after 2024, while the method mix shifts toward context-grounded orchestration.

Figure 8: Evidence realism distribution per application category. Colours indicate evidence type from conceptual to operational deployment. The corpus shows mixed offline evidence in A1, A2, A3, and A5, but A4 remains strongly simulation-heavy; only four pilot studies and no operational deployments appear in the full corpus.

Barrier type	Manifestation in corpus	Affected A $\times$ M cells	Consequence	Candidate mitigation
Semantic gap (executable vs. correct)	Generated models compile and produce plausible outputs while encoding incorrect geometry, schedules, or HVAC topology	A3 $\times$ M1, A3 $\times$ M2	Executability is reported as validation; physical fidelity is not assessed	Physical fidelity checks beyond syntax; calibration against measured data
Temporal / control latency	Agentic systems report response times of seconds to minutes per cycle; HVAC loops require sub-second to low-second decisions	A4 $\times$ M2, A4 $\times$ M3	Direct closed-loop LLM control infeasible; advisory roles only	Latency-constrained RAG designs; hard-coded physics-layer fallback controllers
Governance and data residency	Corpus relies heavily on proprietary cloud APIs; regulated occupant data (presence, occupant feedback text) cannot be sent externally	All cells, especially A5	Deployment blocked in GDPR-constrained or on-premise environments	On-premise open-weight model validation; federated inference designs
Reproducibility (model dependence)	Results tied to API model versions that are updated silently; single-run benchmarks dominate	A1 $\times$ M2, A3 $\times$ M2, A4 $\times$ M2	Single-run results overestimate reliability; cross-version replication fails	Open-weight replication; multi-run consistency reporting as standard
Safety / responsibility boundary	No formal constraint layer between LLM output and physical actuation in A4 prototypes	A4 $\times$ M2, A4 $\times$ M3	High-autonomy deployment requires safety infrastructure absent from current designs	Constraint-enforcing wrappers; independent sensor watchdogs; hard-coded fallback controllers
Security / prompt injection	LLMs ingest manuals, work orders, and operator text; few designs validate input provenance or guard against adversarial or injected instructions	All cells, especially M2	Untrusted ingested content can steer outputs or actions; BAS-connected systems expose an attack surface	Input-provenance checks; output validation; injection-resistant retrieval; model supply-chain trust

Table 4: Cross-cutting deployment barriers identified across the A

\times

M taxonomy. Affected cells refer to primary A

\times

M combinations where the barrier is most acute; it may appear less severely elsewhere. The table is derived from recurring patterns across the corpus rather than from any single paper.

Role	Cat.	Deployment	Evidence basis	Representative papers
EnergyPlus MCP server (tool infrastructure)	A3	Research-only	Example-model retrofit demo; open-source artefact; no real-building validation	[ 55 ]
RAG pipeline for heterogeneous building data	A3	Near-term	13 real buildings; ${\sim}$ $30\text{\,}\mathrm{\%}$ extraction-coverage gain	[ 53 ]
Office HVAC control via multimodal LLM	A4	Near-term	Real office, two periods ( ${\approx}$ 7.5 weeks); $47.9\text{\,}\mathrm{\%}$ energy savings (LLM-only condition)	[ 58 , 13 ]
Chiller FDD with knowledge graph	A1	Research-only	Retrospective field data; $89\text{\,}\mathrm{\%}$ accuracy; single building	[ 21 ]
Chiller maintenance copilot	A1	Research-only	Lab evaluation; real manuals	[ 22 ]
NL-to-IDF automated modelling	A3	Research-only	Simulation-only; $100\text{\,}\mathrm{\%}$ accuracy on benchmark	[ 47 ]

Table 5: Deployment readiness of representative LLM roles in HVAC operations.

Figure 9: LLM–HVAC cyber-physical stack showing the responsibility boundary between semantic and workflow support and operational action. The labels describe architectural responsibility, not the deployment-readiness classifications assigned to corpus studies.

Task type	Better default tool	Where LLMs add value	Deployment boundary
Load forecasting	Time-series ML, transfer learning, probabilistic forecasting	Generating metadata descriptions, building pipeline code, explaining forecast drivers	Human-reviewed analytics; not autonomous scheduling evidence
Closed-loop control and optimisation	MPC, RL, rule-based supervisory control, safety filters	Explaining controller actions, translating operator intent, proposing slow-time-scale tuning candidates	LLM advisory only; physics layer decides
Point naming and metadata alignment	Brick/Haystack, ontologies, graph rules, supervised taggers	Resolving ambiguous labels, extracting context from manuals and drawings, suggesting mappings for review	Human validation for spatial and equipment relations
Fault diagnosis and maintenance support	FDD algorithms, rule libraries, expert systems, knowledge graphs	Linking alarms, manuals, logs, and work orders into understandable diagnostic narratives	Recommendation only; technician or engineer acts
BEM generation and retrofit analysis	EnergyPlus/OpenStudio, calibration routines, optimisation engines	Translating requirements into model edits, orchestrating tools, documenting assumptions, comparing retrofit narratives	Modeller review before investment or control use
Technician training and procedural O&M	Manuals, standard operating procedures, training programmes	Grounded step-by-step explanations, safety reminders, context-specific procedure lookup	Training and decision support; no unsupervised field action

Table 6: Fit-for-purpose boundary between language reasoning and conventional HVAC methods. Apart from the task types, all entries state what current evidence supports rather than permanent capability limits, and may shift as tool-using agents mature.

Requirement	Purpose	Failure mode if omitted
Source grounding via retrieval over trusted documents, metadata, or logs	Anchors outputs in verified, building-specific context and reduces hallucination	FM2, FM3
Human review before execution of safety-adjacent outputs	Keeps outputs advisory, with decision authority retained by qualified humans	FM4
Hard-coded safety constraints outside the LLM layer	Enforces physical limits regardless of LLM failure mode or latency	FM5
Fallback controller or operator override independent of LLM availability	Maintains operational continuity under API outage, rate-limiting, or error	FM5, FM6
Logging of prompts, retrieved sources, outputs, and overrides	Enables auditability and post-hoc failure analysis	FM1, FM6
Task-specific evaluation on real building data before deployment	Validates performance under actual sensor noise, point naming, and topology	FM1, FM3

Table 7: Minimum deployment package for LLM roles in HVAC operations. The failure-mode column references the taxonomy in Section 6.5 .

	FW1	FW2	FW3	FW4
FM1 Reasoning brittleness	$\bullet$	$\bullet$
FM2 Semantic hallucination	$\bullet$	$\bullet$
FM3 Executable but wrong		$\bullet$	$\bullet$
FM4 Unfaithful explanations		$\bullet$
FM5 Latency / real-time control		$\bullet$	$\bullet$
FM6 Proprietary model dependence				$\bullet$

Table 8: Failure modes (FM1–FM6) mapped to future-work priorities (FW1–FW4). Filled circles indicate a direct relationship.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Ref	System scope	LLM methodology	Key result	Primary limitation	Evidence	Deployment
[ 26 ]	Chiller plant, Shenzhen	Workflow automation; itemset interpretation	GPT-4 recall $89.17\text{\,}\mathrm{\%}$ , cost $17.68	API cost; offline audit only	Retrospective field data	research-only
[ 8 ]	AHU/VAV/chiller	LoRA fine-tuning; supervised FDD classifier	Accuracy $29.5\text{\,}\mathrm{\%}$ $\rightarrow$ $100\text{\,}\mathrm{\%}$ after fine-tuning	Correct labels, flawed causal reasoning	Retrospective field data	research-only
[ 23 ]	Buildings’ HVAC fault detection	Fine-tuned DistilBERT benchmark	F1 $82\text{\,}\mathrm{\%}99\text{\,}\mathrm{\%}$	Simulation-data benchmark; no field data	Simulation-only	research-only
[ 20 ]	Facility management work orders	GPT-3.5 advisory NL-to-action workflow	Advisory NL-to-action integration	Work-order completeness dependency	Retrospective field data	research-only
[ 25 ]	Metro station chilled water	LLM-guided ML virtual flowmeter	RMSE $8.84\text{\,}{\mathrm{m}}^{3}\text{\,}{\mathrm{h}}^{-1}9.05\text{\,}{\mathrm{m}}^{3}\text{\,}{\mathrm{h}}^{-1}$	Single-building validation only	Retrospective field data	research-only
[ 22 ]	Industrial chiller systems	RAG-based maintenance copilot	Hallucinations reduced by $31.2\text{\,}\mathrm{\%}$	Knowledge-base curation required	Lab study	research-only

Appendix

Table A1: Primary-coded A1 papers: fault detection and diagnosis. Depl. = deployment readiness.

Ref	System scope	LLM methodology	Key result	Primary limitation	Evidence	Deployment
[ 29 ]	Building Genome Project	Metadata-to-text conditioning for diffusion forecasting	MAE 10.58, CRPS 12.01	No prospective real-building validation	Retrospective field data	research-only
[ 27 ]	Two real buildings (Shenzhen, London)	Bayesian-optimised GPT pipeline	Avg. R 2 = 0.95, cost $0.318 per model trained and deployed	Semantic errors in generated code	Retrospective field data	research-only
[ 28 ]	Fujian commercial centre	Frozen BERT-base; patch reprogramming	MAE reduced $17.3\text{\,}\mathrm{\%}$ , RMSE $19.3\text{\,}\mathrm{\%}$	Short-horizon lag; single dataset	Retrospective field data	research-only

Appendix

Table A2: Primary-coded A2 papers: energy load forecasting.

Ref	System scope	LLM methodology	Key result	Primary limitation	Evidence	Deployment
[ 46 ]	152 synthetic test cases	Fine-tuned T5; NL to IDF	$100\text{\,}\mathrm{\%}$ IDF accuracy, $>$ 95\text{,}\mathrm{%}$$ effort reduction	Synthetic cases; no physical fidelity	Simulation-only	research-only
[ 47 ]	402 modelling cases	LoRA-tuned T5-11B	$100\text{\,}\mathrm{\%}$ accuracy (geometry/parameter/simulation match), $>$ 98\text{,}\mathrm{%}$$ effort reduction	Bounded benchmark setting	Simulation-only	research-only
[ 48 ]	648 LLM case studies	Few-shot prompting across 18 LLMs	${\sim}$ $25\text{\,}\mathrm{\%}$ (one-shot) to $31.5\text{\,}\mathrm{\%}$ (two-shot) of cases rated Perfect	Performance drops on complex cases	Simulation-only	research-only
[ 51 ]	3 real buildings + 1 synthetic	Few-shot BMS-to-Brick tagging	F1 $92\text{\,}\mathrm{\%}98\text{\,}\mathrm{\%}$ for Brick tagging	Spatial relation inference under-tested	Retrospective field data	research-only
[ 93 ]	$>$ 50 validation instances	Text/image to Honeybee/EnergyPlus	$100\text{\,}\mathrm{\%}$ convergence, avg. 7.2 s (text), 13.4 s (image)	Image-to-mesh geometry over-simplification	Simulation-only	research-only
[ 52 ]	building sensor system	LLM-integrated sensor-data protocol	Pilot natural-language interaction	Token limit of 4K constrains use	Pilot deployment	research-only

Appendix

Table A3: Selected primary-coded A3 papers: building energy modelling, simulation, and interoperability.

Ref	System scope	LLM methodology	Key result	Primary limitation	Evidence	Deployment
[ 58 ]	Real office, 30 workers, two periods ( ${\approx}$ 7.5 weeks)	Multimodal prompting; IoT + occupant feedback	$47.9\text{\,}\mathrm{\%}$ energy savings, $26.4\text{\,}\mathrm{\%}$ comfort gain (LLM-only condition)	Prediction divergence in dynamic weather	Pilot deployment	near-term
[ 67 ]	Tertiary hospital HVAC system	Retrieval-grounded optimisation support	$1.9\text{\,}\mathrm{\%}$ energy-saving gain	Output consistency remains fragile	Simulation-only	research-only
[ 66 ]	Building energy-system control design	End-to-end controller synthesis	Comparable control quality without manual design	Simulation-only validation	Simulation-only	research-only
[ 65 ]	Multi-zone office building	RAG-based setpoint-control interface	Interpretable setpoints with energy savings	Uncertain LLM outputs	Simulation-only	research-only
[ 60 ]	Building energy management system	RCoT explanation layer	Improved explanation quality	Simulation-tested; no deployment validation	Simulation-only	research-only
[ 70 ]	HVAC temperature and humidity control	LLM-enhanced MPC strategy	Simulation-only energy reduction claims	No real-building validation	Simulation-only	research-only

Appendix

Table A4: Selected primary-coded A4 papers: HVAC control and optimisation. Rows show the studies discussed in Section 4.4 ; the advisory-MPC wrapper of Liang et al. [63] is characterised in Tables 5 and A8 instead.

Ref	System scope	LLM methodology	Key result	Primary limitation	Evidence	Deployment
[ 72 ]	Office IEQ monitoring	Context-grounded comfort interface	$94\text{\,}\mathrm{\%}$ accuracy, $92\text{\,}\mathrm{\%}$ precision	Limited generalisation beyond the case setting	Retrospective field data	research-only
[ 74 ]	$14\,622$ IEQ feedback entries	Fine-tuned IEQ-BERT	$93\text{\,}\mathrm{\%}$ accuracy, F1 = 0.93	Public web-app deployment; no BAS integration	Retrospective field data	research-only
[ 75 ]	Office data, Hebei and National University of Singapore	Few-shot ICL for dynamic cooling behaviour	$95.8\text{\,}\mathrm{\%}$ binary occupancy-detection accuracy (Gemini)	No longitudinal preference learning	Retrospective field data	research-only
[ 73 ]	Conversational comfort requests	Function-calling LLM with RL interface	$95\text{\,}\mathrm{\%}$ function-call accuracy	Research-only control integration	Lab study	research-only
[ 76 ]	Conversational preference learning	M1 comfort dialogue model	Month-long participant study in a real office yielding 411 trajectories; model trained and evaluated offline on that dataset	Behaviour drift not tested	Lab study	research-only
[ 78 ]	Persona-based comfort simulation	Human-in-the-loop preference modelling	Simulation-only preference proxy	Real physiology not validated	Simulation-only	research-only

Appendix

Table A5: Primary-coded A5 papers: thermal comfort and occupant interaction.

Use case	Typical inputs	Typical outputs	Reported cost / latency	Use horizon	Evidence
Point-name cleanup and metadata alignment	BAS point lists; metadata exports	Normalised names; Brick / Haystack tags		Bounded trial	[ 51 , 81 ]
Grounded document and knowledge-base support	Manuals; BAS documents; retrieved tables and images	Grounded answers; diagnostic support; structured metadata	5–20 s	Bounded trial	[ 72 , 22 , 53 ]
BEM workflow assistance	Natural-language model descriptions; IDF / OpenStudio files	Generated or debugged models; calibration support	7.2 s (text path)	Bounded trial	[ 93 , 49 , 12 ]
Structured NL-to-IDF generation on bounded tasks	Constrained natural-language model descriptions	Executable IDF candidates		Validate first	[ 47 ]
Controller advisory or optimisation support	Forecasts; controller state; comfort and constraint signals	Explanations; tuning suggestions; slow-time-scale advisory setpoints	$0.013–0.032 / call	Validate first	[ 63 , 67 ]
Natural-language tool access to EnergyPlus	Natural-language simulation requests	Tool calls; model queries; workflow automation		Bounded trial	[ 55 ]

Appendix

Table A6: Fit-for-purpose deployment archetypes distilled from this corpus. The Use horizon column reports role-level practitioner guidance synthesised across the corpus; it is not a per-study deployment-readiness code. Empty cost/latency cells indicate that the source papers did not report a direct figure.

Stakeholder	Defensible LLM role	Main risk	Required guardrail
Facility operator	Document-grounded troubleshooting; alarm interpretation; maintenance procedure lookup	Hallucinated diagnosis	Source citation; human approval before action
Controls engineer	BEM or MPC workflow support; automated model calibration assistance	Wrong assumptions propagated to model or controller	Simulation check; qualified review before investment or control use
Building owner / asset manager	Portfolio point-name normalisation; metadata cleanup and alignment	Inconsistent tagging at scale	Ontology alignment check; spot-check audit trail
Occupant	Comfort feedback interface; preference capture	Privacy exposure; data residency	Consent mechanism; on-premise processing
Regulator / auditor	Explanation and traceability of control decisions	Unverifiable or incomplete reasoning chains	Logged evidence chain; human expert countersignature

Appendix

Table A7: Defensible LLM roles, main risks, and required guardrails by stakeholder group.

Paper	Cat.	Use case	Reported cost	Reported latency	Context
[ 9 ]	A1	Energy audit / optimisation workflow	$\sim$ $5 / audit		Multi-agent optimisation from unstructured data
[ 21 ]	A1	Diagnostic reasoning		2–5 s / inference	Real office HVAC operational data
[ 26 ]	A1	Chiller-plant workflow automation	$17.68 / analysis run		Offline audit on real plant data
[ 27 ]	A2	Automated load-prediction pipeline	$0.318 / training and deployment		Two real buildings; Bayesian prompt search
[ 53 ]	A3	RAG building-data grounding		5–20 s	13-building campus-scale pipeline
[ 93 ]	A3	Text/image to Honeybee-EnergyPlus		7.2 s (text)	User-facing model-generation interface

Appendix

Table A8: Reported cost and latency figures in the reviewed corpus. Empty cells indicate that the source paper did not report a direct value.

Explore similar work

Jun 22, 2026

ThermoLLM: Thermodynamics-Aware HVAC Control with Spatial-Semantic Knowledge Graph

Multi-zone HVAC control is a spatial decision problem in which indoor thermal evolution and control decisions depend not only on outdoor conditions and internal heat gains but also on zone layout, physical adjacency, and delayed thermal interactions across the building. Recent LLM-based HVAC controllers have shown that prompt-based control is feasible. However, these methods typically rely on task descriptions, observation values, short textual feedback, or unstructured retrieval, which limits their ability to reason about zone coupling, thermal response, and building dynamics. This paper presents a thermodynamics-aware LLM control framework for a five-zone EnergyPlus building simulation. The controller is grounded in a physics-informed spatial knowledge graph derived from Brick-style building semantics and linked with recent interaction history. At each control step, the model receives the current building state, graph-structured spatial context, and recent environment-controller history, enabling it to make decisions that reflect both building structure and short-term thermal evolution. We evaluate the framework against standard control baselines and several LLM-based alternatives. Results show that the proposed approach achieves the best overall energy-comfort trade-off and the lowest PMV violation while maintaining energy-efficient operation.

Kirtan Bhatt, Xiachong Lin, Matthew Amos +2

UNSW Sydney · Sydney, NSW, Australia · University of New South Wales +2

Language Model-Based Control Building Energy Management Feedback Control Heating, Ventilation, and Air Conditioning Control

Ask PDF

Mar 27, 2026

Experimental study on surveillance video-based indoor occupancy measurement with occupant-centric control

Accurate occupancy information is essential for closed-loop occupant-centric control (OCC) in smart buildings. However, existing vision-based occupancy measurement methods often struggle to provide stable and accurate measurements in real indoor environments, and their implications for downstream HVAC control remain insufficiently studied. To achieve Net Zero emissions by 2050, this paper presents an experimental study of large language models (LLMs)-enhanced vision-based indoor occupancy measurement and its impact on OCC-enabled HVAC operation. Detection-only, tracking-based, and LLM-based refinement pipelines are compared under identical conditions using real surveillance data collected from a research laboratory in China, with frame-level manual ground-truth annotations. Results show that tracking-based methods improve temporal stability over detection-only measurement, while LLM-based refinement further improves occupancy measurement performance and reduces false unoccupied prediction. The best-performing pipeline, YOLOv8+DeepSeek, achieves an accuracy of 0.8824 and an F1-score of 0.9320. This pipeline is then integrated into an HVAC supervisory model predictive control framework in OpenStudio-EnergyPlus. Experimental results demonstrate that the proposed framework can support more efficient OCC operation, achieving a substantial HVAC energy-saving potential of 17.94%. These findings provide an effective methodology and practical foundation for future research in AI-enhanced smart building operations.

Irfan Qaisar, Kailai Sun, Qingshan Jia +1

Center for Intelligent and Networked Systems, Department of Automation, BNRist, Tsinghua University, Beijing, 100084, China · Urban Mobility Lab, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · Singapore-MIT Alliance for Research and Technology Centre, Massachusetts Institute of Technology, 138602, Singapore

Video Surveillance Building Energy Management Heating, Ventilation, and Air Conditioning Control

Ask PDF

Feb 18, 2026

Human-AI Collaboration in Large Language Model-Integrated Building Energy Management Systems: The Role of User Domain Knowledge and AI Literacy

This study aimed to comprehend how user domain knowledge and artificial intelligence (AI) literacy impact the effective use of human-AI interactive building energy management system (BEMS). While prior studies have investigated the potential of integrating large language models (LLMs) into BEMS or building energy modeling, very few studies have examined how user interact with such systems. We conducted a systematic role-playing experiment, where 85 human subjects interacted with an advanced generative pre-trained transformer (OpenAI GPT-4o). Participants were tasked with identifying the top five behavioral changes that could reduce home energy use with the GPT model that functioned as an LLM-integrated BEMS. Then, the collected prompt-response data and participant conclusions were analyzed using an analytical framework that hierarchically assessed and scored human-AI interactions and their home energy analysis approaches. Also, participants were classified into four groups based on their self-evaluated domain knowledge of building energy use and AI literacy, and Kruskal-Wallis H tests with post-hoc pairwise comparisons were conducted across 20 quantifiable metrics. Key takeaways include: most participants employed concise prompts (median: 16.2 words) and relied heavily on GPT's analytical capabilities; and notably, only 1 of 20 metrics, appliance identification rate, showed statistically significant group differences (p=0.037), driven by AI literacy rather than domain knowledge, suggesting an equalizing effect of LLMs across expertise levels. This study provides foundational insights into human-AI collaboration dynamics and promising development directions in the context of LLM-integrated BEMS and contributes to realizing human-centric LLM-integrated energy systems.

Wooyoung Jung, Kahyun Jeon, Prosper Babon-Ayeng

The University of Arizona · Illinois Institute of Technology

Human-AI Collaboration Human-AI Interaction Human-AI Decision Making Building Energy Management

Ask PDF