Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry

Le Qiu; Zhuan Anne Zhang; Hok Ming Adeline Chan; Emmanuele Chersoni

cs.CEOct 12, 2026

Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry

Authors: Le Qiu, Zhuan Anne Zhang, Hok Ming Adeline Chan, Emmanuele Chersoni

Organizations: Language Science and Technology, Hong Hong Polytechnic University, Hong Kong · AAL INNOVATION Limited, Hong Kong · Journalism and Communication, Hong Kong Shue Yan University, Hong Kong · Artificial Intelligence and the Humanities, Hong Hong Polytechnic University, Hong Kong

Abstract

Large Language Models (LLMs) are increasingly explored in the Architecture, Engineering, and Construction (AEC) domain, yet their reliability in regulatory compliance remains uncertain. In this study, we evaluate the performance of LLMs on Hong Kong fire safety codes. In collaboration with domain experts, we create a localized benchmark dataset comprising both multiple-choice questions (MCQs) and open-ended questions (OEQs) to evaluate 12 widely used proprietary and open-source models. Our findings show that top-performing proprietary models achieve 80-90% accuracy on MCQs, whereas open-source models generally lag behind. However, on open-ended evaluation, expert human assessments reveal an accuracy range of only 50-70%, demonstrating that LLMs still have substantial room for improvement in specialized compliance tasks.

Figures & tables

	Identifier	In/ Out Price
Claude Opus	claude-4.5-opus	5/ 25
DeepSeek	deepseek-chat	0.007 $\sim$ 1.32 / 0.66 $\sim$ 3.96
Gemini 3	gemini-3-flash-preview	0.05 $\sim$ 1 / 3
GPT-4	gpt-4.1	0.5 $\sim$ 2 / 8
Grok-4	grok-beta	0.2 / 0.5
Qwen Plus	qwen-plus	0.4 $\sim$ 2 / 1.6 $\sim$ 6

Table 1: A summary of model identifiers and pricing 8 8 8 Prices are quoted per 1 million tokens in USD and provided for reference only (actual costs may vary depending on usage time, mode, cache status and membership tier); The thinking and non - thinking modes for DeepSeek and Gemini 3 are implemented within the same model architecture, with the mode selection controlled by API parameters.

Accuracy
Fully accurate	The response is correct, complete, and aligned with the Hong Kong Fire Safety Code (100%).
Minor inaccuracy	The response is generally correct but contains small inaccuracies or omissions that do not fundamentally affect compliance (90%).
Major issues	The response contains substantial problems. This level is further divided into four sub - categories:
	Irrelevant but real clause cited (80%).
	Omission of critical information (60%).
	Hallucination or fabricated content (30%)

Table 2: Human evaluation rubric: dimensions, descriptors and scoring ranges.

Table 3: A summary of evaluation results. The (*) marker indicates the model operates in a thinking mode, otherwise non-thinking mode by default, time cost is in seconds. The top scores are underlined .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Question:	What is the main purpose of a Qualitative Design Review (QDR) in a performance-based fire safety design?
	A) To confirm building materials meet combustibility limits
	B) To set fire safety objectives, identify design scenarios, choose acceptance criteria, and outline analysis methods
	C) To run computational fluid dynamics (CFD) simulations
	D) To commission fire service installations
Reference Answer:	B
Explanation:	A qualitative design review establishes the fire safety objectives, scenarios, acceptance criteria, and the analysis approach before any detailed quantitative work is carried out.

Appendix

Table 4: Example instances from the dataset.

Table 5: Templates of Evaluation Prompts

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs
Sep 7, 2026Pengfei Li, Naufal Suryanto, Sicheng Zhang +2LLM Safety Benchmarks LLM Safety Evaluation
Khalifa University · University of Western Australia
AI4Fire: Evaluating Large Language Models on Wildfire Tasks
Oct 7, 2026Yue Zhao, Zuobin Xiong, Xiyang Hu +3LLM Evaluation LLM Grounding
University of Southern California · University of Nevada, Las Vegas · Arizona State University +1
Evaluating Large Language Model Performance on International Maritime Dangerous Goods Code Compliance
Aug 21, 2026Alexander Thomas, Hubert P. H. Shum, Darren Nellis +4LLM Safety Benchmarks Language Model Safety Evaluation
NCB Hazcheck Limited, UK · Durham University, UK

Sep 7, 2026J/K move · Enter open · S save

SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs

Pengfei Li, Naufal Suryanto, Sicheng Zhang, Mohammad Alsharid+1

Khalifa University · University of Western Australia

LLM Safety Benchmarks LLM Safety Evaluation Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) show strong progress on vision-language tasks, yet their reliability in safety-critical settings remains underexplored. Fire-smoke understanding is central to public safety and disaster response, but most existing benchmarks lack diverse real-world scenarios and context-aware evaluation. We introduce SAFIRE, a large-scale benchmark for fire-smoke understanding in MLLMs, comprising 83K captioned images from 20 scenarios and 193K multiple-choice VQA (MCVQA) generated from a 9.7K-image subset, spanning 10 evaluation dimensions from basic perception to higher-order reasoning. A GPT-5.4-assisted multi-stage verification pipeline with MLLM majority voting ensures annotation quality. Evaluating ten open-source MLLMs (8B-38B) yields an average accuracy of 61.9%, exposing major gaps in safety-critical reasoning. We further show that adapting vision encoders with only 7% of our domain-specific data boosts fire-scene classification accuracy from 20.1% to 64.5%, indicating that carefully curated data can yield substantial gains even when data volume is limited. All datasets, models, and code are available at https://risys-lab.github.io/SAFIRE/.

Abstract

LLM Evaluation

Explore similar work

SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs

AI4Fire: Evaluating Large Language Models on Wildfire Tasks

Evaluating Large Language Model Performance on International Maritime Dangerous Goods Code Compliance