Can Large Multimodal Models Inspect Buildings? A Hierarchical Benchmark for Structural Pathology Reasoning

Hui Zhong; Yichun Gao; Luyan Liu; Hai Yang; Wang Wang; Haowei Zhang; Xinhu Zheng

cs.CVMar 20, 2026

Can Large Multimodal Models Inspect Buildings? A Hierarchical Benchmark for Structural Pathology Reasoning

Authors: Hui Zhong, Yichun Gao, Luyan Liu, Hai Yang, Wang Wang, Haowei Zhang, Xinhu Zheng

Organizations: Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Hong Kong University of Science and Technology, Hongkong, China · Hong Kong University of Science and Technology, Hong Kong, China · Hong Kong University, Hong Kong, China

Abstract

Automated building facade inspection is a critical component of urban resilience and smart city maintenance. Traditionally, this field has relied on specialized discriminative models (e.g., YOLO, Mask R-CNN) that excel at pixel-level localization but are constrained to passive perception and worse generization without the visual understandng to interpret structural topology. Large Multimodal Models (LMMs) promise a paradigm shift toward active reasoning, yet their application in such high-stakes engineering domains lacks rigorous evaluation standards. To bridge this gap, we introduce a human-in-the-loop semi-automated annotation framework, leveraging expert-proposal verification to unify 12 fragmented datasets into a standardized, hierarchical ontology. Building on this foundation, we present DefectBench, the first multi-dimensional benchmark designed to interrogate LMMs beyond basic semantic recognition. DefectBench evaluates 18 state-of-the-art (SOTA) LMMs across three escalating cognitive dimensions: Semantic Perception, Spatial Localization, and Generative Geometry Segmentation. Extensive experiments reveal that while current LMMs demonstrate exceptional topological awareness and semantic understanding (effectively diagnosing "what" and "how"), they exhibit significant deficiencies in metric localization precision ("where"). Crucially, however, we validate the viability of zero-shot generative segmentation, showing that general-purpose foundation models can rival specialized supervised networks without domain-specific training. This work provides both a rigorous benchmarking standard and a high-quality open-source database, establishing a new baseline for the advancement of autonomous AI agents in civil engineering.

Figures & tables

Figure 1. Overview of the DefectBench construction and evaluation framework.

Primary class	Sub-type	Number
Crack	Linear crack	2042
Crack	Map cracking	80
Material loss	Spalling	906
Material loss	Peeling	51
Surface stain	Corrosion	254
	Rust stain	21

Table 1. Statistical distribution of facade defect categories

Models	Q1: What defects are in the image?			Q2: How many instances of each defect type?
	Q1: What defects are in the image?			Crack		Material loss		Surface stain		External fixings		Average
	P.	R.	F1.	MAE	RE	MAE	RE	MAE	RE	MAE	RE	MAE	RE
Open-Source Large Multimodal Model
GLM-4.5V	0.6698	0.8264	0.7399	0.7454	0.4194	0.4559	0.2435	0.9815	0.5342	0.3285	0.3986	0.6278	0.3989
GLM-4.6V	0.7249	0.7778	0.7504	0.7290	0.4405	0.4435	0.2774	1.0103	0.6125	0.2053	0.3477	0.5970	0.4195
Deepseek-vl2	0.6191	0.5174	0.5637	0.9322	0.4115	0.5544	0.3885	1.1047	0.8788	0.3778	0.8294	0.7423	0.6271

Table 2. Performance benchmark of 16 state-of-the-art LMMs on the Level 1 (Semantic Understanding) task of DefectBench. Evaluation covers defect classification accuracy (Q1) and numerical counting precision (Q2) across four primary categories.

Model	Q3: Object Detection			Q4: Visual Spatial Reasoning
Model	P.	R.	F1.	P.	R.	F1.
Open-Source Large Multimodal Model
GLM-4.5V	0.3157	0.3866	0.3476	0.7131	0.6198	0.6632
GLM-4.6V	0.3587	0.3981	0.3774	0.7093	0.6653	0.6866
LLaVa-34B	0.2871	0.1366	0.1851	0.2246	0.1353	0.1688
InternVL3.5	0.2517	0.1725	0.2047	0.8462	0.2112	0.3380

Table 3. Performance comparison on object detection and visual spatial reasoning tasks.

Model	mIoU	P.	R.	F1.	PA
Qwen-image-edit-plus	0.6852	0.8110	0.7495	0.7265	0.9709
Wan2.6-image	0.6848	0.8121	0.7349	0.7192	0.9717
Doubao-seedream-4.5	0.6920	0.7867	0.7442	0.7344	0.9676
Gemini-3-pro-edit	0.7418	0.8467	0.8072	0.7875	0.9713
FLUX-2-pro-edit	0.6926	0.8032	0.7455	0.7397	0.9693

Table 4. Quantitative comparison of pixel-level segmentation performance across SOTA LMMs.

Figure 2. Visualization of generative instability and instruction alignment failures in zero-shot segmentation.

Figure 3. Cross-model qualitative evaluation of generative geometry segmentation across multiple facade defect categories.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4. UI for plug-and-play annotation platform.

Figure 5. Comparative defect detection results of different LMMs on facade images.

Task	Dataset	Number	Category	Annotation Level
classification	CCIC	20000	surface cracks	Patch-level
	BDD	436	Three categories: roof defect, cracks defect and flakes	Patch-level
	BD3	3965	Six categories: algae, major crack, minor crack, peeling, spalling, and stain	Patch-level
	HS-23K	23,688	Six categories: biological deterioration, chemical deterioration, cracks, human-induced damage, material loss, and undamaged surfaces	Patch-level
Detection	MBDD2025	14,471	Five categories: crack, leakage, abscission, corrosion, bulge	Bounding-box level
	BDW	1417	Five categories: crack, mold, peeling, stairstep crack, water seepage	Bounding-box level

Appendix

Table 5. Statistics of collected open-source datasets for building defects

Question	Prompt
Q1: Defect Identification	You are an expert in building defect analysis. You will see one image. Your task is to answer two questions about visible defects in this image. Only consider the following four primary defect types, and use exactly these English names: Crack; material_loss; Stain; External Fixings. Return your answers strictly as a single JSON object: {"answer1": "...", "answer2": "..."} . The JSON must be valid and parseable by json.loads .
Q2: Defect Counting	Based on the identified defect types, report the number of instances for each defect class. Counts must follow the same order as in Q1 and be separated by commas.
Q3: Object Detection	Detect bounding boxes for each defect instance identified previously. Each object must be returned with its category and bounding box in the format: bbox: "<x_min y_min x_max y_max>" . Return only a JSON array: ["category": "...", "bbox": "..."] . Use exactly the category names: Crack, material_loss, Stain, External Fixings.
Q4: Visual Spatial Reasoning	Analyze spatial relationships between all pairs of detected defects. Possible relations include: inclusion, overlapping, adjacency, and disjoint. Return a JSON object with the structure: {"relationships": [[i#type, relation, j#type]]} . The output must be valid JSON without any additional text.
Q5: Geometry Segmentation	Edit the input image directly to generate a binary segmentation mask for the target defect class (Crack). Crack pixels must be painted white (255); all other pixels must be black (0). All annotations, bounding boxes, labels, and numbers must be completely removed. The output image must preserve the original resolution, geometry, and viewpoint.

Appendix

Table 6. Multi-level evaluation questions and prompt specifications used in DefectBench.

Zero-Shot Referring Image Segmentation

See all 10 papers

Aug 10, 2026

MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence

Multimodal large language models (MLLMs) perform strongly on engineering imagery, yet existing benchmarks mostly test drawing recognition, information extraction, or compliance checking, leaving open whether models can combine distributed visual evidence with engineering principles to reach a conclusion. We introduce MMArch, a benchmark for architecture and civil engineering spanning ten subdomains and built entirely from figures in peer-reviewed papers. Its

1{,}212

short-answer items are produced by a decoupled planner--writer pipeline and validated through automated screening, a blind adversarial audit, and expert review, so that answering requires perceiving the relevant evidence, identifying the governing principle, and applying it, not exploiting textual or single-figure shortcuts. Evaluating

18

open-weight and proprietary MLLMs against a domain-expert panel, we find a wide gap: the strongest open-source model attains about

30\%

and the best proprietary system

52\%

, while human experts reach

95\%

, more than forty points ahead. Our error analysis shows that failures concentrate in applying principles and combining evidence across figures rather than in locating it, pointing to substantial headroom for future research. Code and data are available at https://dcx-swjtu.github.io/MMArch/.

Chenxu Du, Kang An, Tengyue Wang +8

Southwest Jiaotong University · Shanghai Jiao Tong University · South China University of Technology +7

Multimodal Reasoning Visually Grounded Reasoning Multimodal Model Evaluation

Ask PDF

Mar 20, 2026

Synergistic Perception and Generative Recomposition: A Multi-Agent Orchestration for Expert-Level Building Inspection

Building facade defect inspection is fundamental to structural health monitoring and sustainable urban maintenance, yet it remains a formidable challenge due to extreme geometric variability, low contrast against complex backgrounds, and the inherent complexity of composite defects (e.g., cracks co-occurring with spalling). Such characteristics lead to severe pixel imbalance and feature ambiguity, which, coupled with the critical scarcity of high-quality pixel-level annotations, hinder the generalization of existing detection and segmentation models. To address gaps, we propose FacadeFixer, a unified multi-agent framework that treats defect perception as a collaborative reasoning task rather than isolated recognition. Specifically,FacadeFixer orchestrates specialized agents for detection and segmentation to handle multi-type defect interference, working in tandem with a generative agent to enable semantic recomposition. This process decouples intricate defects from noisy backgrounds and realistically synthesizes them onto diverse clean textures, generating high-fidelity augmented data with precise expert-level masks. To support this, we introduce a comprehensive multi-task dataset covering six primary facade categories with pixel-level annotations. Extensive experiments demonstrate that FacadeFixer significantly outperforms state-of-the-art (SOTA) baselines. Specifically, it excels in capturing pixel-level structural anomalies and highlights generative synthesis as a robust solution to data scarcity in infrastructure inspection. Our code and dataset will be made publicly available.

Hui Zhong, Yichun Gao, Luyan Liu +4

The Hong Kong University of Science and Technology (Guangzhou), Systems Hub, China · Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things, Guangdong, China · The Hong Kong University of Science and Technology, Hongkong +1

Image Segmentation Synthetic Data Augmentation Structural Health Monitoring

Ask PDF

Jul 29, 2026

SafeBuild-Bench: A Temporal-Robust Construction Safety Benchmark with Graph-Enhanced Data Mining

Construction-safety models must handle concrete deployment risks, such as a worker standing near a scaffold edge without guardrails, rather than only recognize common objects in curated images. Yet real inspection archives are redundant, long-tailed, and collected across changing sites and months. We introduce SafeBuild-Bench, a metadata-driven benchmark for evaluating multimodal large language models on construction safety under realistic temporal and site variation. It is mined from 100K+ industrial image-text records and contains 3,314 task instances from over 3,000 expert-verified images, covering multiple-choice hazard identification and free-form hazard description. To make expert verification scalable, we develop GEMS, a graph-enhanced multimodal selection pipeline that combines a proxy-model confusion signal with graph-based diversity to identify informative candidates from redundant streams. On public instruction-tuning data, GEMS-selected subsets preserve robustness-oriented performance under small data budgets. On SafeBuild-Bench, current MLLMs remain far from reliable construction-safety understanding, with the best overall score near 60. We release the benchmark, evaluation scripts, and GEMS codebase at https://github.com/safebuild/gems.

Yi Cui, Zilin Wang, Yijie Xu +5

Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China · The Hong Kong University of Science and Technology(Guangzhou) Guangzhou, China · School of Management Northwest Polytechnical University Xi’an Xi’an, China +4

Multimodal Robustness Benchmark Construction Construction Safety Multimodal Model Evaluation

Ask PDF

Abstract

Zero-Shot Referring Image Segmentation

Explore similar work

MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence

Synergistic Perception and Generative Recomposition: A Multi-Agent Orchestration for Expert-Level Building Inspection

SafeBuild-Bench: A Temporal-Robust Construction Safety Benchmark with Graph-Enhanced Data Mining