CivilBench: Benchmarking Vision-Language Models for Multimodal Information Integration in Civil Engineering
Organizations: Faculty of Information Science and Electrical Engineering, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka 819-0395, Japan · School of Civil Engineering, Sichuan University of Science & Engineering, Zigong 643000, China · Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong, China · School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
Abstract
Artificial intelligence (AI), particularly large language models (LLMs) and vision-language models (VLMs), is increasingly applied in civil and structural engineering for disaster assessment, structural analysis, and engineering decision support. However, the extent to which current VLMs possess the reasoning capabilities required for reliable engineering judgments remains unclear. In this work, we introduce CivilBench, a comprehensive benchmark for evaluating VLMs in civil engineering and disaster-related scenarios. CivilBench covers three major task families: Sequential Reasoning, Image-based Structural Understanding, and Open-world Understanding, encompassing numerical reasoning, structural damage assessment, risk inference, causal analysis, and repair planning. The benchmark is constructed through a systematic pipeline of domain-specific data collection, engineering-oriented captioning, expert annotation, question design, and quality verification. Evaluation of representative state-of-the-art VLMs reveals substantial limitations in domain-specific reasoning, with Sequential Reasoning emerging as a major bottleneck, particularly when models must infer numerical relationships from sparse structured observations. Further experiments show that numerical priors and external knowledge can alleviate these limitations, although their effectiveness depends on the model's ability to reliably interpret and use the available evidence. CivilBench provides a capability-oriented testbed for identifying reasoning limitations and evaluating progress toward reliable AI-assisted engineering decision-making. The benchmark will be publicly available at https://huggingface.co/datasets/traceuryuan/CivilBench.
Figures & tables
| Dimension | Description |
| Capability 1: Sequential Reasoning Tasks | |
| Classification | Classify future event occurrence and states from structured context. |
| Ordinal Reasoning | Rank the severity levels of observed damage or hazard events. |
| Change Detection | Identify significant condition changes over time. |
| Capability 2: Image-based Structural Understanding | |
| Event Existence | Determine whether a hazardous structural event is present. |
| Model | A1 | A2 | A3 | B | C | Avg | Total |
| Open Source | |||||||
| Minicpm-V [ 52 ] | 64.55 | 20.00 | 43.53 | 43.36 | 42.14 | 42.72 | 43.13 |
| Qwen2.5-VL-7B [ 4 ] | 47.27 | 26.67 | 54.12 | 47.90 | 88.28 | 52.85 | 63.07 |
| InternVL [ 13 ] | 65.45 | 26.67 | 61.18 | 50.70 | 84.79 | 57.76 | 65.13 |
| LLaVA-OneVision-7B [ 29 ] | 70.00 | 13.33 | 49.41 | 45.45 | 84.54 | 52.55 | 61.73 |
| Phi-3.5-Vision [ 1 ] | 64.55 | 11.11 | 43.53 | 39.51 | 85.04 | 48.75 | 58.85 |
| Dimension | Share | Mean | Std | Min | Max | Range | Open | Closed | # | # | ||
| A1 | 110 | 11.32% | 63.53 | 15.94 | 4.55 | 74.55 | 70.00 | 59.26 | 71.36 | 12.11 | 1 | 0 |
| A2 | 90 | 9.26% | 23.79 | 8.26 | 1.11 | 37.78 | 36.67 | 20.81 | 29.26 | 8.45 | 15 | 0 |
| A3 | 85 | 8.74% | 50.86 | 12.57 | 10.59 | 67.06 | 56.47 | 46.95 | 58.03 | 11.08 | 1 | 0 |
| B | 286 | 29.42% | 48.13 | 6.86 | 36.01 | 60.49 | 24.48 | 44.98 | 53.91 | 8.93 | 0 | 0 |
| C | 401 | 41.26% | 80.67 | 11.68 | 42.14 | 88.28 | 46.14 | 79.08 | 83.58 | 4.51 | 0 | 14 |
| Statistic | Ordinal Reasoning | Change Detection | Interpretation |
| Source schemas | 10 | 17 | Instances come from regression sources , not one rule. |
| Samples per source schema | 9.0 (5–10) | 5.0 (5–5) | Each source relation contributes only a small number of benchmark instances. |
| Labeled observations per instance | 4 (4–4) | 4 (4–4) | Only four support examples are available for inferring the hidden target relation. |
| Query observations per instance | 3 (3–3) | 2 (2–2) | The inferred rule must be applied to unseen candidates or a held-out pair. |
| Unique structured fields per instance | 8.8 3.7 | 15.9 11.9 | The model must select relevant variables from heterogeneous structured inputs. |
| Fields per labeled observation | 2.21 0.93 | 3.97 2.99 | Larger value indicates candidate variables for each observed target label. |
| Configuration | A2 | A3 | A2 (pp) | A3 (pp) |
| Reported baseline | 26.67 | 62.35 | — | — |
| Prior-guided inference | 35.56 | 68.24 | +8.89 | +5.89 |
| Prior-guided inference + retrieval | 46.67 | 71.76 | +20.00 | +9.41 |
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 1,058 papers