Organizations: Faculty of Information Science and Electrical Engineering, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka 819-0395, Japan · School of Civil Engineering, Sichuan University of Science & Engineering, Zigong 643000, China · Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong, China · School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
Artificial intelligence (AI), particularly large language models (LLMs) and vision-language models (VLMs), is increasingly applied in civil and structural engineering for disaster assessment, structural analysis, and engineering decision support. However, the extent to which current VLMs possess the reasoning capabilities required for reliable engineering judgments remains unclear. In this work, we introduce CivilBench, a comprehensive benchmark for evaluating VLMs in civil engineering and disaster-related scenarios. CivilBench covers three major task families: Sequential Reasoning, Image-based Structural Understanding, and Open-world Understanding, encompassing numerical reasoning, structural damage assessment, risk inference, causal analysis, and repair planning. The benchmark is constructed through a systematic pipeline of domain-specific data collection, engineering-oriented captioning, expert annotation, question design, and quality verification. Evaluation of representative state-of-the-art VLMs reveals substantial limitations in domain-specific reasoning, with Sequential Reasoning emerging as a major bottleneck, particularly when models must infer numerical relationships from sparse structured observations. Further experiments show that numerical priors and external knowledge can alleviate these limitations, although their effectiveness depends on the model's ability to reliably interpret and use the available evidence. CivilBench provides a capability-oriented testbed for identifying reasoning limitations and evaluating progress toward reliable AI-assisted engineering decision-making. The benchmark will be publicly available at https://huggingface.co/datasets/traceuryuan/CivilBench.
Figures & tables
Figure 1: Overview of CivilBench and its capability-oriented evaluation framework for vision-language models in civil engineering.
Figure 2: Overview of the CivilBench construction pipeline: (a) data collection, (b) visual necessity test, and (c) final filtering.
Figure 3: CivilBench task taxonomy and representative examples across all capability dimensions.
Dimension
Description
Capability 1: Sequential Reasoning Tasks
Classification
Classify future event occurrence and states from structured context.
Ordinal Reasoning
Rank the severity levels of observed damage or hazard events.
Determine whether a hazardous structural event is present.
Table 1: Capability-oriented evaluation dimensions in CivilBench.
Figure 4: Sample distribution across CivilBench task categories and capability dimensions. Percentages are calculated within each category.
Figure 5: Image resolution distribution in CivilBench across (top) task categories and (bottom) fine-grained capability dimensions.
Model
A1
A2
A3
B
C
Avg
Total
Open Source
Minicpm-V [ 52 ]
64.55
20.00
43.53
43.36
42.14
42.72
43.13
Qwen2.5-VL-7B [ 4 ]
47.27
26.67
54.12
47.90
88.28
52.85
63.07
InternVL [ 13 ]
65.45
26.67
61.18
50.70
84.79
57.76
65.13
LLaVA-OneVision-7B [ 29 ]
70.00
13.33
49.41
45.45
84.54
52.55
61.73
Phi-3.5-Vision [ 1 ]
64.55
11.11
43.53
39.51
85.04
48.75
58.85
Table 2: Performance comparison of different VLMs on CivilBench (%). The best result in each column is highlighted in bold. A1: Classification, A2: Ordinal Reasoning, and A3: Change Detection, respectively; B: Image-based Structural Understanding; C: Open-world Understanding. “Avg” represents the capability-balanced average accuracy across all evaluated dimensions, whereas “Total” represents the distribution-weighted overall accuracy.
Figure 6: Comparison of capability-balanced Avg and distribution-weighted Total accuracy across VLMs on CivilBench. Color indicates Ordinal Reasoning accuracy.
Dimension
n
Share
Mean
Std
Min
Max
Range
Open
Closed
Δ
# <30
# ≥80
A1
110
11.32%
63.53
15.94
4.55
74.55
70.00
59.26
71.36
12.11
1
0
A2
90
9.26%
23.79
8.26
1.11
37.78
36.67
20.81
29.26
8.45
15
0
A3
85
8.74%
50.86
12.57
10.59
67.06
56.47
46.95
58.03
11.08
1
0
B
286
29.42%
48.13
6.86
36.01
60.49
24.48
44.98
53.91
8.93
0
0
C
401
41.26%
80.67
11.68
42.14
88.28
46.14
79.08
83.58
4.51
0
14
Table 3: Capability-level performance statistics across all evaluated models on CivilBench. A1–A3 denote Classification, Ordinal Reasoning, and Change Detection; B and C denote Image-based Structural Understanding and Open-world Understanding. Share indicates the proportion of evaluated samples; Mean, Std, Min, Max, and Range summarize model accuracy. Open and Closed report mean accuracy by model type, with Δ indicating their gap. The final two columns count models with accuracy below 30% and at least 80%. Bold values highlight key diagnostic findings.
Figure 7: Model accuracy distributions across CivilBench capability dimensions. Raincloud plots show individual model accuracies, boxplots, and density distributions; annotated values indicate mean accuracy, and n denotes sample size.
Figure 8: Distribution of event-record cues, missing cells, and long text fields in raw Ordinal Reasoning and Change Detection samples.
Statistic
Ordinal Reasoning
Change Detection
Interpretation
Source schemas
10
17
Instances come from regression sources , not one rule.
Samples per source schema
9.0 (5–10)
5.0 (5–5)
Each source relation contributes only a small number of benchmark instances.
Labeled observations per instance
4 (4–4)
4 (4–4)
Only four support examples are available for inferring the hidden target relation.
Query observations per instance
3 (3–3)
2 (2–2)
The inferred rule must be applied to unseen candidates or a held-out pair.
Unique structured fields per instance
8.8 ± 3.7
15.9 ± 11.9
The model must select relevant variables from heterogeneous structured inputs.
Fields per labeled observation
2.21 ± 0.93
3.97 ± 2.99
Larger value indicates candidate variables for each observed target label.
Table 4: Regression-derived target burden in Ordinal Reasoning and Change Detection. Statistics are computed from the final evaluated Category A samples, with 90 Ordinal Reasoning samples and 85 Change Detection samples. Values in parentheses denote ranges, and values with ± denote the mean and standard deviation. The “fields per labeled observation” value measures how many structured input attributes are associated with each support label, reflecting the variable-selection burden under sparse evidence.
Configuration
A2
A3
Δ A2 (pp)
Δ A3 (pp)
Reported baseline
26.67
62.35
—
—
Prior-guided inference
35.56
68.24
+8.89
+5.89
Prior-guided inference + retrieval
46.67
71.76
+20.00
+9.41
Table 5: Performance comparison of GPT-5-mini on A2 and A3 under different inference configurations. Accuracy is reported in percent (%), and improvements over the baseline are given in percentage points (pp). The retrieval-augmented configuration extends prior-guided inference with web retrieval.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Hong Kong University of Science and Technology, Hongkong, China · Hong Kong University of Science and Technology, Hong Kong, China +1