Organizations: Department of Civil, Construction and Environmental Engineering, North Dakota State University, Fargo, 58102, ND, USA · Department of Civil, Construction and Environmental Engineering, University of Memphis, Memphis, 38152, TN, USA
General-purpose vision-language models demonstrate strong performance in everyday domains but struggle with specialized technical fields requiring precise terminology, structured reasoning, and adherence to engineering standards. This work addresses whether domain-specific instruction tuning can enable comprehensive pavement condition assessment through vision-language models. PaveInstruct, a dataset containing 278,889 image-instruction-response pairs spanning 32 task types, was created by unifying annotations from nine heterogeneous pavement datasets. PaveGPT, a pavement foundation model trained on this dataset, was evaluated against state-of-the-art vision-language models across perception, understanding, and reasoning tasks. Instruction tuning transformed model capabilities, achieving improvements exceeding 20% in spatial grounding, reasoning, and generation tasks while producing ASTM D6433-compliant outputs. These results enable transportation agencies to deploy unified conversational assessment tools that replace multiple specialized systems, simplifying workflows and reducing technical expertise requirements. The approach establishes a pathway for developing instruction-driven AI systems across infrastructure domains including bridge inspection, railway maintenance, and building condition assessment.
Figures & tables
Figure 1: Sample annotated images across each of the individual datasets in PaveInstruct.
Figure 2: Overview of the PaveInstruct task taxonomy
Dataset
Annotation Format
Task Type
PID [ 34 ]
YOLO
Distress Detection
PaveTrack (US) [ 56 ]
YOLO
Distress Detection
UAPD [ 60 ]
Pascal VOC XML
Distress Detection
DSPS23 [ 1 ]
COCO
Severity Estimation
DSPS24 [ 1 ]
CSV
PCI Assessment
RDD2022 [ 2 ]
YOLO
Distress Detection
Table 2: Dataset Annotation Formats and Task Types
Figure 3: PaveInstruct Dataset Creation Pipeline
Figure 4: Instruction count per source dataset.
Figure 5: Number of distress classes per dataset.
Figure 6: (Left) Global distress type frequencies. (Right) Percentage distribution of distress categories by dataset.
Figure 7: Distribution of single-turn vs. multi-turn conversations.
Figure 8: Distribution of conversation turns per dialogue.
Figure 9: Distribution of answer word counts.
Figure 10: Distribution of answer formats in PaveInstruct.
Figure 11: Representative instruction-response pairs from PaveInstruct across multiple task families.
Figure 12: PaveGPT architecture
Parameter
Value
Training epochs
10
Per-device train batch size
16
Gradient accumulation steps
4
Learning rate (LLM)
2×10−7
Learning rate (MM-MLP)
5×10−4
Weight decay
0.01
Table 3: Key hyperparameters used for fine-tuning PaveGPT.
Figure 14: Zero-shot vs. fine-tuned performance on spatial grounding (mIoU). Fine-tuning on PaveInstruct yields substantial improvements across all VLMs.
Figure 15: Performance improvement ( Δ %) from fine-tuning on PaveInstruct across spatial grounding, reasoning, and captioning tasks compared to zero-shot settings.
Figure 16: Heatmap of fine-tuned model performance across evaluation metrics. Darker colors indicate higher scores.
Perception
Understanding
Explanatory
Region Analysis
Spatial Grounding
Classification
VQA
PCI Prediction
Captioning
Reasoning
Method
Setting
Distress ↑
Severity ↑
Repair ↑
mIoU ↑
Accuracy ↑
Severity ↑
Exact Accuracy ↑
Relaxed Accuracy ↑
MAE ↓
RMSE ↓
R 2 ↑
BLEU-4 ↑
ROUGE-L ↑
CIDEr ↑
Judge Score ↑
Pass Rate ↑
PaliGemma-3B
Zero-shot
–
–
–
1.55
1.48
–
–
–
–
–
–
0.00
1.30
0.00
6.38
62.5
PaliGemma-3B
Trained
15.00
29.16
1.71
10.47
8.17
62.04
11.81
33.96
31.13
42.84
-0.73
6.04
18.30
8.05
5.12
30.0
LLaVA-1.5-7B
Zero-shot
–
–
–
7.29
7.28
–
–
–
–
–
–
4.13
12.90
2.39
4.53
30.0
LLaVA-1.5-7B
Trained
33.76
42.55
4.55
23.45
19.39
71.20
17.35
45.11
14.37
22.27
0.54
10.08
22.25
15.50
6.56
56.0
Table 4: Benchmark results comparing state-of-the-art VLMs on pavement condition assessment. Zero-shot (gray rows) shows baseline performance; Trained (white rows) shows performance after fine-tuning on PaveInstruct. Perception evaluates distress detection and spatial localization; Understanding covers VQA and PCI score prediction; Explanatory assesses caption quality and reasoning ability via LLM-based evaluation. Best in bold , second-best underlined . ↑ / ↓ : higher/lower is better. – : not evaluable in zero-shot.
Figure 17: Radar chart comparing fine-tuned VLMs across six key evaluation metrics on PaveInstruct.
Figure 18: Comparison of InternVL-3.5-8B performance before and after fine-tuning on PaveInstruct across diverse task types: one-word classification, distress enumeration, and coordinate-based identification. Red highlights indicate correct alignment with ground truth, demonstrating substantial improvements from domain-specific instruction tuning.
Figure 19: Qualitative comparison of VLM responses for distress listing task. Red highlights indicate correct matches with ground truth.
Figure 20: Results on multiple-choice VQA for maintenance recommendation. Red highlights indicate correct matches with ground truth.
Figure 21: Qualitative comparison on PCI estimation with ASTM D6433-compliant reasoning. Red highlights indicate correct matches.
Figure 22: Model predictions for spatial grounding task. Red boxes indicate localized distresses.
Model
TTFT (ms) ↓
Throughput (tok/s) ↑
Memory (GB) ↓
Efficiency (TPS/GB) ↑
Total Time (s) ↓
PaliGemma-3B
60
64.3
6.02
10.7
1.554
LLaVA-1.5-7B
93
44.2
14.51
3.0
2.263
LLaVA-1.6-7B
347
39.0
15.71
2.5
2.566
InternVL-3.5-8B
307
28.7
17.58
1.6
3.481
LLaMA-3.2-11B
253
30.8
22.31
1.4
3.247
PaveGPT-7B
236
39.7
16.81
2.4
2.518
Table 5: Inference efficiency benchmarks measured on an NVIDIA A6000 GPU (48 GB). TTFT: Time to First Token. Efficiency is computed as Throughput / Memory (TPS/GB). Best results are shown in bold , second best are underlined .
Pavement condition assessment is essential for road safety and maintenance. Existing research has made significant progress. However, most studies focus on conventional computer vision tasks such as classification, detection, and segmentation. In real-world applications, pavement inspection requires more than visual recognition. It also requires quantitative analysis, explanation, and interactive decision support. Current datasets are limited. They focus on unimodal perception. They lack support for multi-turn interaction and fact-grounded reasoning. They also do not connect perception with vision-language analysis. To address these limitations, we introduce PaveBench, a large-scale benchmark for pavement distress perception and interactive vision-language analysis on real-world highway inspection images. PaveBench supports four core tasks: classification, object detection, semantic segmentation, and vision-language question answering. It provides unified task definitions and evaluation protocols. On the visual side, PaveBench provides large-scale annotations and includes a curated hard-distractor subset for robustness evaluation. It contains a large collection of real-world pavement images. On the multimodal side, we introduce PaveVQA, a real-image question answering (QA) dataset that supports single-turn, multi-turn, and expert-corrected interactions. It covers recognition, localization, quantitative estimation, and maintenance reasoning. We evaluate several state-of-the-art methods and provide a detailed analysis. We also present a simple and effective agent-augmented visual question answering framework that integrates domain-specific models as tools alongside vision-language models. The dataset is available at: https://huggingface.co/datasets/MML-Group/PaveBench.
Dexiang Li, Zhenning Che, Haijun Zhang +3
Harbin Institute of Technology Shenzhen, China · Tianjin University Tianjin, China · Hefei University of Technology Hefei, China
Road traffic injuries remain a major challenge in low- and middle-income countries, where proactive road safety auditing is limited by incomplete crash records, shortages of qualified auditors, and the high cost of large-scale field inspections. To address this problem, we propose Expert-Grounded Distillation (EGD), a novel artificial intelligence framework that transfers institutional road safety expertise into a compact vision-language model for scalable visual road safety auditing. The key innovation is a quantified expert-grounding stage in which the teacher vision-language model is calibrated against authoritative field audits. Large-scale annotation is permitted only after the teacher reaches substantial agreement with expert risk assessments (Cohen's kappa = 0.74). The calibrated teacher then generates structured supervision that is distilled into an 8-billion-parameter student vision-language model using Low-Rank Adaptation and a single leakage-free prompt. We also introduce Bangladesh Road Safety Audit (BD-ARSA), the first open, expert-grounded Bangladeshi visual road safety audit dataset containing 21,947 image-audit records with near-national coverage, and Expert-Grounded Road Safety Auditor (EG-ARSA), the first vision-language model developed specifically for this task. Experimental results show that grounded fine-tuning substantially improves ordinal risk assessment over the zero-shot baseline, while blind expert evaluation demonstrates that the compact student outperforms both its 31 billion-parameter teacher and Gemini-2.5-Flash. These findings demonstrate that EGD provides an effective and scalable engineering solution for proactive road safety auditing in resource-constrained environments.
Md Thamed Bin Zaman Chowdhury, Moazzem Hossain
Department of Civil Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka, Bangladesh
Pedestrian intention and trajectory prediction are critical for the safe deployment of autonomous driving systems, directly influencing navigation decisions in complex traffic environments. Recent advances in large vision-language models offer a powerful new paradigm for these tasks by combining high-capacity visual understanding with flexible natural language reasoning. In this work, we introduce PedestrianQA, a large-scale video-based dataset that formulates pedestrian intention and trajectory prediction as question-answering tasks augmented with structured rationales. PedestrianQA expresses richly annotated pedestrian sequences, in natural language, enabling VLMs to learn from visual dynamics, contextual cues, and interactions among traffic agents while generating concise explanations of their predictions without needing specialized architectures tailored for each task. Empirical evaluations across PIE, JAAD, TITAN, and IDD-PeD show that finetuning state-of-the-art VLMs on PedestrianQA significantly improves intention classification, trajectory forecasting accuracy, and the quality of explanatory rationales, demonstrating the strong potential of VLMs as a unified and explainable framework for safety-critical pedestrian behavior modeling.