Vision-Language Foundation Models for Comprehensive Automated Pavement Condition Assessment

Blessing Agyei Kyem; Joshua Kofi Asamoah; Anthony Dontoh; Armstrong Aboah

cs.CVApr 9, 2026

Vision-Language Foundation Models for Comprehensive Automated Pavement Condition Assessment

Authors: Blessing Agyei Kyem, Joshua Kofi Asamoah, Anthony Dontoh, Armstrong Aboah

Organizations: Department of Civil, Construction and Environmental Engineering, North Dakota State University, Fargo, 58102, ND, USA · Department of Civil, Construction and Environmental Engineering, University of Memphis, Memphis, 38152, TN, USA

Abstract

General-purpose vision-language models demonstrate strong performance in everyday domains but struggle with specialized technical fields requiring precise terminology, structured reasoning, and adherence to engineering standards. This work addresses whether domain-specific instruction tuning can enable comprehensive pavement condition assessment through vision-language models. PaveInstruct, a dataset containing 278,889 image-instruction-response pairs spanning 32 task types, was created by unifying annotations from nine heterogeneous pavement datasets. PaveGPT, a pavement foundation model trained on this dataset, was evaluated against state-of-the-art vision-language models across perception, understanding, and reasoning tasks. Instruction tuning transformed model capabilities, achieving improvements exceeding 20% in spatial grounding, reasoning, and generation tasks while producing ASTM D6433-compliant outputs. These results enable transportation agencies to deploy unified conversational assessment tools that replace multiple specialized systems, simplifying workflows and reducing technical expertise requirements. The approach establishes a pathway for developing instruction-driven AI systems across infrastructure domains including bridge inspection, railway maintenance, and building condition assessment.

Figures & tables

Figure 1: Sample annotated images across each of the individual datasets in PaveInstruct.

Figure 2: Overview of the PaveInstruct task taxonomy

Dataset	Annotation Format	Task Type
PID [ 34 ]	YOLO	Distress Detection
PaveTrack (US) [ 56 ]	YOLO	Distress Detection
UAPD [ 60 ]	Pascal VOC XML	Distress Detection
DSPS23 [ 1 ]	COCO	Severity Estimation
DSPS24 [ 1 ]	CSV	PCI Assessment
RDD2022 [ 2 ]	YOLO	Distress Detection

Table 2: Dataset Annotation Formats and Task Types

Figure 3: PaveInstruct Dataset Creation Pipeline

Figure 4: Instruction count per source dataset.

Figure 5: Number of distress classes per dataset.

Figure 6: (Left) Global distress type frequencies. (Right) Percentage distribution of distress categories by dataset.

Figure 7: Distribution of single-turn vs. multi-turn conversations.

Figure 8: Distribution of conversation turns per dialogue.

Figure 9: Distribution of answer word counts.

Figure 10: Distribution of answer formats in PaveInstruct.

Figure 11: Representative instruction-response pairs from PaveInstruct across multiple task families.

Parameter	Value
Training epochs	10
Per-device train batch size	16
Gradient accumulation steps	4
Learning rate (LLM)	$2\times 10^{-7}$
Learning rate (MM-MLP)	$5\times 10^{-4}$
Weight decay	0.01

Table 3: Key hyperparameters used for fine-tuning PaveGPT.

Figure 13: PCI prediction Mean Absolute Error (MAE) comparison. Lower values indicate better performance.

Figure 14: Zero-shot vs. fine-tuned performance on spatial grounding (mIoU). Fine-tuning on PaveInstruct yields substantial improvements across all VLMs.

Figure 15: Performance improvement (

\Delta

%) from fine-tuning on PaveInstruct across spatial grounding, reasoning, and captioning tasks compared to zero-shot settings.

Figure 16: Heatmap of fine-tuned model performance across evaluation metrics. Darker colors indicate higher scores.

		Perception						Understanding					Explanatory
		Region Analysis			Spatial Grounding		Classification	VQA		PCI Prediction			Captioning			Reasoning
Method	Setting	Distress $\uparrow$	Severity $\uparrow$	Repair $\uparrow$	mIoU $\uparrow$	Accuracy $\uparrow$	Severity $\uparrow$	Exact Accuracy $\uparrow$	Relaxed Accuracy $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	R 2 $\uparrow$	BLEU-4 $\uparrow$	ROUGE-L $\uparrow$	CIDEr $\uparrow$	Judge Score $\uparrow$	Pass Rate $\uparrow$
PaliGemma-3B	Zero-shot	–	–	–	1.55	1.48	–	–	–	–	–	–	0.00	1.30	0.00	6.38	62.5
PaliGemma-3B	Trained	15.00	29.16	1.71	10.47	8.17	62.04	11.81	33.96	31.13	42.84	-0.73	6.04	18.30	8.05	5.12	30.0
LLaVA-1.5-7B	Zero-shot	–	–	–	7.29	7.28	–	–	–	–	–	–	4.13	12.90	2.39	4.53	30.0
LLaVA-1.5-7B	Trained	33.76	42.55	4.55	23.45	19.39	71.20	17.35	45.11	14.37	22.27	0.54	10.08	22.25	15.50	6.56	56.0

Table 4: Benchmark results comparing state-of-the-art VLMs on pavement condition assessment. Zero-shot (gray rows) shows baseline performance; Trained (white rows) shows performance after fine-tuning on PaveInstruct. Perception evaluates distress detection and spatial localization; Understanding covers VQA and PCI score prediction; Explanatory assesses caption quality and reasoning ability via LLM-based evaluation. Best in bold , second-best underlined .

\uparrow

\downarrow

: higher/lower is better. – : not evaluable in zero-shot.

Figure 17: Radar chart comparing fine-tuned VLMs across six key evaluation metrics on PaveInstruct.

Figure 18: Comparison of InternVL-3.5-8B performance before and after fine-tuning on PaveInstruct across diverse task types: one-word classification, distress enumeration, and coordinate-based identification. Red highlights indicate correct alignment with ground truth, demonstrating substantial improvements from domain-specific instruction tuning.

Figure 19: Qualitative comparison of VLM responses for distress listing task. Red highlights indicate correct matches with ground truth.

Figure 20: Results on multiple-choice VQA for maintenance recommendation. Red highlights indicate correct matches with ground truth.

Figure 21: Qualitative comparison on PCI estimation with ASTM D6433-compliant reasoning. Red highlights indicate correct matches.

Figure 22: Model predictions for spatial grounding task. Red boxes indicate localized distresses.

Model	TTFT (ms) $\downarrow$	Throughput (tok/s) $\uparrow$	Memory (GB) $\downarrow$	Efficiency (TPS/GB) $\uparrow$	Total Time (s) $\downarrow$
PaliGemma-3B	60	64.3	6.02	10.7	1.554
LLaVA-1.5-7B	93	44.2	14.51	3.0	2.263
LLaVA-1.6-7B	347	39.0	15.71	2.5	2.566
InternVL-3.5-8B	307	28.7	17.58	1.6	3.481
LLaMA-3.2-11B	253	30.8	22.31	1.4	3.247
PaveGPT-7B	236	39.7	16.81	2.4	2.518

Table 5: Inference efficiency benchmarks measured on an NVIDIA A6000 GPU (48 GB). TTFT: Time to First Token. Efficiency is computed as Throughput / Memory (TPS/GB). Best results are shown in bold , second best are underlined .

Explore similar work

PaveBench: A Versatile Benchmark for Pavement Distress Perception and Interactive Vision-Language Analysis
Apr 3, 2026Dexiang Li, Zhenning Che, Haijun Zhang +3Visual Question Answering
Harbin Institute of Technology Shenzhen, China · Tianjin University Tianjin, China · Hefei University of Technology Hefei, China
EG-ARSA: An Expert-Grounded Open Model for Visual Road Safety Auditing in Low-Resource Settings
Aug 24, 2026Md Thamed Bin Zaman Chowdhury, Moazzem HossainVLM Distillation
Department of Civil Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka, Bangladesh
PEDESTRIANQA: A Benchmark for Vision-Language Models on Pedestrian Intention and Trajectory Prediction
May 23, 2026Naman Mishra, Shankar Gangisetty, C. V. JawaharVLM Evaluation Pedestrian Intention Prediction
CVIT, IIIT-Hyderabad, India.

Apr 3, 2026J/K move · Enter open · S save

PaveBench: A Versatile Benchmark for Pavement Distress Perception and Interactive Vision-Language Analysis

Dexiang Li, Zhenning Che, Haijun Zhang, Dongliang Zhou+2

Harbin Institute of Technology Shenzhen, China · Tianjin University Tianjin, China · Hefei University of Technology Hefei, China

Visual Question Answering

Open paper AskPDF arXiv

Pavement condition assessment is essential for road safety and maintenance. Existing research has made significant progress. However, most studies focus on conventional computer vision tasks such as classification, detection, and segmentation. In real-world applications, pavement inspection requires more than visual recognition. It also requires quantitative analysis, explanation, and interactive decision support. Current datasets are limited. They focus on unimodal perception. They lack support for multi-turn interaction and fact-grounded reasoning. They also do not connect perception with vision-language analysis. To address these limitations, we introduce PaveBench, a large-scale benchmark for pavement distress perception and interactive vision-language analysis on real-world highway inspection images. PaveBench supports four core tasks: classification, object detection, semantic segmentation, and vision-language question answering. It provides unified task definitions and evaluation protocols. On the visual side, PaveBench provides large-scale annotations and includes a curated hard-distractor subset for robustness evaluation. It contains a large collection of real-world pavement images. On the multimodal side, we introduce PaveVQA, a real-image question answering (QA) dataset that supports single-turn, multi-turn, and expert-corrected interactions. It covers recognition, localization, quantitative estimation, and maintenance reasoning. We evaluate several state-of-the-art methods and provide a detailed analysis. We also present a simple and effective agent-augmented visual question answering framework that integrates domain-specific models as tools alongside vision-language models. The dataset is available at: https://huggingface.co/datasets/MML-Group/PaveBench.

Abstract

Explore similar work

PaveBench: A Versatile Benchmark for Pavement Distress Perception and Interactive Vision-Language Analysis

EG-ARSA: An Expert-Grounded Open Model for Visual Road Safety Auditing in Low-Resource Settings

PEDESTRIANQA: A Benchmark for Vision-Language Models on Pedestrian Intention and Trajectory Prediction