PaveBench: A Versatile Benchmark for Pavement Distress Perception and Interactive Vision-Language Analysis

Dexiang Li; Zhenning Che; Haijun Zhang; Dongliang Zhou; Zhao Zhang; Yahong Han

cs.CVApr 3, 2026

PaveBench: A Versatile Benchmark for Pavement Distress Perception and Interactive Vision-Language Analysis

Authors: Dexiang Li, Zhenning Che, Haijun Zhang, Dongliang Zhou, Zhao Zhang, Yahong Han

Organizations: Harbin Institute of Technology Shenzhen, China · Tianjin University Tianjin, China · Hefei University of Technology Hefei, China

Abstract

Pavement condition assessment is essential for road safety and maintenance. Existing research has made significant progress. However, most studies focus on conventional computer vision tasks such as classification, detection, and segmentation. In real-world applications, pavement inspection requires more than visual recognition. It also requires quantitative analysis, explanation, and interactive decision support. Current datasets are limited. They focus on unimodal perception. They lack support for multi-turn interaction and fact-grounded reasoning. They also do not connect perception with vision-language analysis. To address these limitations, we introduce PaveBench, a large-scale benchmark for pavement distress perception and interactive vision-language analysis on real-world highway inspection images. PaveBench supports four core tasks: classification, object detection, semantic segmentation, and vision-language question answering. It provides unified task definitions and evaluation protocols. On the visual side, PaveBench provides large-scale annotations and includes a curated hard-distractor subset for robustness evaluation. It contains a large collection of real-world pavement images. On the multimodal side, we introduce PaveVQA, a real-image question answering (QA) dataset that supports single-turn, multi-turn, and expert-corrected interactions. It covers recognition, localization, quantitative estimation, and maintenance reasoning. We evaluate several state-of-the-art methods and provide a detailed analysis. We also present a simple and effective agent-augmented visual question answering framework that integrates domain-specific models as tools alongside vision-language models. The dataset is available at: https://huggingface.co/datasets/MML-Group/PaveBench.

Figures & tables

Figure 1. Comparison between existing benchmarks, general VLM benchmarks, and PaveBench. PaveBench unifies perception and interactive vision-language analysis, providing comprehensive and domain-aware pavement distress understanding.

Figure 2. Data acquisition, annotation, and construction pipeline of the dataset for pavement distress perception tasks.

Figure 3. Overview of PaveVQA, including its construction pipeline, visual question types, and multi-turn dialogue examples.

Figure 4. Overview of data distributions in PaveBench. (a) Pavement distress category distribution. (b) Interactive analysis type distribution. (c) Fine-grained distress-condition distribution, showing a long-tailed and diverse dataset.

Dataset	View	Category	#Images	Visual Perception			Vision–Language Analysis
Dataset	View	Category	#Images	Classification	Detection	Segmentation	Single-turn	Multi-turn	Quant. QA	Expert Corr.
AigleRN ( Amhaz et al., 2016 )	Oblique	Crack only	38	✗	✗	✓	✗	✗	✗	✗
CFD ( Shi et al., 2016 )	Oblique	Crack only	118	✗	✗	✓	✗	✗	✗	✗
CRKWH100 ( Liu et al., 2019 )	Top-down	Crack only	100	✗	✗	✓	✗	✗	✗	✗
CrackTree260 ( Liu et al., 2019 )	Oblique	Crack only	260	✗	✗	✓	✗	✗	✗	✗
CrackLS315 ( Zou et al., 2012 )	Top-down	Crack only	315	✗	✗	✓	✗	✗	✗	✗

Table 1. Comparison of existing pavement and crack benchmarks with PaveBench. Here, ‘language’ denotes paired language supervision or image–text annotations.

Method	Metrics (%)
Method	Acc. ( $\uparrow$ )	Prec. ( $\uparrow$ )	Rec. ( $\uparrow$ )	F1 ( $\uparrow$ )
ConvNeXt v2 ( Woo et al., 2023 )	91.19	91.08	91.19	91.04
FASTERVIT ( Hatamizadeh et al., 2023 )	87.36	87.40	87.36	87.03
TinyNeXt ( Zeng et al., 2025 )	90.78	90.74	90.88	90.71
LSNet ( Wang et al., 2025 )	88.80	88.66	88.78	88.54
OverLoCK-T ( Lou and Yu, 2025 )	93.81	93.81	93.81	93.76

Table 2. Comparison of classification performance on PaveBench.

Method	Metrics (%)
Method	mAP ( $\uparrow$ )	AP 50 ( $\uparrow$ )	AP 75 ( $\uparrow$ )	AR ( $\uparrow$ )
YOLO2026 ( Sapkota et al., 2025 )	64.52	83.60	68.53	74.76
RemDet ( Li et al., 2025 )	68.30	73.80	62.10	77.77
MI-DETR ( Nan et al., 2025 )	64.80	72.96	66.96	87.70
DEIM ( Huang et al., 2025 )	71.84	85.87	76.91	89.82

Table 3. Comparison of detection performance on PaveBench.

Method	Metrics (%)
Method	mPrec. ( $\uparrow$ )	mRec. ( $\uparrow$ )	mF1 ( $\uparrow$ )	mIoU ( $\uparrow$ )
DeepLabV3+ ( Chen et al., 2018 )	70.26	69.57	69.79	54.10
SegFormer ( Xie et al., 2021 )	69.69	66.61	70.59	55.44
SOSNet ( Liu et al., 2025b )	70.02	70.18	69.92	54.85
SCSegamba ( Liu et al., 2025a )	71.80	70.78	71.18	56.59

Table 4. Comparison of segmentation performance on PaveBench.

VLM Model	Numerical Metrics (%)			Language Metrics (%)
VLM Model	Cls. Acc. ( $\uparrow$ )	Loc. Token-F1 ( $\uparrow$ )	Quant. MAPE ( $\downarrow$ )	ROUGE-L ( $\uparrow$ )	BLEU ( $\uparrow$ )	METEOR ( $\uparrow$ )	BERT Score ( $\uparrow$ )
Qwen2.5-VL-3B ( Bai et al., 2023 )
Base (zero-shot)	65.18	16.39	116.20	8.32	0.79	14.39	83.69
+ LoRA FT	88.24	43.77	47.01	52.40	20.82	44.66	92.86
+ Agent Aug. ( ours )	89.68	42.66	35.40	53.41	21.95	44.22	92.78
DeepSeek-VL2-small ( Lu et al., 2024 )

Table 5. Comparison of VQA paradigms on PaveBench, including zero-shot, LoRA fine-tuning, and the proposed agent-augmented framework. Results are reported on numerical and language metrics.

Explore similar work

Apr 9, 2026

Vision-Language Foundation Models for Comprehensive Automated Pavement Condition Assessment

General-purpose vision-language models demonstrate strong performance in everyday domains but struggle with specialized technical fields requiring precise terminology, structured reasoning, and adherence to engineering standards. This work addresses whether domain-specific instruction tuning can enable comprehensive pavement condition assessment through vision-language models. PaveInstruct, a dataset containing 278,889 image-instruction-response pairs spanning 32 task types, was created by unifying annotations from nine heterogeneous pavement datasets. PaveGPT, a pavement foundation model trained on this dataset, was evaluated against state-of-the-art vision-language models across perception, understanding, and reasoning tasks. Instruction tuning transformed model capabilities, achieving improvements exceeding 20% in spatial grounding, reasoning, and generation tasks while producing ASTM D6433-compliant outputs. These results enable transportation agencies to deploy unified conversational assessment tools that replace multiple specialized systems, simplifying workflows and reducing technical expertise requirements. The approach establishes a pathway for developing instruction-driven AI systems across infrastructure domains including bridge inspection, railway maintenance, and building condition assessment.

Blessing Agyei Kyem, Joshua Kofi Asamoah, Anthony Dontoh +1

Department of Civil, Construction and Environmental Engineering, North Dakota State University, Fargo, 58102, ND, USA · Department of Civil, Construction and Environmental Engineering, University of Memphis, Memphis, 38152, TN, USA

Multimodal Instruction Tuning VLM Evaluation

Ask PDF

May 25, 2026

Pixel-Level Pavement Distress Assessment Using Instance Segmentation

Automated pavement distress assessment requires more than image-level classification or coarse bounding box detection, demanding precise localization of thin, branching, and irregular cracks to achieve the geometric precision necessary for maintenance-relevant quantification. This paper presents a vision-based pavement distress analysis system based on Mask R-CNN instance segmentation and evaluates it on UWGB-StreetCrack, a custom field-collected roadway image dataset acquired with a vehicle-mounted smartphone and manually annotated with polygon labels for longitudinal cracks, transverse cracks, alligator cracks, and potholes. Five Detectron2-based Mask R-CNN backbone variants were considered under a consistent fine-tuning protocol. The best-performing model, Mask R-CNN with a ResNet-101 FPN backbone, achieved 84.23% precision, 90.04% recall, and an F1 score of 87.04% under the project-specific bounding-box matching protocol. The same model produced an aggregate predicted crack-area fraction of 2.164%, closely matching the 2.170% ground-truth crack-area fraction. To contextualize the segmentation system against a detector-oriented alternative, a CSPDarknet53-based YOLO detector was also adapted and retrained on the dataset, reaching 27.5% precision and 20.7% recall on the validation protocol. The results show that instance segmentation is a practical direction for field pavement imagery and aggregate crack-area estimation, while also exposing open challenges in annotation consistency, class imbalance, confounder rejection, and mask-level benchmarking.

Logan Dewick, Bibesh Pyakurel, Kong Pheng Yang +2

University of Wisconsin - Green Bay · Computer Science Department, University of Wisconsin–Green Bay, Green Bay, WI, USA

Image Segmentation Instance Segmentation Crack Segmentation

Ask PDF

Apr 9, 2026

CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning

Cooperative autonomous driving requires traffic scene understanding from both vehicle and infrastructure perspectives. While vision-language models (VLMs) show strong general reasoning capabilities, their performance in safety-critical traffic scenarios remains insufficiently evaluated due to the ego-vehicle focus of existing benchmarks. To bridge this gap, we present CrashSight, a large-scale vision-language benchmark for roadway crash understanding using real-world roadside camera data. The dataset comprises 250 crash videos, annotated with 13K multiple-choice question-answer pairs organized under a two-tier taxonomy. Tier 1 evaluates the visual grounding of scene context and involved parties, while Tier 2 probes higher-level reasoning, including crash mechanics, causal attribution, temporal progression, and post-crash outcomes. We benchmark 8 state-of-the-art VLMs and show that, despite strong scene description capabilities, current models struggle with temporal and causal reasoning in safety-critical scenarios. We provide a detailed analysis of failure scenarios and discuss directions for improving VLM crash understanding. The benchmark provides a standardized evaluation framework for infrastructure-assisted perception in cooperative autonomous driving. The CrashSight benchmark, including the full dataset and code, is accessible at https://mcgrche.github.io/crashsight.

Rui Gan, Junyi Ma, Pei Li +4

University of Wisconsin–Madison · University of Wyoming · Columbia University

Autonomous Driving Benchmarks Autonomous Driving Safety Evaluation Autonomous Driving Temporal Video Understanding

Ask PDF