Potholes impose billions of dollars in annual vehicle damage, yet detectors report only where potholes are, not how severe they are or what to do. Judging severity from a single RGB image is ill-posed, and ungrounded vision-language model (VLM) assessments tend to be over-conservative. We present a five-stage modular framework for the offline assessment of already-detected potholes that combines geometric measurements with VLM reasoning: perspective transformation, zero-shot SAM segmentation, geometric feature extraction, Gemini 2.5 Pro severity classification with composite risk scoring, and natural-language warning generation. We evaluate it on 3,092 potholes in 1,119 dashcam scenes against independent references. Severity agreement with a prediction-free three-annotator panel is weak (Cohen's κ=0.139, 95% CI [0.00, 0.28]; n=101) and far below the annotators' agreement with one another (κ=0.605); against physically measured depth on an independent RGB-D dataset, it is also weak (κ=0.098), although the predicted classes preserve the order of measured depths. The severity classes are therefore perceived visual categories, not depth measurements. Geometric grounding does change the warnings: relative to an image-only baseline, it moves 26.7% of scenes out of the high-urgency tier and raises critical-tier assignments by 109% (Stuart-Maxwell p<10−15), and in a blind study, three raters preferred the grounded warnings 3.5:1. A six-arm prompt ablation attributes the hybrid prompt's rule adherence to the per-pothole measurements rather than to the rubric or worked examples. At USD 0.105 per scene, it suits large-scale offline infrastructure assessment; its maneuver suggestions are advisory research outputs, not validated vehicle guidance.
Figures & tables
Fig. 1: Overview of the five-stage modular pothole hazard assessment framework. Stage 1 applies perspective transformation to raw dashcam imagery. Stage 2 performs zero-shot pothole segmentation using SAM. Stage 3 extracts deterministic geometric features from segmentation masks. Stage 4 performs VLM-based severity classification grounded in geometric context and computes composite risk scores. Stage 5 generates natural language warnings with maneuver recommendations. Structured outputs are passed sequentially between stages.
Fig. 2: Illustration of the perspective transformation process. The transformation maps the pothole-containing road region to fill the full frame, reducing foreshortening and improving pothole visibility across distances for downstream segmentation and analysis.
Fig. 3: Example SAM segmentation output on a perspective-transformed road scene. Left: transformed image with ground truth bounding box prompt. Right: Generated masks in different colors, and the bottom row shows two mask samples where the left one has a score of 0.970, and the right one has a score of 0.891.
#
Feature
Description
1
Area ( A )
Foreground pixel count of SAM mask
2
Perimeter ( P )
Arc length of external contour
3
Circularity ( C )
Shape compactness: 4πA/P2
4
Aspect Ratio ( AR )
Bounding box width-to-height ratio
5
Solidity ( S )
Mask area / convex hull area
6
Position ( pos )
Joint encoding of longitudinal distance (Near / Medium / Far) and lateral zone (Far-Left / In-Path Left / In-Path Right / Far-Right)
Risk categories: Low ( R<40 ), Medium ( 40≤R<60 ),
High ( 60≤R<80 ), Critical ( R≥80 ).
TABLE III: Composite Risk Score Components, Weights, and Thresholds
Urgency
Risk Score
Trigger Conditions
Critical
R≥80
High depth + Near distance + In wheel path, or composite risk ≥80
High
60 – 79
Medium/high depth + Near distance, or composite risk 60–79
Medium
40 – 59
Low/medium depth + Near, or high depth + Far, or composite risk 40–59
Low
R<40
Low depth + Medium/far distance, or composite risk <40
TABLE IV: Decision Rubric Mapping Risk Scores to Urgency Levels (the numeric band is primary; trigger conditions can only escalate)
Attribute
Category
Count
%
Depth
Low
2,316
74.9
Medium
603
19.5
High
173
5.6
Distance
Near
2,562
82.9
Medium
158
5.1
Far
372
12.0
TABLE V: ENPAD Dataset Statistics
Model
Low P/R
Medium P/R
High P/R
Gemini 2.5 Pro
0.65/1.00
0.73/0.53
0.57/0.40
Gemini 2.5 Flash
0.69/0.69
0.47/0.60
0.17/0.10
GPT-5
0.42/1.00
0.86/0.40
0.00/0.00
Gemini 2.5 Pro (released)
0.57/0.92
0.60/0.40
0.43/0.30
TABLE VI: Per-class precision/recall against the first panel’s consensus ( n=38 ). The first three rows are the validation run; the last row scores the released full-dataset labels. Confusion matrices (rows: consensus low/medium/high; columns: model): validation run [13,0,0;4,8,3;3,3,4] , released labels [12,1,0;5,6,4;4,3,3] .
Fig. 4: Depth maps produced by foundation monocular depth estimation models on pothole scenes. All three models smooth over pothole depressions, treating them as flat road surface. The potholes are effectively invisible in the predicted depth maps, illustrating that, on these examples, the models do not capture the localized depth discontinuities required for pothole severity assessment.
Annotator Pair
Cohen’s κ
Agreement (%)
H1 vs. H2
0.495
66.7
H1 vs. H3
0.471
64.3
H2 vs. H3
0.394
59.5
Average
0.453
63.5
TABLE VII: Human Inter-Annotator Agreement on Depth Classification ( n=42 )
Model
κ
Weighted κ (95% CI)
Agreement (%)
Gemini 2.5 Pro
0.479
0.541 [0.28,0.77]
65.8
Gemini 2.5 Flash
0.224
0.315 [0.02,0.58]
50.0
GPT-5
0.229
0.063 [−0.02,0.22]
50.0
Gemini 2.5 Pro (released labels)
0.321
0.406 [0.14,0.65]
55.3
Mean pairwise human κ=0.453 (not an upper bound; see text).
Weighted κ is quadratic; CIs are bootstrap percentile intervals.
TABLE VIII: VLM Agreement with the First Panel’s Consensus ( n=38 )
Urgency Level
Image-Only
Hybrid
Low
78
(7.0%)
58
(5.2%)
Medium
306
(27.3%)
596
(53.3%)
High
658
(58.8%)
304
(27.2%)
Critical
77
(6.9%)
161
(14.4%)
Stuart–Maxwell χ2=260.1 , df =3 , p=4.2×10−56
TABLE IX: Warning Urgency Distribution Comparison ( n=1,119 scenes)
Fig. 5: Warning urgency distribution across all 1,119 scenes for the image-only and hybrid methods. The hybrid pipeline shifts a substantial portion of high-urgency assignments to medium while increasing critical-tier assignments.
Metric
Value
Cohen’s Kappa ( κ )
0.186 (Slight)
Exact Agreement
44.6%
Adjacent Agreement
95.4%
Stuart–Maxwell χ2
260.1 (df =3 ), p<10−15
TABLE X: Agreement Between Warning Methods ( n=1,119 scenes)
Method / Criterion
κ
Interpretation
Image-only – Clarity
0.590
Moderate
Image-only – Safety
0.561
Moderate
Hybrid – Clarity
0.458
Moderate
Hybrid – Safety
0.603
Moderate
Average
0.553
Moderate
TABLE XI: Inter-Rater Reliability (Fleiss’ Kappa)
Criterion
Image-Only
Hybrid
Diff. [95% CI]
p
Clarity
3.21±1.04
4.04±0.87
+0.83[0.54,1.11]
6.6×10−6
Safety
3.25±1.12
3.98±1.01
+0.73[0.36,1.11]
7.2×10−4
TABLE XII: Human Evaluation Results ( n=50 scenes, 3 annotators, 150 total ratings). CI: scene-clustered bootstrap; p : scene-level Wilcoxon signed-rank test.
Preference
Votes
Hybrid (Method B)
92
(61.3%)
Image-Only (Method A)
26
(17.3%)
Tie
32
(21.3%)
Ratio 3.5×[2.0,7.5] ; scene-level binomial p=1.7×10−4
Fig. 6: Distribution of annotator preferences across 150 evaluator-scene pairs. The hybrid method was preferred in 61.3% of comparisons, with a 3.5:1 preference ratio over the image-only baseline.
Fig. 7: Qualitative warning-generation example for a high-hazard scene. Top: dashcam image with annotated pothole bounding boxes. Middle: warning produced by the image-only baseline. Bottom: warning produced by the multimodal hybrid pipeline, which combines the same visual input with the deterministic geometric feature vector. The hybrid output localizes hazards to wheel paths, recommends a specific steering offset, and reports a composite risk score.
Fig. 8: Qualitative warning-generation example for a minor-hazard scene with a single small defect. Top: dashcam image with annotated pothole bounding box. Middle: warning produced by the image-only baseline, which describes the scene in generic terms. Bottom: warning produced by the multimodal hybrid pipeline, which identifies the defect’s wheel-path location, suggests a slight lane-internal steering adjustment, and assigns a low composite risk score.
TABLE XV: Incremental prompt ablation with Gemini 2.5 Pro ( n=300 scenes, one generation configuration for all arms). Rubric consistency is agreement between the generated urgency and the deterministic risk-score rubric.
Reference
κ
weighted κ
human κˉ
First panel, 38 items
0.321
0.406
0.453
Second panel, same 42 items
0.218
0.342
0.759
Second panel, 59 new items
0.090
0.328
0.485
Second panel, all 101 items
0.139
0.320
0.605
All 101 items: κ 95% CI [0.00,0.28] ; human κˉ over all 102 items.
TABLE XVI: Pipeline severity agreement (released labels) against the two human panels. The second row is the like-for-like comparison with the first panel.
Stage
Compute
Time (s)
Cost ($)
1: Perspective Transform
CPU
< 0.01
—
2: SAM Segmentation
GPU
∼ 3.20
—
3: Geometric Extraction
CPU
∼ 0.05
—
4: Depth + Risk Scoring
API
∼ 13.30
∼ 0.080
5: Warning Generation
API
∼ 3.50
∼ 0.025
Total per scene
∼ 20.1
∼ 0.105
TABLE XVII: Per-Scene Processing Time and Cost Breakdown
Visually grounded reasoning links visual evidence to multi-step conclusions and actions, aiming to make multimodal systems accurate, auditable, and robust.