Vision-Language Feature Alignment for Road Anomaly Segmentation

Zhuolin He; Jiacheng Tang; Jian Pu; Xiangyang Xue

cs.CVMar 1, 2026

Vision-Language Feature Alignment for Road Anomaly Segmentation

Authors: Zhuolin He, Jiacheng Tang, Jian Pu, Xiangyang Xue

Organizations: School of Computer Science, Fudan University, Shanghai 200433, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, 200433, China

Abstract

Safe autonomous systems in complex environments require robust road anomaly segmentation to identify unknown obstacles. However, existing approaches often rely on pixel-level statistics to determine whether a region appears anomalous. This reliance leads to high false-positive rates on semantically normal background regions such as sky or vegetation, and poor recall of true Out-of-distribution (OOD) instances, thereby posing safety risks for robotic perception and decision-making. To address these challenges, we propose VL-Anomaly, a vision-language anomaly segmentation framework that incorporates semantic priors from pre-trained Vision-Language Models (VLMs). Specifically, we design a prompt learning-driven alignment module that adapts Mask2Forme's visual features to CLIP text embeddings of known categories, effectively suppressing spurious anomaly responses in background regions. At inference time, we further introduce a multi-source inference strategy that integrates text-guided similarity, CLIP-based image-text similarity and detector confidence, enabling more reliable anomaly prediction by leveraging complementary information sources. Extensive experiments demonstrate that VL-Anomaly achieves state-of-the-art performance on benchmark datasets including RoadAnomaly, SMIYC and Fishyscapes.Code is released on https://github.com/NickHezhuolin/VL-aligner-Road-anomaly-segment.

Figures & tables

Fig. 1: Comparison of Anomaly Score Maps. The first row shows the original images, the second row presents anomaly score maps generated by Mask2Anomaly, and the third row illustrates the results of our method. Our approach yields cleaner maps by suppressing false positives on semantically normal background regions such as road surface and vegetation, while more precisely highlighting true anomalies like animals.

Fig. 2: Overall architecture of VL-Anomaly. The framework integrates a segmentation backbone with CLIP-based vision–language modules. During training, the Prompt Learning-Driven Aligner (PL-Aligner) first performs pixel-level alignment between the backbone’s visual features and CLIP text embeddings of known categories, and then further establishes mask-level alignment with the decoder’s mask queries. During inference, multi-source scores from the segmentation model outputs, text-guided similarity and CLIP-based image-text similarity are fused to produce robust anomaly segmentation results.

Fig. 3: Architecture of PL-Aligner. The first layer aligns pixel-level visual features from the backbone with text embeddings, while the second layer aligns mask queries from the decoder with the pixel-level features from the first layer to achieve mask-level alignment. Standard operations such as normalization and activation functions are omitted for clarity.

Methods	RoadAnomaly			SMIYC-RA21			SMIYC-RO21			Average
Methods	FPR 95 $\downarrow$	AuPRC $\uparrow$	AuROC $\uparrow$	FPR 95 $\downarrow$	AuPRC $\uparrow$	AuROC $\uparrow$	FPR 95 $\downarrow$	AuPRC $\uparrow$	AuROC $\uparrow$	FPR 95 $\downarrow$	AuPRC $\uparrow$	AuROC $\uparrow$
MSP [ 21 ]	71.4	15.7	-	72.0	28.0	-	16.6	15.7	-	53.3	19.8	-
Entropy [ 21 ]	68.2	15.7	-	-	-	-	-	-	-	68.2	15.7	-
Mahalanobis [ 32 ]	81.1	14.4	-	87.0	20.0	-	13.1	20.1	-	60.4	18.2	-
SML [ 23 ]	70.7	17.5	-	39.5	46.8	-	36.8	3.4	-	49.0	22.6	-
JSRNet [ 33 ]	9.2	94.4	-	43.9	33.6	-	28.9	28.1	-	27.3	52.0	-

TABLE I: Evaluation on RoadAnomaly [ 18 ] , SMIYC-RA21 [ 19 ] and SMIYC-RO21 [ 19 ] .

Methods	FS Static			FS L&F
Methods	FPR 95 $\downarrow$	AuPRC $\uparrow$	AuROC $\uparrow$	FPR 95 $\downarrow$	AuPRC $\uparrow$	AuROC $\uparrow$
MSP [ 21 ]	39.8	12.9	-	44.8	1.3	-
Entropy [ 21 ]	39.7	15.4	-	44.8	2.9	-
SML [ 23 ]	20.5	52.1	-	21.9	31.7	-
SynBoost [ 9 ]	15.8	43.2	-	18.8	72.6	-
Max Entropy [ 34 ]	8.6	86.5	-	35.1	29.9	-

TABLE II: Evaluation on Fishyscapes Static and Fishyscapes Lost & Found validation datasets.

Methods	SMIYC-RA21			SMIYC-RO21
Methods	sIoU $\uparrow$	PPV $\uparrow$	F1 ${}^{*}\uparrow$	sIoU $\uparrow$	PPV $\uparrow$	F1 ${}^{*}\uparrow$
MSP [ 21 ]	15.5	15.3	5.4	19.7	15.9	6.3
Mahalanobis [ 32 ]	14.8	10.2	2.7	13.5	21.8	4.7
SML [ 23 ]	26.0	24.7	12.2	5.1	13.3	3.0
JSRNet [ 33 ]	20.2	29.3	13.7	18.6	24.5	11.0
Mask2Former [ 5 ]	25.2	18.2	15.3	5.0	21.9	4.8

TABLE III: Results on the SMIYC [ 19 ] benchmark.

Setting	$\text{FPR}_{95}$ $\downarrow$	AuPRC $\uparrow$	AuROC $\uparrow$	FPS $\uparrow$
Baseline	4.4	46.0	93.6	8.3
+ MLP & Hand-crafted prompt	13.4	60.1	95.1	8.3
$\rightarrow$ PL-Aligner & S ${}_{\text{text}}$	9.2	64.0	95.1	7.9
$\rightarrow$ Learnable prompt [ 30 ]	9.2	69.7	95.4	7.9
+ S ${}_{\text{img}}$ (Full design)	8.4	69.5	96.0	6.7
Note: Starting from the baseline, we progressively add prompts, PL-Aligner and multi-source inference components (S ${}_{\text{text}}$ :text-guided similarity, S ${}_{\text{img}}$ : CLIP-based image-text similarity). $\uparrow$ indicates higher is better, $\downarrow$ indicates lower is better.

TABLE IV: Ablation on pipeline design.

Alignment	Aligner	$\text{FPR}_{95}$ $\downarrow$	AuPRC $\uparrow$	AuROC $\uparrow$
Pixel-only	MLP	13.4	60.1	93.6
Pixel-only	Cross-attn. [ 26 ]	9.2	69.4	95.4
Pixel-only	PL-Aligner	9.1	69.4	95.6
Mask-only	PL-Aligner	16.7	55.4	78.6
Pixel+Mask	PL-Aligner	8.4	69.5	96.0
Note: We compare aligning only with pixel-level, only with mask-level and using both layers (pixel+mask). $\uparrow$ indicates higher is better, $\downarrow$ indicates lower is better.

TABLE V: Ablation on pl-aligner layers.

Fig. 4: Qualitative comparison of anomaly segmentation results on the Road Anomaly dataset [ 18 ] . We compare the outlier score maps predicted by our method with those generated by MSP [ 21 ] and Mask2Anomaly [ 12 ] , using the same backbone for a fair comparison. For visualization, all scores are normalized to the same range. Our method more effectively suppresses false positives in semantically normal background regions, while competing approaches often yield blurred or spurious activations in these areas.

Fig. 5: Visualization of the similarity between image features and the constructed text prompts. The highlighted areas show where the model associates image regions with specific semantic categories, demonstrating the effectiveness of our prompt learning strategy in guiding cross-modal alignment.

Image Segmentation

Image segmentation assigns pixels or voxels to objects and regions, supporting measurement, scene understanding, and decisions across image domains.

See all 399 papers

Road-Aware Anomaly Segmentation with Query-Guided Polygons and CLIP in Autonomous Driving
Jul 5, 2026Zhiran Yan, Gordon ElgerImage Segmentation Autonomous Driving Perception
Institute of Innovative Mobility (IIMo), Technical University Ingolstadt of Applied Sciences, Ingolstadt, 85049, Germany. · Fraunhofer Institute for Transportation and Infrastructure Systems IVI, Ingolstadt, 85051, Germany.
LARAD: Layout-Aware Road Anomaly Detection via Spatial-Logic Reasoning
Jul 14, 2026Shiyi Mu, Xujie Chen, Shugong XuAutonomous Driving Perception Anomaly Localization
Shanghai University, Shanghai, China · Xi’an Jiaotong-Liverpool University, Suzhou, China
AD4AD: Benchmarking Visual Anomaly Detection Models for Safer Autonomous Driving
Apr 16, 2026Fabrizio Genilotti, Arianna Stropeni, Gionata Grotto +4Autonomous Driving Anomaly Localization
University of Padova, Italy

Jul 5, 2026J/K move · Enter open · S save

Road-Aware Anomaly Segmentation with Query-Guided Polygons and CLIP in Autonomous Driving

Zhiran Yan, Gordon Elger

Institute of Innovative Mobility (IIMo), Technical University Ingolstadt of Applied Sciences, Ingolstadt, 85049, Germany. · Fraunhofer Institute for Transportation and Infrastructure Systems IVI, Ingolstadt, 85051, Germany.

Image Segmentation Autonomous Driving Perception OOD Detection Zero-Shot Anomaly Detection

Open paper AskPDF arXiv

Traditional semantic segmentation models operate under a closed-set assumption and struggle to recognize unknown or unexpected objects-an essential capability for autonomous driving. As a result, such models often misclassify or overlook out-of-distribution (OOD) road anomalies, posing safety risks in open-world environments. We present a lightweight, postprocessing, road-aware anomaly segmentation framework that requires no retraining, no OOD data, and no auxiliary supervision. Our approach builds on a mask transformer-based segmentation network by exploiting query-level mask confidence and deriving a polygonal road prior to detect gap regions that may correspond to anomalies. To further suppress false positives, we introduce a CLIP-based zero-shot semantic filtering module using in-distribution prompts, with optional generalized OOD prompts. By jointly leveraging spatial priors and semantic verification, our framework produces robust and interpretable anomaly predictions. Evaluation on three public benchmarks-Fishyscapes, SMIYC, and RoadAnomaly-shows consistently strong performance. In particular, our method outperforms the training-free baseline Maskomaly on most metrics and achieves the highest AP on Fishyscapes LostAndFound. These results demonstrate the practicality and deployability of our approach for real-world autonomous driving systems.

Abstract

Image Segmentation

Explore similar work

Road-Aware Anomaly Segmentation with Query-Guided Polygons and CLIP in Autonomous Driving

LARAD: Layout-Aware Road Anomaly Detection via Spatial-Logic Reasoning

AD4AD: Benchmarking Visual Anomaly Detection Models for Safer Autonomous Driving