Organizations: School of Computer Science, Fudan University, Shanghai 200433, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, 200433, China
Safe autonomous systems in complex environments require robust road anomaly segmentation to identify unknown obstacles. However, existing approaches often rely on pixel-level statistics to determine whether a region appears anomalous. This reliance leads to high false-positive rates on semantically normal background regions such as sky or vegetation, and poor recall of true Out-of-distribution (OOD) instances, thereby posing safety risks for robotic perception and decision-making. To address these challenges, we propose VL-Anomaly, a vision-language anomaly segmentation framework that incorporates semantic priors from pre-trained Vision-Language Models (VLMs). Specifically, we design a prompt learning-driven alignment module that adapts Mask2Forme's visual features to CLIP text embeddings of known categories, effectively suppressing spurious anomaly responses in background regions. At inference time, we further introduce a multi-source inference strategy that integrates text-guided similarity, CLIP-based image-text similarity and detector confidence, enabling more reliable anomaly prediction by leveraging complementary information sources. Extensive experiments demonstrate that VL-Anomaly achieves state-of-the-art performance on benchmark datasets including RoadAnomaly, SMIYC and Fishyscapes.Code is released on https://github.com/NickHezhuolin/VL-aligner-Road-anomaly-segment.
Figures & tables
Fig. 1: Comparison of Anomaly Score Maps. The first row shows the original images, the second row presents anomaly score maps generated by Mask2Anomaly, and the third row illustrates the results of our method. Our approach yields cleaner maps by suppressing false positives on semantically normal background regions such as road surface and vegetation, while more precisely highlighting true anomalies like animals.
Fig. 2: Overall architecture of VL-Anomaly. The framework integrates a segmentation backbone with CLIP-based vision–language modules. During training, the Prompt Learning-Driven Aligner (PL-Aligner) first performs pixel-level alignment between the backbone’s visual features and CLIP text embeddings of known categories, and then further establishes mask-level alignment with the decoder’s mask queries. During inference, multi-source scores from the segmentation model outputs, text-guided similarity and CLIP-based image-text similarity are fused to produce robust anomaly segmentation results.
Fig. 3: Architecture of PL-Aligner. The first layer aligns pixel-level visual features from the backbone with text embeddings, while the second layer aligns mask queries from the decoder with the pixel-level features from the first layer to achieve mask-level alignment. Standard operations such as normalization and activation functions are omitted for clarity.
Methods
RoadAnomaly
SMIYC-RA21
SMIYC-RO21
Average
FPR 95 ↓
AuPRC ↑
AuROC ↑
FPR 95 ↓
AuPRC ↑
AuROC ↑
FPR 95 ↓
AuPRC ↑
AuROC ↑
FPR 95 ↓
AuPRC ↑
AuROC ↑
MSP [ 21 ]
71.4
15.7
-
72.0
28.0
-
16.6
15.7
-
53.3
19.8
-
Entropy [ 21 ]
68.2
15.7
-
-
-
-
-
-
-
68.2
15.7
-
Mahalanobis [ 32 ]
81.1
14.4
-
87.0
20.0
-
13.1
20.1
-
60.4
18.2
-
SML [ 23 ]
70.7
17.5
-
39.5
46.8
-
36.8
3.4
-
49.0
22.6
-
JSRNet [ 33 ]
9.2
94.4
-
43.9
33.6
-
28.9
28.1
-
27.3
52.0
-
TABLE I: Evaluation on RoadAnomaly [ 18 ] , SMIYC-RA21 [ 19 ] and SMIYC-RO21 [ 19 ] .
Methods
FS Static
FS L&F
FPR 95 ↓
AuPRC ↑
AuROC ↑
FPR 95 ↓
AuPRC ↑
AuROC ↑
MSP [ 21 ]
39.8
12.9
-
44.8
1.3
-
Entropy [ 21 ]
39.7
15.4
-
44.8
2.9
-
SML [ 23 ]
20.5
52.1
-
21.9
31.7
-
SynBoost [ 9 ]
15.8
43.2
-
18.8
72.6
-
Max Entropy [ 34 ]
8.6
86.5
-
35.1
29.9
-
TABLE II: Evaluation on Fishyscapes Static and Fishyscapes Lost & Found validation datasets.
Methods
SMIYC-RA21
SMIYC-RO21
sIoU ↑
PPV ↑
F1 ∗↑
sIoU ↑
PPV ↑
F1 ∗↑
MSP [ 21 ]
15.5
15.3
5.4
19.7
15.9
6.3
Mahalanobis [ 32 ]
14.8
10.2
2.7
13.5
21.8
4.7
SML [ 23 ]
26.0
24.7
12.2
5.1
13.3
3.0
JSRNet [ 33 ]
20.2
29.3
13.7
18.6
24.5
11.0
Mask2Former [ 5 ]
25.2
18.2
15.3
5.0
21.9
4.8
TABLE III: Results on the SMIYC [ 19 ] benchmark.
Setting
FPR95↓
AuPRC ↑
AuROC ↑
FPS ↑
Baseline
4.4
46.0
93.6
8.3
+ MLP & Hand-crafted prompt
13.4
60.1
95.1
8.3
→ PL-Aligner & S text
9.2
64.0
95.1
7.9
→ Learnable prompt [ 30 ]
9.2
69.7
95.4
7.9
+ S img (Full design)
8.4
69.5
96.0
6.7
Note: Starting from the baseline, we progressively add prompts, PL-Aligner and multi-source inference components (S text :text-guided similarity, S img : CLIP-based image-text similarity). ↑ indicates higher is better, ↓ indicates lower is better.
TABLE IV: Ablation on pipeline design.
Alignment
Aligner
FPR95↓
AuPRC ↑
AuROC ↑
Pixel-only
MLP
13.4
60.1
93.6
Pixel-only
Cross-attn. [ 26 ]
9.2
69.4
95.4
Pixel-only
PL-Aligner
9.1
69.4
95.6
Mask-only
PL-Aligner
16.7
55.4
78.6
Pixel+Mask
PL-Aligner
8.4
69.5
96.0
Note: We compare aligning only with pixel-level, only with mask-level and using both layers (pixel+mask). ↑ indicates higher is better, ↓ indicates lower is better.
TABLE V: Ablation on pl-aligner layers.
Fig. 4: Qualitative comparison of anomaly segmentation results on the Road Anomaly dataset [ 18 ] . We compare the outlier score maps predicted by our method with those generated by MSP [ 21 ] and Mask2Anomaly [ 12 ] , using the same backbone for a fair comparison. For visualization, all scores are normalized to the same range. Our method more effectively suppresses false positives in semantically normal background regions, while competing approaches often yield blurred or spurious activations in these areas.
Fig. 5: Visualization of the similarity between image features and the constructed text prompts. The highlighted areas show where the model associates image regions with specific semantic categories, demonstrating the effectiveness of our prompt learning strategy in guiding cross-modal alignment.
Traditional semantic segmentation models operate under a closed-set assumption and struggle to recognize unknown or unexpected objects-an essential capability for autonomous driving. As a result, such models often misclassify or overlook out-of-distribution (OOD) road anomalies, posing safety risks in open-world environments. We present a lightweight, postprocessing, road-aware anomaly segmentation framework that requires no retraining, no OOD data, and no auxiliary supervision. Our approach builds on a mask transformer-based segmentation network by exploiting query-level mask confidence and deriving a polygonal road prior to detect gap regions that may correspond to anomalies. To further suppress false positives, we introduce a CLIP-based zero-shot semantic filtering module using in-distribution prompts, with optional generalized OOD prompts. By jointly leveraging spatial priors and semantic verification, our framework produces robust and interpretable anomaly predictions. Evaluation on three public benchmarks-Fishyscapes, SMIYC, and RoadAnomaly-shows consistently strong performance. In particular, our method outperforms the training-free baseline Maskomaly on most metrics and achieves the highest AP on Fishyscapes LostAndFound. These results demonstrate the practicality and deployability of our approach for real-world autonomous driving systems.
Zhiran Yan, Gordon Elger
Institute of Innovative Mobility (IIMo), Technical University Ingolstadt of Applied Sciences, Ingolstadt, 85049, Germany. · Fraunhofer Institute for Transportation and Infrastructure Systems IVI, Ingolstadt, 85051, Germany.
Accurate open-world obstacle detection is critical for autonomous driving. Current anomaly segmentation methods suffer from a fundamental blind spot: they over-rely on texture novelty to identify out-of-distribution (OoD) objects while ignoring contextual spatial logic. Furthermore, mitigating the resulting false positives often requires cascading massive vision models, introducing unacceptable inference latency. To address these issues, we propose Layout-Aware Road Anomaly Detection (LARAD), shifting the paradigm from appearance matching to spatial-logic reasoning. First, we introduce the Spatial-Logic Violation Synthesis (SLVS) pipeline, which generates training samples that are texture-consistent yet spatially invalid, forcing the model to learn contextual violations. Second, we augment a standard closed-set segmentation network with a lightweight, OoD-guided attention branch. Extensive experiments demonstrate that LARAD significantly enhances robustness against logical anomalies and establishes a new state-of-the-art, all while retaining the high efficiency of a single-model architecture.
Shiyi Mu, Xujie Chen, Shugong Xu
Shanghai University, Shanghai, China · Xi’an Jiaotong-Liverpool University, Suzhou, China
The reliability of a machine vision system for autonomous driving depends heavily on its training data distribution. When a vehicle encounters significantly different conditions, such as atypical obstacles, its perceptual capabilities can degrade substantially. Unlike many domains where errors carry limited consequences, failures in autonomous driving translate directly into physical risk for passengers, pedestrians, and other road users. To address this challenge, we explore Visual Anomaly Detection (VAD) as a solution. VAD enables the identification of anomalous objects not present during training, allowing the system to alert the driver when an unfamiliar situation is detected. Crucially, VAD models produce pixel-level anomaly maps that can guide driver attention to specific regions of concern without requiring any prior assumptions about the nature or form of the hazard. We benchmark eight state-of-the-art VAD methods on AnoVox, the largest synthetic dataset for anomaly detection in autonomous driving. In particular, we evaluate performance across four backbone architectures spanning from large networks to lightweight ones such as MobileNet and DeiT-Tiny. Our results demonstrate that VAD transfers effectively to road scenes. Notably, Tiny-Dinomaly achieves the best accuracy-efficiency trade-off for edge deployment, matching full-scale localization performance at a fraction of the memory cost. This study represents a concrete step toward safer, more responsible deployment of autonomous vehicles, ultimately improving protection for passengers, pedestrians, and all road users.