Recent advances in 3D semantic segmentation have significantly improved scene understanding in autonomous driving. However, existing methods are typically developed under a closed-set assumption and evaluated under standard driving conditions, where all categories are known in advance. In real-world driving, autonomous vehicles inevitably encounter previously unseen objects, making closed-set perception insufficient for safe deployment. To address this challenge, we introduce DriveOSS, which, to the best of our knowledge, is the first real-world multimodal dataset and benchmark specifically designed for 3D anomaly segmentation and open-set semantic segmentation. It contains 12,661 synchronized and annotated frames captured using six surround-view RGB cameras, a 128-beam LiDAR, and a GPS/IMU unit. The dataset comprises 52 real-world driving scenes collected under clear, nighttime, cloudy, rainy, and snowy conditions and is organized into ID (in-distribution) and OOD (out-of-distribution) subsets. The ID scenes contain only known classes and are used for training, whereas the OOD scenes additionally contain unknown objects annotated as the outlier class and are used for validation and testing. Extensive experiments show that unseen objects remain a major challenge for current 3D perception systems. The dataset, benchmark protocols, and code are available at https://github.com/chrisyan/DriveOSS.
Figures & tables
Figure 1: Overview of the proposed DriveOSS dataset and benchmark. Top: Data collected under diverse weather and illumination conditions, including clear, nighttime, cloudy, rainy, and snowy scenes. The left shows the in-distribution (ID) subset containing normal driving scenarios, while the right shows the out-of-distribution (OOD) subset containing unknown objects that are not used during training. Bottom: Different perception settings illustrated using the same scene. From left to right: RGB image, closed-set semantic segmentation prediction, anomaly segmentation ground truth, and open-set semantic segmentation ground truth.
Dataset
Year
Weather
Modality
#Beam
Camera
#Scan
ID Evaluation Classes
OOD Scene Labels
Supported 3D Tasks
SemanticKITTI [ 2 ]
2019
Clear
L+C
64
Front-view
43,552
19 a
–
SS, PS
Waymo [ 34 ]
2019
Clear/Night/Rainy
L+C
64
Surround-view
46,000
23
–
SS
Toronto-3D [ 35 ]
2020
Clear
L+C
32
Surround-view
–
8
–
SS
A2D2 [ 12 ]
2020
Cloudy/Rainy/Sunny
L+C
16
Surround-view
41,277
38
–
2D SS, detection
SemanticPOSS [ 27 ]
2020
Clear
L+C
40
Surround-view
2,988
14
–
SS, PS
PandaSet [ 41 ]
2021
Clear/Night
L+C
64/150
Surround-view
16,480
37
–
SS, detection
Table 1: Comparison of representative automotive 3D segmentation datasets. SS, IS, PS, AS, OSS, and SSC denote semantic, instance, panoptic, anomaly, and open-set semantic segmentation, and semantic scene completion, respectively. L and C denote LiDAR and camera, while #Beam indicates the number of LiDAR beams. The ID Evaluation Classes column reports the number of ID semantic classes used for benchmark evaluation. a SemanticKITTI provides 34 raw labels, which are merged or ignored to obtain 19 ID evaluation classes. b DriveOSS provides 32 raw annotation labels, including outlier and unlabeled . After merging semantically equivalent labels and excluding ignored labels, 19 ID semantic classes are retained for training and evaluation. The outlier label is additionally used for anomaly segmentation and open-set semantic segmentation.
Category
Representative Objects
Characteristics
Containers & Packaging
Cardboard boxes, plastic boxes, foam blocks, garbage bags, and small trash bins
Common roadside obstacles with simple geometries and diverse materials
Bulky objects with complex structures and varying heights
Tools & Industrial Equipment
Ladders, shovels, metal barriers, and maintenance equipment
Rigid structures with distinctive geometries and occasionally high reflectivity
Personal Belongings
Backpacks, bicycle helmets, bags, and other personal items
Predominantly small objects with irregular shapes and mixed materials
Utility & Service Equipment
Shopping carts, waste-collection carts, and mobile bins
Movable service-related objects commonly encountered in urban environments
Roadside Debris & Miscellaneous
Wooden boards, fallen signs, and other uncommon objects
Unstructured obstacles with diverse appearances and physical properties
Table 2: Taxonomy of OOD objects in DriveOSS.
Figure 2: DriveOSS data collection platform. Top: the sensor vehicle. Bottom: six surround-view cameras, one 360 ∘ LiDAR, and one GPS/IMU unit.
Figure 3: LiDAR–camera calibration. LiDAR points projected onto the six camera views using the estimated intrinsic and extrinsic parameters.
Figure 4: Representative anomalies in DriveOSS. Physically present objects with diverse scales, geometries, and materials, captured under different environmental conditions.
ID Train
ID Validation
OOD Validation
OOD Test
Condition
Seq.
Frames
Seq.
Frames
Seq.
Frames
OOD Points
Seq.
Frames
OOD Points
Clear
4
809
6
1,701
9
2,315
252,606
8
1,766
885,729
Nighttime
1
184
1
79
2
287
22,132
2
357
42,979
Cloudy
2
636
1
186
2
742
43,548
2
666
71,440
Rainy
1
399
1
165
2
696
35,104
2
618
66,509
Snowy
1
212
1
253
2
238
12,898
2
352
89,803
Table 3: Distribution across environmental conditions. Numbers of sequences and LiDAR frames for each split, together with annotated OOD points for DriveOSS-OOD.
Figure 5: Label distributions in DriveOSS-ID and DriveOSS-OOD. The outlier label occurs exclusively in DriveOSS-OOD, while labels marked with 1 are excluded from evaluation.
Figure 6: Spatial distribution of OOD points. Normalized OOD-point frequency relative to the ego vehicle over all DriveOSS-OOD validation and test sequences.
Method
M
ID-val
OOD-test
Δ
P3Former
L
43.27
29.58
−13.69
LCPS-lidar
L
51.94
33.75
−18.19
M4F3D
L
34.98
31.11
−3.87
LIDO
L
58.35
40.99
−17.36
LCPS
L+C
54.05
34.00
−20.05
M4F3D*
L+C
48.44
28.46
−19.98
Table 4: Closed-set 3D semantic segmentation. mIoU (%) on DriveOSS-ID validation and the ID-labeled points of DriveOSS-OOD test. Δ denotes the change from ID to OOD.
Method
M
AUROC ↑
FPR@95 ↓
AP ↑
P3Former + MSP
L
26.35
99.50
0.27
P3Former + ML
L
34.60
99.80
0.35
P3Former + Energy
L
37.94
99.79
0.65
P3Former + Entropy
L
26.63
99.57
0.37
P3Former + RbA
L
89.80
45.94
3.53
LCPS-lidar + MSP
L
78.76
47.32
1.11
Table 5: 3D anomaly segmentation on DriveOSS-OOD test. We report AUROC ↑ , FPR@95 ↓ , and AP ↑ . ML denotes MaxLogit, and M denotes the input modality. M4F3D* is the modified LiDAR–camera fusion model.
Method
M
car
bicycle
motorcycle
truck
other-veh.
person
bicyclist
motorcyclist
road
parking
sidewalk
other-ground
building
fence
vegetation
trunk
terrain
pole
traffic-sign
outlier
mOpen-IoU
P3Former + MSP
L
24.69
0.0
0.0
6.56
0.0
0.14
1.00
0.0
0.03
0.0
0.0
0.0
0.05
0.13
1.26
0.05
0.0
0.03
0.06
0.28
1.79
P3Former + ML
L
22.96
0.0
0.0
6.33
0.0
0.95
0.28
0.0
0.02
0.0
0.0
0.0
1.37
1.08
0.48
0.28
0.01
0.07
0.27
0.27
1.80
P3Former + Energy
L
25.45
0.0
0.0
6.55
0.0
1.14
0.30
0.0
0.02
0.0
0.0
0.0
1.70
1.16
0.49
0.32
0.0
0.07
0.30
0.27
1.98
P3Former + Entropy
L
24.65
0.0
0.0
6.63
0.0
0.15
0.85
0.0
0.02
0.0
0.0
0.0
0.05
0.16
1.07
0.05
0.0
0.03
0.05
0.27
1.78
P3Former + RbA
L
2.13
0.0
0.0
0.0
0.0
0.0
0.0
0.0
16.62
1.03
0.51
0.72
65.18
53.47
0.08
13.10
4.49
0.38
1.14
0.51
8.36
LCPS-lidar + MSP
L
41.78
9.74
1.46
11.21
0.0
29.56
9.91
0.0
31.21
0.13
8.55
0.0
74.63
35.17
54.87
5.60
0.33
13.55
64.90
0.61
20.66
Table 6: 3D open-set semantic segmentation on DriveOSS-OOD test. We report class-wise open-IoU for the 19 ID classes, outlier IoU, and mOpen-IoU over ID classes. M denotes the input modality. M4F3D* is the modified LiDAR–camera fusion model.
Segmentation
Post-hoc Scoring Overhead
Total Time
Method
M
Network
+MSP
+ML
+Energy
+Entropy
+RbA
(ms)
P3Former
L
51.29
0.07
0.10
0.13
0.10
0.11
[51.36, 51.42]
LCPS-lidar
L
85.44
0.12
0.17
0.25
0.19
0.18
[85.56, 85.69]
M4F3D
L
104.21
0.11
0.17
0.23
0.18
0.13
[104.32, 104.44]
LIDO
L
–
–
–
–
–
–
68.01
LCPS
L+C
179.80
0.11
0.16
0.21
0.15
0.16
[179.91, 180.01]
Table 7: Inference time on the DriveOSS-OOD test set. Per-frame runtime in ms, measured on a single NVIDIA RTX 4090 GPU and decomposed into the base segmentation network and post-hoc anomaly scoring overhead. Since LIDO jointly performs semantic and anomaly segmentation, only its end-to-end runtime is reported.
> 600 Hz update rate, dual-antenna GNSS/INS, 0.01 m horizontal accuracy with RTK, 0.015 m vertical accuracy with RTK, and 0.1 ∘ roll/pitch/heading accuracy
Appendix
Table S1: Sensor specifications of the DriveOSS platform.
Method
M
car
bicycle
motorcycle
truck
other-veh.
person
bicyclist
motorcyclist
road
parking
sidewalk
other-ground
building
fence
vegetation
trunk
terrain
pole
traffic-sign
mIoU
P3Former
L
83.03
1.45
0.20
45.10
44.08
24.12
30.29
4.65
64.77
30.19
19.69
2.34
91.31
57.76
87.92
65.99
46.78
57.73
64.64
43.27
LCPS-lidar
L
85.52
23.06
8.15
48.59
41.16
34.51
54.58
21.42
79.49
34.22
59.63
2.62
94.51
47.55
91.55
70.93
54.05
58.69
76.60
51.94
M4F3D
L
73.96
6.51
22.04
8.69
26.69
15.52
40.65
1.86
63.21
22.79
17.01
2.29
66.02
42.00
87.04
43.74
40.38
34.53
49.69
34.98
LIDO
L
90.93
19.87
29.64
73.91
63.43
51.10
80.35
25.32
81.29
37.98
57.06
8.63
93.52
55.53
88.91
70.43
56.49
50.39
73.92
58.35
LCPS
L+C
91.80
15.22
27.37
50.12
64.59
31.53
55.11
12.66
78.32
32.91
56.54
2.58
96.29
55.64
91.73
71.48
57.63
64.19
71.18
54.05
M4F3D*
L+C
84.25
9.41
27.01
31.98
20.73
46.02
68.33
42.07
72.51
16.85
48.63
1.69
86.78
44.35
88.61
67.21
43.50
49.85
70.55
48.44
Appendix
Table S2 : Closed-set 3D semantic segmentation results on the DriveOSS-ID validation set. Class-wise IoU and mIoU are reported. “M” denotes the input modality, where “L” and “C” represent LiDAR and camera, respectively. M4F3D* denotes the modified LiDAR–camera fusion model.
Method
M
car
bicycle
motorcycle
truck
other-veh.
person
bicyclist
motorcyclist
road
parking
sidewalk
other-ground
building
fence
vegetation
trunk
terrain
pole
traffic-sign
mIoU
P3Former
L
81.50
0.20
0.13
18.31
37.36
33.70
2.81
2.76
52.11
4.80
6.48
0.34
91.88
59.06
85.07
75.38
28.51
36.49
46.12
34.90
M4F3D
L
85.28
30.02
3.47
59.85
54.47
75.57
31.46
23.47
49.90
12.59
14.42
6.98
91.26
40.31
85.04
75.99
33.98
32.90
62.08
45.74
LCPS-lidar
L
86.02
5.02
0.0
64.99
13.03
53.16
21.61
5.74
61.36
9.18
10.41
4.42
91.47
48.72
86.15
76.54
34.45
42.20
57.10
40.61
LIDO
L
88.95
23.22
3.41
73.95
20.92
77.69
40.93
19.48
55.17
29.29
15.71
12.01
92.14
36.93
84.56
65.24
40.66
27.23
58.64
45.59
LCPS
L+C
86.64
1.48
0.48
36.23
21.75
50.84
12.73
4.12
58.26
14.20
8.68
6.39
92.74
51.89
86.42
78.06
33.84
35.93
55.99
38.78
M4F3D*
L+C
73.57
3.28
10.76
10.42
24.50
59.25
0.00
18.53
45.59
0.07
7.99
2.11
77.74
25.48
85.07
67.30
29.49
23.71
34.76
31.56
Appendix
Table S3 : Closed-set 3D semantic segmentation results on the DriveOSS-OOD validation set. Class-wise IoU and mIoU over the 19 known classes are reported. “M” denotes the input modality, where “L” and “C” indicate LiDAR and camera, respectively. M4F3D* represents the modified fusion model.
Method
M
car
bicycle
motorcycle
truck
other-veh.
person
bicyclist
motorcyclist
road
parking
sidewalk
other-ground
building
fence
vegetation
trunk
terrain
pole
traffic-sign
mIoU
P3Former
L
61.64
0.34
0.15
7.42
3.22
22.47
14.27
0.0
53.58
1.88
13.27
3.03
94.60
53.74
83.44
46.87
29.07
18.84
54.13
29.58
LCPS-lidar
L
61.46
21.19
2.83
13.49
0.0
36.86
17.84
0.0
54.59
6.14
19.48
2.71
95.63
52.44
85.76
55.92
22.49
20.31
72.12
33.75
M4F3D
L
59.62
11.83
8.66
10.02
0.60
54.11
43.22
0.00
44.65
16.62
26.85
3.19
82.82
33.25
80.63
38.60
13.87
9.41
53.19
31.11
LIDO
L
68.18
28.94
0.55
15.03
36.02
62.09
58.62
3.47
56.52
25.19
22.48
14.61
96.77
45.39
85.76
42.54
34.06
12.29
70.31
40.99
LCPS
L+C
60.91
9.81
7.50
14.03
0.43
33.93
24.87
0.0
52.71
14.87
22.10
6.32
96.74
58.67
85.36
53.60
28.23
16.54
59.30
34.00
M4F3D*
L+C
60.58
5.40
7.80
7.66
2.52
51.12
35.81
0.50
43.87
0.03
18.70
0.70
83.11
25.50
80.53
38.48
14.26
12.64
51.56
28.46
Appendix
Table S4 : Closed-set 3D semantic segmentation results on the DriveOSS-OOD test set. Class-wise IoU and mIoU are reported. "M" means modality, and "L" and "C" denote LiDAR and Camera, respectively. M4F3D* represents the modified fusion model.
Method
M
AUROC ↑
FPR@95 ↓
AP ↑
P3Former + MSP
L
35.34
99.02
0.13
P3Former + ML
L
41.72
98.55
0.14
P3Former + Energy
L
46.53
98.42
0.26
P3Former + Entropy
L
32.64
99.03
0.14
P3Former + RbA
L
86.43
61.27
1.57
LCPS-lidar + MSP
L
72.51
55.61
0.31
Appendix
Table S5: 3D anomaly segmentation results on the DriveOSS-OOD validation set. We report AUROC ↑ , FPR@95 ↓ , and AP ↑ . M4F3D* represents the modified fusion model.
Method
M
car
bicycle
motorcycle
truck
other-veh.
person
bicyclist
motorcyclist
road
parking
sidewalk
other-ground
building
fence
vegetation
trunk
terrain
pole
traffic-sign
outlier
mOpen-IoU
P3former + MSP
L
20.46
0.0
0.08
17.26
0.73
0.03
2.07
0.09
0.05
0.0
0.0
0.0
0.01
0.03
0.15
0.0
0.0
0.0
0.82
0.10
2.20
P3former + ML
L
30.49
0.0
0.16
16.68
12.48
1.40
2.77
0.09
0.0
0.0
0.0
0.0
0.44
3.58
0.10
0.0
0.01
0.01
0.53
0.10
3.62
P3former + Energy
L
33.03
0.0
0.15
17.08
14.80
1.64
2.51
0.10
0.0
0.0
0.0
0.0
0.60
4.35
0.10
0.0
0.01
0.01
0.52
0.10
3.94
P3former + Entropy
L
21.07
0.0
0.10
17.04
0.89
0.05
3.52
0.10
0.02
0.0
0.0
0.0
0.01
0.04
0.13
0.0
0.01
0.0
0.76
0.10
2.30
P3former + RbA
L
3.35
0.0
0.0
0.0
0.0
0.0
0.0
0.0
34.14
3.51
0.01
0.01
69.96
31.04
0.03
39.03
3.92
3.16
0.84
0.14
9.95
LCPS-lidar + MSP
L
69.58
1.87
0.0
44.93
12.12
42.17
25.34
5.18
29.48
0.16
12.79
0.01
72.53
12.50
63.89
21.76
1.94
35.77
46.73
0.17
26.25
Appendix
Table S6 : 3D open-set segmentation results on the DriveOSS-OOD validation set. Class-wise open-IoU is reported for the 19 known classes, along with outlier IoU and mOpen-IoU over known classes. "M" denotes the input modality: L for LiDAR-only and L+C for LiDAR-camera fusion. M4F3D* represents the modified fusion model.
Figure S1 : Qualitative segmentation results. From left to right: RGB image, closed-set semantic segmentation prediction, anomaly segmentation prediction, open-set semantic segmentation prediction, and open-set semantic segmentation ground truth.
Figure S2 : Point cloud-to-image projection under different weather conditions. From left to right, the columns show clear, night, cloudy, rainy, and snowy conditions. From top to bottom, the rows correspond to DriveOSS-ID, DriveOSS-OOD validation, and DriveOSS-OOD test scenes. The projected points show weather-dependent LiDAR distributions, with notably fewer road-surface reflections in rainy scenes.
Figure S3 : Snapshot of the point cloud labeling interface, showing temporally registered 3D point clouds and RGB images.
Figure S4 : Visual comparison between automatic pre-labeling and final human-refined ground truth across DriveOSS subsets. Rows show representative samples from ID , OOD Validation , and OOD Test . Columns show: (1) RGB image, (2) raw Mask4Former pre-labels, and (3) final manual ground truth.
Understanding the surrounding environment is fundamental in autonomous driving and robotic perception. Distinguishing between known classes and previously unseen objects is crucial in real-world environments, as done in Anomaly Segmentation. However, research in the 3D field remains limited, with most existing approaches applying post-processing techniques from 2D vision. To cover this lack, we propose a new efficient approach that directly operates in the feature space, modeling the feature distribution of inlier classes to constrain anomalous samples. Moreover, the only publicly available 3D LiDAR anomaly segmentation dataset contains simple scenarios, with few anomaly instances, and exhibits a severe domain gap due to its sensor resolution. To bridge this gap, we introduce a set of mixed real-synthetic datasets for 3D LiDAR anomaly segmentation, built upon established semantic segmentation benchmarks, with multiple out-of-distribution objects and diverse, complex environments. Extensive experiments demonstrate that our approach achieves state-of-the-art and competitive results on the existing real-world dataset and the newly introduced mixed datasets, respectively, validating the effectiveness of our method and the utility of the proposed datasets. Code and datasets are available at https://simom0.github.io/lido-page/.
Stereo-based 3D obstacle perception for autonomous driving is currently constrained by an imbalanced triplet: deployment cost, detection accuracy, and open-set adaptability. While existing methods struggle to balance these three competing objectives, there is an urgent demand for high-precision, real-time algorithms capable of detecting arbitrary obstacles in the wild. In this paper, we present DDStereo, a novel Dual-Decoder Stereo Transformer that achieves a synergistic integration of 3D object detection and Out-of-Distribution (OoD) road anomaly detection. Leveraging the geometric priors of stereo disparity, our approach effectively couples 3D attribute regression with open-set foreground detection within a streamlined dual-branch decoder architecture. Conventional methods rely on complex feature-level fusion; DDStereo maintains execution efficiency by employing a decoupled decoding strategy and shared object-level queries to ensure cross-modal target alignment. Extensive evaluations of public benchmarks demonstrate that DDStereo not only achieves state-of-the-art accuracy under open-set and closed-set protocols. Our method delivers real-time performance comparable to monocular 3D detection baselines, providing a cost-effective solution for the perception of obstacles of the normal and OoD category. Code and models are available at https://github.com/shiyi-mu/DDStereo.
Shiyi Mu, Zichong Gu, Zhiqi Ai +2
Shanghai University, Shanghai, China · Xi’an Jiaotong-Liverpool University, Suzhou, China
Traditional semantic segmentation models operate under a closed-set assumption and struggle to recognize unknown or unexpected objects-an essential capability for autonomous driving. As a result, such models often misclassify or overlook out-of-distribution (OOD) road anomalies, posing safety risks in open-world environments. We present a lightweight, postprocessing, road-aware anomaly segmentation framework that requires no retraining, no OOD data, and no auxiliary supervision. Our approach builds on a mask transformer-based segmentation network by exploiting query-level mask confidence and deriving a polygonal road prior to detect gap regions that may correspond to anomalies. To further suppress false positives, we introduce a CLIP-based zero-shot semantic filtering module using in-distribution prompts, with optional generalized OOD prompts. By jointly leveraging spatial priors and semantic verification, our framework produces robust and interpretable anomaly predictions. Evaluation on three public benchmarks-Fishyscapes, SMIYC, and RoadAnomaly-shows consistently strong performance. In particular, our method outperforms the training-free baseline Maskomaly on most metrics and achieves the highest AP on Fishyscapes LostAndFound. These results demonstrate the practicality and deployability of our approach for real-world autonomous driving systems.
Zhiran Yan, Gordon Elger
Institute of Innovative Mobility (IIMo), Technical University Ingolstadt of Applied Sciences, Ingolstadt, 85049, Germany. · Fraunhofer Institute for Transportation and Infrastructure Systems IVI, Ingolstadt, 85051, Germany.