Safety-Aligned 3D Object Detection: Single-Vehicle, Cooperative, and End-to-End Perspectives
Authors: Brian Hsuan-Cheng Liao, Chih-Hong Cheng, Hasan Esen, Alois Knoll
Organizations: DENSO AUTOMOTIVE Deutschland GmbH, 85386 Eching, Germany. · Department of Computer Engineering, Technical University of Munich, 85748 Garching, Germany. · Department of Computer Science, Carl von Ossietzky University of Oldenburg, 26129 Oldenburg, Germany.
Perception plays a central role in connected and autonomous vehicles (CAVs), underpinning not only conventional modular driving stacks, but also cooperative perception systems and recent end-to-end driving models. While deep learning has greatly improved perception performance, its statistical nature makes perfect predictions difficult to attain. Meanwhile, standard training objectives and evaluation benchmarks treat all perception errors equally, even though only a subset is safety-critical. In this paper, we investigate safety-aligned evaluation and optimization for 3D object detection that explicitly characterize high-impact errors. Building on our previously proposed safety-oriented metric, NDS-USC, and safety-aware loss function, EC-IoU, we make three contributions. First, we present an expanded study of single-vehicle 3D object detection models across diverse neural network architectures and sensing modalities, showing that gains under standard metrics such as mAP and NDS may not translate to safety-oriented criteria represented by NDS-USC. With EC-IoU, we reaffirm the benefit of safety-aware fine-tuning for improving safety-critical detection performance. Second, we conduct an ego-centric, safety-oriented evaluation of AV-infrastructure cooperative object detection models, underscoring its superiority over vehicle-only models and demonstrating a safety impact analysis that illustrates the potential contribution of cooperative models to "Vision Zero." Third, we integrate EC-IoU into SparseDrive and show that safety-aware perception hardening can reduce collision rate by nearly 30% and improve system-level safety directly in an end-to-end perception-to-planning framework. Overall, our results indicate that safety-aligned perception evaluation and optimization offer a practical path toward enhancing CAV safety across single-vehicle, cooperative, and end-to-end autonomy settings.
Figures & tables
Fig. 1: Asymmetry in the safety consequences of perception errors: The red prediction overestimates the truck’s longitudinal distance and under-covers it relative to the blue prediction, making it more safety-critical. Whereas standard metrics such as IoU and translation error (TE) assign identical values to both predictions, our safety-aligned metrics—USC and EC-IoU—favor the blue prediction as the safer one.
Projects 3D coordinates onto 2D feature maps and adapts a fully convolutional one-stage detector for 3D box regression.
PGD [ 19 ]
Extends FCOS3D with probabilistic depth estimation and geometric relation graphs, enabling uncertainty-aware depth refinement for more accurate localization.
DETR3D [ 31 ]
Adopts a Transformer architecture in which sparse 3D queries retrieve encoded 2D image features and decode them directly into 3D bounding boxes.
PETR [ 32 ]
Improves DETR3D by injecting 3D coordinate information into 2D image features, yielding 3D position-aware feature maps for more accurate localization.
PointPillars [ 33 ]
Converts point clouds into pseudo-images using vertical pillars, enabling efficient processing by CNN backbones.
SSN [ 34 ]
Extends PointPillars by incorporating explicit shape information from point clouds through an additional shape-aware loss.
TABLE II: Representative models included in our benchmark and their key contributions.
Fig. 4: Benchmarking results of representative 3D object detectors on nuScenes, labeled by the sensor modality (C: camera, L: lidar, and F: fusion). Top: true-positive error measures (mAIoU, mATE ′ , mAUSC). Bottom: overall metrics (mAP, NDS, NDS-USC). Safety-oriented metrics tolerate certain non-critical errors but emphasize safety-critical mislocalizations, changing the performance picture compared with standard metrics.
Model
TP measures ↑
Overall (%) ↑
IoU
EC-IoU
TE ′
USC
mAP
NDS
NDS-USC
PGD
0.404
0.397
0.374
0.801
46.86
48.29
64.19
w/ EC-IoU-2
0.419
0.408
0.408
0.817
47.34
48.87
65.28
w/ EC-IoU-4
0.417
0.405
0.381
0.819
47.62
48.82
65.36
PETR
0.389
0.360
0.340
0.761
52.83
52.07
64.10
w/ EC-IoU-2
0.393
0.363
0.347
0.770
53.25
52.40
64.68
TABLE III: Fine-tuning PGD and PETR with standard and EC-IoU losses (higher is better).
Fig. 5: Qualitative comparison of PETR fine-tuned with the standard loss (red) and our safety-aware EC-IoU loss (blue). EC-IoU shifts predictions toward the ego vehicle, improving coverage of safety-critical regions.
TABLE IV: CoopDet3D performance ( % ) under roadside-view, ego-view, and ego-centric safety-oriented evaluation on TUMTraf [ 8 ] .
Fig. 7: Top: vehicle-only lidar-based model. Bottom: cooperative lidar-based model. The cooperative model reduces false positives, but slightly overestimates the lead vehicle’s longitudinal distance from the ego perspective. In the bird’s-eye view, the roadside lidar point cloud is shown in red, while the ego vehicle appears on the right in white and is moving leftward.
Stage
Collisions/year
HDVs only
10.95
HDVs + AVs
5.48
HDVs + AVs + CP
2.67
TABLE V: Estimated annual collisions at an intersection with human-driven vehicles (HDVs) only, mixed traffic with HDVs and AVs, and mixed traffic with AVs supported by cooperative perception (CP).
TABLE VI: End-to-end driving performance with standard ℓ1 loss or EC-IoU loss applied to SparseDrive fine-tuning. UniAD is included for reference. Performance is evaluated for detection, tracking, prediction, and planning. Lower is better for ADE, Col., and L2.
Fig. 9: Qualitative comparison of baseline and EC-IoU-optimized SparseDrive variants. In the same cut-in scenario as Fig. 5 , the EC-IoU-based model decelerates earlier and accelerates more decisively, more closely matching the human reference.
Zhejiang University, Hangzhou, China · Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing, Hangzhou, China · China Jiliang University, Hangzhou, China