From Leaderboard to Deployment: Code Quality Challenges in AV Perception Repositories

Mateus Karvat; Bram Adams; Sidney Givigi

cs.CVMar 2, 2026

From Leaderboard to Deployment: Code Quality Challenges in AV Perception Repositories

Authors: Mateus Karvat, Bram Adams, Sidney Givigi

Organizations: School of Computing, Queen’s University. Kingston, ON, Canada.

Abstract

Autonomous vehicle (AV) perception models are typically evaluated solely on benchmark performance metrics, with limited attention to code quality, production readiness and long-term maintainability. This creates a significant gap between research excellence and real-world deployment in safety-critical systems subject to international safety standards. To address this gap, we present the first large-scale empirical study of software quality in AV perception repositories, systematically analyzing 178 unique models from the KITTI and NuScenes 3D Object Detection leaderboards. Using static analysis tools (Pylint, Bandit, and Radon), we evaluated code errors, security vulnerabilities, maintainability, and development practices. Our findings revealed that only 7.3% of the studied repositories meet basic production-readiness criteria, defined as having zero critical errors and no high-severity security vulnerabilities. Security issues are highly concentrated, with the top five issues responsible for almost 80% of occurrences, which prompted us to develop a set of actionable guidelines to prevent them. Additionally, the adoption of Continuous Integration/Continuous Deployment pipelines was correlated with better code maintainability. Our findings highlight that leaderboard performance does not reflect production readiness and that targeted interventions could substantially improve the quality and safety of AV perception code.

Figures & tables

Fig. 1: Relationship between errors and repository size (measured in SLOC).

Fig. 2: Correlation of repository size and number of security issues.

Fig. 3: Correlation between maintainability and security issue density.

Fig. 4: Concentration of security issues in our dataset. Ties broken by severity.

ID	Severity	Category	Issue	Occurrences	Affected repositories	Percentage of repositories
B614	MEDIUM	Serialization	Use of unsafe PyTorch load	$713$	$144$	$80.9$
B110	LOW	Error Handling	Try, Except, Pass	$273$	$69$	$38.8$
B605	HIGH	Shell Injection	Starting a process with a shell, possible injection detected	$239$	$79$	$44.4$
B307	MEDIUM	Input Validation	Unsafe eval() allowing code injection	$222$	$106$	$59.6$
B602	HIGH	Shell Injection	Subprocess call with shell=True	$163$	$28$	$15.7$
B506	MEDIUM	Serialization	Use of unsafe yaml load	$163$	$64$	$36.0$

TABLE I: Security issues identified by Bandit with severity classification and security categories.

Fig. 5: Repositories that adopt CI/CD have a significantly higher mean maintainability index.

Software Security

Software security studies how to find, prevent, validate, and repair vulnerabilities in code, systems, dependencies, and software supply chains.

See all 95 papers

Safety-Aligned 3D Object Detection: Single-Vehicle, Cooperative, and End-to-End Perspectives
Apr 2, 2026Brian Hsuan-Cheng Liao, Chih-Hong Cheng, Hasan Esen +13D Object Detection Autonomous Driving Safety Evaluation
DENSO AUTOMOTIVE Deutschland GmbH, 85386 Eching, Germany. · Department of Computer Engineering, Technical University of Munich, 85748 Garching, Germany. · Department of Computer Science, Carl von Ossietzky University of Oldenburg, 26129 Oldenburg, Germany.
Behavioral Safety Assessment towards Large-scale Deployment of Autonomous Vehicles, Part II: Assessment Results
Oct 6, 2026Henry X. Liu, Tinghan Wang, Xintao Yan +6Autonomous Driving Safety Evaluation Autonomous Driving
University of Michigan Transportation Research Institute, Ann Arbor, MI 48109 USA · Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI 48109 USA · Department of Civil Engineering, The University of Hong Kong, Hong Kong 999077, China +2
Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks
Mar 3, 2026Junjie Chu, Xinyue Shen, Ye Leng +3Benchmark Reproducibility LLM Safety Benchmarks
CISPA Helmholtz Center for Information Security · University of Waterloo · Flexera

Apr 2, 2026J/K move · Enter open · S save

Safety-Aligned 3D Object Detection: Single-Vehicle, Cooperative, and End-to-End Perspectives

Brian Hsuan-Cheng Liao, Chih-Hong Cheng, Hasan Esen, Alois Knoll

DENSO AUTOMOTIVE Deutschland GmbH, 85386 Eching, Germany. · Department of Computer Engineering, Technical University of Munich, 85748 Garching, Germany. · Department of Computer Science, Carl von Ossietzky University of Oldenburg, 26129 Oldenburg, Germany.

3D Object Detection Autonomous Driving Safety Evaluation Cooperative Perception End-to-End Autonomous Driving

Open paper AskPDF arXiv

Perception plays a central role in connected and autonomous vehicles (CAVs), underpinning not only conventional modular driving stacks, but also cooperative perception systems and recent end-to-end driving models. While deep learning has greatly improved perception performance, its statistical nature makes perfect predictions difficult to attain. Meanwhile, standard training objectives and evaluation benchmarks treat all perception errors equally, even though only a subset is safety-critical. In this paper, we investigate safety-aligned evaluation and optimization for 3D object detection that explicitly characterize high-impact errors. Building on our previously proposed safety-oriented metric, NDS-USC, and safety-aware loss function, EC-IoU, we make three contributions. First, we present an expanded study of single-vehicle 3D object detection models across diverse neural network architectures and sensing modalities, showing that gains under standard metrics such as mAP and NDS may not translate to safety-oriented criteria represented by NDS-USC. With EC-IoU, we reaffirm the benefit of safety-aware fine-tuning for improving safety-critical detection performance. Second, we conduct an ego-centric, safety-oriented evaluation of AV-infrastructure cooperative object detection models, underscoring its superiority over vehicle-only models and demonstrating a safety impact analysis that illustrates the potential contribution of cooperative models to "Vision Zero." Third, we integrate EC-IoU into SparseDrive and show that safety-aware perception hardening can reduce collision rate by nearly 30% and improve system-level safety directly in an end-to-end perception-to-planning framework. Overall, our results indicate that safety-aligned perception evaluation and optimization offer a practical path toward enhancing CAV safety across single-vehicle, cooperative, and end-to-end autonomy settings.

Abstract

Software Security

Explore similar work

Safety-Aligned 3D Object Detection: Single-Vehicle, Cooperative, and End-to-End Perspectives

Behavioral Safety Assessment towards Large-scale Deployment of Autonomous Vehicles, Part II: Assessment Results

Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks