Effort-Based Criticality Metrics for Evaluating 3D Perception Errors in Autonomous Driving

Sharang Kaul; Simon Bultmann; Mario Berk; Abhinav Valada

cs.CVMar 30, 2026

Effort-Based Criticality Metrics for Evaluating 3D Perception Errors in Autonomous Driving

Authors: Sharang Kaul, Simon Bultmann, Mario Berk, Abhinav Valada

Organizations: CARIAD SE, Germany · Department of Computer Science, University of Freiburg, Germany

Abstract

Criticality metrics such as time-to-collision (TTC) quantify collision urgency but do not distinguish the operational consequences of false-positive (FP) and false-negative (FN) perception errors. We formulate two error-specific effort metrics: False Speed Reduction (FSR), the cumulative velocity loss associated with persistent phantom detections, and Maximum Deceleration Rate (MDR), the peak braking demand associated with missed objects under a longitudinal kinematic model. These longitudinal metrics are complemented by Lateral Evasion Acceleration (LEA), adapted from prior lateral-evasion kinematics and coupled with reachability-based collision timing. The collision check quantifies the minimum steering effort required to avoid a predicted collision. A dynamically conservative, semantically unfiltered reachability gate selects candidate interactions before frame-level scoring and track-level aggregation. Evaluation on nuScenes and Argoverse 2 shows that 65% to 93% of errors fall below the chosen criticality thresholds. Correlation and threshold analysis indicate that the proposed metrics provide complementary rankings for screening and mining perception failures and are not substitutes for closed-loop safety validation.

Figures & tables

Fig. 1 : Effort-based criticality assessment of perception errors. The pipeline identifies false positives (FPs) and false negatives (FNs) by matching tracker outputs against ground-truth (GT) annotations, performs reachable-set collision analysis, and computes effort metrics. Regular GT boxes are shown in green. A missed GT box is recoloured red to indicate its FN status. False Speed Reduction (FSR) captures the cumulative velocity loss from persistent FPs, and Maximum Deceleration Rate (MDR) captures the peak braking demand from missed objects. Left side: FP from AB3DMOT on nuScenes where a phantom detection (yellow) persists for $19$ $\mathrm{s}$ ( $39$ frames) at ${\sim}20$ $\mathrm{m}$ ahead with no matching GT object. FSR accumulates to $7.2$ $\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$ . Right side: FN from BEVFusion on Argoverse 2. The red box is the GT annotation of a real vehicle in the ego lane ( $R_{\text{lat}}\!\approx\!0.3$ $\mathrm{m}$ ). It is an FN because no BEVFusion prediction is matched to this GT object for ${\sim}16$ $\mathrm{s}$ . MDR peaks at $2.4$ $\mathrm{m}\text{\,}{\mathrm{s}}^{-2}$ . LiDAR bird’s-eye-view panels show multi-sweep point clouds in gray. Metrics are computed only during frames with perception errors.

Fig. 2 : Collision avoidance strategies between the ego vehicle (gray) and a perception error (red). Left: longitudinal braking ( $a_{\text{brake}}$ ) and lateral evasion are evaluated independently. Right: lateral-evasion geometry. The signed candidates target relative offsets $d_{y}=+w_{c}$ and $d_{y}=-w_{c}$ under the ego-frame convention $+x$ forward and $+y$ left.

Parameter	Symbol	Value	Parameter	Symbol	Value
Reachable set bounds			Capability limits
Forward acc.	$a_{\text{lon,max}}$	$2.0$	Max braking	$a_{\text{brake,max}}$	$10.0$
Braking	$a_{\text{lon,min}}$	$-$ $3.0$	Max lateral acc.	$a_{\text{lat,cap}}$	$5.0$
Lateral acc.	$a_{\text{lat,max}}$	$2.0$
Vehicle & scenario			Timing
Length	$L$	$4.5$ $\mathrm{m}$	Reaction time	$t_{\text{react}}$	$0.3$ $\mathrm{s}$

TABLE I : Parameters for reachability-based collision filtering and criticality computation

				Track Errors		Critical			Mean Effort			Cum. Effort			Worst Event			Detection
Dataset	Pipeline	Mod.	Class	FN	FP	FN	FP	TC	MDR	FSR	LEA	MDR	FSR	LEA	MDR	FSR	LEA	Prec.	Rec.
nuScenes	AB3DMOT	L	Car	$3252$	$36\,239$	$696$	$4816$	$5632$	$2.32$	$1.66$	$0.05$	$7550$	$60\,242$	$2061$	$10.0$	$200.0$	$5.0$	$0.32$	$0.70$
	AB3DMOT	L	Truck	$644$	$23\,262$	$163$	$2616$	$2799$	$2.73$	$1.39$	$0.04$	$1759$	$32\,247$	$872$	$10.0$	$165.0$	$5.0$	$0.11$	$0.57$
	CenterPoint	L	Car	$3554$	$40\,172$	$764$	$3468$	$5101$	$2.32$	$0.99$	$0.04$	$8247$	$39\,955$	$1670$	$10.0$	$155.0$	$5.0$	$0.41$	$0.67$
	CenterPoint	L	Truck	$654$	$12\,740$	$182$	$1112$	$1410$	$2.93$	$1.01$	$0.04$	$1915$	$12\,873$	$503$	$10.0$	$80.0$	$5.0$	$0.24$	$0.55$
	BEVFusion	L+C	Car	$3865$	$3440$	$919$	$454$	$1372$	$2.53$	$1.37$	$0.08$	$9766$	$4723$	$599$	$10.0$	$49.9$	$5.0$	$0.86$	$0.60$

TABLE II : Safety-critical evaluation of object detection models with track-level aggregation on nuScenes and Argoverse 2 validation sets. Bold represents best value per dataset-class group, and underline represents second best.

	Time-Based		Decel.-Based		Effort-Based (Proposed)
Severity	TTC	TTB	BTN	DST	MDR	FSR	LEA
	( $\mathrm{s}$ )	( $\mathrm{s}$ )	(-)	( $\mathrm{m}$ )	( $\mathrm{m}$ )	( $\mathrm{m}$ )	( $\mathrm{m}$ )
Safe	${>}3.0$	${>}1.0$	${\leq}0.4$	${<}1.0$	${\leq}2.0$	${\leq}1.0$	${\leq}1.0$
Moderate	$2.0$ - $3.0$	-	$0.4$ - $0.7$	$1.0$ - $4.0$	$2.0$ - $4.0$	$1.0$ - $2.5$	$1.0$ - $2.0$
Critical	$1.0$ - $2.0$	$0.4$ - $1.0$	$0.7$ - $1.0$	$4.0$ - $6.0$	$4.0$ - $6.0$	$2.5$ - $5.0$	$2.0$ - $4.0$
Imminent	${<}1.0$	${<}0.4$	${>}1.0$	${\geq}6.0$	${>}6.0$	${>}5.0$	${>}4.0$

TABLE III : Criticality Metric Thresholds for Perception Error Assessment

Fig. 3 : Distribution of non-safe perception errors (absolute track counts) on the nuScenes validation set. Rows: Car / Truck. Columns: FN by MDR, FP by FSR, all by LEA. Severity thresholds follow Tab. III .

	Established					Proposed
Property	TTC	DRAC	BTN	DST	TET	MDR	FSR	LEA
FP/FN specific	-	-	-	-	-	FN	FP	Both
Temporal scope	inst.	inst.	inst.	inst.	cum.	peak	cum.	peak
Duration-sensitive	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$
Lateral dimension	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$
Collision gate	-	-	-	-	-	RSB	RSB	RSB

TABLE IV : Established vs proposed effort-based criticality metrics for assessing perception error.

		TTC		DRAC		THW		TET		$\boldsymbol{d_{y}}$
		RSB	SAT	RSB	SAT	RSB	SAT	RSB	SAT	RSB	SAT
FN	MDR	$\text{-}0.30$	$\text{-}0.34$	$0.30$	$\mathbf{0.68}$	$\text{-}0.39$	$0.07$	$\mathbf{0.41}$	$0.15$	$0.11$	$0.16$
FN	LEA	$0.14$	$0.31$	$\text{-}0.12$	$\text{-}0.36$	$0.10$	$0.16$	$\text{-}0.13$	$\text{-}0.10$	$\mathbf{\text{-}0.55}$	$\mathbf{\text{-}0.58}$
FP	FSR	$\mathbf{\text{-}0.93}$	$\mathbf{\text{-}0.65}$	$\mathbf{0.89}$	$\mathbf{0.80}$	$\mathbf{\text{-}0.90}$	$\text{-}0.25$	$\mathbf{0.73}$	$0.31$	$0.06$	$0.11$
FP	LEA	$0.08$	$\text{-}0.02$	$\text{-}0.07$	$0.02$	$0.06$	$0.04$	$\text{-}0.08$	$\text{-}0.06$	$\mathbf{\text{-}0.41}$	$\mathbf{\text{-}0.57}$

TABLE V : Spearman correlation (

\rho

) between the proposed and established metrics on nuScenes, aggregated across all pipelines under both collision checks. Bold:

|\rho|\geq 0.40

Fig. 4 : Scenario-level analysis for scene 26a6b03c (nuScenes, BEVFusion). Each row shows, from left to right: bird’s-eye trajectory evolution with start/end markers, reachability-based TTC (

\text{TTC}_{\text{RSB}}

), braking effort (

a_{\text{brake}}

), lateral evasion effort (LEA), and the cumulative criticality metric (MDR for FNs, FSR for FPs). Top : three FN objects (

\text{FN}_{1\text{-}3}

) illustrate how MDR saturates for both in-path and laterally distant objects, while LEA disambiguates true collision risk. Bottom : three FP objects (

\text{FP}_{1\text{-}3}

) demonstrate FSR’s persistence-sensitive accumulation.

\text{FP}_{1}

(24 frames, low per-frame braking) scores highest, while short-lived FPs with higher intensity receive lower FSR.

Panel of Figure 10 — Fig. 5 : Classical TTC ( $d/v_{\text{rel}}$ ) vs. minimum distance for effort-critical car-category tracks on the nuScenes validation set. Dashed line: TTC $=2$ $\mathrm{s}$ . A substantial fraction of MDR-critical FN tracks cluster at TTC ${>}3$ $\mathrm{s}$ or at the dataset cap of $10$ $\mathrm{s}$ , indicating that classical TTC rates them as safe despite high braking demand.

Autonomous Driving Safety Evaluation

Autonomous driving safety evaluation studies how to measure, test, and improve the safety of self-driving systems in routine and hazardous conditions.

See all 158 papers

Scenario-Independent Criticality Assessment and Prediction for Vulnerable Road Users in Autonomous Driving
Sep 14, 2026Jörg Gamerdinger, Victor Schwarzenberger, Philipp Schmid +2Autonomous Driving Safety Evaluation
University of Tübingen, Faculty of Science, Department of Computer Science, Embedded Systems Group
Expressiveness, Equivalence, and Uncertainty in Velocity Obstacles and Closest Point of Approach Metrics
Oct 5, 2026Elizabeth Dietrich, Liam M. Imagawa, Hanna Krasowski +3Collision Avoidance Uncertainty Quantification
University of California, Berkeley, USA · Norwegian University of Science and Technology, Trondheim, Norway
Driving risk emerges from the required two-dimensional joint evasive acceleration
Apr 20, 2026Hao Cheng, Yanbo Jiang, Wenhao Yu +9Collision Avoidance Autonomous Driving Safety Evaluation
School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China · State Key Laboratory of Intelligent Green Vehicle and Mobility, Beijing 100084, China · School of Traffic & Transportation Engineering, Central South University, Changsha 410000, China +2

Sep 14, 2026J/K move · Enter open · S save

Scenario-Independent Criticality Assessment and Prediction for Vulnerable Road Users in Autonomous Driving

Jörg Gamerdinger, Victor Schwarzenberger, Philipp Schmid, Sven Teufel+1

University of Tübingen, Faculty of Science, Department of Computer Science, Embedded Systems Group

Autonomous Driving Safety Evaluation

Open paper AskPDF arXiv

Increasing safety is the primary objective of automated vehicles. Achieving this goal requires reliable safety metrics that incorporate safety-relevant factors such as object type, velocity, and criticality. A key capability of such metrics is the distinction between critical and non-critical objects, which is addressed through criticality or relevance estimation. Existing criticality metrics are typically designed for specific scenarios and primarily focus on vehicle-to-vehicle interactions. In this paper, we therefore propose a novel criticality metric tailored to vulnerable road users (VRUs), which require special consideration due to their less predictable motion behavior. Furthermore, to avoid the complexity introduced by scenario-specific metrics, we introduce a scenario-independent criticality prediction framework applicable to all traffic participant classes. The effectiveness of both the proposed VRU-centric criticality metric and the criticality prediction framework is evaluated using the DeepAccident dataset, which contains a diverse set of safety-critical traffic scenarios. The proposed VRU-centric criticality metric improves pedestrian criticality classification performance by up to 50 %. In addition, the proposed criticality prediction framework outperforms state-of-the-art metrics by 275 %, achieving an F1-score of 0.96 and enabling scenario-independent criticality assessment across all object classes. These results demonstrate the strong potential of the proposed approaches to enhance criticality assessment for safety evaluation in automated driving systems.

Abstract

Autonomous Driving Safety Evaluation

Explore similar work

Scenario-Independent Criticality Assessment and Prediction for Vulnerable Road Users in Autonomous Driving

Expressiveness, Equivalence, and Uncertainty in Velocity Obstacles and Closest Point of Approach Metrics

Driving risk emerges from the required two-dimensional joint evasive acceleration