Uncertainty Evaluation to Support Safety of the Intended Functionality Analysis for Determining the Applicability of Performance Targets Across Triggering Conditions in ML-Based LiDAR Object Detection

Milin Patel; Rolf Jung

cs.CVOct 26, 2026

Uncertainty Evaluation to Support Safety of the Intended Functionality Analysis for Determining the Applicability of Performance Targets Across Triggering Conditions in ML-Based LiDAR Object Detection

Authors: Milin Patel, Rolf Jung

Organizations: Institute for Driver Assistance and Connected Mobility, Kempten University of Applied Sciences, Benningen, Germany · Faculty of Computer Science, Kempten University of Applied Sciences, Kempten, Germany

Abstract

For safety of the intended functionality (SOTIF) analysis according to ISO 21448, a machine learning object detector for light detection and ranging (LiDAR) point clouds can meet a performance target, a limit on the mean per-frame share of wrong detections among accepted detections, by rejecting detections above an uncertainty threshold. A threshold selected on data of one condition carries a statistical guarantee only for data distributed like its selection data, whereas SOTIF analysis covers every triggering condition. This paper presents an uncertainty evaluation in which the threshold is transferred into a triggering condition, a test decides at a stated error probability whether the target is applicable there, not applicable or not determined, and a lower bound counts the independent units an applicable outcome requires. For PointPillars ensembles on KITTI and nuScenes, counted in recorded sequences, as the dependence between frames requires, neither dataset held enough independent units for any target limit, and no transfer had a not applicable outcome. Counted in frames, the same data gave not applicable outcomes at limits fixed in advance after transfers into far distance, into rain and between two separately trained detectors, thresholds that met the target limit after a transfer rejected most correct detections, and in rain the area under the receiver operating characteristic curve did not decrease. A performance target met under one condition therefore counts as evidence under a triggering condition only after a test there, reported with its share of accepted correct detections and its number of independent recorded sequences.

Figures & tables

Figure 1: A wrong detection that the transferred threshold accepts and the threshold selected in the condition of use rejects. (a) KITTI, far distance, spatial indicator $u_{\mathrm{s}}$ , threshold transferred from near distance. (b) nuScenes, rain, existence indicator $u_{\mathrm{e}}$ , threshold transferred from clear weather. Bottom strips: indicator value of each detection. Open markers: rejected by the transferred threshold. IoU: intersection over union. Both thresholds were selected at the exploratory limit 0.1 † , which defines no performance target.

work	quantity that is set or estimated	set or estimated per condition	limit tested in another condition	finite-sample guarantee
Salay [ 30 ]	misperception rate, estimated	yes	no	no
CUBICS [ 31 ]	performance per situation, Bayesian estimate weighted by situation frequency	yes	no	no (Bayesian)
Kumar [ 32 ]	abstention threshold, selected once	no (reported per severity)	no	no (coverage of the prediction set only)
Lee [ 33 ]	confidence threshold, adapted to distance	yes	no	no
Li [ 34 ]	confidence threshold, per sensor degradation	yes	no	no
this paper	uncertainty threshold, selected with LTT in each condition	yes	yes, tested in the condition of use	yes (LTT, independent units)

Table 1: Works closest to the question whether a threshold selected under one condition meets its limit under another.

Figure 2: The applicability test. (a) Threshold

\tau_{c}

selected in the condition of selection

c

, for example near distance. (b) The same threshold in the condition of use

c^{\prime}

, for example far distance. (c) Outcome in

c^{\prime}

from the interval

[L,U]

of the FAP, or no threshold selected if the lowest candidate threshold fails in

c

u

: indicator value;

A_{c}(\tau)

: FAP in

c

. The distributions are schematic.

$c$ , $c^{\prime}$	condition of selection, condition of use	$\tau$	uncertainty threshold
$f$ , $f_{c}$	frame; car detections of $c$ in $f$	$\tau_{j}$	candidate threshold
$d$ , $y(d)$	detection; 1 if correct, 0 if wrong	$\tau_{c}$	threshold selected in $c$
$M$ , $m(d)$	number of members of the ensemble; number of members that contribute a box to $d$	$a_{f}$	per-frame share of false acceptances among the accepted detections in $f_{c}$
$b_{k}$ , $s_{k}$	box of member $k$ ; its confidence	$A_{c}$ , $\hat{A}_{c}$	FAP in $c$ ; its estimate
$u_{\mathrm{s}}$ , $u_{\mathrm{e}}$	spatial and existence indicator	$A$ , $\pi_{c}$	FAP over all frames with a car detection; share of them in $c$
$u(d)$	indicator value of $d$	$N_{f}$	number of accepted detections in $f_{c}$

Table 2: Symbols.

pair of conditions	recorded frames and car detections	conditions	frames, selection / test
KITTI distance	3,769 validation frames; 27,175 car detections (KITTI rule); 0.858 of 14,385 annotated cars matched	near / far, boundary 34.8 m	1,484 / 1,500 1,514 / 1,545
nuScenes distance	6,019 validation frames from 150 scenes; 40,036 car detections; 0.398 of 65,465 annotated cars matched	near / far, boundary 21.4 m	2,207 / 2,209 2,270 / 2,276
nuScenes weather	4,449 clear and 968 rain frames, all in daylight; 602 night frames excluded	clear / rain	1,917 / 1,900 392 / 414
change of dataset and detector	all validation frames of KITTI and of nuScenes	KITTI with the KITTI detector (KITTI rule) / nuScenes with the nuScenes detector	1,844 / 1,852 2,849 / 2,850

Table 3: Pairs of conditions. Last column: frames per condition in the selection and test parts of split 0 (one of five random assignments of the frames to the parts), first line for the first condition. Boundaries: split 0.

Figure 3: Division of the recorded frames into parts within one dataset (left) and for the change of dataset and detector (right). Arrows: a threshold selected in

c

is tested in

c^{\prime}

(one transfer direction) and in

c

(check of the selection). Bottom: 36 example frames divided by frame or by sequence.

Figure 4: Outcome of every transfer in the frame-level evaluation (left) and retention of the transferred thresholds with an applicable outcome at

10^{-2}

(right). † : exploratory limit, which defines no performance target. ‘

<

’: below 0.01 %. No threshold was selected at

10^{-3}

and

10^{-4}

. Frames of one recorded sequence are not independent, so the stated error probability is not guaranteed.

transfer (indicator)	AUROC	share of wrong detections	further change	shown by AUROC
near $\to$ far, KITTI ( $u_{\mathrm{s}}$ )	0.915 $\to$ 0.845	0.34 $\to$ 0.76	–	weaker separation
near $\to$ far, nuScenes ( $u_{\mathrm{s}}$ )	0.729 $\to$ 0.655	0.19 $\to$ 0.51	–	weaker separation
clear $\to$ rain, nuScenes ( $u_{\mathrm{e}}$ )	0.844 $\to$ 0.856	0.33 $\to$ 0.42	different spread over frames: FAP 0.09 $\to$ 0.17, pooled share 0.12 $\to$ 0.17	none
nuScenes $\to$ KITTI ( $u_{\mathrm{s}}$ )	0.749 $\to$ 0.905	0.35 $\to$ 0.55	shifted values: median $u_{\mathrm{s}}$ of the correct detections 0.23 $\to$ 0.12	none

Table 4: Changes in the detections from the condition of selection to the condition of use (

\to

) for the transfers into a condition with more wrong detections that gave not applicable outcomes, one indicator each. Shares of wrong detections: all validation frames; other values: test part of split 0, FAP and pooled share at the transferred threshold selected at 0.1 † . –: no further change reported.

Figure 5: ROC curves (top; legend: AUROC) and estimated FAP against the threshold (bottom) on the test part of split 0, in the condition of selection (solid) and the condition of use (dashed). Squares and vertical lines: transferred threshold selected at 0.1 † ; circles: transferred threshold selected at

10^{-2}

; dotted horizontal lines: limits 0.1 † and

10^{-2}

; bars: interval

[L,U]

Figure 6: Lowest limit that

n

independent units without a false acceptance can show to be met (Eq. ( 5 )), with the test units of each condition in split 0. Marker heights are computed from

n

, not measured. Whole set: the KITTI frames of the change of dataset and detector.

outcome	meaning for the performance target	measure that follows
applicable	evidence, at error probability at most $\delta/2$ if the units are independent, that the target is met in $c^{\prime}$ with the transferred threshold	use the threshold in $c^{\prime}$ if the share of annotated cars that reach the motion planner meets a second target on missed objects
not applicable	evidence, at error probability at most $\delta/2$ if the units are independent, that the target is not met in $c^{\prime}$ with the transferred threshold	detector supplier: select a threshold in $c^{\prime}$ with a run-time signal for $c^{\prime}$ , or use a common threshold for several conditions; vehicle manufacturer: restrict the ODD or the functionality, or add a measure in tracking, fusion or motion planning
not determined	no statement, because the data of $c^{\prime}$ do not suffice or the FAP lies close to $\alpha$	record further independent units (Eq. ( 5 ) gives the minimum) and treat $c^{\prime}$ as not verified
no threshold selected	the lowest candidate threshold fails in $c$ with the available data	record further independent units in $c$ or request a reallocation of $\alpha$ at vehicle level

Table 5: Meaning of each outcome for the performance target in

c^{\prime}

and the measure that follows.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 652 papers

Jun 29, 2026

Explainability-Aware Frustum Attack: Exposing Structural Vulnerabilities in LiDAR-Based 3D Object Detectors

The structural vulnerabilities of point cloud-based 3D object detectors remain poorly understood. Prior work has studied adversarial robustness primarily on isolated 3D object models, while recent LiDAR spoofing attacks target richer and more realistic driving scenes but focus mainly on physical realizability rather than understanding detector behavior or attack efficiency. In this work, we investigate how LiDAR-based detectors rely on spatial evidence in complex scenes and whether these reliance patterns can be exploited to induce failures more efficiently. To this end, we propose an explainability-guided adversarial analysis methodology. We introduce the Saliency-LiDAR (SALL) method, which aggregates Integrated Gradient attributions across scenes to produce universal saliency maps for LiDAR-based 3D object detectors. Guided by these maps, we design the Explainability-aware Frustum Attack (EFA), which selectively perturbs only the most influential frustums rather than uniformly attacking entire object regions. Experiments on KITTI and nuScenes, across detectors such as PointPillars and SECOND, show that EFA reduces detection recall by more than 15 percentage points while requiring 25-50% fewer perturbed frustums than the state-of-the-art non-saliency-aware baseline. These findings reveal that modern 3D detectors concentrate discriminative evidence in a small subset of spatial regions, exposing a structural robustness vulnerability in current LiDAR perception systems. Our code is released at https://github.com/SecMindLab/Saliency_LiDAR.

Chengzeng You, Binbin Xu, Soteris Demetriou

Imperial College London, UK · Now at Google.

Feature Attribution Lidar-Based 3D Object Detection Adversarial Attacks on Object Detectors Adversarial Robustness 3D Object Detection

Ask PDF

Jan 30, 2026

A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions

Reliable environmental perception remains one of the main obstacles for safe operation of automated vehicles. Safety of the Intended Functionality (SOTIF) concerns safety risks from perception insufficiencies, particularly under adverse conditions where conventional detectors often falter. While Large Vision-Language Models (LVLMs) demonstrate promising semantic reasoning, their quantitative effectiveness for safety-critical 2D object detection is underexplored. This paper presents a systematic evaluation of ten representative LVLMs using the PeSOTIF dataset, a benchmark specifically curated for long-tail traffic scenarios and environmental degradations. Performance is quantitatively compared against two specialized detectors: the anchor-based YOLOv5 and the transformer-based RT-DETRv4. Experimental results reveal a critical trade-off: top-performing LVLMs (e.g., Gemini 3) surpass the YOLOv5 in recall by over 25% and closely match RT-DETRv4 under natural visual degradation, while specialized detectors retain an advantage in geometric precision for handcrafted perturbations. These findings highlight the complementary strengths of semantic reasoning versus geometric regression, supporting the use of LVLMs as high-level safety validators in SOTIF-oriented automated driving systems.

Ji Zhou, Yilin Ding, Yongqi Zhao +4

Institute of Automotive Engineering, Graz University of Technology, 8010 Graz, Austria · Independent Researcher, 200000 Shanghai, China · School of Intelligent Connected Vehicle, Hubei University of Automotive Technology, 442002 Shiyan, China +1

VLM Evaluation Large Vision-Language Models Autonomous Driving Perception Object Detection

Ask PDF

Jun 8, 2026

Taming Perception Jitter: Uncertainty-Aware LiDAR Object Detection for Reliable Motion Classification

Reliable motion classification is critical for autonomous driving, as false dynamic predictions of static objects can cascade into unnecessary planner interventions. Unstable bounding box predictions can lead to spurious velocity estimates in tracking and falsely predicted trajectories. We present a deployment-friendly mitigation strategy that augments a 3D object detector with aleatoric uncertainty estimates and applies a two-sample z-test over short observation windows to separate true motion from jitter. Integrated into Autoware with minimal changes, the approach reuses existing data association for minimal compute overhead. Empirical results show parity with velocity thresholding on nuScenes, but substantially fewer false dynamic predictions and unnecessary stops in real-world test drives, explained by the presence of an intermediate jitter band in the recorded data that speed-only rules misclassify. This demonstrates that uncertainty-aware detection and lightweight statistical testing can deliver practical performance gains for autonomous driving in noisier real-world settings.

Cornelius Schröder, Žygimantas Marcinkus, Markus Lienkamp

Institute for Automotive Engineering, Munich Institute of Robotics and Machine Intelligence, School of Engineering and Design, Technical University of Munich

Lidar-Based 3D Object Detection Uncertainty Quantification Autonomous Driving 3D Object Detection Aleatoric Uncertainty Statistical Hypothesis Testing

Ask PDF

Abstract

Uncertainty Quantification

Explore similar work

Explainability-Aware Frustum Attack: Exposing Structural Vulnerabilities in LiDAR-Based 3D Object Detectors

A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions

Taming Perception Jitter: Uncertainty-Aware LiDAR Object Detection for Reliable Motion Classification