C-DFF: Concept-Level Diagnostics for Mechanism Parity in Simulation-Based Validation of Autonomous Driving Perception

Danial Safaei; Siddartha Khastgir; Matthew Higgins; Jeroen Ploeg; Mohsen Alirezaei; Son Tong; Ludo Gielen

cs.CVOct 24, 2026

C-DFF: Concept-Level Diagnostics for Mechanism Parity in Simulation-Based Validation of Autonomous Driving Perception

Authors: Danial Safaei, Siddartha Khastgir, Matthew Higgins, Jeroen Ploeg, Mohsen Alirezaei, Son Tong, Ludo Gielen

Organizations: WMG, University of Warwick, Coventry, U.K. · 2getthere, Utrecht, The Netherlands · Eindhoven University of Technology (TU/e), Eindhoven, The Netherlands · Siemens Digital Industries Software, Leuven, Belgium

Abstract

Simulation is widely used in autonomous-driving validation, but matching task outputs between real and synthetic images does not guarantee that a system relies on the same visual evidence. This creates a validation gap for transfer, especially in safety-critical perception functions. We present Concept-Decisive Feature Fidelity (C-DFF), a concept-level mechanism-parity diagnostic for paired real-synthetic evaluation. Starting from decisive evidence extracted from a frozen system-under-test (SUT), C-DFF projects that evidence into a shared semantic concept space using off-the-shelf segmentation and compares real and synthetic concept-importance vectors. The result is not only a scalar mechanism-parity score, but also a concept-shift diagnostic that identifies which scene concepts become more or less decisive in simulation. We evaluate C-DFF on KITTI $\leftrightarrow$ Virtual KITTI 2 with 2126 matched pairs across three frozen SUTs (steering regression, YOLOP drivable-area, YOLOP lane-line). Across these tasks, C-DFF is SUT-discriminative and yields interpretable concept-level summaries that support validation triage and simulator debugging. When reported alongside task-specific output-value context metrics, C-DFF shows that output agreement does not by itself establish mechanism parity. These results position C-DFF as a practical semantic diagnostic layer for validation and verification workflows in simulation-based AV testing.

Figures & tables

Fig. 1: C-DFF pipeline. For an image $x$ , compute (1) a decisive map $M(x)=H(F,x)$ on the frozen SUT, (2) semantic segmentation $S(x)$ via an off-the-shelf segmenter, (3) aligned maps $\tilde{M}(x),\tilde{S}(x)$ by applying the SUT geometry and then pooling to a compact shared grid, using average pooling for decisive maps and nearest-neighbor resizing for labels, (4) a concept-importance vector $\bm{c}(x)$ by aggregating decisive evidence within semantic concept masks, and (5) C-DFF as a distance between $\bm{c}(x_{r})$ and $\bm{c}(x_{s})$ .

SUT	Valid-run rate	Median $\lvert s(x_{\mathrm{cf}}^{\star})-s_{\mathrm{tgt}}\rvert$
Steering	100.0%	$4.04\text{\times}{10}^{-1}$
YOLOP–DA	100.0%	$1.45\text{\times}{10}^{-3}$
YOLOP–LL	100.0%	$8.83\text{\times}{10}^{-3}$

TABLE I: CF-XAI valid-run diagnostics on the synthetic-side held-out cache subset (50 runs per SUT). “Valid” means the optimization completed without numerical failure and produced a finite decisive map. The residual is reported only as a secondary convergence diagnostic; this subset-level diagnostic does not replace the full held-out C-DFF results.

SUT	Mean	Med.	Std.	P@ $\varepsilon_{90}$	P@ $\varepsilon_{95}$	OV (ctx.)
Steering	$3.12\text{\times}{10}^{-4}$	$2.94\text{\times}{10}^{-4}$	$1.71\text{\times}{10}^{-4}$	77.1%	92.8%	$0.3742$
YOLOP–DA	$1.25\text{\times}{10}^{-4}$	$1.00\text{\times}{10}^{-4}$	$1.03\text{\times}{10}^{-4}$	97.0%	100.0%	$0.4518$
YOLOP–LL	$3.90\text{\times}{10}^{-5}$	$3.20\text{\times}{10}^{-5}$	$3.30\text{\times}{10}^{-5}$	66.9%	74.7%	$0.0730$

TABLE II: Main C-DFF results on the held-out evaluation split. Lower C-DFF is better. Thresholds

\varepsilon_{90},\varepsilon_{95}

are calibrated on the calibration split as percentiles of C-DFF. Pass rates are computed on the held-out split. OV is a task-specific output-value context metric, not a mechanism-parity metric, and is not intended for cross-SUT ranking.

SUT	Pred labels	GT labels	$\bm{\varepsilon_{90}}$	$\bm{\varepsilon_{95}}$
Steering	$2.87\text{\times}{10}^{-4}$	$2.41\text{\times}{10}^{-4}$	$4.06\text{\times}{10}^{-4}$	$5.48\text{\times}{10}^{-4}$
YOLOP–DA	$7.11\text{\times}{10}^{-5}$	$6.05\text{\times}{10}^{-5}$	$3.99\text{\times}{10}^{-4}$	$5.23\text{\times}{10}^{-4}$
YOLOP–LL	$2.28\text{\times}{10}^{-5}$	$2.52\text{\times}{10}^{-5}$	$4.40\text{\times}{10}^{-5}$	$5.10\text{\times}{10}^{-5}$

TABLE III: Synthetic-side label-source diagnostic and calibrated C-DFF thresholds. Label-source C-DFF compares Mask2Former predictions against VKITTI2 ground-truth labels on the held-out cache subset. Thresholds are calibrated on the calibration split.

Fig. 2: C-DFF distributions by SUT on the held-out split. Empirical CDFs

P(\mathrm{C\text{-}DFF}\leq\varepsilon)

are shown for each SUT; dashed vertical lines indicate calibration-split

\varepsilon_{95}

values from Table III , not universal safety bounds.

SUT	Top- $k$ increases (synthetic more decisive)	Top- $k$ decreases (synthetic less decisive)
Steering	terrain (+0.0097), wall (+0.0094), sky (+0.0076)	traffic sign (-0.0090), truck (-0.0062), building (-0.0048)
YOLOP–DA	car (+0.0030), pole (+0.0030), wall (+0.0025)	building (-0.0021), fence (-0.0016), truck (-0.0016)
YOLOP–LL	wall (+0.0020), sky (+0.0018), pole (+0.0007)	terrain (-0.0013), fence (-0.0011), traffic sign (-0.0007)

TABLE IV: Top concept shifts per SUT. Each cell lists concept names with signed magnitudes from the dataset-level mean shift

\bar{\Delta\bm{c}}

Fig. 3: Mean concept shift $\bar{\Delta\bm{c}}$ by SUT and concept. Positive values indicate that a concept becomes more decisive in synthetic imagery, negative values indicate that it becomes less decisive. Table IV lists the largest shifts numerically for readability.

Fig. 4: Illustrative qualitative diagnostics. For each exemplar, we show real and synthetic inputs, decisive overlays, semantic segmentation, and top-6 concept bars. These selected examples are semantic triage aids rather than prevalence estimates. Quantitative results are reported in Tables II – IV .

Step	What you do / what you log
1. Pairing	Construct $(x_{r},x_{s})$ with tight correspondence. Log pairing quality flags.
2. Scalar gate	Compute C-DFF and compare to calibrated $\varepsilon$ for regression screening. Log pass rate.
3. Diagnose	Compute $\Delta\bm{c}$ and report top- $k$ increases and decreases per SUT for debugging.
4. Requirement-style checks	Track envelopes for critical concepts and alarm on drift beyond tolerances.
5. Act	Map dominant shifted concepts to simulator or data-generation knobs and re-test.

TABLE V: Practical C-DFF protocol for paired real–synthetic validation. The output is a scalar gate plus a ranked diagnostic of concept shifts.

Autonomous Driving Simulation

Autonomous driving simulation models roads, traffic, sensors, and vehicle behavior to train and evaluate driving systems safely and at scale.

See all 228 papers

A Causal Probabilistic Framework for Perception-Informed Closed-Loop Simulation of Autonomous Driving
Jun 5, 2026Zhennan Fei, Rickard Johansson, Mikael Andersson +12Autonomous Driving Perception Scenario-Based Testing
Volvo Cars, Sweden · Chalmers University of Technology, Sweden · TNO, The Netherlands +2
The Reality Gap in Automated Driving Systems Testing: Landscape, Challenges, and Research Directions
Oct 28, 2026Stefano Carlo Lambertenghi, Oliver Weissl, Johannes Betz +1Autonomous Driving Simulation Autonomous Driving
Technical University of Munich, Boltzmannstraße 3, Garching, Bavaria, Germany, 85748 · fortiss GmbH, Boltzmannstraße 3, Garching, Bavaria, Germany, 85748 · NorCom GmbH & Co. KGaA, Boltzmannstraße 3, Garching, Bavaria, Germany, 85748 +1
From Nominal Intensity to Equivalent Rainfall: A Path-Based Credibility Evaluation Framework for Simulated Rainfall in Autonomous-Driving Perception Tests
Jun 10, 2026Tian Xia, Xin Zhao, Shaolingfeng Ye +1Autonomous Driving Perception Autonomous Driving Simulation
College of Automotive and Energy Engineering, Tongji University, Shanghai, China · Tsinghua University, Beijing, China

Jun 5, 2026J/K move · Enter open · S save

A Causal Probabilistic Framework for Perception-Informed Closed-Loop Simulation of Autonomous Driving

Zhennan Fei, Rickard Johansson, Mikael Andersson, Matthias Eng+11

Volvo Cars, Sweden · Chalmers University of Technology, Sweden · TNO, The Netherlands+2

Autonomous Driving Perception Scenario-Based Testing Autonomous Driving Simulation

Open paper AskPDF arXiv

Software-in-the-loop (SIL) simulation is a cornerstone for the validation of modern automotive safety functions. However, many current frameworks utilize ideal sensing, which bypasses the functional insufficiencies of perception algorithms, leading to over-optimistic safety assessments. This paper proposes a perception-informed SIL testing methodology that bridges the gap between ground-truth simulation and real-world perception behavior. We present a framework for incorporating causal probabilistic models into standardized, scenario-based simulation toolchains, applicable to both Advanced Driver Assistance Systems (ADAS) and Autonomous Driving Systems (ADS). Our approach enables the systematic injection of realistic perception errors, such as loss of detection, sizing inaccuracies, and positioning offsets, derived from physical triggering conditions like fog, rain, and object-merging scenarios. By evaluating these ``faults'' within a standardized simulation environment, we demonstrate that perception-informed testing reveals latent operational risks that ideal SIL environments fail to capture, providing a scalable pathway for SOTIF (ISO 21448) validation.

Abstract

Autonomous Driving Simulation

Explore similar work

A Causal Probabilistic Framework for Perception-Informed Closed-Loop Simulation of Autonomous Driving

The Reality Gap in Automated Driving Systems Testing: Landscape, Challenges, and Research Directions

From Nominal Intensity to Equivalent Rainfall: A Path-Based Credibility Evaluation Framework for Simulated Rainfall in Autonomous-Driving Perception Tests