The Reality Gap in Automated Driving Systems Testing: Landscape, Challenges, and Research Directions

Stefano Carlo Lambertenghi; Oliver Weissl; Johannes Betz; Andrea Stocco

cs.SEOct 28, 2026

The Reality Gap in Automated Driving Systems Testing: Landscape, Challenges, and Research Directions

Authors: Stefano Carlo Lambertenghi, Oliver Weissl, Johannes Betz, Andrea Stocco

Organizations: Technical University of Munich, Boltzmannstraße 3, Garching, Bavaria, Germany, 85748 · fortiss GmbH, Boltzmannstraße 3, Garching, Bavaria, Germany, 85748 · NorCom GmbH & Co. KGaA, Boltzmannstraße 3, Garching, Bavaria, Germany, 85748 · Technical University of Munich, Boltzmannstraße 15, Garching, Bavaria, Germany, 85748

Abstract

Simulation is essential for testing automated driving systems (ADS), but discrepancies between simulation and reality can undermine testing conclusions. It remains unclear which discrepancies the literature addresses and what evidence supports correspondence claims. We present a systematic mapping of 235 primary studies that measure or mitigate the reality gap in ADS testing, selected from 4,195 publications. We characterise objectives, targets, approaches, experimental setups, and evaluation evidence. Our classification distinguishes simulated and real-world execution across six ADS test components, separately for each approach and its evaluation reference. This distinguishes real-world information used to construct an approach from evidence used to evaluate its claims. Of the included studies, 76% address mitigation, primarily targeting scenario and actor realism and perception, while 94% report validation or measurement procedures. Only 61 studies report metrics from multiple families covering sensor/perception correspondence, ADS behaviour, vehicle dynamics, or safety outcomes; this coverage does not establish causal links between discrepancies and test verdicts. The synthesis highlights that real-world content and physical execution must be interpreted against the claim evaluated. The findings motivate claim-specific evaluation and component-level reporting to establish which discrepancies affect testing conclusions and when simulation evidence supports conclusions about real-world ADS behaviour.

Figures & tables

Figure 1. Four visual discrepancies between a real camera recording and a comparable simulation: (1) image appearance, (2) vehicle model, (3) road markings, and (4) road-user representation. Boxes highlight the differences; their impact on ADS behaviour and test verdicts requires separate investigation.

Category	Research Question	What it captures
Context	RQ 1.1 What is the primary objective of the study?	Whether the study measures the gap, mitigates it, or does both.
	RQ 1.2 Which aspect(s) of the simulation–reality gap are addressed?	The aspect of the simulation–reality gap addressed by the study, including combinations of aspects.
	RQ 1.3 What is the direction of the simulation–reality relationship?	Whether the study addresses the gap from simulation to reality, from reality to simulation, or in both directions.
Approach	RQ 2.1 Which research target does the study’s simulation–reality gap concern?	The specific capability, component, or phenomenon considered in relation to the simulation–reality gap.
	RQ 2.2 How is the proposed simulation–reality mitigation implemented?	How the gap is mitigated, from the specific technique to its broader family.
	RQ 2.3 What are the execution levels of the proposed approach?	The execution level of each component and functional group across simulation and reality.

Table 1. Research questions and what each one captures from a study.

Study	Objective	Direction	Gap aspect
VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles ( Amini et al., 2022 )	Mitigate	Real2sim	Perception, Behaviour
The Effect of Spray Water on an Automotive LIDAR Sensor: A Real-Time Simulation Study ( Vargas Rivero et al., 2022 )	Mitigate	Real2sim	Perception
Synthetic to Real Gap Estimation of Autonomous Driving Datasets using Feature Embedding ( Gadipudi et al., 2022 )	Measure	Sim2real	Perception
Sim-on-Wheels: Physical World in the Loop Simulation for Self-Driving ( Shen et al., 2023 )	Mitigate	Both	Perception
Reinforcement Learning with Human Feedback for Realistic Traffic Simulation ( Cao et al., 2024b )	Mitigate	Sim2real	Scenario
Quantifying the LiDAR Sim-to-Real Domain Shift: A Detailed Investigation Using Object Detectors and Analysing Point Clouds at Target-Level ( Huch et al., 2023 )	Measure	Sim2real	Perception

Table 2. The 10 pilot studies used to calibrate the review protocol and the coded values that motivated their selection.

Panel of Figure 2 — Figure 2. Study identification, preliminary selection, and full selection process.

RQ	Data type	Data content
RQ 1.1	Predefined categorical	Measure only, Mitigate only, Measure and mitigate
RQ 1.2	Predefined categorical, combinable	Scenario, Behaviour, Perception, Localization, Actuation, Entire; a study addressing several aspects equally is recorded as their combination
RQ 1.3	Predefined categorical	Sim to real, Real to sim, Both
RQ 2.1	Free text, categorised	Research target, grouped into macro categories and subcategories
RQ 2.2	Free text, categorised	Mitigation technique, grouped into macro categories, subcategories, and optional subsubcategories
RQ 2.3	Predefined categorical, one value per component	Scenario, Actors, Perception, Localization, Actuation, ECU/control, each recorded as Not modelled, Simulated, Pseudo-sim, Pseudo-real, Mixed, Both, or Real; Partially real additionally arises at group level; each setup also carries one or more execution-source tags (real vehicle, real hardware, real dataset, simulator, simulated dataset), or None

Table 3. Type and content of the data used for each research question.

Category	Definition
Not modelled	The component is not present.
Simulated	The component is provided by a general-purpose simulation platform or physics model. This holds even if the platform is calibrated, tuned, or partly trained using real-world data.
Pseudo-sim	The component is provided by a standalone model or generator, not a general-purpose simulation platform, whose representation is not constructed from real-world observations or measured real-world characteristics.
Pseudo-real	The component is provided by a standalone model, reconstruction, or generator whose representation is constructed from real-world observations, measurements, or measured real-world characteristics, but the real-world component is not present live during execution.
Mixed	Real and simulated elements are combined within the same execution.
Both	Separate simulated and real-world executions are performed in parallel.

Table 4. Categorical execution levels used for RQ 2.3 and RQ 3.2 .

Research target	Studies	Primary studies
Scenario and actor realism	109
Traffic/NPC realism	50	( Liu et al., 2021 ; Langer et al., 2021 ; Kim et al., 2024 ; Lichtle et al., 2024 ; Shuguang et al., 2022 ; Amini et al., 2020 ; Bergamini et al., 2021 ; Cao et al., 2024a ; Gu et al., 2023 ; Tettamanti et al., 2018 ; Yang et al., 2021a ; Wagner et al., 2019 ; Tenbrock et al., 2021 ; Shi et al., 2022 ; Zhang et al., 2022b ; Xu et al., 2023 ; Sun et al., 2024 ; Stadler et al., 2022b ; Queiroz et al., 2024 ; Barker et al., 2018 ; Che et al., 2021 ; Liu et al., 2024 ; Samak et al., 2024a ; Suo et al., 2023 ; Guo et al., 2025 ; Huang et al., 2025 ; Neis et al., 2025 ; Ameli et al., 2025 ; Yan et al., 2025 ; Gao et al., 2026a ; Zhang et al., 2025c ; Naing et al., 2025 ; Wu and Ma, 2025 ; Li et al., 2025b ; Buzdugan et al., 2025 ; Wang et al., 2025c ; Jing et al., 2025 ; Bi et al., 2025 ; Wang et al., 2026 ; Huang et al., 2026 ; Cao et al., 2024b ; Khaire et al., 2025 ; Wang et al., 2022b ; Liang et al., 2025 ; Drechsler et al., 2022c ; Gao et al., 2025a ; Coppola et al., 2025 ; Daza et al., 2023 ; Leung et al., 2025 ; Sigl et al., 2021 )
Test-scenario validity	26	( Liu et al., 2021 ; Nalic et al., 2019 ; Peintner et al., 2021 ; Kim et al., 2022 ; Bai et al., 2024 ; Chen et al., 2020 ; Fremont et al., 2020 ; Gao et al., 2020 ; Gupta et al., 2023 ; Schyr et al., 2022 ; Zofka et al., 2018a ; Wagner et al., 2019 ; Stadler et al., 2022a ; Wu et al., 2024 ; Solmaz et al., 2020 ; Lan et al., 2024 ; Reckenzaun et al., 2024 ; Chen et al., 2025 ; Wu et al., 2025 ; Kolb et al., 2025 ; Shen et al., 2023 ; Arango et al., 2026 ; Toth and Szalay, 2023 ; Tian et al., 2026 ; Elgharbawy et al., 2019 ; Guo et al., 2021 )
Crash-scenario validity	13	( Oladimeji et al., 2024 ; Bashetty et al., 2020 ; Gambi et al., 2019a ; Xinxin et al., 2020 ; Zhang and Cai, 2023 ; Zhou et al., 2024 ; Zhang et al., 2025a ; Wei et al., 2025b ; Zhao et al., 2025 ; Abbasi et al., 2025 ; Hu and Zhao, 2025 ; Shoukat et al., 2024 ; Wei et al., 2025a )
Pedestrian/VRU realism	12	( Zofka et al., 2018b ; Drechsler et al., 2022a ; Larter et al., 2022 ; Long et al., 2024 ; Fu et al., 2024 ; Fu et al., 2025 ; Liu et al., 2026 ; Li et al., 2025e ; Martín Serrano et al., 2025 ; Wei et al., 2025a ; Steimle et al., 2025 ; Yang et al., 2021b )
Road/environment realism	11	( Bateman et al., 2024 ; Brogle et al., 2019 ; Szalai et al., 2020 ; Zhang et al., 2022c ; Rezaldi et al., 2021 ; Li et al., 2025d ; Sun et al., 2025a ; Schulz et al., 2025 ; Wang et al., 2025a ; Jekl et al., 2025 ; Park et al., 2020 )

Table 5. RQ 2.1 research targets and supporting primary studies. Bars show the number of studies assigned to each target, and the final column lists the corresponding primary studies. Studies may be assigned to several targets, so counts across categories are not mutually exclusive and subcategory counts may not sum to the parent count.

Mitigation technique	Studies
Real-equipment integration (ViL/HiL/MR/XiL)	56
Vehicle-in-the-loop (on-road)	28
Mixed reality	21
Controller/ECU hardware-in-the-loop	6
Co-simulation	5
Vehicle-in-the-loop (testbench)	4

Table 6. RQ 2.2 mitigation technique families and techniques, among the studies with a mitigation objective. Bars show the number of studies using each family or technique. A study can use several families and several techniques within a family, so counts are not mutually exclusive and technique counts may exceed the family total.

Figure 3. RQ 2.3 execution level of the proposed approach: execution levels of the six individual components (top), and the observed combinations for Scene, Sensing, and Control ranked by frequency (bottom). In the lower panel, colours encode study objective and icons encode execution level.

Evaluation technique	Studies	Primary studies
Direct or matched sim-to-real comparison	132	( Rosenberger et al., 2018 ; Rosenberger et al., 2019 ; Langer et al., 2021 ; Ngo et al., 2021b ; Pahk et al., 2023 ; Jayapal Gowdu et al., 2022 ; Lichtle et al., 2024 ; Pahk et al., 2024 ; Peintner et al., 2021 ; Kim et al., 2022 ; Reway et al., 2020 ; Manivasagam et al., 2023 ; Wachtel et al., 2021 ; Parra et al., 2020 ; Krecht et al., 2023 ; Shuguang et al., 2022 ; Amini et al., 2020 ; Argui et al., 2023 ; Bashetty et al., 2020 ; Bateman et al., 2024 ; Bergamini et al., 2021 ; Brogle et al., 2019 ; Dona et al., 2022 ; Drechsler et al., 2022b ; Fremont et al., 2020 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Haq et al., 2021 ; Stocco et al., 2023b ; Hoppe et al., 2022 ; Hsiang et al., 2022 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Zofka et al., 2018a ; Wagner et al., 2019 ; Xu et al., 2021 ; Wallace et al., 2022 ; Shi et al., 2022 ; Zhu et al., 2022 ; Xu et al., 2023 ; Yuan et al., 2024 ; Stadler et al., 2022b ; Durukal et al., 2022 ; Queiroz et al., 2024 ; Huang et al., 2024 ; Larter et al., 2022 ; Jiang et al., 2021 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Ngo et al., 2021a ; Chen et al., 2019 ; Oh et al., 2024a ; Che et al., 2021 ; Solmaz et al., 2020 ; Lan et al., 2024 ; Amini and Nejati, 2024 ; Fu et al., 2024 ; Hwang et al., 2024 ; Suo et al., 2023 ; Ulreich et al., 2024 ; Wachtel et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Chen et al., 2025 ; Guo et al., 2025 ; Ganesan et al., 2025 ; Li et al., 2025a ; Neis et al., 2025 ; Ameli et al., 2025 ; Li et al., 2025f ; Zhou et al., 2026b ; Yan et al., 2025 ; Gao et al., 2026a ; Verma et al., 2025 ; Liu et al., 2026 ; Nosheen et al., 2025 ; Naing et al., 2025 ; Wu and Ma, 2025 ; Wei et al., 2025b ; Allamaa et al., 2025 ; Gaddam et al., 2025 ; Zuo et al., 2025 ; Jung et al., 2025 ; Saad et al., 2025 ; Lambertenghi et al., 2025a ; Zhao et al., 2025 ; Haghighi et al., 2026 ; Harhammer et al., 2025 ; Samak et al., 2026 ; Bi et al., 2025 ; Anand et al., 2026 ; Wang et al., 2026 ; Lee et al., 2025 ; Waheed et al., 2025 ; Oh et al., 2024b ; Stocco et al., 2023a ; Manivasagam et al., 2020 ; Shen et al., 2023 ; Huch et al., 2023 ; Vargas Rivero et al., 2022 ; Magosi and Eichberger, 2023 ; Wang et al., 2025a ; Khaire et al., 2025 ; Wang et al., 2022b ; Balım et al., 2025 ; Cardozo and Hlavác, 2025 ; Arango et al., 2026 ; Liang et al., 2025 ; Toth and Szalay, 2023 ; Drechsler et al., 2022c ; Magosi et al., 2022 ; Argui et al., 2025 ; Wei et al., 2025a ; Tian et al., 2026 ; Steimle et al., 2025 ; Hofrichter et al., 2025 ; Gao et al., 2025a ; Li et al., 2024 ; Coppola et al., 2025 ; Elgharbawy et al., 2019 ; Segonne and Duthon, 2023 ; Daza et al., 2023 ; Jekl et al., 2025 ; Park et al., 2020 ; Wang et al., 2022a ; Yang et al., 2021b ; Leung et al., 2025 ; Guo et al., 2021 ; Sigl et al., 2021 ; Deter et al., 2021 )
Task/SUT performance comparison	130	( Nalic et al., 2019 ; Saleh et al., 2019 ; Pahk et al., 2023 ; Oladimeji et al., 2024 ; Jayapal Gowdu et al., 2022 ; Pahk et al., 2024 ; Peintner et al., 2021 ; Reway et al., 2022 ; Reway et al., 2020 ; Zofka et al., 2018b ; Wachtel et al., 2021 ; Shuguang et al., 2022 ; Amini et al., 2020 ; Argui et al., 2023 ; Bai et al., 2024 ; Bashetty et al., 2020 ; Bateman et al., 2024 ; Brogle et al., 2019 ; Bui et al., 2024 ; Chen and Muller, 2022 ; Chen et al., 2020 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Fremont et al., 2020 ; Gambi et al., 2019a ; Gao et al., 2020 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Haq et al., 2021 ; Stocco et al., 2023b ; Hoppe et al., 2022 ; Hsiang et al., 2022 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Yang et al., 2021a ; Von Bernuth et al., 2018 ; Tim et al., 2020 ; Talwar et al., 2020 ; Tenbrock et al., 2021 ; Xu et al., 2021 ; Zhang et al., 2022c ; Shi et al., 2022 ; Zhang and Cai, 2023 ; Stadler et al., 2022b ; Queiroz et al., 2024 ; Huang et al., 2024 ; Jiang et al., 2021 ; Barker et al., 2018 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Ngo et al., 2021a ; Chen et al., 2019 ; Oh et al., 2024a ; Che et al., 2021 ; Solmaz et al., 2020 ; Lan et al., 2024 ; Amini and Nejati, 2024 ; Fu et al., 2024 ; Suo et al., 2023 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Chen et al., 2025 ; Fu et al., 2025 ; Zeng et al., 2025 ; Steinhauser et al., 2025 ; Li et al., 2025a ; Zhou et al., 2026b ; Wang et al., 2025d ; Hong, 2025 ; Simakov et al., 2025 ; Baresi et al., 2025 ; Zhang et al., 2025c ; Pasios and Nikolaidis, 2025 ; Nosheen et al., 2025 ; Li et al., 2025b ; Wang et al., 2025b ; Zhang et al., 2025a ; Zhou et al., 2026a ; Allamaa et al., 2025 ; Gaddam et al., 2025 ; Peng and Wang, 2025 ; Da Rocha et al., 2025 ; Zuo et al., 2025 ; Gao et al., 2025b ; Saad et al., 2025 ; Elmquist et al., 2025 ; Lambertenghi et al., 2025a ; Zhang et al., 2025b ; Haghighi et al., 2026 ; Drechsler et al., 2025 ; Li et al., 2025d ; Kim et al., 2025 ; Harhammer et al., 2025 ; Sun et al., 2025a ; Stangherlin et al., 2025 ; Li et al., 2025e ; Wang et al., 2026 ; Ulreich et al., 2025 ; Otto et al., 2025 ; Huang et al., 2026 ; Schulz et al., 2025 ; Haider et al., 2025 ; Schabus et al., 2025 ; Petrovic and Muhammad, 2025 ; Stocco et al., 2023a ; Manivasagam et al., 2020 ; Lambertenghi and Stocco, 2024 ; Shen et al., 2023 ; Huch et al., 2023 ; Amini et al., 2022 ; Wang et al., 2025a ; Shoukat et al., 2024 ; Martín Serrano et al., 2025 ; Balım et al., 2025 ; Arango et al., 2026 ; Toth and Szalay, 2023 ; Argui et al., 2025 ; Tian et al., 2026 ; Gao et al., 2025a ; Segonne and Duthon, 2023 ; Daza et al., 2023 ; Jekl et al., 2025 ; Park et al., 2020 ; Song et al., 2026 ; Dauner et al., 2024 ; Wang et al., 2022a ; Yang et al., 2021b ; Reisgys et al., 2021 ; Deter et al., 2021 )
Quantitative or distributional comparison	65	( Langer et al., 2021 ; Kettelgerdes and Elger, 2023 ; Reway et al., 2020 ; Dona et al., 2022 ; Duminil et al., 2024b ; Gambi et al., 2019a ; Gu et al., 2023 ; Hsiang et al., 2022 ; Wagner et al., 2019 ; Tim et al., 2020 ; Stadler et al., 2022a ; Shi et al., 2022 ; Zhang et al., 2022b ; Sun et al., 2024 ; Stadler et al., 2022b ; Durukal et al., 2022 ; Jiang et al., 2021 ; Haider et al., 2024 ; Oh et al., 2024a ; Lan et al., 2024 ; Amini and Nejati, 2024 ; Fei et al., 2024 ; Suo et al., 2023 ; Ulreich et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Guo et al., 2025 ; Ganesan et al., 2025 ; Huang et al., 2025 ; Steinhauser et al., 2025 ; Neis et al., 2025 ; Yan et al., 2025 ; Gao et al., 2026a ; Simakov et al., 2025 ; Zhang et al., 2025c ; Naing et al., 2025 ; Li et al., 2025b ; Wang et al., 2025b ; Zhang et al., 2025a ; Wang et al., 2025c ; Jung et al., 2025 ; Elmquist et al., 2025 ; Wu et al., 2025 ; Jing et al., 2025 ; Kolb et al., 2025 ; Lee et al., 2025 ; Oh et al., 2024b ; Lambertenghi and Stocco, 2024 ; Cao et al., 2024b ; Duminil et al., 2024a ; Magosi and Eichberger, 2023 ; Wang et al., 2025a ; Martín Serrano et al., 2025 ; Liang et al., 2025 ; Drechsler et al., 2022c ; Magosi et al., 2022 ; Gao et al., 2025a ; Li et al., 2024 ; Daza et al., 2023 ; Song et al., 2026 ; Dauner et al., 2024 ; Wang et al., 2022a ; Yang et al., 2021b ; Reisgys et al., 2021 ; Deter et al., 2021 )
Feature or image-quality comparison	24	( Oladimeji et al., 2024 ; Reway et al., 2022 ; Cao et al., 2024a ; Duminil et al., 2024b ; Hsiang et al., 2022 ; Thieling et al., 2019 ; Soni and Nair, 2021 ; Chen et al., 2025 ; Steinhauser et al., 2025 ; Baresi et al., 2025 ; Zhou et al., 2026a ; Kuroda et al., 2025 ; Eberhardt et al., 2025 ; Hess et al., 2025 ; Yu and Wang, 2026 ; Schulz et al., 2025 ; Li et al., 2025c ; Gadipudi et al., 2022 ; Argui et al., 2025 ; Tian et al., 2026 ; Wang et al., 2022a ; Leung et al., 2025 ; Reisgys et al., 2021 ; Guo et al., 2021 )
Human or qualitative assessment	21	( Langer et al., 2021 ; Lichtle et al., 2024 ; Kim et al., 2022 ; Zofka et al., 2018b ; Yeo et al., 2019 ; Yuan et al., 2024 ; Zhou et al., 2024 ; Ulhas et al., 2024 ; Sun et al., 2024 ; Wu et al., 2024 ; Durukal et al., 2022 ; Che et al., 2021 ; Fu et al., 2024 ; Baresi et al., 2025 ; Zhang et al., 2025c ; Buzdugan et al., 2025 ; Abbasi et al., 2025 ; Yu and Wang, 2026 ; Cao et al., 2024b ; Khaire et al., 2025 ; Argui et al., 2025 )
No substantive evaluation	16	( Liu et al., 2021 ; Lu et al., 2022 ; Kim et al., 2024 ; Samak et al., 2024b ; Szalai et al., 2020 ; Szalay et al., 2019 ; Xinxin et al., 2020 ; Zofka et al., 2023 ; Rezaldi et al., 2021 ; Long et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Capdevila et al., 2024 ; Hu and Zhao, 2025 ; Wang et al., 2023 ; Heikkinen et al., 2019 )

Table 7. RQ 3.1 evaluation techniques and supporting primary studies. Bars show the number of studies using each technique, and the final column lists the corresponding primary studies. Studies may use several techniques, so counts are not mutually exclusive.

Figure 4. RQ 3.2 execution level of the evaluation: six-component execution-state counts (top), and the observed functional-group state combinations ranked by frequency (bottom). In the lower panel, colours encode study objective, and icons encode execution category.

Metric category / metric	Studies	Primary studies
Sensor/perception comparison	107
Detection accuracy/rate	30	( Rosenberger et al., 2019 ; Oladimeji et al., 2024 ; Reway et al., 2020 ; Manivasagam et al., 2023 ; Bashetty et al., 2020 ; Chen and Muller, 2022 ; Hsiang et al., 2022 ; Schyr et al., 2022 ; Von Bernuth et al., 2018 ; Stadler et al., 2022b ; Barker et al., 2018 ; Yang et al., 2023 ; Li et al., 2025a ; Simakov et al., 2025 ; Peng and Wang, 2025 ; Saad et al., 2025 ; Wang et al., 2026 ; Otto et al., 2025 ; Haider et al., 2025 ; Schabus et al., 2025 ; Manivasagam et al., 2020 ; Vargas Rivero et al., 2022 ; Duminil et al., 2024a ; Balım et al., 2025 ; Argui et al., 2025 ; Tian et al., 2026 ; Segonne and Duthon, 2023 ; Wang et al., 2022a ; Yang et al., 2021b ; Deter et al., 2021 )
Average precision (mAP/AP)	27	( Manivasagam et al., 2023 ; Bateman et al., 2024 ; Chen and Muller, 2022 ; Hsiang et al., 2022 ; Von Bernuth et al., 2018 ; Talwar et al., 2020 ; Xu et al., 2021 ; Zhang and Cai, 2023 ; Amini and Nejati, 2024 ; Yang et al., 2023 ; Guo et al., 2025 ; Fu et al., 2025 ; Zeng et al., 2025 ; Steinhauser et al., 2025 ; Li et al., 2025a ; Zhou et al., 2026b ; Nosheen et al., 2025 ; Peng and Wang, 2025 ; Zuo et al., 2025 ; Gao et al., 2025b ; Haider et al., 2025 ; Petrovic and Muhammad, 2025 ; Manivasagam et al., 2020 ; Huch et al., 2023 ; Tian et al., 2026 ; Yang et al., 2021b ; Leung et al., 2025 )
IoU/segmentation overlap	20	( Argui et al., 2023 ; Tim et al., 2020 ; Talwar et al., 2020 ; Zhang and Cai, 2023 ; Ngo et al., 2021a ; Yang et al., 2023 ; Steinhauser et al., 2025 ; Pasios and Nikolaidis, 2025 ; Zhou et al., 2026a ; Gaddam et al., 2025 ; Zuo et al., 2025 ; Elmquist et al., 2025 ; Lambertenghi et al., 2025a ; Haghighi et al., 2026 ; Manivasagam et al., 2020 ; Shen et al., 2023 ; Tian et al., 2026 ; Wang et al., 2022a ; Yang et al., 2021b ; Leung et al., 2025 )
Precision/recall/F1	19	( Saleh et al., 2019 ; Reway et al., 2022 ; Reway et al., 2020 ; Manivasagam et al., 2023 ; Bateman et al., 2024 ; Brogle et al., 2019 ; Chen and Muller, 2022 ; Drechsler et al., 2022a ; Hsiang et al., 2022 ; Tim et al., 2020 ; Barker et al., 2018 ; Steinhauser et al., 2025 ; Hong, 2025 ; Nosheen et al., 2025 ; Li et al., 2025b ; Gao et al., 2025b ; Huch et al., 2023 ; Yang et al., 2021b ; Leung et al., 2025 )
Distributional image/video realism (FID/FVD/KID)	18	( Pahk et al., 2023 ; Pahk et al., 2024 ; Cao et al., 2024a ; Soni and Nair, 2021 ; Sun et al., 2024 ; Yang et al., 2023 ; Chen et al., 2025 ; Zeng et al., 2025 ; Steinhauser et al., 2025 ; Zhou et al., 2026b ; Wang et al., 2025d ; Nosheen et al., 2025 ; Wang et al., 2025b ; Zhou et al., 2026a ; Haghighi et al., 2026 ; Yu and Wang, 2026 ; Tian et al., 2026 ; Wang et al., 2022a )

Table 8. RQ 3.3 evaluation metric categories and their eight most frequent specific metrics (ties alphabetical), with primary studies.

Simulation platform / dataset	Studies
Simulation platforms	182
CARLA	45
IPG CarMaker	16
Unity	16
MATLAB/Simulink	13
SUMO	12

Table 9. RQ 4.1 simulation platforms and reference datasets reported by at least five studies.

Hardware platform	Studies	Primary studies
No hardware or unspecified	91	( Liu et al., 2021 ; Langer et al., 2021 ; Kim et al., 2024 ; Nalic et al., 2019 ; Saleh et al., 2019 ; Pahk et al., 2024 ; Kim et al., 2022 ; Argui et al., 2023 ; Bashetty et al., 2020 ; Bateman et al., 2024 ; Bergamini et al., 2021 ; Cao et al., 2024a ; Duminil et al., 2024b ; Gambi et al., 2019a ; Gu et al., 2023 ; Haq et al., 2021 ; Hoppe et al., 2022 ; Hsiang et al., 2022 ; Yang et al., 2021a ; Von Bernuth et al., 2018 ; Szalai et al., 2020 ; Wagner et al., 2019 ; Thieling et al., 2019 ; Tim et al., 2020 ; Xinxin et al., 2020 ; Talwar et al., 2020 ; Tenbrock et al., 2021 ; Xu et al., 2021 ; Soni and Nair, 2021 ; Zhang et al., 2022c ; Stadler et al., 2022a ; Shi et al., 2022 ; Zhu et al., 2022 ; Zhang et al., 2022b ; Xu et al., 2023 ; Zhang and Cai, 2023 ; Zhou et al., 2024 ; Sun et al., 2024 ; Wu et al., 2024 ; Stadler et al., 2022b ; Queiroz et al., 2024 ; Lan et al., 2024 ; Amini and Nejati, 2024 ; Long et al., 2024 ; Suo et al., 2023 ; Chen et al., 2025 ; Guo et al., 2025 ; Huang et al., 2025 ; Fu et al., 2025 ; Steinhauser et al., 2025 ; Zhou et al., 2026b ; Yan et al., 2025 ; Wang et al., 2025d ; Simakov et al., 2025 ; Pasios and Nikolaidis, 2025 ; Naing et al., 2025 ; Wu and Ma, 2025 ; Wang et al., 2025c ; Gao et al., 2025b ; Elmquist et al., 2025 ; Zhang et al., 2025b ; Wu et al., 2025 ; Haghighi et al., 2026 ; Abbasi et al., 2025 ; Jing et al., 2025 ; Kolb et al., 2025 ; Eberhardt et al., 2025 ; Harhammer et al., 2025 ; Samak et al., 2026 ; Hess et al., 2025 ; Sun et al., 2025a ; Bi et al., 2025 ; Yu and Wang, 2026 ; Huang et al., 2026 ; Schulz et al., 2025 ; Lee et al., 2025 ; Schabus et al., 2025 ; Lambertenghi and Stocco, 2024 ; Cao et al., 2024b ; Gadipudi et al., 2022 ; Hu and Zhao, 2025 ; Wang et al., 2022b ; Liang et al., 2025 ; Argui et al., 2025 ; Wei et al., 2025a ; Gao et al., 2025a ; Dauner et al., 2024 ; Wang et al., 2022a ; Leung et al., 2025 ; Deter et al., 2021 ; Wang et al., 2023 )
Full-size vehicle	76	( Ngo et al., 2021b ; Jayapal Gowdu et al., 2022 ; Lichtle et al., 2024 ; Peintner et al., 2021 ; Reway et al., 2020 ; Samak et al., 2024b ; Manivasagam et al., 2023 ; Zofka et al., 2018b ; Wachtel et al., 2021 ; Parra et al., 2020 ; Krecht et al., 2023 ; Shuguang et al., 2022 ; Bai et al., 2024 ; Brogle et al., 2019 ; Chen et al., 2020 ; Dona et al., 2022 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Fremont et al., 2020 ; Gao et al., 2020 ; Gutiérrez-Moreno et al., 2024 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Szalay et al., 2019 ; Yeo et al., 2019 ; Zofka et al., 2023 ; Durukal et al., 2022 ; Huang et al., 2024 ; Jiang et al., 2021 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Ngo et al., 2021a ; Chen et al., 2019 ; Oh et al., 2024a ; Che et al., 2021 ; Solmaz et al., 2020 ; Fei et al., 2024 ; Hwang et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Capdevila et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Ganesan et al., 2025 ; Ameli et al., 2025 ; Allamaa et al., 2025 ; Peng and Wang, 2025 ; Zuo et al., 2025 ; Jung et al., 2025 ; Zhao et al., 2025 ; Drechsler et al., 2025 ; Oh et al., 2024b ; Manivasagam et al., 2020 ; Shen et al., 2023 ; Vargas Rivero et al., 2022 ; Amini et al., 2022 ; Duminil et al., 2024a ; Magosi and Eichberger, 2023 ; Khaire et al., 2025 ; Shoukat et al., 2024 ; Cardozo and Hlavác, 2025 ; Arango et al., 2026 ; Drechsler et al., 2022c ; Magosi et al., 2022 ; Tian et al., 2026 ; Steimle et al., 2025 ; Li et al., 2024 ; Coppola et al., 2025 ; Daza et al., 2023 ; Jekl et al., 2025 ; Park et al., 2020 ; Yang et al., 2021b ; Reisgys et al., 2021 ; Sigl et al., 2021 )
Camera	48	( Pahk et al., 2023 ; Oladimeji et al., 2024 ; Lichtle et al., 2024 ; Reway et al., 2022 ; Reway et al., 2020 ; Shuguang et al., 2022 ; Amini et al., 2020 ; Bai et al., 2024 ; Brogle et al., 2019 ; Chen et al., 2020 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Fremont et al., 2020 ; Gupta et al., 2023 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Barker et al., 2018 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Chen et al., 2019 ; Hwang et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Ulreich et al., 2024 ; Capdevila et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Ganesan et al., 2025 ; Li et al., 2025a ; Ameli et al., 2025 ; Hong, 2025 ; Jung et al., 2025 ; Lambertenghi et al., 2025a ; Zhao et al., 2025 ; Li et al., 2025d ; Li et al., 2025e ; Wang et al., 2026 ; Ulreich et al., 2025 ; Otto et al., 2025 ; Li et al., 2025c ; Petrovic and Muhammad, 2025 ; Shen et al., 2023 ; Amini et al., 2022 ; Wang et al., 2025a ; Cardozo and Hlavác, 2025 ; Song et al., 2026 ; Yang et al., 2021b )
LiDAR	44	( Rosenberger et al., 2018 ; Rosenberger et al., 2019 ; Kettelgerdes and Elger, 2023 ; Manivasagam et al., 2023 ; Zofka et al., 2018b ; Wachtel et al., 2021 ; Bai et al., 2024 ; Brogle et al., 2019 ; Chen and Muller, 2022 ; Chen et al., 2020 ; Fremont et al., 2020 ; Gupta et al., 2023 ; Tettamanti et al., 2018 ; Zofka et al., 2018a ; Szalay et al., 2019 ; Wallace et al., 2022 ; Yuan et al., 2024 ; Huang et al., 2024 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Chen et al., 2019 ; Hwang et al., 2024 ; Samak et al., 2024a ; Capdevila et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Ganesan et al., 2025 ; Peng and Wang, 2025 ; Saad et al., 2025 ; Lambertenghi et al., 2025a ; Anand et al., 2026 ; Wang et al., 2026 ; Petrovic and Muhammad, 2025 ; Oh et al., 2024b ; Shen et al., 2023 ; Huch et al., 2023 ; Vargas Rivero et al., 2022 ; Amini et al., 2022 ; Wang et al., 2025a ; Balım et al., 2025 ; Hofrichter et al., 2025 ; Li et al., 2024 ; Yang et al., 2021b )
Localization/positioning hardware	44	( Ngo et al., 2021b ; Lu et al., 2022 ; Pahk et al., 2023 ; Peintner et al., 2021 ; Reway et al., 2020 ; Parra et al., 2020 ; Krecht et al., 2023 ; Shuguang et al., 2022 ; Amini et al., 2020 ; Bai et al., 2024 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Fremont et al., 2020 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Zofka et al., 2023 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Ngo et al., 2021a ; Chen et al., 2019 ; Che et al., 2021 ; Solmaz et al., 2020 ; Fu et al., 2024 ; Hwang et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Drechsler et al., 2024 ; Ganesan et al., 2025 ; Verma et al., 2025 ; Jung et al., 2025 ; Zhao et al., 2025 ; Petrovic and Muhammad, 2025 ; Oh et al., 2024b ; Shen et al., 2023 ; Magosi and Eichberger, 2023 ; Wang et al., 2025a ; Shoukat et al., 2024 ; Toth and Szalay, 2023 ; Steimle et al., 2025 ; Li et al., 2024 ; Guo et al., 2021 ; Sigl et al., 2021 )
Compute hardware	25	( Barker et al., 2018 ; Zeng et al., 2025 ; Neis et al., 2025 ; Ameli et al., 2025 ; Li et al., 2025f ; Gao et al., 2026a ; Hong, 2025 ; Baresi et al., 2025 ; Zhang et al., 2025c ; Liu et al., 2026 ; Nosheen et al., 2025 ; Li et al., 2025b ; Wang et al., 2025b ; Zhang et al., 2025a ; Zhou et al., 2026a ; Buzdugan et al., 2025 ; Wei et al., 2025b ; Gaddam et al., 2025 ; Da Rocha et al., 2025 ; Zuo et al., 2025 ; Stangherlin et al., 2025 ; Petrovic and Muhammad, 2025 ; Tian et al., 2026 ; Steimle et al., 2025 ; Heikkinen et al., 2019 )

Table 10. RQ 4.2 hardware platforms and supporting primary studies.

System under test	Studies
No SUT	89
Camera perception	40
YOLO-family detector	15
Object detector	12
Segmentation model	7
BEV-based detector/feature extractor	4

Table 11. RQ 4.3 systems under test and specific technologies.

Test-case origin	Studies	Primary studies
Simulation-generated	157	( Rosenberger et al., 2019 ; Lu et al., 2022 ; Nalic et al., 2019 ; Saleh et al., 2019 ; Pahk et al., 2023 ; Oladimeji et al., 2024 ; Jayapal Gowdu et al., 2022 ; Pahk et al., 2024 ; Peintner et al., 2021 ; Reway et al., 2022 ; Reway et al., 2020 ; Samak et al., 2024b ; Manivasagam et al., 2023 ; Zofka et al., 2018b ; Parra et al., 2020 ; Amini et al., 2020 ; Bashetty et al., 2020 ; Bergamini et al., 2021 ; Cao et al., 2024a ; Chen et al., 2020 ; Drechsler et al., 2022b ; Duminil et al., 2024b ; Fremont et al., 2020 ; Gu et al., 2023 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Hoppe et al., 2022 ; Hsiang et al., 2022 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Yang et al., 2021a ; Von Bernuth et al., 2018 ; Zofka et al., 2018a ; Szalai et al., 2020 ; Wagner et al., 2019 ; Thieling et al., 2019 ; Szalay et al., 2019 ; Yeo et al., 2019 ; Tim et al., 2020 ; Xinxin et al., 2020 ; Talwar et al., 2020 ; Xu et al., 2021 ; Soni and Nair, 2021 ; Zhang et al., 2022c ; Wallace et al., 2022 ; Shi et al., 2022 ; Zhu et al., 2022 ; Zhang et al., 2022b ; Zofka et al., 2023 ; Zhang and Cai, 2023 ; Yuan et al., 2024 ; Zhou et al., 2024 ; Ulhas et al., 2024 ; Sun et al., 2024 ; Huang et al., 2024 ; Larter et al., 2022 ; Barker et al., 2018 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Ngo et al., 2021a ; Chen et al., 2019 ; Oh et al., 2024a ; Solmaz et al., 2020 ; Lan et al., 2024 ; Amini and Nejati, 2024 ; Fei et al., 2024 ; Long et al., 2024 ; Fu et al., 2024 ; Hwang et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Suo et al., 2023 ; Ulreich et al., 2024 ; Wachtel et al., 2024 ; Capdevila et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Chen et al., 2025 ; Guo et al., 2025 ; Fu et al., 2025 ; Li et al., 2025a ; Neis et al., 2025 ; Zhou et al., 2026b ; Yan et al., 2025 ; Wang et al., 2025d ; Hong, 2025 ; Simakov et al., 2025 ; Zhang et al., 2025c ; Liu et al., 2026 ; Pasios and Nikolaidis, 2025 ; Nosheen et al., 2025 ; Li et al., 2025b ; Wang et al., 2025b ; Zhang et al., 2025a ; Zhou et al., 2026a ; Buzdugan et al., 2025 ; Wei et al., 2025b ; Gaddam et al., 2025 ; Peng and Wang, 2025 ; Wang et al., 2025c ; Zuo et al., 2025 ; Jung et al., 2025 ; Gao et al., 2025b ; Elmquist et al., 2025 ; Lambertenghi et al., 2025a ; Zhang et al., 2025b ; Wu et al., 2025 ; Haghighi et al., 2026 ; Abbasi et al., 2025 ; Jing et al., 2025 ; Kolb et al., 2025 ; Kim et al., 2025 ; Eberhardt et al., 2025 ; Harhammer et al., 2025 ; Samak et al., 2026 ; Hess et al., 2025 ; Sun et al., 2025a ; Stangherlin et al., 2025 ; Bi et al., 2025 ; Anand et al., 2026 ; Yu and Wang, 2026 ; Huang et al., 2026 ; Haider et al., 2025 ; Schabus et al., 2025 ; Waheed et al., 2025 ; Shen et al., 2023 ; Huch et al., 2023 ; Cao et al., 2024b ; Gadipudi et al., 2022 ; Vargas Rivero et al., 2022 ; Amini et al., 2022 ; Duminil et al., 2024a ; Wang et al., 2025a ; Khaire et al., 2025 ; Shoukat et al., 2024 ; Balım et al., 2025 ; Cardozo and Hlavác, 2025 ; Arango et al., 2026 ; Liang et al., 2025 ; Drechsler et al., 2022c ; Magosi et al., 2022 ; Tian et al., 2026 ; Hofrichter et al., 2025 ; Coppola et al., 2025 ; Elgharbawy et al., 2019 ; Segonne and Duthon, 2023 ; Daza et al., 2023 ; Jekl et al., 2025 ; Song et al., 2026 ; Wang et al., 2022a ; Yang et al., 2021b ; Leung et al., 2025 ; Guo et al., 2021 ; Deter et al., 2021 ; Wang et al., 2023 ; Heikkinen et al., 2019 )
Real-world executed	87	( Rosenberger et al., 2018 ; Ngo et al., 2021b ; Kim et al., 2024 ; Lichtle et al., 2024 ; Kettelgerdes and Elger, 2023 ; Reway et al., 2020 ; Wachtel et al., 2021 ; Parra et al., 2020 ; Krecht et al., 2023 ; Shuguang et al., 2022 ; Amini et al., 2020 ; Bateman et al., 2024 ; Brogle et al., 2019 ; Chen and Muller, 2022 ; Chen et al., 2020 ; Dona et al., 2022 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Fremont et al., 2020 ; Gao et al., 2020 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Stocco et al., 2023b ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Zofka et al., 2018a ; Szalai et al., 2020 ; Wagner et al., 2019 ; Yeo et al., 2019 ; Xu et al., 2021 ; Zhang et al., 2022b ; Zofka et al., 2023 ; Yuan et al., 2024 ; Stadler et al., 2022b ; Durukal et al., 2022 ; Queiroz et al., 2024 ; Jiang et al., 2021 ; Barker et al., 2018 ; Haider et al., 2024 ; Ngo et al., 2021a ; Oh et al., 2024a ; Che et al., 2021 ; Solmaz et al., 2020 ; Fei et al., 2024 ; Hwang et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Suo et al., 2023 ; Ulreich et al., 2024 ; Capdevila et al., 2024 ; Reckenzaun et al., 2024 ; Drechsler et al., 2024 ; Ganesan et al., 2025 ; Huang et al., 2025 ; Hong, 2025 ; Verma et al., 2025 ; Zuo et al., 2025 ; Jung et al., 2025 ; Lambertenghi et al., 2025a ; Zhao et al., 2025 ; Abbasi et al., 2025 ; Bi et al., 2025 ; Ulreich et al., 2025 ; Otto et al., 2025 ; Lee et al., 2025 ; Haider et al., 2025 ; Petrovic and Muhammad, 2025 ; Waheed et al., 2025 ; Oh et al., 2024b ; Stocco et al., 2023a ; Manivasagam et al., 2020 ; Shen et al., 2023 ; Vargas Rivero et al., 2022 ; Magosi and Eichberger, 2023 ; Hu and Zhao, 2025 ; Shoukat et al., 2024 ; Arango et al., 2026 ; Liang et al., 2025 ; Toth and Szalay, 2023 ; Magosi et al., 2022 ; Wei et al., 2025a ; Tian et al., 2026 ; Hofrichter et al., 2025 ; Li et al., 2024 ; Park et al., 2020 ; Reisgys et al., 2021 ; Sigl et al., 2021 )
Real-world reconstruction	73	( Liu et al., 2021 ; Langer et al., 2021 ; Ngo et al., 2021b ; Lu et al., 2022 ; Pahk et al., 2023 ; Oladimeji et al., 2024 ; Jayapal Gowdu et al., 2022 ; Peintner et al., 2021 ; Reway et al., 2020 ; Samak et al., 2024b ; Manivasagam et al., 2023 ; Zofka et al., 2018b ; Wachtel et al., 2021 ; Bashetty et al., 2020 ; Cao et al., 2024a ; Dona et al., 2022 ; Fremont et al., 2020 ; Gambi et al., 2019a ; Haq et al., 2021 ; Stocco et al., 2023b ; Schyr et al., 2022 ; Xinxin et al., 2020 ; Tenbrock et al., 2021 ; Zhang et al., 2022c ; Stadler et al., 2022a ; Zhang and Cai, 2023 ; Yuan et al., 2024 ; Zhou et al., 2024 ; Rezaldi et al., 2021 ; Queiroz et al., 2024 ; Larter et al., 2022 ; Haider et al., 2024 ; Chen et al., 2019 ; Che et al., 2021 ; Solmaz et al., 2020 ; Hwang et al., 2024 ; Yang et al., 2023 ; Chen et al., 2025 ; Steinhauser et al., 2025 ; Liu et al., 2026 ; Li et al., 2025b ; Zhang et al., 2025a ; Zhou et al., 2026a ; Wei et al., 2025b ; Allamaa et al., 2025 ; Gaddam et al., 2025 ; Peng and Wang, 2025 ; Zuo et al., 2025 ; Saad et al., 2025 ; Drechsler et al., 2025 ; Li et al., 2025d ; Stangherlin et al., 2025 ; Li et al., 2025e ; Wang et al., 2026 ; Ulreich et al., 2025 ; Otto et al., 2025 ; Schulz et al., 2025 ; Lee et al., 2025 ; Oh et al., 2024b ; Stocco et al., 2023a ; Lambertenghi and Stocco, 2024 ; Vargas Rivero et al., 2022 ; Amini et al., 2022 ; Wang et al., 2025a ; Wang et al., 2022b ; Martín Serrano et al., 2025 ; Magosi et al., 2022 ; Argui et al., 2025 ; Wang et al., 2022a ; Yang et al., 2021b ; Leung et al., 2025 ; Sigl et al., 2021 ; Deter et al., 2021 )
Recorded/replayed	63	( Kim et al., 2024 ; Saleh et al., 2019 ; Lichtle et al., 2024 ; Kim et al., 2022 ; Krecht et al., 2023 ; Amini et al., 2020 ; Bergamini et al., 2021 ; Dona et al., 2022 ; Haq et al., 2021 ; Hsiang et al., 2022 ; Von Bernuth et al., 2018 ; Tim et al., 2020 ; Talwar et al., 2020 ; Soni and Nair, 2021 ; Stadler et al., 2022a ; Zhang et al., 2022b ; Xu et al., 2023 ; Yuan et al., 2024 ; Zhou et al., 2024 ; Ulhas et al., 2024 ; Sun et al., 2024 ; Wu et al., 2024 ; Queiroz et al., 2024 ; Larter et al., 2022 ; Jiang et al., 2021 ; Haider et al., 2024 ; Ngo et al., 2021a ; Chen et al., 2019 ; Oh et al., 2024a ; Solmaz et al., 2020 ; Lan et al., 2024 ; Fu et al., 2024 ; Suo et al., 2023 ; Wachtel et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Guo et al., 2025 ; Ganesan et al., 2025 ; Steinhauser et al., 2025 ; Ameli et al., 2025 ; Li et al., 2025f ; Verma et al., 2025 ; Naing et al., 2025 ; Wu and Ma, 2025 ; Zhang et al., 2025a ; Zuo et al., 2025 ; Saad et al., 2025 ; Zhang et al., 2025b ; Kolb et al., 2025 ; Eberhardt et al., 2025 ; Hess et al., 2025 ; Huang et al., 2026 ; Li et al., 2025c ; Gadipudi et al., 2022 ; Amini et al., 2022 ; Wang et al., 2025a ; Balım et al., 2025 ; Tian et al., 2026 ; Gao et al., 2025a ; Elgharbawy et al., 2019 ; Dauner et al., 2024 ; Yang et al., 2021b ; Guo et al., 2021 )
Manual/predefined	37	( Rosenberger et al., 2018 ; Rosenberger et al., 2019 ; Bateman et al., 2024 ; Brogle et al., 2019 ; Bui et al., 2024 ; Chen and Muller, 2022 ; Dona et al., 2022 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Gao et al., 2020 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Hoppe et al., 2022 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Stadler et al., 2022b ; Jiang et al., 2021 ; Barker et al., 2018 ; Haider et al., 2024 ; Hwang et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Ulreich et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Hong, 2025 ; Buzdugan et al., 2025 ; Da Rocha et al., 2025 ; Lambertenghi et al., 2025a ; Drechsler et al., 2025 ; Kuroda et al., 2025 ; Otto et al., 2025 ; Lee et al., 2025 ; Oh et al., 2024b ; Deter et al., 2021 )
Hybrid/mixed	16	( Argui et al., 2023 ; Bai et al., 2024 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Gao et al., 2020 ; Szalay et al., 2019 ; Shi et al., 2022 ; Haider et al., 2024 ; Fu et al., 2024 ; Ulreich et al., 2024 ; Zeng et al., 2025 ; Gao et al., 2026a ; Baresi et al., 2025 ; Arango et al., 2026 ; Steimle et al., 2025 ; Coppola et al., 2025 )

Table 12. RQ 4.4 test-case origin and supporting primary studies.

Category	Subcategory	Citations
Testing artefacts	None/unspecified (188)	( Rosenberger et al., 2018 ; Liu et al., 2021 ; Rosenberger et al., 2019 ; Langer et al., 2021 ; Ngo et al., 2021b ; Lu et al., 2022 ; Kim et al., 2024 ; Nalic et al., 2019 ; Saleh et al., 2019 ; Pahk et al., 2023 ; Oladimeji et al., 2024 ; Jayapal Gowdu et al., 2022 ; Lichtle et al., 2024 ; Kettelgerdes and Elger, 2023 ; Pahk et al., 2024 ; Peintner et al., 2021 ; Kim et al., 2022 ; Reway et al., 2022 ; Reway et al., 2020 ; Manivasagam et al., 2023 ; Zofka et al., 2018b ; Wachtel et al., 2021 ; Parra et al., 2020 ; Krecht et al., 2023 ; Shuguang et al., 2022 ; Bai et al., 2024 ; Bashetty et al., 2020 ; Bateman et al., 2024 ; Brogle et al., 2019 ; Bui et al., 2024 ; Cao et al., 2024a ; Chen and Muller, 2022 ; Chen et al., 2020 ; Dona et al., 2022 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Duminil et al., 2024b ; Fremont et al., 2020 ; Gao et al., 2020 ; Gu et al., 2023 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Hoppe et al., 2022 ; Hsiang et al., 2022 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Yang et al., 2021a ; Von Bernuth et al., 2018 ; Zofka et al., 2018a ; Szalai et al., 2020 ; Wagner et al., 2019 ; Thieling et al., 2019 ; Szalay et al., 2019 ; Yeo et al., 2019 ; Tim et al., 2020 ; Xinxin et al., 2020 ; Talwar et al., 2020 ; Xu et al., 2021 ; Soni and Nair, 2021 ; Zhang et al., 2022c ; Wallace et al., 2022 ; Stadler et al., 2022a ; Shi et al., 2022 ; Zhu et al., 2022 ; Zhang et al., 2022b ; Zofka et al., 2023 ; Yuan et al., 2024 ; Zhou et al., 2024 ; Ulhas et al., 2024 ; Sun et al., 2024 ; Rezaldi et al., 2021 ; Stadler et al., 2022b ; Durukal et al., 2022 ; Queiroz et al., 2024 ; Huang et al., 2024 ; Larter et al., 2022 ; Jiang et al., 2021 ; Barker et al., 2018 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Chen et al., 2019 ; Oh et al., 2024a ; Che et al., 2021 ; Solmaz et al., 2020 ; Lan et al., 2024 ; Long et al., 2024 ; Fu et al., 2024 ; Hwang et al., 2024 ; Liu et al., 2024 ; Ulreich et al., 2024 ; Wachtel et al., 2024 ; Capdevila et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Chen et al., 2025 ; Guo et al., 2025 ; Huang et al., 2025 ; Fu et al., 2025 ; Zeng et al., 2025 ; Steinhauser et al., 2025 ; Ameli et al., 2025 ; Li et al., 2025f ; Zhou et al., 2026b ; Yan et al., 2025 ; Wang et al., 2025d ; Hong, 2025 ; Simakov et al., 2025 ; Verma et al., 2025 ; Zhang et al., 2025c ; Liu et al., 2026 ; Wu and Ma, 2025 ; Li et al., 2025b ; Wang et al., 2025b ; Buzdugan et al., 2025 ; Gaddam et al., 2025 ; Peng and Wang, 2025 ; Wang et al., 2025c ; Jung et al., 2025 ; Gao et al., 2025b ; Saad et al., 2025 ; Elmquist et al., 2025 ; Zhang et al., 2025b ; Zhao et al., 2025 ; Drechsler et al., 2025 ; Abbasi et al., 2025 ; Jing et al., 2025 ; Kolb et al., 2025 ; Li et al., 2025d ; Kim et al., 2025 ; Eberhardt et al., 2025 ; Harhammer et al., 2025 ; Samak et al., 2026 ; Sun et al., 2025a ; Stangherlin et al., 2025 ; Anand et al., 2026 ; Li et al., 2025e ; Yu and Wang, 2026 ; Ulreich et al., 2025 ; Otto et al., 2025 ; Schulz et al., 2025 ; Lee et al., 2025 ; Haider et al., 2025 ; Schabus et al., 2025 ; Petrovic and Muhammad, 2025 ; Waheed et al., 2025 ; Oh et al., 2024b ; Manivasagam et al., 2020 ; Shen et al., 2023 ; Gadipudi et al., 2022 ; Vargas Rivero et al., 2022 ; Duminil et al., 2024a ; Magosi and Eichberger, 2023 ; Wang et al., 2025a ; Hu and Zhao, 2025 ; Khaire et al., 2025 ; Wang et al., 2022b ; Shoukat et al., 2024 ; Martín Serrano et al., 2025 ; Balım et al., 2025 ; Cardozo and Hlavác, 2025 ; Liang et al., 2025 ; Toth and Szalay, 2023 ; Drechsler et al., 2022c ; Magosi et al., 2022 ; Wei et al., 2025a ; Tian et al., 2026 ; Steimle et al., 2025 ; Hofrichter et al., 2025 ; Gao et al., 2025a ; Li et al., 2024 ; Coppola et al., 2025 ; Elgharbawy et al., 2019 ; Segonne and Duthon, 2023 ; Daza et al., 2023 ; Jekl et al., 2025 ; Park et al., 2020 ; Song et al., 2026 ; Wang et al., 2022a ; Yang et al., 2021b ; Leung et al., 2025 ; Reisgys et al., 2021 ; Guo et al., 2021 ; Sigl et al., 2021 ; Deter et al., 2021 ; Wang et al., 2023 ; Heikkinen et al., 2019 )
	Code (26)	( Amini et al., 2020 ; Argui et al., 2023 ; Bergamini et al., 2021 ; Gambi et al., 2019a ; Stocco et al., 2023b ; Zhang and Cai, 2023 ; Wu et al., 2024 ; Samak et al., 2024a ; Ganesan et al., 2025 ; Li et al., 2025a ; Pasios and Nikolaidis, 2025 ; Nosheen et al., 2025 ; Naing et al., 2025 ; Zhang et al., 2025a ; Zhou et al., 2026a ; Wei et al., 2025b ; Da Rocha et al., 2025 ; Zuo et al., 2025 ; Wu et al., 2025 ; Haghighi et al., 2026 ; Hess et al., 2025 ; Wang et al., 2026 ; Huang et al., 2026 ; Stocco et al., 2023a ; Lambertenghi and Stocco, 2024 ; Dauner et al., 2024 )
	Other artefact (14)	( Samak et al., 2024b ; Xu et al., 2023 ; Ngo et al., 2021a ; Amini and Nejati, 2024 ; Fei et al., 2024 ; Suo et al., 2023 ; Neis et al., 2025 ; Gao et al., 2026a ; Allamaa et al., 2025 ; Lambertenghi et al., 2025a ; Kuroda et al., 2025 ; Bi et al., 2025 ; Amini et al., 2022 ; Arango et al., 2026 )
	Dataset (5)	( Haq et al., 2021 ; Tenbrock et al., 2021 ; Huch et al., 2023 ; Cao et al., 2024b ; Argui et al., 2025 )
	Code and dataset (2)	( Baresi et al., 2025 ; Li et al., 2025c )

Table 13. RQ 4.5 testing artefacts and citations.

Secondary study scope	reality gap focus	Gap targets	Mitigation	Evaluation	Execution	Component evidence
ADS testing surveys ( Tang et al., 2023 ; Zhao et al., 2024 )	–	Partial	Partial	Partial	–	–
Scenario-based testing surveys ( Zhong et al., 2021 )	–	Partial	Partial	–	–	–
Testing ML-based systems mappings ( Riccio et al., 2020 ; Zhang et al., 2022a )	–	Partial	–	Partial	–	–
Simulation platform surveys ( Kaur et al., 2021 ; Rosique et al., 2019 )	–	–	Partial	–	–	–
Sim-to-real and digital-twin surveys ( Hu et al., 2023 )	Partial	Partial	Yes	Partial	–	–
This mapping	Yes	Yes	Yes	Yes	Yes	Yes

Table 14. Scope of related secondary studies and this mapping. “Yes” indicates systematic classification, “Partial” discussion without classification, and “–” a dimension outside the scope or not systematically addressed.

Autonomous Driving Simulation

Autonomous driving simulation models roads, traffic, sensors, and vehicle behavior to train and evaluate driving systems safely and at scale.

See all 228 papers

Jul 6, 2026

Real-World Perturbation Testing of Autonomous Driving Systems

Autonomous Driving Systems (ADS) must operate reliably under diverse conditions, yet representative data for rare or adverse scenarios is difficult to obtain. Perturbation-based testing is widely used to assess robustness, but most studies focus on offline datasets or simulation, leaving open questions about how such results translate to real-world driving. We present a large-scale study of 72 camera and LiDAR perturbations, evaluated across three testing modalities: offline model-level analysis, hardware-in-the-loop execution, and closed-loop system-level testing on a full-scale autonomous vehicle. The study covers both an end-to-end vision-based driving model and a modular LiDAR-based perception and planning stack. Our results reveal a clear gap between testing levels. For camera-based systems, perturbations with limited offline impact can still induce unstable control and failures in real-world driving. For LiDAR-based systems, degradation is more consistent at the perception level but weakly predictive of system-level failures. Across both modalities, model-level metrics alone are insufficient to identify the most harmful perturbations. We further show that real-time feasibility is a key constraint in real-world testing, and that robustness observations obtained from recorded data do not consistently transfer to closed-loop behavior on a physical vehicle, highlighting the importance of complementary real-world, system-level evaluation.

Stefano Carlo Lambertenghi, Matthias Weil, Andrea Stocco

Technical University of Munich · fortiss GmbH · Germany

Autonomous Driving Safety Evaluation Adversarial Robustness Autonomous Driving Adversarial Perturbations

Ask PDF

Sep 3, 2026

Virtual Testing of Automated Driving Systems through Credible Simulations

Simulation is increasingly used to support safety-related decision-making in road transport, particularly for the assessment and approval of automated driving systems (ADS). The complexity of ADS behavior and size of their operational design domains make exclusive reliance on physical testing impractical, leading to extensive use of virtual testing (VT) during the approval phase. This shift raises critical questions regarding the credibility of modelling and simulation (M&S) results used to support road safety decisions. Current VT accreditation approaches in the ADS domain typically rely on validation-only practices, which have been shown to scale poorly when applied to complex, multi-tool simulation environments. To address this limitation, this paper proposes a risk-based framework for assessing the credibility of simulation toolchains used in ADS safety evaluation, drawing inspiration from established practices in other safety-critical domains, notably NASA's STD-7009 for models and simulations. The framework extends traditional verification and validation (V&V) by explicitly linking credibility requirements to the intended use of simulation outputs and to the safety criticality of the decisions they support within the approval process. It provides a lifecycle-oriented assessment scheme integrating toolchain management, modelling assumptions and limitations, verification, validation, and sensitivity analysis. Credibility acceptance thresholds are defined proportionally, allowing differentiated requirements depending on whether simulation is used for exploratory safety analysis, partial decision support, or as a substitute for physical testing. While demonstrated for ADS, the proposed approach is directly applicable to road safety and simulation studies where VT plays a central role in safety assessment and regulatory decision-making.

Riccardo Dona, Espedito Rusciano, Biagio Ciuffo

Joint Research Centre for the European Commission, Ispra (VA) 21027, Italy · Joint Research Centre for the European Commission, Petten 1755-ZG, Netherlands

Autonomous Driving Safety Evaluation Autonomous Driving Autonomous Driving Simulation

Ask PDF

Mar 11, 2025

Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing

Scenario-based testing with driving simulators is extensively used to identify failing conditions of automated driving assistance systems (ADAS). However, existing studies have shown that repeated test execution in the same as well as in distinct simulators can yield different outcomes, which can be attributed to sources of flakiness or different implementations of the physics. In this paper, we present MultiSim, a novel approach to multi-simulation ADAS testing based on a search-based testing approach that leverages an ensemble of simulators to identify failure-inducing, simulator-agnostic test scenarios. During the search, each scenario is evaluated jointly on multiple simulators. Scenarios that produce consistent results across simulators are prioritized for further exploration, while those that fail on only a subset of simulators are given less priority, as they may reflect simulator-specific issues rather than generalizable failures. Our empirical study, which involves testing three lane-keeping ADAS on different pairs of three widely used simulators, demonstrates that MultiSim outperforms single-simulator testing by achieving, on average, a higher rate of simulator-agnostic failures by 66%. Compared to a state-of-the-art multi-simulator approach that combines the outcome of independent test generation campaigns obtained in different simulators, MultiSim identifies, on average, up to 3.4X more simulator-agnostic failing tests and higher failure rates. To avoid the costly execution of test inputs on which simulators disagree, we propose to predict simulator disagreements and bypass test executions. Our results show that utilizing a surrogate model during the search retains the average number of valid failures and also improves efficiency. Our findings indicate that combining an ensemble of simulators is a promising approach for the automated cross-replication in ADAS testing.

Lev Sorokin, Matteo Biagiola, Andrea Stocco

Technical University of Munich, Munich, Germany. · University of St. Gallen and Universit`a della Svizzera italiana, St. Gallen / Lugano, Switzerland. · fortiss GmbH, Munich, Germany.

Automated Software Testing Autonomous Driving Simulation Automated Test Generation

Ask PDF

Abstract

Autonomous Driving Simulation

Explore similar work

Real-World Perturbation Testing of Autonomous Driving Systems

Virtual Testing of Automated Driving Systems through Credible Simulations

Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing