cs.SEOct 28, 2026
SaveThe Reality Gap in Automated Driving Systems Testing: Landscape, Challenges, and Research Directions
Organizations: Technical University of Munich, Boltzmannstraße 3, Garching, Bavaria, Germany, 85748 · fortiss GmbH, Boltzmannstraße 3, Garching, Bavaria, Germany, 85748 · NorCom GmbH & Co. KGaA, Boltzmannstraße 3, Garching, Bavaria, Germany, 85748 · Technical University of Munich, Boltzmannstraße 15, Garching, Bavaria, Germany, 85748
Abstract
Simulation is essential for testing automated driving systems (ADS), but discrepancies between simulation and reality can undermine testing conclusions. It remains unclear which discrepancies the literature addresses and what evidence supports correspondence claims. We present a systematic mapping of 235 primary studies that measure or mitigate the reality gap in ADS testing, selected from 4,195 publications. We characterise objectives, targets, approaches, experimental setups, and evaluation evidence. Our classification distinguishes simulated and real-world execution across six ADS test components, separately for each approach and its evaluation reference. This distinguishes real-world information used to construct an approach from evidence used to evaluate its claims. Of the included studies, 76% address mitigation, primarily targeting scenario and actor realism and perception, while 94% report validation or measurement procedures. Only 61 studies report metrics from multiple families covering sensor/perception correspondence, ADS behaviour, vehicle dynamics, or safety outcomes; this coverage does not establish causal links between discrepancies and test verdicts. The synthesis highlights that real-world content and physical execution must be interpreted against the claim evaluated. The findings motivate claim-specific evaluation and component-level reporting to establish which discrepancies affect testing conclusions and when simulation evidence supports conclusions about real-world ADS behaviour.
Figures & tables
Figure 1. Four visual discrepancies between a real camera recording and a comparable simulation: (1) image appearance, (2) vehicle model, (3) road markings, and (4) road-user representation. Boxes highlight the differences; their impact on ADS behaviour and test verdicts requires separate investigation.
| Category | Research Question | What it captures |
| Context | RQ 1.1 What is the primary objective of the study? | Whether the study measures the gap, mitigates it, or does both. |
| RQ 1.2 Which aspect(s) of the simulation–reality gap are addressed? | The aspect of the simulation–reality gap addressed by the study, including combinations of aspects. | |
| RQ 1.3 What is the direction of the simulation–reality relationship? | Whether the study addresses the gap from simulation to reality, from reality to simulation, or in both directions. | |
| Approach | RQ 2.1 Which research target does the study’s simulation–reality gap concern? | The specific capability, component, or phenomenon considered in relation to the simulation–reality gap. |
| RQ 2.2 How is the proposed simulation–reality mitigation implemented? | How the gap is mitigated, from the specific technique to its broader family. | |
| RQ 2.3 What are the execution levels of the proposed approach? | The execution level of each component and functional group across simulation and reality. |
Table 1. Research questions and what each one captures from a study.
| Study | Objective | Direction | Gap aspect |
|---|---|---|---|
| VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles ( Amini et al., 2022 ) | Mitigate | Real2sim | Perception, Behaviour |
| The Effect of Spray Water on an Automotive LIDAR Sensor: A Real-Time Simulation Study ( Vargas Rivero et al., 2022 ) | Mitigate | Real2sim | Perception |
| Synthetic to Real Gap Estimation of Autonomous Driving Datasets using Feature Embedding ( Gadipudi et al., 2022 ) | Measure | Sim2real | Perception |
| Sim-on-Wheels: Physical World in the Loop Simulation for Self-Driving ( Shen et al., 2023 ) | Mitigate | Both | Perception |
| Reinforcement Learning with Human Feedback for Realistic Traffic Simulation ( Cao et al., 2024b ) | Mitigate | Sim2real | Scenario |
| Quantifying the LiDAR Sim-to-Real Domain Shift: A Detailed Investigation Using Object Detectors and Analysing Point Clouds at Target-Level ( Huch et al., 2023 ) | Measure | Sim2real | Perception |
Table 2. The 10 pilot studies used to calibrate the review protocol and the coded values that motivated their selection.
Figure 2. Study identification, preliminary selection, and full selection process.
| RQ | Data type | Data content |
| RQ 1.1 | Predefined categorical | Measure only, Mitigate only, Measure and mitigate |
| RQ 1.2 | Predefined categorical, combinable | Scenario, Behaviour, Perception, Localization, Actuation, Entire; a study addressing several aspects equally is recorded as their combination |
| RQ 1.3 | Predefined categorical | Sim to real, Real to sim, Both |
| RQ 2.1 | Free text, categorised | Research target, grouped into macro categories and subcategories |
| RQ 2.2 | Free text, categorised | Mitigation technique, grouped into macro categories, subcategories, and optional subsubcategories |
| RQ 2.3 | Predefined categorical, one value per component | Scenario, Actors, Perception, Localization, Actuation, ECU/control, each recorded as Not modelled, Simulated, Pseudo-sim, Pseudo-real, Mixed, Both, or Real; Partially real additionally arises at group level; each setup also carries one or more execution-source tags (real vehicle, real hardware, real dataset, simulator, simulated dataset), or None |
Table 3. Type and content of the data used for each research question.
| Category | Definition |
|---|---|
| Not modelled | The component is not present. |
| Simulated | The component is provided by a general-purpose simulation platform or physics model. This holds even if the platform is calibrated, tuned, or partly trained using real-world data. |
| Pseudo-sim | The component is provided by a standalone model or generator, not a general-purpose simulation platform, whose representation is not constructed from real-world observations or measured real-world characteristics. |
| Pseudo-real | The component is provided by a standalone model, reconstruction, or generator whose representation is constructed from real-world observations, measurements, or measured real-world characteristics, but the real-world component is not present live during execution. |
| Mixed | Real and simulated elements are combined within the same execution. |
| Both | Separate simulated and real-world executions are performed in parallel. |
Table 4. Categorical execution levels used for RQ 2.3 and RQ 3.2 .
| Research target | Studies | Primary studies |
|---|---|---|
| Scenario and actor realism | 109 | |
| Traffic/NPC realism | 50 | ( Liu et al., 2021 ; Langer et al., 2021 ; Kim et al., 2024 ; Lichtle et al., 2024 ; Shuguang et al., 2022 ; Amini et al., 2020 ; Bergamini et al., 2021 ; Cao et al., 2024a ; Gu et al., 2023 ; Tettamanti et al., 2018 ; Yang et al., 2021a ; Wagner et al., 2019 ; Tenbrock et al., 2021 ; Shi et al., 2022 ; Zhang et al., 2022b ; Xu et al., 2023 ; Sun et al., 2024 ; Stadler et al., 2022b ; Queiroz et al., 2024 ; Barker et al., 2018 ; Che et al., 2021 ; Liu et al., 2024 ; Samak et al., 2024a ; Suo et al., 2023 ; Guo et al., 2025 ; Huang et al., 2025 ; Neis et al., 2025 ; Ameli et al., 2025 ; Yan et al., 2025 ; Gao et al., 2026a ; Zhang et al., 2025c ; Naing et al., 2025 ; Wu and Ma, 2025 ; Li et al., 2025b ; Buzdugan et al., 2025 ; Wang et al., 2025c ; Jing et al., 2025 ; Bi et al., 2025 ; Wang et al., 2026 ; Huang et al., 2026 ; Cao et al., 2024b ; Khaire et al., 2025 ; Wang et al., 2022b ; Liang et al., 2025 ; Drechsler et al., 2022c ; Gao et al., 2025a ; Coppola et al., 2025 ; Daza et al., 2023 ; Leung et al., 2025 ; Sigl et al., 2021 ) |
| Test-scenario validity | 26 | ( Liu et al., 2021 ; Nalic et al., 2019 ; Peintner et al., 2021 ; Kim et al., 2022 ; Bai et al., 2024 ; Chen et al., 2020 ; Fremont et al., 2020 ; Gao et al., 2020 ; Gupta et al., 2023 ; Schyr et al., 2022 ; Zofka et al., 2018a ; Wagner et al., 2019 ; Stadler et al., 2022a ; Wu et al., 2024 ; Solmaz et al., 2020 ; Lan et al., 2024 ; Reckenzaun et al., 2024 ; Chen et al., 2025 ; Wu et al., 2025 ; Kolb et al., 2025 ; Shen et al., 2023 ; Arango et al., 2026 ; Toth and Szalay, 2023 ; Tian et al., 2026 ; Elgharbawy et al., 2019 ; Guo et al., 2021 ) |
| Crash-scenario validity | 13 | ( Oladimeji et al., 2024 ; Bashetty et al., 2020 ; Gambi et al., 2019a ; Xinxin et al., 2020 ; Zhang and Cai, 2023 ; Zhou et al., 2024 ; Zhang et al., 2025a ; Wei et al., 2025b ; Zhao et al., 2025 ; Abbasi et al., 2025 ; Hu and Zhao, 2025 ; Shoukat et al., 2024 ; Wei et al., 2025a ) |
| Pedestrian/VRU realism | 12 | ( Zofka et al., 2018b ; Drechsler et al., 2022a ; Larter et al., 2022 ; Long et al., 2024 ; Fu et al., 2024 ; Fu et al., 2025 ; Liu et al., 2026 ; Li et al., 2025e ; Martín Serrano et al., 2025 ; Wei et al., 2025a ; Steimle et al., 2025 ; Yang et al., 2021b ) |
| Road/environment realism | 11 | ( Bateman et al., 2024 ; Brogle et al., 2019 ; Szalai et al., 2020 ; Zhang et al., 2022c ; Rezaldi et al., 2021 ; Li et al., 2025d ; Sun et al., 2025a ; Schulz et al., 2025 ; Wang et al., 2025a ; Jekl et al., 2025 ; Park et al., 2020 ) |
Table 5. RQ 2.1 research targets and supporting primary studies. Bars show the number of studies assigned to each target, and the final column lists the corresponding primary studies. Studies may be assigned to several targets, so counts across categories are not mutually exclusive and subcategory counts may not sum to the parent count.
| Mitigation technique | Studies |
|---|---|
| Real-equipment integration (ViL/HiL/MR/XiL) | 56 |
| Vehicle-in-the-loop (on-road) | 28 |
| Mixed reality | 21 |
| Controller/ECU hardware-in-the-loop | 6 |
| Co-simulation | 5 |
| Vehicle-in-the-loop (testbench) | 4 |
Table 6. RQ 2.2 mitigation technique families and techniques, among the studies with a mitigation objective. Bars show the number of studies using each family or technique. A study can use several families and several techniques within a family, so counts are not mutually exclusive and technique counts may exceed the family total.
Figure 3. RQ 2.3 execution level of the proposed approach: execution levels of the six individual components (top), and the observed combinations for Scene, Sensing, and Control ranked by frequency (bottom). In the lower panel, colours encode study objective and icons encode execution level.
| Evaluation technique | Studies | Primary studies |
|---|---|---|
| Direct or matched sim-to-real comparison | 132 | ( Rosenberger et al., 2018 ; Rosenberger et al., 2019 ; Langer et al., 2021 ; Ngo et al., 2021b ; Pahk et al., 2023 ; Jayapal Gowdu et al., 2022 ; Lichtle et al., 2024 ; Pahk et al., 2024 ; Peintner et al., 2021 ; Kim et al., 2022 ; Reway et al., 2020 ; Manivasagam et al., 2023 ; Wachtel et al., 2021 ; Parra et al., 2020 ; Krecht et al., 2023 ; Shuguang et al., 2022 ; Amini et al., 2020 ; Argui et al., 2023 ; Bashetty et al., 2020 ; Bateman et al., 2024 ; Bergamini et al., 2021 ; Brogle et al., 2019 ; Dona et al., 2022 ; Drechsler et al., 2022b ; Fremont et al., 2020 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Haq et al., 2021 ; Stocco et al., 2023b ; Hoppe et al., 2022 ; Hsiang et al., 2022 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Zofka et al., 2018a ; Wagner et al., 2019 ; Xu et al., 2021 ; Wallace et al., 2022 ; Shi et al., 2022 ; Zhu et al., 2022 ; Xu et al., 2023 ; Yuan et al., 2024 ; Stadler et al., 2022b ; Durukal et al., 2022 ; Queiroz et al., 2024 ; Huang et al., 2024 ; Larter et al., 2022 ; Jiang et al., 2021 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Ngo et al., 2021a ; Chen et al., 2019 ; Oh et al., 2024a ; Che et al., 2021 ; Solmaz et al., 2020 ; Lan et al., 2024 ; Amini and Nejati, 2024 ; Fu et al., 2024 ; Hwang et al., 2024 ; Suo et al., 2023 ; Ulreich et al., 2024 ; Wachtel et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Chen et al., 2025 ; Guo et al., 2025 ; Ganesan et al., 2025 ; Li et al., 2025a ; Neis et al., 2025 ; Ameli et al., 2025 ; Li et al., 2025f ; Zhou et al., 2026b ; Yan et al., 2025 ; Gao et al., 2026a ; Verma et al., 2025 ; Liu et al., 2026 ; Nosheen et al., 2025 ; Naing et al., 2025 ; Wu and Ma, 2025 ; Wei et al., 2025b ; Allamaa et al., 2025 ; Gaddam et al., 2025 ; Zuo et al., 2025 ; Jung et al., 2025 ; Saad et al., 2025 ; Lambertenghi et al., 2025a ; Zhao et al., 2025 ; Haghighi et al., 2026 ; Harhammer et al., 2025 ; Samak et al., 2026 ; Bi et al., 2025 ; Anand et al., 2026 ; Wang et al., 2026 ; Lee et al., 2025 ; Waheed et al., 2025 ; Oh et al., 2024b ; Stocco et al., 2023a ; Manivasagam et al., 2020 ; Shen et al., 2023 ; Huch et al., 2023 ; Vargas Rivero et al., 2022 ; Magosi and Eichberger, 2023 ; Wang et al., 2025a ; Khaire et al., 2025 ; Wang et al., 2022b ; Balım et al., 2025 ; Cardozo and Hlavác, 2025 ; Arango et al., 2026 ; Liang et al., 2025 ; Toth and Szalay, 2023 ; Drechsler et al., 2022c ; Magosi et al., 2022 ; Argui et al., 2025 ; Wei et al., 2025a ; Tian et al., 2026 ; Steimle et al., 2025 ; Hofrichter et al., 2025 ; Gao et al., 2025a ; Li et al., 2024 ; Coppola et al., 2025 ; Elgharbawy et al., 2019 ; Segonne and Duthon, 2023 ; Daza et al., 2023 ; Jekl et al., 2025 ; Park et al., 2020 ; Wang et al., 2022a ; Yang et al., 2021b ; Leung et al., 2025 ; Guo et al., 2021 ; Sigl et al., 2021 ; Deter et al., 2021 ) |
| Task/SUT performance comparison | 130 | ( Nalic et al., 2019 ; Saleh et al., 2019 ; Pahk et al., 2023 ; Oladimeji et al., 2024 ; Jayapal Gowdu et al., 2022 ; Pahk et al., 2024 ; Peintner et al., 2021 ; Reway et al., 2022 ; Reway et al., 2020 ; Zofka et al., 2018b ; Wachtel et al., 2021 ; Shuguang et al., 2022 ; Amini et al., 2020 ; Argui et al., 2023 ; Bai et al., 2024 ; Bashetty et al., 2020 ; Bateman et al., 2024 ; Brogle et al., 2019 ; Bui et al., 2024 ; Chen and Muller, 2022 ; Chen et al., 2020 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Fremont et al., 2020 ; Gambi et al., 2019a ; Gao et al., 2020 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Haq et al., 2021 ; Stocco et al., 2023b ; Hoppe et al., 2022 ; Hsiang et al., 2022 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Yang et al., 2021a ; Von Bernuth et al., 2018 ; Tim et al., 2020 ; Talwar et al., 2020 ; Tenbrock et al., 2021 ; Xu et al., 2021 ; Zhang et al., 2022c ; Shi et al., 2022 ; Zhang and Cai, 2023 ; Stadler et al., 2022b ; Queiroz et al., 2024 ; Huang et al., 2024 ; Jiang et al., 2021 ; Barker et al., 2018 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Ngo et al., 2021a ; Chen et al., 2019 ; Oh et al., 2024a ; Che et al., 2021 ; Solmaz et al., 2020 ; Lan et al., 2024 ; Amini and Nejati, 2024 ; Fu et al., 2024 ; Suo et al., 2023 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Chen et al., 2025 ; Fu et al., 2025 ; Zeng et al., 2025 ; Steinhauser et al., 2025 ; Li et al., 2025a ; Zhou et al., 2026b ; Wang et al., 2025d ; Hong, 2025 ; Simakov et al., 2025 ; Baresi et al., 2025 ; Zhang et al., 2025c ; Pasios and Nikolaidis, 2025 ; Nosheen et al., 2025 ; Li et al., 2025b ; Wang et al., 2025b ; Zhang et al., 2025a ; Zhou et al., 2026a ; Allamaa et al., 2025 ; Gaddam et al., 2025 ; Peng and Wang, 2025 ; Da Rocha et al., 2025 ; Zuo et al., 2025 ; Gao et al., 2025b ; Saad et al., 2025 ; Elmquist et al., 2025 ; Lambertenghi et al., 2025a ; Zhang et al., 2025b ; Haghighi et al., 2026 ; Drechsler et al., 2025 ; Li et al., 2025d ; Kim et al., 2025 ; Harhammer et al., 2025 ; Sun et al., 2025a ; Stangherlin et al., 2025 ; Li et al., 2025e ; Wang et al., 2026 ; Ulreich et al., 2025 ; Otto et al., 2025 ; Huang et al., 2026 ; Schulz et al., 2025 ; Haider et al., 2025 ; Schabus et al., 2025 ; Petrovic and Muhammad, 2025 ; Stocco et al., 2023a ; Manivasagam et al., 2020 ; Lambertenghi and Stocco, 2024 ; Shen et al., 2023 ; Huch et al., 2023 ; Amini et al., 2022 ; Wang et al., 2025a ; Shoukat et al., 2024 ; Martín Serrano et al., 2025 ; Balım et al., 2025 ; Arango et al., 2026 ; Toth and Szalay, 2023 ; Argui et al., 2025 ; Tian et al., 2026 ; Gao et al., 2025a ; Segonne and Duthon, 2023 ; Daza et al., 2023 ; Jekl et al., 2025 ; Park et al., 2020 ; Song et al., 2026 ; Dauner et al., 2024 ; Wang et al., 2022a ; Yang et al., 2021b ; Reisgys et al., 2021 ; Deter et al., 2021 ) |
| Quantitative or distributional comparison | 65 | ( Langer et al., 2021 ; Kettelgerdes and Elger, 2023 ; Reway et al., 2020 ; Dona et al., 2022 ; Duminil et al., 2024b ; Gambi et al., 2019a ; Gu et al., 2023 ; Hsiang et al., 2022 ; Wagner et al., 2019 ; Tim et al., 2020 ; Stadler et al., 2022a ; Shi et al., 2022 ; Zhang et al., 2022b ; Sun et al., 2024 ; Stadler et al., 2022b ; Durukal et al., 2022 ; Jiang et al., 2021 ; Haider et al., 2024 ; Oh et al., 2024a ; Lan et al., 2024 ; Amini and Nejati, 2024 ; Fei et al., 2024 ; Suo et al., 2023 ; Ulreich et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Guo et al., 2025 ; Ganesan et al., 2025 ; Huang et al., 2025 ; Steinhauser et al., 2025 ; Neis et al., 2025 ; Yan et al., 2025 ; Gao et al., 2026a ; Simakov et al., 2025 ; Zhang et al., 2025c ; Naing et al., 2025 ; Li et al., 2025b ; Wang et al., 2025b ; Zhang et al., 2025a ; Wang et al., 2025c ; Jung et al., 2025 ; Elmquist et al., 2025 ; Wu et al., 2025 ; Jing et al., 2025 ; Kolb et al., 2025 ; Lee et al., 2025 ; Oh et al., 2024b ; Lambertenghi and Stocco, 2024 ; Cao et al., 2024b ; Duminil et al., 2024a ; Magosi and Eichberger, 2023 ; Wang et al., 2025a ; Martín Serrano et al., 2025 ; Liang et al., 2025 ; Drechsler et al., 2022c ; Magosi et al., 2022 ; Gao et al., 2025a ; Li et al., 2024 ; Daza et al., 2023 ; Song et al., 2026 ; Dauner et al., 2024 ; Wang et al., 2022a ; Yang et al., 2021b ; Reisgys et al., 2021 ; Deter et al., 2021 ) |
| Feature or image-quality comparison | 24 | ( Oladimeji et al., 2024 ; Reway et al., 2022 ; Cao et al., 2024a ; Duminil et al., 2024b ; Hsiang et al., 2022 ; Thieling et al., 2019 ; Soni and Nair, 2021 ; Chen et al., 2025 ; Steinhauser et al., 2025 ; Baresi et al., 2025 ; Zhou et al., 2026a ; Kuroda et al., 2025 ; Eberhardt et al., 2025 ; Hess et al., 2025 ; Yu and Wang, 2026 ; Schulz et al., 2025 ; Li et al., 2025c ; Gadipudi et al., 2022 ; Argui et al., 2025 ; Tian et al., 2026 ; Wang et al., 2022a ; Leung et al., 2025 ; Reisgys et al., 2021 ; Guo et al., 2021 ) |
| Human or qualitative assessment | 21 | ( Langer et al., 2021 ; Lichtle et al., 2024 ; Kim et al., 2022 ; Zofka et al., 2018b ; Yeo et al., 2019 ; Yuan et al., 2024 ; Zhou et al., 2024 ; Ulhas et al., 2024 ; Sun et al., 2024 ; Wu et al., 2024 ; Durukal et al., 2022 ; Che et al., 2021 ; Fu et al., 2024 ; Baresi et al., 2025 ; Zhang et al., 2025c ; Buzdugan et al., 2025 ; Abbasi et al., 2025 ; Yu and Wang, 2026 ; Cao et al., 2024b ; Khaire et al., 2025 ; Argui et al., 2025 ) |
| No substantive evaluation | 16 | ( Liu et al., 2021 ; Lu et al., 2022 ; Kim et al., 2024 ; Samak et al., 2024b ; Szalai et al., 2020 ; Szalay et al., 2019 ; Xinxin et al., 2020 ; Zofka et al., 2023 ; Rezaldi et al., 2021 ; Long et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Capdevila et al., 2024 ; Hu and Zhao, 2025 ; Wang et al., 2023 ; Heikkinen et al., 2019 ) |
Table 7. RQ 3.1 evaluation techniques and supporting primary studies. Bars show the number of studies using each technique, and the final column lists the corresponding primary studies. Studies may use several techniques, so counts are not mutually exclusive.
Figure 4. RQ 3.2 execution level of the evaluation: six-component execution-state counts (top), and the observed functional-group state combinations ranked by frequency (bottom). In the lower panel, colours encode study objective, and icons encode execution category.
| Metric category / metric | Studies | Primary studies |
|---|---|---|
| Sensor/perception comparison | 107 | |
| Detection accuracy/rate | 30 | ( Rosenberger et al., 2019 ; Oladimeji et al., 2024 ; Reway et al., 2020 ; Manivasagam et al., 2023 ; Bashetty et al., 2020 ; Chen and Muller, 2022 ; Hsiang et al., 2022 ; Schyr et al., 2022 ; Von Bernuth et al., 2018 ; Stadler et al., 2022b ; Barker et al., 2018 ; Yang et al., 2023 ; Li et al., 2025a ; Simakov et al., 2025 ; Peng and Wang, 2025 ; Saad et al., 2025 ; Wang et al., 2026 ; Otto et al., 2025 ; Haider et al., 2025 ; Schabus et al., 2025 ; Manivasagam et al., 2020 ; Vargas Rivero et al., 2022 ; Duminil et al., 2024a ; Balım et al., 2025 ; Argui et al., 2025 ; Tian et al., 2026 ; Segonne and Duthon, 2023 ; Wang et al., 2022a ; Yang et al., 2021b ; Deter et al., 2021 ) |
| Average precision (mAP/AP) | 27 | ( Manivasagam et al., 2023 ; Bateman et al., 2024 ; Chen and Muller, 2022 ; Hsiang et al., 2022 ; Von Bernuth et al., 2018 ; Talwar et al., 2020 ; Xu et al., 2021 ; Zhang and Cai, 2023 ; Amini and Nejati, 2024 ; Yang et al., 2023 ; Guo et al., 2025 ; Fu et al., 2025 ; Zeng et al., 2025 ; Steinhauser et al., 2025 ; Li et al., 2025a ; Zhou et al., 2026b ; Nosheen et al., 2025 ; Peng and Wang, 2025 ; Zuo et al., 2025 ; Gao et al., 2025b ; Haider et al., 2025 ; Petrovic and Muhammad, 2025 ; Manivasagam et al., 2020 ; Huch et al., 2023 ; Tian et al., 2026 ; Yang et al., 2021b ; Leung et al., 2025 ) |
| IoU/segmentation overlap | 20 | ( Argui et al., 2023 ; Tim et al., 2020 ; Talwar et al., 2020 ; Zhang and Cai, 2023 ; Ngo et al., 2021a ; Yang et al., 2023 ; Steinhauser et al., 2025 ; Pasios and Nikolaidis, 2025 ; Zhou et al., 2026a ; Gaddam et al., 2025 ; Zuo et al., 2025 ; Elmquist et al., 2025 ; Lambertenghi et al., 2025a ; Haghighi et al., 2026 ; Manivasagam et al., 2020 ; Shen et al., 2023 ; Tian et al., 2026 ; Wang et al., 2022a ; Yang et al., 2021b ; Leung et al., 2025 ) |
| Precision/recall/F1 | 19 | ( Saleh et al., 2019 ; Reway et al., 2022 ; Reway et al., 2020 ; Manivasagam et al., 2023 ; Bateman et al., 2024 ; Brogle et al., 2019 ; Chen and Muller, 2022 ; Drechsler et al., 2022a ; Hsiang et al., 2022 ; Tim et al., 2020 ; Barker et al., 2018 ; Steinhauser et al., 2025 ; Hong, 2025 ; Nosheen et al., 2025 ; Li et al., 2025b ; Gao et al., 2025b ; Huch et al., 2023 ; Yang et al., 2021b ; Leung et al., 2025 ) |
| Distributional image/video realism (FID/FVD/KID) | 18 | ( Pahk et al., 2023 ; Pahk et al., 2024 ; Cao et al., 2024a ; Soni and Nair, 2021 ; Sun et al., 2024 ; Yang et al., 2023 ; Chen et al., 2025 ; Zeng et al., 2025 ; Steinhauser et al., 2025 ; Zhou et al., 2026b ; Wang et al., 2025d ; Nosheen et al., 2025 ; Wang et al., 2025b ; Zhou et al., 2026a ; Haghighi et al., 2026 ; Yu and Wang, 2026 ; Tian et al., 2026 ; Wang et al., 2022a ) |
Table 8. RQ 3.3 evaluation metric categories and their eight most frequent specific metrics (ties alphabetical), with primary studies.
| Simulation platform / dataset | Studies |
|---|---|
| Simulation platforms | 182 |
| CARLA | 45 |
| IPG CarMaker | 16 |
| Unity | 16 |
| MATLAB/Simulink | 13 |
| SUMO | 12 |
Table 9. RQ 4.1 simulation platforms and reference datasets reported by at least five studies.
| Hardware platform | Studies | Primary studies |
|---|---|---|
| No hardware or unspecified | 91 | ( Liu et al., 2021 ; Langer et al., 2021 ; Kim et al., 2024 ; Nalic et al., 2019 ; Saleh et al., 2019 ; Pahk et al., 2024 ; Kim et al., 2022 ; Argui et al., 2023 ; Bashetty et al., 2020 ; Bateman et al., 2024 ; Bergamini et al., 2021 ; Cao et al., 2024a ; Duminil et al., 2024b ; Gambi et al., 2019a ; Gu et al., 2023 ; Haq et al., 2021 ; Hoppe et al., 2022 ; Hsiang et al., 2022 ; Yang et al., 2021a ; Von Bernuth et al., 2018 ; Szalai et al., 2020 ; Wagner et al., 2019 ; Thieling et al., 2019 ; Tim et al., 2020 ; Xinxin et al., 2020 ; Talwar et al., 2020 ; Tenbrock et al., 2021 ; Xu et al., 2021 ; Soni and Nair, 2021 ; Zhang et al., 2022c ; Stadler et al., 2022a ; Shi et al., 2022 ; Zhu et al., 2022 ; Zhang et al., 2022b ; Xu et al., 2023 ; Zhang and Cai, 2023 ; Zhou et al., 2024 ; Sun et al., 2024 ; Wu et al., 2024 ; Stadler et al., 2022b ; Queiroz et al., 2024 ; Lan et al., 2024 ; Amini and Nejati, 2024 ; Long et al., 2024 ; Suo et al., 2023 ; Chen et al., 2025 ; Guo et al., 2025 ; Huang et al., 2025 ; Fu et al., 2025 ; Steinhauser et al., 2025 ; Zhou et al., 2026b ; Yan et al., 2025 ; Wang et al., 2025d ; Simakov et al., 2025 ; Pasios and Nikolaidis, 2025 ; Naing et al., 2025 ; Wu and Ma, 2025 ; Wang et al., 2025c ; Gao et al., 2025b ; Elmquist et al., 2025 ; Zhang et al., 2025b ; Wu et al., 2025 ; Haghighi et al., 2026 ; Abbasi et al., 2025 ; Jing et al., 2025 ; Kolb et al., 2025 ; Eberhardt et al., 2025 ; Harhammer et al., 2025 ; Samak et al., 2026 ; Hess et al., 2025 ; Sun et al., 2025a ; Bi et al., 2025 ; Yu and Wang, 2026 ; Huang et al., 2026 ; Schulz et al., 2025 ; Lee et al., 2025 ; Schabus et al., 2025 ; Lambertenghi and Stocco, 2024 ; Cao et al., 2024b ; Gadipudi et al., 2022 ; Hu and Zhao, 2025 ; Wang et al., 2022b ; Liang et al., 2025 ; Argui et al., 2025 ; Wei et al., 2025a ; Gao et al., 2025a ; Dauner et al., 2024 ; Wang et al., 2022a ; Leung et al., 2025 ; Deter et al., 2021 ; Wang et al., 2023 ) |
| Full-size vehicle | 76 | ( Ngo et al., 2021b ; Jayapal Gowdu et al., 2022 ; Lichtle et al., 2024 ; Peintner et al., 2021 ; Reway et al., 2020 ; Samak et al., 2024b ; Manivasagam et al., 2023 ; Zofka et al., 2018b ; Wachtel et al., 2021 ; Parra et al., 2020 ; Krecht et al., 2023 ; Shuguang et al., 2022 ; Bai et al., 2024 ; Brogle et al., 2019 ; Chen et al., 2020 ; Dona et al., 2022 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Fremont et al., 2020 ; Gao et al., 2020 ; Gutiérrez-Moreno et al., 2024 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Szalay et al., 2019 ; Yeo et al., 2019 ; Zofka et al., 2023 ; Durukal et al., 2022 ; Huang et al., 2024 ; Jiang et al., 2021 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Ngo et al., 2021a ; Chen et al., 2019 ; Oh et al., 2024a ; Che et al., 2021 ; Solmaz et al., 2020 ; Fei et al., 2024 ; Hwang et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Capdevila et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Ganesan et al., 2025 ; Ameli et al., 2025 ; Allamaa et al., 2025 ; Peng and Wang, 2025 ; Zuo et al., 2025 ; Jung et al., 2025 ; Zhao et al., 2025 ; Drechsler et al., 2025 ; Oh et al., 2024b ; Manivasagam et al., 2020 ; Shen et al., 2023 ; Vargas Rivero et al., 2022 ; Amini et al., 2022 ; Duminil et al., 2024a ; Magosi and Eichberger, 2023 ; Khaire et al., 2025 ; Shoukat et al., 2024 ; Cardozo and Hlavác, 2025 ; Arango et al., 2026 ; Drechsler et al., 2022c ; Magosi et al., 2022 ; Tian et al., 2026 ; Steimle et al., 2025 ; Li et al., 2024 ; Coppola et al., 2025 ; Daza et al., 2023 ; Jekl et al., 2025 ; Park et al., 2020 ; Yang et al., 2021b ; Reisgys et al., 2021 ; Sigl et al., 2021 ) |
| Camera | 48 | ( Pahk et al., 2023 ; Oladimeji et al., 2024 ; Lichtle et al., 2024 ; Reway et al., 2022 ; Reway et al., 2020 ; Shuguang et al., 2022 ; Amini et al., 2020 ; Bai et al., 2024 ; Brogle et al., 2019 ; Chen et al., 2020 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Fremont et al., 2020 ; Gupta et al., 2023 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Barker et al., 2018 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Chen et al., 2019 ; Hwang et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Ulreich et al., 2024 ; Capdevila et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Ganesan et al., 2025 ; Li et al., 2025a ; Ameli et al., 2025 ; Hong, 2025 ; Jung et al., 2025 ; Lambertenghi et al., 2025a ; Zhao et al., 2025 ; Li et al., 2025d ; Li et al., 2025e ; Wang et al., 2026 ; Ulreich et al., 2025 ; Otto et al., 2025 ; Li et al., 2025c ; Petrovic and Muhammad, 2025 ; Shen et al., 2023 ; Amini et al., 2022 ; Wang et al., 2025a ; Cardozo and Hlavác, 2025 ; Song et al., 2026 ; Yang et al., 2021b ) |
| LiDAR | 44 | ( Rosenberger et al., 2018 ; Rosenberger et al., 2019 ; Kettelgerdes and Elger, 2023 ; Manivasagam et al., 2023 ; Zofka et al., 2018b ; Wachtel et al., 2021 ; Bai et al., 2024 ; Brogle et al., 2019 ; Chen and Muller, 2022 ; Chen et al., 2020 ; Fremont et al., 2020 ; Gupta et al., 2023 ; Tettamanti et al., 2018 ; Zofka et al., 2018a ; Szalay et al., 2019 ; Wallace et al., 2022 ; Yuan et al., 2024 ; Huang et al., 2024 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Chen et al., 2019 ; Hwang et al., 2024 ; Samak et al., 2024a ; Capdevila et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Ganesan et al., 2025 ; Peng and Wang, 2025 ; Saad et al., 2025 ; Lambertenghi et al., 2025a ; Anand et al., 2026 ; Wang et al., 2026 ; Petrovic and Muhammad, 2025 ; Oh et al., 2024b ; Shen et al., 2023 ; Huch et al., 2023 ; Vargas Rivero et al., 2022 ; Amini et al., 2022 ; Wang et al., 2025a ; Balım et al., 2025 ; Hofrichter et al., 2025 ; Li et al., 2024 ; Yang et al., 2021b ) |
| Localization/positioning hardware | 44 | ( Ngo et al., 2021b ; Lu et al., 2022 ; Pahk et al., 2023 ; Peintner et al., 2021 ; Reway et al., 2020 ; Parra et al., 2020 ; Krecht et al., 2023 ; Shuguang et al., 2022 ; Amini et al., 2020 ; Bai et al., 2024 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Fremont et al., 2020 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Zofka et al., 2023 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Ngo et al., 2021a ; Chen et al., 2019 ; Che et al., 2021 ; Solmaz et al., 2020 ; Fu et al., 2024 ; Hwang et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Drechsler et al., 2024 ; Ganesan et al., 2025 ; Verma et al., 2025 ; Jung et al., 2025 ; Zhao et al., 2025 ; Petrovic and Muhammad, 2025 ; Oh et al., 2024b ; Shen et al., 2023 ; Magosi and Eichberger, 2023 ; Wang et al., 2025a ; Shoukat et al., 2024 ; Toth and Szalay, 2023 ; Steimle et al., 2025 ; Li et al., 2024 ; Guo et al., 2021 ; Sigl et al., 2021 ) |
| Compute hardware | 25 | ( Barker et al., 2018 ; Zeng et al., 2025 ; Neis et al., 2025 ; Ameli et al., 2025 ; Li et al., 2025f ; Gao et al., 2026a ; Hong, 2025 ; Baresi et al., 2025 ; Zhang et al., 2025c ; Liu et al., 2026 ; Nosheen et al., 2025 ; Li et al., 2025b ; Wang et al., 2025b ; Zhang et al., 2025a ; Zhou et al., 2026a ; Buzdugan et al., 2025 ; Wei et al., 2025b ; Gaddam et al., 2025 ; Da Rocha et al., 2025 ; Zuo et al., 2025 ; Stangherlin et al., 2025 ; Petrovic and Muhammad, 2025 ; Tian et al., 2026 ; Steimle et al., 2025 ; Heikkinen et al., 2019 ) |
Table 10. RQ 4.2 hardware platforms and supporting primary studies.
| System under test | Studies |
|---|---|
| No SUT | 89 |
| Camera perception | 40 |
| YOLO-family detector | 15 |
| Object detector | 12 |
| Segmentation model | 7 |
| BEV-based detector/feature extractor | 4 |
Table 11. RQ 4.3 systems under test and specific technologies.
| Test-case origin | Studies | Primary studies |
|---|---|---|
| Simulation-generated | 157 | ( Rosenberger et al., 2019 ; Lu et al., 2022 ; Nalic et al., 2019 ; Saleh et al., 2019 ; Pahk et al., 2023 ; Oladimeji et al., 2024 ; Jayapal Gowdu et al., 2022 ; Pahk et al., 2024 ; Peintner et al., 2021 ; Reway et al., 2022 ; Reway et al., 2020 ; Samak et al., 2024b ; Manivasagam et al., 2023 ; Zofka et al., 2018b ; Parra et al., 2020 ; Amini et al., 2020 ; Bashetty et al., 2020 ; Bergamini et al., 2021 ; Cao et al., 2024a ; Chen et al., 2020 ; Drechsler et al., 2022b ; Duminil et al., 2024b ; Fremont et al., 2020 ; Gu et al., 2023 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Hoppe et al., 2022 ; Hsiang et al., 2022 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Yang et al., 2021a ; Von Bernuth et al., 2018 ; Zofka et al., 2018a ; Szalai et al., 2020 ; Wagner et al., 2019 ; Thieling et al., 2019 ; Szalay et al., 2019 ; Yeo et al., 2019 ; Tim et al., 2020 ; Xinxin et al., 2020 ; Talwar et al., 2020 ; Xu et al., 2021 ; Soni and Nair, 2021 ; Zhang et al., 2022c ; Wallace et al., 2022 ; Shi et al., 2022 ; Zhu et al., 2022 ; Zhang et al., 2022b ; Zofka et al., 2023 ; Zhang and Cai, 2023 ; Yuan et al., 2024 ; Zhou et al., 2024 ; Ulhas et al., 2024 ; Sun et al., 2024 ; Huang et al., 2024 ; Larter et al., 2022 ; Barker et al., 2018 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Ngo et al., 2021a ; Chen et al., 2019 ; Oh et al., 2024a ; Solmaz et al., 2020 ; Lan et al., 2024 ; Amini and Nejati, 2024 ; Fei et al., 2024 ; Long et al., 2024 ; Fu et al., 2024 ; Hwang et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Suo et al., 2023 ; Ulreich et al., 2024 ; Wachtel et al., 2024 ; Capdevila et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Chen et al., 2025 ; Guo et al., 2025 ; Fu et al., 2025 ; Li et al., 2025a ; Neis et al., 2025 ; Zhou et al., 2026b ; Yan et al., 2025 ; Wang et al., 2025d ; Hong, 2025 ; Simakov et al., 2025 ; Zhang et al., 2025c ; Liu et al., 2026 ; Pasios and Nikolaidis, 2025 ; Nosheen et al., 2025 ; Li et al., 2025b ; Wang et al., 2025b ; Zhang et al., 2025a ; Zhou et al., 2026a ; Buzdugan et al., 2025 ; Wei et al., 2025b ; Gaddam et al., 2025 ; Peng and Wang, 2025 ; Wang et al., 2025c ; Zuo et al., 2025 ; Jung et al., 2025 ; Gao et al., 2025b ; Elmquist et al., 2025 ; Lambertenghi et al., 2025a ; Zhang et al., 2025b ; Wu et al., 2025 ; Haghighi et al., 2026 ; Abbasi et al., 2025 ; Jing et al., 2025 ; Kolb et al., 2025 ; Kim et al., 2025 ; Eberhardt et al., 2025 ; Harhammer et al., 2025 ; Samak et al., 2026 ; Hess et al., 2025 ; Sun et al., 2025a ; Stangherlin et al., 2025 ; Bi et al., 2025 ; Anand et al., 2026 ; Yu and Wang, 2026 ; Huang et al., 2026 ; Haider et al., 2025 ; Schabus et al., 2025 ; Waheed et al., 2025 ; Shen et al., 2023 ; Huch et al., 2023 ; Cao et al., 2024b ; Gadipudi et al., 2022 ; Vargas Rivero et al., 2022 ; Amini et al., 2022 ; Duminil et al., 2024a ; Wang et al., 2025a ; Khaire et al., 2025 ; Shoukat et al., 2024 ; Balım et al., 2025 ; Cardozo and Hlavác, 2025 ; Arango et al., 2026 ; Liang et al., 2025 ; Drechsler et al., 2022c ; Magosi et al., 2022 ; Tian et al., 2026 ; Hofrichter et al., 2025 ; Coppola et al., 2025 ; Elgharbawy et al., 2019 ; Segonne and Duthon, 2023 ; Daza et al., 2023 ; Jekl et al., 2025 ; Song et al., 2026 ; Wang et al., 2022a ; Yang et al., 2021b ; Leung et al., 2025 ; Guo et al., 2021 ; Deter et al., 2021 ; Wang et al., 2023 ; Heikkinen et al., 2019 ) |
| Real-world executed | 87 | ( Rosenberger et al., 2018 ; Ngo et al., 2021b ; Kim et al., 2024 ; Lichtle et al., 2024 ; Kettelgerdes and Elger, 2023 ; Reway et al., 2020 ; Wachtel et al., 2021 ; Parra et al., 2020 ; Krecht et al., 2023 ; Shuguang et al., 2022 ; Amini et al., 2020 ; Bateman et al., 2024 ; Brogle et al., 2019 ; Chen and Muller, 2022 ; Chen et al., 2020 ; Dona et al., 2022 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Fremont et al., 2020 ; Gao et al., 2020 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Stocco et al., 2023b ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Zofka et al., 2018a ; Szalai et al., 2020 ; Wagner et al., 2019 ; Yeo et al., 2019 ; Xu et al., 2021 ; Zhang et al., 2022b ; Zofka et al., 2023 ; Yuan et al., 2024 ; Stadler et al., 2022b ; Durukal et al., 2022 ; Queiroz et al., 2024 ; Jiang et al., 2021 ; Barker et al., 2018 ; Haider et al., 2024 ; Ngo et al., 2021a ; Oh et al., 2024a ; Che et al., 2021 ; Solmaz et al., 2020 ; Fei et al., 2024 ; Hwang et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Suo et al., 2023 ; Ulreich et al., 2024 ; Capdevila et al., 2024 ; Reckenzaun et al., 2024 ; Drechsler et al., 2024 ; Ganesan et al., 2025 ; Huang et al., 2025 ; Hong, 2025 ; Verma et al., 2025 ; Zuo et al., 2025 ; Jung et al., 2025 ; Lambertenghi et al., 2025a ; Zhao et al., 2025 ; Abbasi et al., 2025 ; Bi et al., 2025 ; Ulreich et al., 2025 ; Otto et al., 2025 ; Lee et al., 2025 ; Haider et al., 2025 ; Petrovic and Muhammad, 2025 ; Waheed et al., 2025 ; Oh et al., 2024b ; Stocco et al., 2023a ; Manivasagam et al., 2020 ; Shen et al., 2023 ; Vargas Rivero et al., 2022 ; Magosi and Eichberger, 2023 ; Hu and Zhao, 2025 ; Shoukat et al., 2024 ; Arango et al., 2026 ; Liang et al., 2025 ; Toth and Szalay, 2023 ; Magosi et al., 2022 ; Wei et al., 2025a ; Tian et al., 2026 ; Hofrichter et al., 2025 ; Li et al., 2024 ; Park et al., 2020 ; Reisgys et al., 2021 ; Sigl et al., 2021 ) |
| Real-world reconstruction | 73 | ( Liu et al., 2021 ; Langer et al., 2021 ; Ngo et al., 2021b ; Lu et al., 2022 ; Pahk et al., 2023 ; Oladimeji et al., 2024 ; Jayapal Gowdu et al., 2022 ; Peintner et al., 2021 ; Reway et al., 2020 ; Samak et al., 2024b ; Manivasagam et al., 2023 ; Zofka et al., 2018b ; Wachtel et al., 2021 ; Bashetty et al., 2020 ; Cao et al., 2024a ; Dona et al., 2022 ; Fremont et al., 2020 ; Gambi et al., 2019a ; Haq et al., 2021 ; Stocco et al., 2023b ; Schyr et al., 2022 ; Xinxin et al., 2020 ; Tenbrock et al., 2021 ; Zhang et al., 2022c ; Stadler et al., 2022a ; Zhang and Cai, 2023 ; Yuan et al., 2024 ; Zhou et al., 2024 ; Rezaldi et al., 2021 ; Queiroz et al., 2024 ; Larter et al., 2022 ; Haider et al., 2024 ; Chen et al., 2019 ; Che et al., 2021 ; Solmaz et al., 2020 ; Hwang et al., 2024 ; Yang et al., 2023 ; Chen et al., 2025 ; Steinhauser et al., 2025 ; Liu et al., 2026 ; Li et al., 2025b ; Zhang et al., 2025a ; Zhou et al., 2026a ; Wei et al., 2025b ; Allamaa et al., 2025 ; Gaddam et al., 2025 ; Peng and Wang, 2025 ; Zuo et al., 2025 ; Saad et al., 2025 ; Drechsler et al., 2025 ; Li et al., 2025d ; Stangherlin et al., 2025 ; Li et al., 2025e ; Wang et al., 2026 ; Ulreich et al., 2025 ; Otto et al., 2025 ; Schulz et al., 2025 ; Lee et al., 2025 ; Oh et al., 2024b ; Stocco et al., 2023a ; Lambertenghi and Stocco, 2024 ; Vargas Rivero et al., 2022 ; Amini et al., 2022 ; Wang et al., 2025a ; Wang et al., 2022b ; Martín Serrano et al., 2025 ; Magosi et al., 2022 ; Argui et al., 2025 ; Wang et al., 2022a ; Yang et al., 2021b ; Leung et al., 2025 ; Sigl et al., 2021 ; Deter et al., 2021 ) |
| Recorded/replayed | 63 | ( Kim et al., 2024 ; Saleh et al., 2019 ; Lichtle et al., 2024 ; Kim et al., 2022 ; Krecht et al., 2023 ; Amini et al., 2020 ; Bergamini et al., 2021 ; Dona et al., 2022 ; Haq et al., 2021 ; Hsiang et al., 2022 ; Von Bernuth et al., 2018 ; Tim et al., 2020 ; Talwar et al., 2020 ; Soni and Nair, 2021 ; Stadler et al., 2022a ; Zhang et al., 2022b ; Xu et al., 2023 ; Yuan et al., 2024 ; Zhou et al., 2024 ; Ulhas et al., 2024 ; Sun et al., 2024 ; Wu et al., 2024 ; Queiroz et al., 2024 ; Larter et al., 2022 ; Jiang et al., 2021 ; Haider et al., 2024 ; Ngo et al., 2021a ; Chen et al., 2019 ; Oh et al., 2024a ; Solmaz et al., 2020 ; Lan et al., 2024 ; Fu et al., 2024 ; Suo et al., 2023 ; Wachtel et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Guo et al., 2025 ; Ganesan et al., 2025 ; Steinhauser et al., 2025 ; Ameli et al., 2025 ; Li et al., 2025f ; Verma et al., 2025 ; Naing et al., 2025 ; Wu and Ma, 2025 ; Zhang et al., 2025a ; Zuo et al., 2025 ; Saad et al., 2025 ; Zhang et al., 2025b ; Kolb et al., 2025 ; Eberhardt et al., 2025 ; Hess et al., 2025 ; Huang et al., 2026 ; Li et al., 2025c ; Gadipudi et al., 2022 ; Amini et al., 2022 ; Wang et al., 2025a ; Balım et al., 2025 ; Tian et al., 2026 ; Gao et al., 2025a ; Elgharbawy et al., 2019 ; Dauner et al., 2024 ; Yang et al., 2021b ; Guo et al., 2021 ) |
| Manual/predefined | 37 | ( Rosenberger et al., 2018 ; Rosenberger et al., 2019 ; Bateman et al., 2024 ; Brogle et al., 2019 ; Bui et al., 2024 ; Chen and Muller, 2022 ; Dona et al., 2022 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Gao et al., 2020 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Hoppe et al., 2022 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Stadler et al., 2022b ; Jiang et al., 2021 ; Barker et al., 2018 ; Haider et al., 2024 ; Hwang et al., 2024 ; Liu et al., 2024 ; Samak et al., 2024a ; Ulreich et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Hong, 2025 ; Buzdugan et al., 2025 ; Da Rocha et al., 2025 ; Lambertenghi et al., 2025a ; Drechsler et al., 2025 ; Kuroda et al., 2025 ; Otto et al., 2025 ; Lee et al., 2025 ; Oh et al., 2024b ; Deter et al., 2021 ) |
| Hybrid/mixed | 16 | ( Argui et al., 2023 ; Bai et al., 2024 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Gao et al., 2020 ; Szalay et al., 2019 ; Shi et al., 2022 ; Haider et al., 2024 ; Fu et al., 2024 ; Ulreich et al., 2024 ; Zeng et al., 2025 ; Gao et al., 2026a ; Baresi et al., 2025 ; Arango et al., 2026 ; Steimle et al., 2025 ; Coppola et al., 2025 ) |
Table 12. RQ 4.4 test-case origin and supporting primary studies.
| Category | Subcategory | Citations |
|---|---|---|
| Testing artefacts | None/unspecified (188) | ( Rosenberger et al., 2018 ; Liu et al., 2021 ; Rosenberger et al., 2019 ; Langer et al., 2021 ; Ngo et al., 2021b ; Lu et al., 2022 ; Kim et al., 2024 ; Nalic et al., 2019 ; Saleh et al., 2019 ; Pahk et al., 2023 ; Oladimeji et al., 2024 ; Jayapal Gowdu et al., 2022 ; Lichtle et al., 2024 ; Kettelgerdes and Elger, 2023 ; Pahk et al., 2024 ; Peintner et al., 2021 ; Kim et al., 2022 ; Reway et al., 2022 ; Reway et al., 2020 ; Manivasagam et al., 2023 ; Zofka et al., 2018b ; Wachtel et al., 2021 ; Parra et al., 2020 ; Krecht et al., 2023 ; Shuguang et al., 2022 ; Bai et al., 2024 ; Bashetty et al., 2020 ; Bateman et al., 2024 ; Brogle et al., 2019 ; Bui et al., 2024 ; Cao et al., 2024a ; Chen and Muller, 2022 ; Chen et al., 2020 ; Dona et al., 2022 ; Drechsler et al., 2022a ; Drechsler et al., 2022b ; Duminil et al., 2024b ; Fremont et al., 2020 ; Gao et al., 2020 ; Gu et al., 2023 ; Gupta et al., 2023 ; Gutiérrez-Moreno et al., 2024 ; Hoppe et al., 2022 ; Hsiang et al., 2022 ; Schyr et al., 2022 ; Tettamanti et al., 2018 ; Yang et al., 2021a ; Von Bernuth et al., 2018 ; Zofka et al., 2018a ; Szalai et al., 2020 ; Wagner et al., 2019 ; Thieling et al., 2019 ; Szalay et al., 2019 ; Yeo et al., 2019 ; Tim et al., 2020 ; Xinxin et al., 2020 ; Talwar et al., 2020 ; Xu et al., 2021 ; Soni and Nair, 2021 ; Zhang et al., 2022c ; Wallace et al., 2022 ; Stadler et al., 2022a ; Shi et al., 2022 ; Zhu et al., 2022 ; Zhang et al., 2022b ; Zofka et al., 2023 ; Yuan et al., 2024 ; Zhou et al., 2024 ; Ulhas et al., 2024 ; Sun et al., 2024 ; Rezaldi et al., 2021 ; Stadler et al., 2022b ; Durukal et al., 2022 ; Queiroz et al., 2024 ; Huang et al., 2024 ; Larter et al., 2022 ; Jiang et al., 2021 ; Barker et al., 2018 ; Haider et al., 2024 ; Hasirlioglu and Riener, 2018 ; Chen et al., 2019 ; Oh et al., 2024a ; Che et al., 2021 ; Solmaz et al., 2020 ; Lan et al., 2024 ; Long et al., 2024 ; Fu et al., 2024 ; Hwang et al., 2024 ; Liu et al., 2024 ; Ulreich et al., 2024 ; Wachtel et al., 2024 ; Capdevila et al., 2024 ; Lee and Won, 2024 ; Reckenzaun et al., 2024 ; Yang et al., 2023 ; Drechsler et al., 2024 ; Chen et al., 2025 ; Guo et al., 2025 ; Huang et al., 2025 ; Fu et al., 2025 ; Zeng et al., 2025 ; Steinhauser et al., 2025 ; Ameli et al., 2025 ; Li et al., 2025f ; Zhou et al., 2026b ; Yan et al., 2025 ; Wang et al., 2025d ; Hong, 2025 ; Simakov et al., 2025 ; Verma et al., 2025 ; Zhang et al., 2025c ; Liu et al., 2026 ; Wu and Ma, 2025 ; Li et al., 2025b ; Wang et al., 2025b ; Buzdugan et al., 2025 ; Gaddam et al., 2025 ; Peng and Wang, 2025 ; Wang et al., 2025c ; Jung et al., 2025 ; Gao et al., 2025b ; Saad et al., 2025 ; Elmquist et al., 2025 ; Zhang et al., 2025b ; Zhao et al., 2025 ; Drechsler et al., 2025 ; Abbasi et al., 2025 ; Jing et al., 2025 ; Kolb et al., 2025 ; Li et al., 2025d ; Kim et al., 2025 ; Eberhardt et al., 2025 ; Harhammer et al., 2025 ; Samak et al., 2026 ; Sun et al., 2025a ; Stangherlin et al., 2025 ; Anand et al., 2026 ; Li et al., 2025e ; Yu and Wang, 2026 ; Ulreich et al., 2025 ; Otto et al., 2025 ; Schulz et al., 2025 ; Lee et al., 2025 ; Haider et al., 2025 ; Schabus et al., 2025 ; Petrovic and Muhammad, 2025 ; Waheed et al., 2025 ; Oh et al., 2024b ; Manivasagam et al., 2020 ; Shen et al., 2023 ; Gadipudi et al., 2022 ; Vargas Rivero et al., 2022 ; Duminil et al., 2024a ; Magosi and Eichberger, 2023 ; Wang et al., 2025a ; Hu and Zhao, 2025 ; Khaire et al., 2025 ; Wang et al., 2022b ; Shoukat et al., 2024 ; Martín Serrano et al., 2025 ; Balım et al., 2025 ; Cardozo and Hlavác, 2025 ; Liang et al., 2025 ; Toth and Szalay, 2023 ; Drechsler et al., 2022c ; Magosi et al., 2022 ; Wei et al., 2025a ; Tian et al., 2026 ; Steimle et al., 2025 ; Hofrichter et al., 2025 ; Gao et al., 2025a ; Li et al., 2024 ; Coppola et al., 2025 ; Elgharbawy et al., 2019 ; Segonne and Duthon, 2023 ; Daza et al., 2023 ; Jekl et al., 2025 ; Park et al., 2020 ; Song et al., 2026 ; Wang et al., 2022a ; Yang et al., 2021b ; Leung et al., 2025 ; Reisgys et al., 2021 ; Guo et al., 2021 ; Sigl et al., 2021 ; Deter et al., 2021 ; Wang et al., 2023 ; Heikkinen et al., 2019 ) |
| Code (26) | ( Amini et al., 2020 ; Argui et al., 2023 ; Bergamini et al., 2021 ; Gambi et al., 2019a ; Stocco et al., 2023b ; Zhang and Cai, 2023 ; Wu et al., 2024 ; Samak et al., 2024a ; Ganesan et al., 2025 ; Li et al., 2025a ; Pasios and Nikolaidis, 2025 ; Nosheen et al., 2025 ; Naing et al., 2025 ; Zhang et al., 2025a ; Zhou et al., 2026a ; Wei et al., 2025b ; Da Rocha et al., 2025 ; Zuo et al., 2025 ; Wu et al., 2025 ; Haghighi et al., 2026 ; Hess et al., 2025 ; Wang et al., 2026 ; Huang et al., 2026 ; Stocco et al., 2023a ; Lambertenghi and Stocco, 2024 ; Dauner et al., 2024 ) | |
| Other artefact (14) | ( Samak et al., 2024b ; Xu et al., 2023 ; Ngo et al., 2021a ; Amini and Nejati, 2024 ; Fei et al., 2024 ; Suo et al., 2023 ; Neis et al., 2025 ; Gao et al., 2026a ; Allamaa et al., 2025 ; Lambertenghi et al., 2025a ; Kuroda et al., 2025 ; Bi et al., 2025 ; Amini et al., 2022 ; Arango et al., 2026 ) | |
| Dataset (5) | ( Haq et al., 2021 ; Tenbrock et al., 2021 ; Huch et al., 2023 ; Cao et al., 2024b ; Argui et al., 2025 ) | |
| Code and dataset (2) | ( Baresi et al., 2025 ; Li et al., 2025c ) |
Table 13. RQ 4.5 testing artefacts and citations.
| Secondary study scope | reality gap focus | Gap targets | Mitigation | Evaluation | Execution | Component evidence |
|---|---|---|---|---|---|---|
| ADS testing surveys ( Tang et al., 2023 ; Zhao et al., 2024 ) | – | Partial | Partial | Partial | – | – |
| Scenario-based testing surveys ( Zhong et al., 2021 ) | – | Partial | Partial | – | – | – |
| Testing ML-based systems mappings ( Riccio et al., 2020 ; Zhang et al., 2022a ) | – | Partial | – | Partial | – | – |
| Simulation platform surveys ( Kaur et al., 2021 ; Rosique et al., 2019 ) | – | – | Partial | – | – | – |
| Sim-to-real and digital-twin surveys ( Hu et al., 2023 ) | Partial | Partial | Yes | Partial | – | – |
| This mapping | Yes | Yes | Yes | Yes | Yes | Yes |
Table 14. Scope of related secondary studies and this mapping. “Yes” indicates systematic classification, “Partial” discussion without classification, and “–” a dimension outside the scope or not systematically addressed.
More in this topic
Autonomous Driving Simulation
Autonomous driving simulation models roads, traffic, sensors, and vehicle behavior to train and evaluate driving systems safely and at scale.
See all 228 papers