cs.AIMar 18, 2026
SaveFrom Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving
Organizations: Department of Computer Science, Edge Hill University, United Kingdom.
Abstract
Autonomous driving technologies have achieved significant advances in recent years, yet their real-world deployment remains constrained by data scarcity, safety requirements, and the need for generalization across diverse environments. In response, synthetic data and virtual environments have emerged as powerful enablers, offering scalable, controllable, and richly annotated scenarios for training and evaluation. This survey presents a comprehensive review of recent developments at the intersection of autonomous driving, simulation technologies, and synthetic datasets. We organize the landscape across three core dimensions: (i) the use of synthetic data for perception and planning, (ii) digital twin-based simulation for system validation, and (iii) domain adaptation strategies bridging synthetic and real-world data. We also highlight the role of vision-language models and simulation realism in enhancing scene understanding and generalization. A detailed taxonomy of datasets, tools, and simulation platforms is provided, alongside an analysis of trends in benchmark design. Finally, we discuss critical challenges and open research directions, including Sim2Real transfer, scalable safety validation, cooperative autonomy, and simulation-driven policy learning, that must be addressed to accelerate the path toward safe, generalizable, and globally deployable autonomous driving systems.
Figures & tables
Fig. 1: Illustration of a traditional AV perception-control pipeline. Sensor inputs (cameras, radar, LiDAR, GPS) capture environmental data, which is filtered and processed in a preprocessing stage. The refined data feeds into deep learning models for object detection and prediction. The outputs then guide high-level decision-making modules for path planning and behavior prediction, which are executed by the control system. A feedback loop continuously updates the system for real-time adjustments.
| Paper Title | Synthetic Data | Simulators | VLMs | Sim2Real/ Real2Sim Transfer | Digital Twins | AV Trials |
|---|---|---|---|---|---|---|
| How Simulation Helps Autonomous Driving: A Survey of Sim2real, Digital Twins, and Parallel Intelligence [ 61 ] | ✓ | ✓ | ✓ | ✓ | ||
| Synthetic Datasets for Autonomous Driving: A Survey [ 62 ] | ✓ | ✓ | ||||
| Vision Language Models in Autonomous Driving: A Survey and Outlook [ 63 ] | ✓ | |||||
| A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook [ 64 ] | ✓ | |||||
| A Comprehensive Review on Traffic Datasets and Simulators for Autonomous Vehicles [ 65 ] | ✓ | ✓ |
TABLE I: Comparison of Review Papers on Autonomous Driving Topics
Fig. 2: A unified framework for driving scene understanding where several critical components, namely, sensor fusion, object recognition, semantic segmentation, motion prediction, mapping, and localization – merge synergistically to enable robust and safe driving. Each component addresses specific perception, prediction, or decision-making challenges. Together, they allow the vehicle to navigate dynamic environments with precision and reliability.
Fig. 3: Showcasing a DriveGPT [ 75 ] workflow in which natural language inputs from users are interpreted by LLM agents to allocate 3D models, gather relevant contextual elements like the surroundings and movement dynamics, and execute rendering functions. It also includes view modification and background creation, ultimately producing a photorealistic driving scene with dynamic simulation. This workflow demonstrates the capability of integrating advanced language models with simulation tools to enable precise and flexible autonomous vehicle testing and scenario visualization, bridging the gap between human intent and machine-generated environments.
Fig. 4: Illustration of the integration of static, dynamic, and external factors into a unified framework for driving scene understanding. Together, they inform perception tasks, guide decision-making processes, and enable domain adaptation strategies to handle the complexities of real-world environments. By simultaneously considering these three aspects, AVs can gain a comprehensive insight into their environment, leading to safer and more efficient navigation.
| Paper, Year | Env. | Input Modality | Output Modality | Learning Type | Technique | Key point | Versatility | Viability | Data |
| Talk2Car (2019) [ 76 ] | Real-world | RGB camera, language instructions | Navigation adjustments | Supervised learning | Visual-language grounding | Language-based interaction safety | High | Moderate | Talk2Car dataset |
| DiLu (2024) [ 77 ] | Simulated, real-world | multimodal inputs (text, visual) | Decision-making commands | Knowledge-driven reasoning | Explainability via reasoning modules | Handles human-like reasoning in safety-critical tasks | Very High | High | Knowledge-based datasets |
| LanguageMPC (2023) [ 78 ] | Various simulations | Language instructions, visual data | Control actions, navigation | Language-to-control mapping | Attention maps from language to control | Multi-vehicle coordination | High | Very High | Custom multi-agent datasets |
| LimSim++ (2024) [ 84 ] | Urban simulations | RGB, language-based commands | Adaptive control actions | Closed-loop multimodal learning | Insights through continuous adaptation | Robustness across diverse scenarios | Moderate | Moderate | Multi-scenario simulated datasets |
| NaVid (2024) [ 79 ] | Real-world, simulation | Continuous RGB streams | Waypoints, navigational cues | Vision-language navigation | Visual-textual mappings for navigation | Navigates unseen environments | High | Moderate | Continuous RGB-based dataset |
| CarLLaVA (2024) [ 80 ] | CARLA, real-world | Camera (RGB), language input | Steering, throttle | Vision-language integration | Auxiliary depth outputs, semantic maps | High performance in safety-critical tasks | High | High | CARLA driving data |
TABLE II: Evaluation of the different VLMs designed for scene understanding. State-of-the-art methods have been compared with respect to the datasets they use, the input modalities, learning types, target tasks, computational complexity, robustness, and generalization capabilities. All the models provide unique insights on using language as a medium to deepen the context of a scene. The only caveat is that there is no standardized application nor an appropriate metric that has been developed to accurately estimate the efficiency and usefulness of these models.
| Paper, Year | Env. | Input Modality | Output Modality | Learning Type | Technique | Key point | Versatility | Viability | Data |
| DRAMA (2023) [ 88 ] | Real-world | Camera, LiDAR, language | Risk localization | Joint risk explanation | Text-based explanations for risks | Identifies safety-critical objects | High | Moderate | Risk annotation datasets |
| VLP (2024) [ 89 ] | Simulated environments | Visual data, text | Planning tasks | Vision-language planning | Text-embedded reasoning | Supports challenging scenario planning | High | Moderate | Planning datasets |
| VLFM (2024) [ 90 ] | Custom scenarios | Vision-language mappings, spatial information | Zero-shot navigation | Zero-shot navigation via language integration | Spatial-textual integration reasoning | Effectiveness in unfamiliar environments | High | Moderate | Custom scenarios data |
| Natural Language Can Facilitate Sim2Real Transfer (2024) [ 91 ] | Various environments | Language, RGB camera | Navigation adjustments | Language-guided domain adaptation | Explainability through language-based cues | Bridging Sim2Real gap | Very High | High | Domain adaptation datasets |
| DriveVLM-Dual (2023) [ 92 ] | Real-world, simulated | RGB, multimodal data | Trajectory planning | multimodal integration | Explainability through trajectory reasoning | Adaptability in dynamic driving scenarios | Very High | High | DriveVLM datasets |
| Vision-Language Frontier Maps (2024) [ 90 ] | Custom, real-world | Vision-language mappings, semantic context | Zero-shot object localization | Semantic navigation via language | Spatial reasoning using language prompts | Robustness in real-world navigation | Very High | High | Boston Dynamics Spot dataset |
TABLE III: (Continued) Additional VLMs for Scene Understanding
Fig. 5: The Venn diagram shows the distribution of recent driving datasets across annotation types . About 50–60% of datasets provide only 2D bounding boxes; 20–30% include 3D bounding boxes along with other annotations; nearly 20% include segmentation masks (again typically alongside other modalities). Only a small fraction (5–10%) of datasets contain all three annotation types. Moreover, accounting for static, dynamic, and external factors makes data collection and labeling even more challenging. This underscores the need to supplement real datasets with simulation-based pipelines to cover rare or complex scenarios.
| Simulator | Physics / Graphics | Sensor Fidelity / Realism | Scenario & Traffic | Weather & Env. Variation | ROS Integration | Multi-Agent & Parallelism | Open Source | Certification / Compliance | Key Technical Highlights |
| CARLA [ 5 ] | Unreal Engine, custom physics pipeline | Camera (RGB, depth), LiDAR, Radar, GPS, IMU | Built-in Traffic Manager; Scenario Runner | Dynamic weather, time-of-day changes, varying road conditions (wet/dry, fog) | Native ROS bridge (ROS1, partial ROS2) | Multiple vehicles/agents in synchronous or real-time mode; large-scale HPC possible with manual setup | Yes | N/A (Research) | High-fidelity urban maps, realistic traffic logic, flexible Python APIs, large open-source community |
| SVL Simulator ( formerly LGSVL ) [ 105 ] | Unity with modular physics plug-ins | Camera, LiDAR, Radar, GPS, IMU | Scenario Editor | Dynamic weather and lighting changes | Native support (ROS1 and ROS2) | Scenario Editor | Yes | N/A (Research) | Integrated Autoware and Apollo AD stacks; user-friendly scenario creation and sensor configuration; active community |
| NVIDIA DRIVE Sim [ 106 ] | Omniverse and PhysX | Photorealistic camera feeds (with sensor artifacts), LiDAR, Radar | Scenario authoring via Omniverse | Physically based sky system, dynamic lighting, advanced reflections and refractions for weather variability | Closed-source but adapters exist for ROS/ROS2 integration | Highly parallelizable on GPU clusters; enterprise HPC pipelines for large-scale synthetic data generation | Partial | Enterprise (not certified) | Real-time photorealism powered by GPU ray tracing, robust sensor simulation, strong integration with NVIDIA’s AD software stack |
| rFpro [ 107 ] | Proprietary motorsport-grade engine | High-fidelity camera feeds, LiDAR, Radar | AI traffic vehicles and real-time scenario definition, SUMO integration | Physically accurate weather and surface conditions (wet track, ice, mud), day/night transitions | Limited (via proprietary bridges) | Optimized for real-time HPC and Hardware-in-the-Loop (HIL); multi-vehicle concurrency common in OEM testing | No | OEM validation (not ISO 26262) | Ultra-accurate vehicle dynamics (e.g. tire modeling), widely used by automotive OEMs; ideal for track-level precision and HIL |
| dSPACE [ 108 ] | ISO 26262-certified simulation platform | High-fidelity sensor models | Scenario-based testing | Configurable weather models | Native support | Hardware-in-the-Loop (HiL/SiL) | No | ISO 26262 certified | Industry standard for functional safety validation, OEM-grade HiL workflows |
| aiSim [ 109 ] | Certified simulation framework | Camera, LiDAR, Radar | Traffic scenario generation | Weather variation support | Integration available | Scalable multi-agent | No | ISO 26262 certified | Used for ADAS validation, regulatory compliance testing |
TABLE IV: Technical Comparison of Modern Autonomous-Driving Simulators
Fig. 6: A cyclical pipeline of Real2Sim and Sim2Real bridges the domain gap between synthetic training environments and real‐world conditions. In the phase of Real2Sim, the physical parameters and expert knowledge have been encoded into dynamic models that are enriched by domain randomization. Similarly, in the Sim2Real, the adaptation of the models back to real‐world sensor inputs is done using curriculum learning, transfer learning, and knowledge distillation. The iteration of this loop progressively reduces the gap, enhancing robustness and operational safety in autonomous driving.
| Study | Digital Twin | Scenario Complexity | Performance | Applications | ||
|---|---|---|---|---|---|---|
| Type | Sync Capability | Real-Time Processing | Computational Req. | |||
| Duan et al. [ 119 ] | PanoSim-Based Twin | Integration of real vehicles and virtual environments | High: End-to-end testing for AV systems | High | Moderate: Requires edge computing | Autonomous vehicle testing and evaluation |
| Samak et al. [ 124 ] | Open Ecosystem Twin | Synchronization of simulated and real-world data | Medium: Autonomous parking and driving tasks | Moderate | Moderate: Lightweight models | Development and deployment of autonomous driving algorithms |
| Zhang et al. [ 125 ] | Test Platform Digital Twin | Combined use of simulation and real-world testing tools | High: Scenario-specific testing environments | High | Moderate: Optimized for validation setups | Testing and validation of intelligent vehicles |
| Liang et al. [ 126 ] | Platform-Agnostic Digital Twin | Deep RL framework supporting various platforms | High: Lane keeping, multi-lane decision-making | High | High: GPU-accelerated training | Reinforcement learning-based decision-making |
| Klar et al. [ 127 ] | Monitoring and Control Twin | Data-driven vehicle condition monitoring | Medium: Predictive analysis for AV control | Medium | Moderate: Minimal hardware requirements | Vehicle monitoring, fault prediction, and autonomous driving |
TABLE V: Comprehensive Technical Comparison of Digital Twin Implementations in Autonomous Driving
| City/Case Study | AV Technology Used | Observed Impacts | Challenges Addressed |
|---|---|---|---|
| San Francisco [ 130 ] | Shared AVs | Reduced vehicle ownership, altered parking needs, improved traffic flow | Congestion, urban sprawl, pollution |
| Singapore [ 131 ] | Integrated AV and Public Transportation | Enhanced first-mile service, improved public transport efficiency | Congestion, high public transport demand |
| Toronto [ 132 , 133 ] | AV Ridehailing Operations | Legal adaptation to new transport modes, better city planning | Regulatory frameworks, urban design |
| Phoenix, Arizona [ 134 ] | Waymo One Robo-Taxi Service | Increased accessibility, large-scale real-world testing, user acceptance | Safety concerns, insurance policies, liability issues |
| Pittsburgh, Pennsylvania [ 135 ] | Autonomous Shuttles and Ridesharing Pilots | Expanded coverage in underserved areas, boosted local robotics ecosystem | Public acceptance, legal complexities, inclement weather adaptations |
| Hamburg, Germany [ 136 ] | Autonomous Shuttle Trials (e.g., HEAT) | Improved integration with public transport, data-driven traffic management | Infrastructure readiness, cybersecurity, operational costs |
TABLE VI: Impact of AV Technologies on Urban Mobility and Infrastructure
| Standard/Metric | Organization | Domain | Purpose & Scope | Key Applications |
| Safety and Functional Standards | ||||
| ISO 26262:2018 [ 170 ] | ISO TC22/SC32 | Functional Safety | Road vehicle functional safety for E/E systems. Addresses systematic and random hardware failures through ASIL (A-D) risk classification | ADAS development, ECU validation, safety-critical system design |
| ISO 21448:2022 (SOTIF) [ 171 ] | ISO TC22/SC32 | Safety of Intended Functionality | Addresses hazards from functional insufficiencies and foreseeable misuse without system failures. Complements ISO 26262 | AI/ML validation, autonomous driving systems, edge case scenarios |
| ISO/SAE 21434:2021 [ 172 ] | ISO/SAE | Cybersecurity | Automotive cybersecurity engineering lifecycle. Risk assessment (TARA), security goals, and continuous monitoring | Connected vehicles, OTA updates, cyber threat mitigation |
| ANSI/UL 4600:2023 [ 173 ] | UL | Autonomous Product Safety | Safety evaluation for autonomous products beyond traditional functional safety | Autonomous vehicles, robotics, safety certification |
| Automation Level Classification | ||||
TABLE VII: Current Standards and Metrics for Autonomous Vehicle Development and Simulation