From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving

A. Humnabadkar; A. Sikdar; B. Cave; H. Zhang; N. Bessis; A. Behera

cs.AIMar 18, 2026

From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving

Authors: A. Humnabadkar, A. Sikdar, B. Cave, H. Zhang, N. Bessis, A. Behera

Organizations: Department of Computer Science, Edge Hill University, United Kingdom.

Abstract

Autonomous driving technologies have achieved significant advances in recent years, yet their real-world deployment remains constrained by data scarcity, safety requirements, and the need for generalization across diverse environments. In response, synthetic data and virtual environments have emerged as powerful enablers, offering scalable, controllable, and richly annotated scenarios for training and evaluation. This survey presents a comprehensive review of recent developments at the intersection of autonomous driving, simulation technologies, and synthetic datasets. We organize the landscape across three core dimensions: (i) the use of synthetic data for perception and planning, (ii) digital twin-based simulation for system validation, and (iii) domain adaptation strategies bridging synthetic and real-world data. We also highlight the role of vision-language models and simulation realism in enhancing scene understanding and generalization. A detailed taxonomy of datasets, tools, and simulation platforms is provided, alongside an analysis of trends in benchmark design. Finally, we discuss critical challenges and open research directions, including Sim2Real transfer, scalable safety validation, cooperative autonomy, and simulation-driven policy learning, that must be addressed to accelerate the path toward safe, generalizable, and globally deployable autonomous driving systems.

Figures & tables

Fig. 1: Illustration of a traditional AV perception-control pipeline. Sensor inputs (cameras, radar, LiDAR, GPS) capture environmental data, which is filtered and processed in a preprocessing stage. The refined data feeds into deep learning models for object detection and prediction. The outputs then guide high-level decision-making modules for path planning and behavior prediction, which are executed by the control system. A feedback loop continuously updates the system for real-time adjustments.

Paper Title	Synthetic Data	Simulators	VLMs	Sim2Real/ Real2Sim Transfer	Digital Twins	AV Trials
	Synthetic Data	Simulators	VLMs	Sim2Real/ Real2Sim Transfer	Digital Twins	AV Trials
How Simulation Helps Autonomous Driving: A Survey of Sim2real, Digital Twins, and Parallel Intelligence [ 61 ]	✓	✓		✓	✓
Synthetic Datasets for Autonomous Driving: A Survey [ 62 ]	✓			✓
Vision Language Models in Autonomous Driving: A Survey and Outlook [ 63 ]			✓
A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook [ 64 ]	✓
A Comprehensive Review on Traffic Datasets and Simulators for Autonomous Vehicles [ 65 ]	✓	✓

TABLE I: Comparison of Review Papers on Autonomous Driving Topics

Fig. 2: A unified framework for driving scene understanding where several critical components, namely, sensor fusion, object recognition, semantic segmentation, motion prediction, mapping, and localization – merge synergistically to enable robust and safe driving. Each component addresses specific perception, prediction, or decision-making challenges. Together, they allow the vehicle to navigate dynamic environments with precision and reliability.

Fig. 3: Showcasing a DriveGPT [ 75 ] workflow in which natural language inputs from users are interpreted by LLM agents to allocate 3D models, gather relevant contextual elements like the surroundings and movement dynamics, and execute rendering functions. It also includes view modification and background creation, ultimately producing a photorealistic driving scene with dynamic simulation. This workflow demonstrates the capability of integrating advanced language models with simulation tools to enable precise and flexible autonomous vehicle testing and scenario visualization, bridging the gap between human intent and machine-generated environments.

Fig. 4: Illustration of the integration of static, dynamic, and external factors into a unified framework for driving scene understanding. Together, they inform perception tasks, guide decision-making processes, and enable domain adaptation strategies to handle the complexities of real-world environments. By simultaneously considering these three aspects, AVs can gain a comprehensive insight into their environment, leading to safer and more efficient navigation.

Paper, Year	Env.	Input Modality	Output Modality	Learning Type	Technique	Key point	Versatility	Viability	Data
Talk2Car (2019) [ 76 ]	Real-world	RGB camera, language instructions	Navigation adjustments	Supervised learning	Visual-language grounding	Language-based interaction safety	High	Moderate	Talk2Car dataset
DiLu (2024) [ 77 ]	Simulated, real-world	multimodal inputs (text, visual)	Decision-making commands	Knowledge-driven reasoning	Explainability via reasoning modules	Handles human-like reasoning in safety-critical tasks	Very High	High	Knowledge-based datasets
LanguageMPC (2023) [ 78 ]	Various simulations	Language instructions, visual data	Control actions, navigation	Language-to-control mapping	Attention maps from language to control	Multi-vehicle coordination	High	Very High	Custom multi-agent datasets
LimSim++ (2024) [ 84 ]	Urban simulations	RGB, language-based commands	Adaptive control actions	Closed-loop multimodal learning	Insights through continuous adaptation	Robustness across diverse scenarios	Moderate	Moderate	Multi-scenario simulated datasets
NaVid (2024) [ 79 ]	Real-world, simulation	Continuous RGB streams	Waypoints, navigational cues	Vision-language navigation	Visual-textual mappings for navigation	Navigates unseen environments	High	Moderate	Continuous RGB-based dataset
CarLLaVA (2024) [ 80 ]	CARLA, real-world	Camera (RGB), language input	Steering, throttle	Vision-language integration	Auxiliary depth outputs, semantic maps	High performance in safety-critical tasks	High	High	CARLA driving data

TABLE II: Evaluation of the different VLMs designed for scene understanding. State-of-the-art methods have been compared with respect to the datasets they use, the input modalities, learning types, target tasks, computational complexity, robustness, and generalization capabilities. All the models provide unique insights on using language as a medium to deepen the context of a scene. The only caveat is that there is no standardized application nor an appropriate metric that has been developed to accurately estimate the efficiency and usefulness of these models.

Paper, Year	Env.	Input Modality	Output Modality	Learning Type	Technique	Key point	Versatility	Viability	Data
DRAMA (2023) [ 88 ]	Real-world	Camera, LiDAR, language	Risk localization	Joint risk explanation	Text-based explanations for risks	Identifies safety-critical objects	High	Moderate	Risk annotation datasets
VLP (2024) [ 89 ]	Simulated environments	Visual data, text	Planning tasks	Vision-language planning	Text-embedded reasoning	Supports challenging scenario planning	High	Moderate	Planning datasets
VLFM (2024) [ 90 ]	Custom scenarios	Vision-language mappings, spatial information	Zero-shot navigation	Zero-shot navigation via language integration	Spatial-textual integration reasoning	Effectiveness in unfamiliar environments	High	Moderate	Custom scenarios data
Natural Language Can Facilitate Sim2Real Transfer (2024) [ 91 ]	Various environments	Language, RGB camera	Navigation adjustments	Language-guided domain adaptation	Explainability through language-based cues	Bridging Sim2Real gap	Very High	High	Domain adaptation datasets
DriveVLM-Dual (2023) [ 92 ]	Real-world, simulated	RGB, multimodal data	Trajectory planning	multimodal integration	Explainability through trajectory reasoning	Adaptability in dynamic driving scenarios	Very High	High	DriveVLM datasets
Vision-Language Frontier Maps (2024) [ 90 ]	Custom, real-world	Vision-language mappings, semantic context	Zero-shot object localization	Semantic navigation via language	Spatial reasoning using language prompts	Robustness in real-world navigation	Very High	High	Boston Dynamics Spot dataset

TABLE III: (Continued) Additional VLMs for Scene Understanding

Fig. 5: The Venn diagram shows the distribution of recent driving datasets across annotation types . About 50–60% of datasets provide only 2D bounding boxes; 20–30% include 3D bounding boxes along with other annotations; nearly 20% include segmentation masks (again typically alongside other modalities). Only a small fraction (5–10%) of datasets contain all three annotation types. Moreover, accounting for static, dynamic, and external factors makes data collection and labeling even more challenging. This underscores the need to supplement real datasets with simulation-based pipelines to cover rare or complex scenarios.

Simulator	Physics / Graphics	Sensor Fidelity / Realism	Scenario & Traffic	Weather & Env. Variation	ROS Integration	Multi-Agent & Parallelism	Open Source	Certification / Compliance	Key Technical Highlights
CARLA [ 5 ]	Unreal Engine, custom physics pipeline	Camera (RGB, depth), LiDAR, Radar, GPS, IMU	Built-in Traffic Manager; Scenario Runner	Dynamic weather, time-of-day changes, varying road conditions (wet/dry, fog)	Native ROS bridge (ROS1, partial ROS2)	Multiple vehicles/agents in synchronous or real-time mode; large-scale HPC possible with manual setup	Yes	N/A (Research)	High-fidelity urban maps, realistic traffic logic, flexible Python APIs, large open-source community
SVL Simulator ( formerly LGSVL ) [ 105 ]	Unity with modular physics plug-ins	Camera, LiDAR, Radar, GPS, IMU	Scenario Editor	Dynamic weather and lighting changes	Native support (ROS1 and ROS2)	Scenario Editor	Yes	N/A (Research)	Integrated Autoware and Apollo AD stacks; user-friendly scenario creation and sensor configuration; active community
NVIDIA DRIVE Sim [ 106 ]	Omniverse and PhysX	Photorealistic camera feeds (with sensor artifacts), LiDAR, Radar	Scenario authoring via Omniverse	Physically based sky system, dynamic lighting, advanced reflections and refractions for weather variability	Closed-source but adapters exist for ROS/ROS2 integration	Highly parallelizable on GPU clusters; enterprise HPC pipelines for large-scale synthetic data generation	Partial	Enterprise (not certified)	Real-time photorealism powered by GPU ray tracing, robust sensor simulation, strong integration with NVIDIA’s AD software stack
rFpro [ 107 ]	Proprietary motorsport-grade engine	High-fidelity camera feeds, LiDAR, Radar	AI traffic vehicles and real-time scenario definition, SUMO integration	Physically accurate weather and surface conditions (wet track, ice, mud), day/night transitions	Limited (via proprietary bridges)	Optimized for real-time HPC and Hardware-in-the-Loop (HIL); multi-vehicle concurrency common in OEM testing	No	OEM validation (not ISO 26262)	Ultra-accurate vehicle dynamics (e.g. tire modeling), widely used by automotive OEMs; ideal for track-level precision and HIL
dSPACE [ 108 ]	ISO 26262-certified simulation platform	High-fidelity sensor models	Scenario-based testing	Configurable weather models	Native support	Hardware-in-the-Loop (HiL/SiL)	No	ISO 26262 certified	Industry standard for functional safety validation, OEM-grade HiL workflows
aiSim [ 109 ]	Certified simulation framework	Camera, LiDAR, Radar	Traffic scenario generation	Weather variation support	Integration available	Scalable multi-agent	No	ISO 26262 certified	Used for ADAS validation, regulatory compliance testing

TABLE IV: Technical Comparison of Modern Autonomous-Driving Simulators

Fig. 6: A cyclical pipeline of Real2Sim and Sim2Real bridges the domain gap between synthetic training environments and real‐world conditions. In the phase of Real2Sim, the physical parameters and expert knowledge have been encoded into dynamic models that are enriched by domain randomization. Similarly, in the Sim2Real, the adaptation of the models back to real‐world sensor inputs is done using curriculum learning, transfer learning, and knowledge distillation. The iteration of this loop progressively reduces the gap, enhancing robustness and operational safety in autonomous driving.

Study	Digital Twin		Scenario Complexity	Performance		Applications
	Type	Sync Capability		Real-Time Processing	Computational Req.
Duan et al. [ 119 ]	PanoSim-Based Twin	Integration of real vehicles and virtual environments	High: End-to-end testing for AV systems	High	Moderate: Requires edge computing	Autonomous vehicle testing and evaluation
Samak et al. [ 124 ]	Open Ecosystem Twin	Synchronization of simulated and real-world data	Medium: Autonomous parking and driving tasks	Moderate	Moderate: Lightweight models	Development and deployment of autonomous driving algorithms
Zhang et al. [ 125 ]	Test Platform Digital Twin	Combined use of simulation and real-world testing tools	High: Scenario-specific testing environments	High	Moderate: Optimized for validation setups	Testing and validation of intelligent vehicles
Liang et al. [ 126 ]	Platform-Agnostic Digital Twin	Deep RL framework supporting various platforms	High: Lane keeping, multi-lane decision-making	High	High: GPU-accelerated training	Reinforcement learning-based decision-making
Klar et al. [ 127 ]	Monitoring and Control Twin	Data-driven vehicle condition monitoring	Medium: Predictive analysis for AV control	Medium	Moderate: Minimal hardware requirements	Vehicle monitoring, fault prediction, and autonomous driving

TABLE V: Comprehensive Technical Comparison of Digital Twin Implementations in Autonomous Driving

City/Case Study	AV Technology Used	Observed Impacts	Challenges Addressed
San Francisco [ 130 ]	Shared AVs	Reduced vehicle ownership, altered parking needs, improved traffic flow	Congestion, urban sprawl, pollution
Singapore [ 131 ]	Integrated AV and Public Transportation	Enhanced first-mile service, improved public transport efficiency	Congestion, high public transport demand
Toronto [ 132 , 133 ]	AV Ridehailing Operations	Legal adaptation to new transport modes, better city planning	Regulatory frameworks, urban design
Phoenix, Arizona [ 134 ]	Waymo One Robo-Taxi Service	Increased accessibility, large-scale real-world testing, user acceptance	Safety concerns, insurance policies, liability issues
Pittsburgh, Pennsylvania [ 135 ]	Autonomous Shuttles and Ridesharing Pilots	Expanded coverage in underserved areas, boosted local robotics ecosystem	Public acceptance, legal complexities, inclement weather adaptations
Hamburg, Germany [ 136 ]	Autonomous Shuttle Trials (e.g., HEAT)	Improved integration with public transport, data-driven traffic management	Infrastructure readiness, cybersecurity, operational costs

TABLE VI: Impact of AV Technologies on Urban Mobility and Infrastructure

Standard/Metric	Organization	Domain	Purpose & Scope	Key Applications
Safety and Functional Standards
ISO 26262:2018 [ 170 ]	ISO TC22/SC32	Functional Safety	Road vehicle functional safety for E/E systems. Addresses systematic and random hardware failures through ASIL (A-D) risk classification	ADAS development, ECU validation, safety-critical system design
ISO 21448:2022 (SOTIF) [ 171 ]	ISO TC22/SC32	Safety of Intended Functionality	Addresses hazards from functional insufficiencies and foreseeable misuse without system failures. Complements ISO 26262	AI/ML validation, autonomous driving systems, edge case scenarios
ISO/SAE 21434:2021 [ 172 ]	ISO/SAE	Cybersecurity	Automotive cybersecurity engineering lifecycle. Risk assessment (TARA), security goals, and continuous monitoring	Connected vehicles, OTA updates, cyber threat mitigation
ANSI/UL 4600:2023 [ 173 ]	UL	Autonomous Product Safety	Safety evaluation for autonomous products beyond traditional functional safety	Autonomous vehicles, robotics, safety certification
Automation Level Classification

TABLE VII: Current Standards and Metrics for Autonomous Vehicle Development and Simulation

Explore similar work

Nov 27, 2025

HybridWorldSim: A Scalable and Controllable High-fidelity Simulator for Autonomous Driving

Realistic and controllable simulation is critical for advancing end-to-end autonomous driving, yet existing approaches often struggle to support novel view synthesis under large viewpoint changes or to ensure geometric consistency. We introduce HybridWorldSim, a hybrid simulation framework that integrates multi-traversal neural reconstruction for static backgrounds with generative modeling for dynamic agents. This unified design addresses key limitations of previous methods, enabling the creation of diverse and high-fidelity driving scenarios with reliable visual and spatial consistency. To facilitate robust benchmarking, we further release a new multi-traversal dataset MIRROR that captures a wide range of routes and environmental conditions across different cities. Extensive experiments demonstrate that HybridWorldSim surpasses previous state-of-the-art methods, providing a practical and scalable solution for high-fidelity simulation and a valuable resource for research and development in autonomous driving.

Qiang Li, Yingwenqi Jiang, Tuoxi Li +17

1XPeng Motors · 2ShanghaiTech University · University of Science and Technology of China

3D Reconstruction Novel View Synthesis Autonomous Driving Simulation Autonomous Driving Scenario Generation

Ask PDF

May 20, 2026

Closed Loop Dynamic Driving Data Mixture for Real-Synthetic Co-Training

Data scaling is fundamental to modern deep learning, and grows increasingly critical as autonomous driving shifts to end-to-end learning. Real-world driving data is expensive to annotate and scene-biased, making real-synthetic co-training with near-infinite synthetic data a promising direction. However, naively incorporating all available synthetic data is inefficient and leads to distribution shifts, and optimizing data mixture under practical training budgets remains a critical yet under-explored problem. In this sense, we claim that the mixture of training data requires clear guidance in terms of scene types and quantities. Particularly in this work, we conceptualize the data mixture approximately as a dynamic optimization process that iteratively adjusts the training data mixture to maximize model performance, guided by closed-loop evaluation feedback, and propose AutoScale, a fully automated closed-loop data engine unifying scene representation, data mixture optimization and retrieval, as well as model training and evaluation. Specifically, we propose Graph Regularized AutoEncoder (Graph-RAE) for driving scene representations, introduce Cluster-aware Gradient Ascent (Cluster-GA) for cluster-wise importance estimation and reweighting, and perform cluster-guided vector retrieval to select high-value samples. Experiments on NavSim demonstrate that AutoScale outperforms vanilla co-training and cross-domain baselines, achieving better performance with fewer synthetic samples under constrained budgets.

Hongzhi Ruan, Pei Liu, Weiliang Ma +5

Li Auto · HKUST (GZ) · HKUST

Autonomous Driving Datasets Synthetic Data Augmentation End-to-End Autonomous Driving Adaptive Sampling Data Mixture Optimization

Ask PDF

May 13, 2025

Generative AI for Autonomous Driving: Frontiers and Opportunities

Generative Artificial Intelligence (GenAI) constitutes a transformative technological wave that reconfigures industries through its unparalleled capabilities for content creation, reasoning, planning, and multimodal understanding. This revolutionary force offers the most promising path yet toward solving one of engineering's grandest challenges: achieving reliable, fully autonomous driving, particularly the pursuit of Level 5 autonomy. This survey delivers a comprehensive and critical synthesis of the emerging role of GenAI across the autonomous driving stack. We begin by distilling the principles and trade-offs of modern generative modeling, encompassing VAEs, GANs, Diffusion Models, and Large Language Models (LLMs). We then map their frontier applications in image, LiDAR, trajectory, occupancy, video generation as well as LLM-guided reasoning and decision making. We categorize practical applications, such as synthetic data workflows, end-to-end driving strategies, high-fidelity digital twin systems, smart transportation networks, and cross-domain transfer to embodied AI. We identify key obstacles and possibilities such as comprehensive generalization across rare cases, evaluation and safety checks, budget-limited implementation, regulatory compliance, ethical concerns, and environmental effects, while proposing research plans across theoretical assurances, trust metrics, transport integration, and socio-technical influence. By unifying these threads, the survey provides a forward-looking reference for researchers, engineers, and policymakers navigating the convergence of generative AI and advanced autonomous mobility. An actively maintained repository of cited works is available at https://github.com/taco-group/GenAI4AD.

Yuping Wang, Shuo Xing, Cui Can +46

Autonomous Driving Autonomous Driving Scenario Generation

Ask PDF