cs.ROMar 18, 2026
SaveMultisource human-in-the-loop digital twin testbed for connected and autonomous vehicles in mixed traffic flow
Organizations: School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China. · Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI 48109, USA.
Abstract
In the emerging mixed traffic environments, Connected and Autonomous Vehicles (CAVs) have to interact with surrounding human-driven vehicles (HDVs). This paper introduces MSH-MCCT (Multi-Source Human-in-the-Loop Mixed Cloud Control Testbed), a novel CAV testbed that captures complex interactions between various CAVs and HDVs. Utilizing the Mixed Digital Twin concept, which combines Mixed Reality with Digital Twin, MSH-MCCT integrates physical, virtual, and mixed platforms, along with multi-source control inputs. Bridged by the mixed platform, MSH-MCCT allows human drivers and CAV algorithms to operate both physical and virtual vehicles within multiple fields of view. Particularly, this testbed facilitates the coexistence and real-time interaction of physical and virtual CAVs & HDVs, significantly enhancing the experimental flexibility and scalability. Experiments on vehicle platooning in mixed traffic showcase the potential of MSH-MCCT to conduct CAV testing with multi-source real human drivers in the loop through driving simulators of diverse fidelity. The videos for the experiments are available at our project website: https://dongjh20.github.io/MSH-MCCT.
Figures & tables
Fig. 1: Schematics for classical DT and mixedDT. (a) In classical DT, the virtual space is typically a digital replica of the physical space. (b) In mixedDT, the virtual space and the physical space are integrated into the mixed space, where physical and virtual entities could coexist and interact with each other.
| Metrics | Platforms | |||||||||||||||||
| SUMO | CARLA | NVIDIA Drive Sim | VeHIL | Mcity | MSH-MCCT | |||||||||||||
| L | M | H | L | M | H | L | M | H | L | M | H | L | M | H | L | M | H | |
| Support for Multiple Human Drivers | ||||||||||||||||||
| Physical-Virtual Integration | ||||||||||||||||||
| Scalability | ||||||||||||||||||
| Affordability | ||||||||||||||||||
TABLE I: Comparison of CAV Testing Platforms Highlighting Multi-Human-in-the-Loop Capabilities
Fig. 2: Schematic of MSH-MCCT. In the mixed platform, the physical vehicles in the physical platform (outlined in gold), the virtual vehicles in the virtual platform (outlined in cyan) and the virtual vehicle in the driving environment of a driving simulator (outlined in red) coexist and interact with each other in real-time. In contrast, in existing platforms [ 48 , 49 ] , physical and virtual vehicles operate exclusively within their respective environments, without effective integration or interactive mechanisms established between them. The multi-source control inputs includes human drivers via various-fidelity levels of driving simulators and diverse CAV algorithms. The visualization of the mixed platform shown in (c) is provided by a MR device. The demonstration videos are available at our project website: https://dongjh20.github.io/MSH-MCCT .
Fig. 3: The methodological framework for conducting CAV testing with multi-source human drivers in the loop via driving simulators in MSH-MCCT. Both physical and virtual vehicles could be controlled by human drivers and CAV algorithms through a cloud-based intermediary. Consequently, the four types of vehicles, i.e. , physical HDVs, virtual HDVs, physical CAVs and virtual CAVs, coexist and interact with each other simultaneously within the mixed platform.
Fig. 4: Driving field-of-view and prompt panel. In (a) and (b), the snapshots of human-in-the-loop experiments based on physical and virtual field-of-view are presented respectively; see the screens for the specific view. In (c), the real-time speed prompt of the preceding vehicle in the virtual driving field-of-view is displayed on the white panel on the top of the preceding vehicle.
Fig. 5: Control modes diagram. Human drivers and CAV algorithms could control both the physical vehicles and the virtual vehicles via a cloud-based intermediary.
Fig. 6: Physical and virtual driving environments and vehicles. (a) The physical sand table, i.e. , the physical driving environment, having a size of . (b) The virtual sand table, i.e. , the virtual driving environment, developed based on the Unity game engine. (c) The miniature physical vehicle. A color block card is pasted to the top for localization. (d) Two commonly used vehicle body models and the default dynamics model of the virtual vehicles, i.e. , the typical bicycle model [ 67 ] .
| Component | Setup |
| Physical sand table | , with respect to the real-world driving environment |
| Roadside sensors in the physical platform | Four identical cameras in a manner with overlap |
| Roadside facilities in the physical platform | street lamps, traffic lights and lifting rods |
| Miniature physical vehicles | , powered by Raspberry Pi 4B |
| Host of the virtual platform | Processor: Intel Core i7-10700K CPU @ 3.80 GHz 16, Memory: 16 GB |
| Host of the G29 simulator | Processor: Intel Core i7-10700 CPU@ 2.90 GHz 16, Memory: 16 GB |
TABLE II: Platform Setups
Fig. 7: Driving simulators with various-fidelity levels. (a) The three Logitech G29 Driving Force simulators, each of which comprises one steering wheel, an accelerator pedal, a brake pedal and a host computer. (b) The high-fidelity commercial driving simulator from InnoSimulation, consisting of a driver cabin, three screens, a motion platform and a host computer.
Fig. 8: A demonstration diagram of practical MSH-MCCT operation. Multiple vehicles in diverse platforms and environments, controlled by multiple human drivers and CAV algorithms, interact with each other simultaneously via the mixed platform. The visualization of the mixed platform projected to virtual space by Unity is presented on the middle panel. Precisely, there are three miniature physical vehicles from the physical platform (outlined in gold), four virtual vehicles from the virtual platform (outlined in cyan), and one virtual vehicle from the InnoSimulation environment (outlined in red). For the multi-source control inputs, three human drivers on the G29 simulators and one human driver on the InnoSimulation simulator collectively control the four vehicles, while the other four vehicles are controlled by CAV algorithms. The detailed driving mode setups are illustrated in Table III .
Fig. 9: The vehicles drive along the direction . In Experiment A (traffic wave scenario), when reaching point , the velocity of the head vehicle suffers from a half-sinusoidal perturbation. In Experiment B (safety-critical scenario), when reaching point , a braking perturbation is imposed on the head vehicle.
| ID | Type | Entity | Control input | Field-of-View |
| 1 | CAV | Physical vehicle | CACC controller | / |
| 2 | HDV | Physical vehicle | Human via G29 simulator | Physical view from the sand table |
| 3 | CAV | Virtual vehicle | CACC controller | / |
| 4 | HDV | Physical vehicle | Human via G29 simulator | Virtual view from the Unity environment |
| 5 | HDV | Virtual vehicle | Human via G29 simulator | Virtual view from the Unity environment |
| 6 | CAV | Virtual vehicle | CACC controller | / |
TABLE III: Driving Mode Setups
Fig. 10: Snapshots of the platform effectiveness experiment. In (a), the image captured by the roadside cameras in the physical platform displays running physical vehicles. In (b), the visualization of the mixed platoon by Unity is presented, with the coexistence of four HDVs and four CAVs. The corresponding driving mode setups of the platoon are shown in Table III . In (c) and (d), three male and one female drivers with various levels of driving experiences operate vehicles via different-fidelity driving simulators. Particularly, as can be observed from the screens in (c), the driver in the middle is driving based on physical field-of-view, while the other two are driving based on virtual field-of-view. In (d), the driver is driving based on virtual field-of-view from the InnoSimulation environment, whose overview is shown in the lower left corner of (d). The corresponding videos can be found in https://dongjh20.github.io/MSH-MCCT .
Fig. 11: Velocity profiles of all the vehicles in the platform effectiveness experiments. The physical vehicle is in solid line, while the virtual vehicle is in dashed line. The CAV is in blue line, while the HDV is in black line. In Experiment A, a half-sinusoidal perturbation is imposed to the head vehicle, while in Experiment B, a braking perturbation is imposed. The velocity values have been unified to the virtual platform, which has a consistent size as real-world roads.
Fig. 12: Spacing profiles of all the following vehicles in the platform effectiveness experiments. The inter-vehicle spacing here represents the distance between the centroids of the two vehicles. The physical vehicle is in solid line, while the virtual vehicle is in dashed line. The CAV is in blue line, while the HDV is in black line. Particularly, three collisions indeed occur, marked with red crosses. However, since at least one of the involved participants is a virtual vehicle, no actual damage is incurred by any vehicle. This shows that our platform supports safety-critical scenarios when testing CAV technologies.