cs.CVNov 15, 2026

Beyond Vehicle-Centric Data: Scaling Autonomous Driving through Roadside-to-Vehicle Generation

Authors: Yichen Zhang, Zihan Wang, Hao Chen, Runyi Huang, Ruixue Zong, Jing Wang, Yunong Mao, Lei He, 

Organizations: The School of Vehicle and Mobility, Tsinghua University · The State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University · College of AI, Tsinghua University · Shanghai Jiao Tong University

Abstract

Training autonomous driving models requires large and diverse datasets, collected primarily by vehicles. Scaling them entails investment in fleet operation, data processing and annotation. Road infrastructure records traffic and multi-vehicle interactions, yet the viewpoint gap between roadside and onboard sensors limits its direct use for training. Here we introduce AnyR2V, a framework that transforms multimodal roadside observations into seven-camera surround-view videos for different vehicle perspectives. It aggregates sequential observations into a shared scene representation of geometry and road-user motion, then renders appearance, depth and layout conditions along target-vehicle trajectories. Supervised by spatiotemporally aligned roadside-vehicle pairs, AnyR2V generates surround-view videos guided by scene structure and road-user motion. Existing roadside 3D bounding boxes are transformed into target-vehicle coordinates and paired with generated images, allowing one traffic recording and its annotations to yield training data from multiple vehicle perspectives. We construct R2V360, a dataset of 89 real traffic sequences at an urban intersection, with aligned multimodal roadside data and onboard surround-view videos. AnyR2V outperforms six scene reconstruction methods in image fidelity and in depth and semantic agreement between generated and real onboard images. We further evaluate generated videos along other vehicles' trajectories by adding them to real training data. Adding generated frames equal to 50% of the original real training frame count improves mean average precision (mAP) by 3.3-3.4 AP-points across three 3D object detectors on the real onboard test set, relative to real-data-only baselines. These results demonstrate that roadside traffic records can provide training data from multiple vehicle perspectives, offering a new data source for autonomous driving.

Figures & tables

Explore similar work

CardsList
  1. RS2AD-LiDAR: End-to-End Autonomous Driving LiDAR Data Generation from Roadside Sensor Observations

    May 22, 2026Runyi Huang, Ni Ding, Ruidan Xing +3Synthetic Data AugmentationAutonomous Driving Perception

  2. From Driving Videos to Simulatable Scenarios

    Jun 20, 2026Alexandre Levy, Ernest Valveny Llobet, Antonio Manuel LópezAutonomous Driving SimulationAutonomous Driving Scenario Generation

  3. AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond

    May 25, 2026Haiming Zhang, Junfei Zhou, Feng Jiang +6Long-Horizon Video Generation3D Scene Generation