VistaGEN: Consistent Driving Video Generation with Fine-Grained Control Using Multiview Visual-Language Reasoning

Li-Heng Chen; Ke Cheng; Yahui Liu; Lei Shi; Shi-Sheng Huang; Hongbo Fu

cs.CVMar 30, 2026

VistaGEN: Consistent Driving Video Generation with Fine-Grained Control Using Multiview Visual-Language Reasoning

Authors: Li-Heng Chen, Ke Cheng, Yahui Liu, Lei Shi, Shi-Sheng Huang, Hongbo Fu

Organizations: Hong Kong University of Science and Technology · Meituan, Inc. · Beijing Normal University

Abstract

Driving video generation has achieved much progress in controllability, video resolution, and length, but fails to support fine-grained object-level controllability for diverse driving videos, while preserving the spatiotemporal consistency, especially in long video generation. In this paper, we present a new driving video generation technique, called VistaGEN, which enables fine-grained control of specific entities, including 3D objects, images, and text descriptions, while maintaining spatiotemporal consistency in long video sequences. Our key innovation is the incorporation of multiview visual-language reasoning into the long driving video generation. To this end, we inject visual-language features into a multiview video generator to enable fine-grained controllability. More importantly, we propose a multiview vision-language evaluator (MV-VLM) to intelligently and automatically evaluate spatiotemporal consistency of the generated content, thus formulating a novel generation-evaluation-regeneration closed-loop generation mechanism. This mechanism ensures high-quality, coherent outputs, facilitating the creation of complex and reliable driving scenarios. Besides, within the closed-loop generation, we introduce an object-level refinement module to refine the unsatisfied results evaluated from the MV-VLM and then feed them back to the video generator for regeneration. Extensive evaluation shows that our VistaGEN achieves diverse driving video generation results with fine-grained controllability, especially for long-tail objects, and much better spatiotemporal consistency than previous approaches.

Figures & tables

Figure 1: This paper introduces a new driving video generation technique, VistaGEN, which enables fine-grained control, like specifying multiple objects’ visual-language conditions (roadmaps, text descriptors and visual appearance in (a) and (c), where the index numbers match the corresponding roadmaps in the right), with high-quality and coherent video generation outputs ((b) and (d)). Moreover, our VistaGEN can achieve spatiotemporal consistent generation (e) with both multiview consistency (colored red and orange in the left) and long-range temporal consistency (colored pink and cyan across different frames respectively) in the long video sequences. Please refer to our demo video for more details.

Figure 2: The pipeline of VistaGEN. Given frame descriptors as input, we hierarchically inject these signals as global and local scene control guidance via visual-language feature fusion for fine-grained controllability. Subsequently, we perform driving video generation via a multiview video generator $\mathcal{D}$ . The generation is automatically evaluated by a multiview vision-language evaluator $\mathcal{E}$ , followed by an object-level refinement module $\mathcal{R}$ . This formulates a closed-loop mechanism to maintain spatiotemporal consistency for long video sequence generation.

Figure 3: The illustration of MV-VLM structure.

Figure 4: The workflow of our object-level refinement module.

Figure 5: Qualitative results of fine-grained video generation by VistaGEN in complex scenes. Our model seamlessly integrates multimodal conditions, including BEV layouts, textual descriptions, and reference images. The colored bounding boxes track specific controlled instances across the generated multi-view videos, demonstrating precise object-level controllability.

Method	Perceptual Quality		Layout Fidelity
Method	FVD ( $\downarrow$ )	FID ( $\downarrow$ )	mAP ( $\uparrow$ )	IoU ( $\uparrow$ )
BEVControl [ 22 ]	112.3	12.01	N/A	52.37
MagicDrive-V2 [ 26 ]	105.2	11.54	43.50	53.80
DriveDreamer [ 23 ]	113.8	11.85	38.45	47.20
DriveDreamer-2 [ 29 ]	110.3	11.20	41.30	50.10
Ours	107.7	11.03	53.48	65.70

Table I: Comparison of Video Quality and Layout Control. We evaluate perceptual quality using FVD/FID and geometric precision using mAP/IoU. Assessments are conducted on the validation sets of both the nuScenes dataset and our self-collected dataset.

\downarrow

indicates lower is better,

\uparrow

indicates higher is better.

Metric	BEVControl	MagicDrive-V2	DriveDreamer	DriveDreamer-2	Ours	Real Data (Ref)
Metric	[ 22 ]	[ 26 ]	[ 23 ]	[ 29 ]	(Generated)	(Oracle)
Global Scene Level
Weather Accuracy ( $\uparrow$ )	73.2%	75.4%	72.8%	74.3%	75.6%	76.6%
Time Accuracy ( $\uparrow$ )	75.14%	77.39%	76.50%	78.31%	80.9%	83.0%
Object Level (Box-Guided)
Category Accuracy ( $\uparrow$ )	76.5%	79.9%	75.9%	78.3%	85.1%	88.5%

Table II: Quantitative Evaluation using MV-VLM Evaluator. We report the accuracy and consistency scores, where ”Real Data” serves as the reference upper bound.

Figure 6: Qualitative video generation comparison results evaluated on our self-collected real-world dataset from different comparing approaches, using prompt: Generating the same scene and objects as GT. The GT reference images are also demonstrated.

Figure 7: Qualitative video generation comparison results evaluated on nuScenes dataset from different comparing approaches, including DriveDreamer-2 [ 29 ] , MagicDrive-V2 [ 26 ] and ours, using the same text prompt: A white sedan, clean, facing camera. The GT reference images are also demonstrated.

Figure 8: Qualitative video generation comparison results evaluated on our dataset from different comparing approaches, including MagicDrive-V2 [ 26 ] and ours, using the same GT input video.

Figure 9: More visual generation results by our VistaGEN, which preserve multiview spatial-temporal consistency in complex scene.

Figure 10: More visual results of VistaGEN using different prompts. Our VistaGEN can easily achieve intra-class long-tail generation with reasonable and coherent generation given different prompts while maintaining spatiotemporal consistency.

Figure 11: Visual comparison results without (middle) or with (right) using the visual-language feature condition.

Condition Setting	Layout (mAP) $\uparrow$	Text Alignment $\uparrow$	Image Alignment $\uparrow$
Global Only ( $\mathbf{c}_{global}$ )	53.03	15.7	17.4
Global + Local ( $\mathbf{c}_{global}+\mathbf{c}_{local}$ )	53.48	50.5	57.1

Table III: Ablation on Object-Level Conditioning. We compare multiview video generation performance with and without the local object prompt

\mathbf{c}_{local}

. While geometric accuracy (mAP) remains high in both settings due to box constraints, semantic alignment improves significantly with

\mathbf{c}_{local}

Settings	Video Quality		Layout Fidelity		Alignment
Settings	FVD ( $\downarrow$ )	FID ( $\downarrow$ )	mAP ( $\uparrow$ )	IoU ( $\uparrow$ )	CLIP-T ( $\uparrow$ )	CLIP-I ( $\uparrow$ )
Direct Generation	110.4	11.53	45.52	555.73	0.667	0.686
Closed-Loop Generation	107.7	11.03	53.48	65.70	0.778	0.797

Table IV: Effectiveness of the MV-VLM Evaluator. Qualitative comparison between direct open-loop generation and our proposed closed-loop paradigm.

Settings	Video Quality		Layout Fidelity		Alignment
Settings	FVD ( $\downarrow$ )	FID ( $\downarrow$ )	mAP ( $\uparrow$ )	IoU ( $\uparrow$ )	CLIP-T ( $\uparrow$ )	CLIP-I ( $\uparrow$ )
w/o Refinement	108.3	11.01	48.78	58.82	0.672	0.691
w/ Refinement	107.7	11.03	53.48	65.70	0.778	0.797

Table V: Ablation study of the Refinement Module. We compare the generation quality with and without the refinement loop. The full model demonstrates superior performance in video quality, geometric precision, and semantic alignment.

Figure 12: The visual comparison results with (left) or without (right) using the object-level refinement module for a fast-moving vehicle generation.

Figure 13: Inpainting capability to highly articulated or rapidly deforming objects (e.g., pedestrians in dense crowds) remains an open challenge.

Controllable Video Generation

Controllable video generation steers how videos look, move, and change over time using user-provided instructions and structured controls.

See all 321 papers

DriveCtrl: Conditioned Sim-to-Real Driving Video Generation
May 14, 2026Haonan Zhao, Yiting Wang, Jingkun Chen +3Autonomous Driving Datasets Synthetic Data Augmentation
University of Warwick · 2Northwestern Polytechnical University · 3Queen Mary University of London
M $^\text{4}$ World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming
Jul 15, 2026Ke Cheng, Hanqiao Ye, Lei Shi +8Multi-View Video Generation Long-Horizon Video Generation
1Meituan · Institute of Automation, Chinese Academy of Sciences · 3Beijing Institute of Technology
OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation
Jun 16, 2026Zijie Meng, Yufei Liu, Chengqian Ma +8Multi-View Consistency Multi-View Video Generation
Peking University · Xiamen University · Korea Advanced Institute of Science and Technology (KAIST) +5

May 14, 2026J/K move · Enter open · S save

DriveCtrl: Conditioned Sim-to-Real Driving Video Generation

Haonan Zhao, Yiting Wang, Jingkun Chen, Valentina Donzella+2

University of Warwick · 2Northwestern Polytechnical University · 3Queen Mary University of London

Autonomous Driving Datasets Synthetic Data Augmentation Video Generation Controllable Video Generation Autonomous Driving Simulation Video Generation Evaluation

Open paper AskPDF arXiv

Large-scale labelled driving video data is essential for training autonomous driving systems. Although simulation offers scalable and fully annotated data, the domain gap between synthetic and real-world driving videos significantly limits its utility for downstream deployment. Existing video generation methods are not well-suited for this task, as they fail to simultaneously preserve scene structure, object dynamics, temporal consistency, and visual realism, all of which are critical for maintaining annotation validity in generated data. In this paper, we present DriveCtrl, a depth-conditioned controllable sim-to-real video generation framework for realistic driving video synthesis. Built upon a pretrained video foundation model, DriveCtrl introduces a structure-aware adapter that enables depth-guided generation while preserving the scene layout and motion patterns of the source simulation, producing temporally coherent driving videos that remain aligned with the original simulated sequences. We further introduce a scalable data generation pipeline that transforms simulator videos into realistic driving footage matching the visual style of a target real-world dataset. The pipeline supports three conditioning signals: structural depth, reference-dataset style, and text prompts, while preserving frame-level annotations for downstream perception tasks. To better assess this task, we propose a driving-domain-specific knowledge-informed evaluation metric called Driving Video Realism Score (DVRS) that assesses the realism of generated videos. Experiments demonstrate that DriveCtrl consistently outperforms the base model and competing alternatives in realism, temporal quality, and perception task performance, substantially narrowing the sim-to-real gap for driving video generation.

Abstract

Controllable Video Generation

Explore similar work

DriveCtrl: Conditioned Sim-to-Real Driving Video Generation

M4^\text{4}4World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming

OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation

M $^\text{4}$ World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming