Despite rapid progress in autonomous driving, reliable training and evaluation of driving systems remain fundamentally constrained by the lack of scalable and interactive simulation environments. Recent generative video models achieve remarkable visual fidelity, yet most operate in open-loop settings and fail to support fine-grained frame-level interaction between agent actions and environment evolution. Building a learning-based closed-loop simulator for autonomous driving poses three major challenges: maintaining long-horizon temporal and cross-view consistency, mitigating autoregressive degradation under iterative self-conditioning, and satisfying low-latency inference constraints. In this work, we propose FAR-Drive, a frame-level autoregressive video generation framework for autonomous driving. We introduce a multi-view diffusion transformer with fine-grained structured control, enabling geometrically consistent multi-camera generation. To address long-horizon consistency and iterative degradation, we design a two-stage training strategy consisting of adaptive reference horizon conditioning and blend-forcing autoregressive training, which progressively improves consistency and robustness under self-conditioning. To meet low-latency interaction requirements, we further integrate system-level efficiency optimizations for inference acceleration. Experiments on the nuScenes dataset demonstrate that our method achieves state-of-the-art performance among existing closed-loop autonomous driving simulation approaches, while maintaining sub-second latency on a single GPU.
Figures & tables
Figure 1: Overview of FAR-Drive framework for autonomous driving closed-loop simulation. Given driving agent actions, the simulator generates temporally and geometrically consistent multi-view video in an interactive rollout manner. Three core challenges of closed-loop generation are highlighted in different colors: long-horizon consistency, autoregressive degradation, and low-latency interaction. Adaptive reference-horizon conditioning, blend-forcing training, and system-level efficiency optimizations correspond to these challenges, with colors indicating the associated solutions.
Figure 2: Overview of the proposed multi-view MMDiT architecture. Reference frames and noise latents are fed into the backbone DiT while structured controls including projected 3D bounding boxes and BEV maps are first encoded by a convolutional encoder and then processed by a control DiT conditioned on the same latent inputs. All control signals are additionally aggregated into a unified scene prompt, which is injected into every layer of both the backbone and control DiTs. The outputs of each Control DiT block are injected into the corresponding backbone DiT blocks through zero-initialized projection layers (dashed arrows). Since the Control DiT has fewer layers than the backbone, control injection is applied only to the early backbone blocks.
Method
AR Unit
FVD ↓
mAP ↑
mIoU ↑
MagicDrive [ 15 ]
60
217.94
11.49
18.27
MagicDrive [ 15 ]
16
218.12
11.86
18.34
MagicDrive3D [ 13 ]
16
210.40
12.05
18.27
MagicDrive-V2 [ 14 ]
241
94.84
18.17
20.40
DreamForge (DiT) [ 36 ]
5
103.61
19.17
34.36
DreamForge (SD1.5) [ 36 ]
5
233.20
22.52
32.98
Table 1: Quantitative comparison on multi-view autonomous driving video generation under different autoregressive (AR) units. AR Unit denotes the number of frames generated per forward pass: AR Unit = 1 corresponds to strict frame-level autoregressive rollout, while larger values indicate chunk-wise AR generation. Our methods generate 16 frames so AR Unit = 16 corresponds to single-shot generation. For AR generation we use 3 sampling steps per frame, while single-shot generation uses 30 sampling steps.
Method
Volumetric Condition-Free
Multi-view
Video
1st-Frame Condition-Free
FID ↓
FVD ↓
X-Scene [ 50 ]
✗
✓
✗
✓
11.29
-
OccScene [ 32 ]
✗
✓
✓
✓
11.87
-
WoVoGen [ 35 ]
✗
✓
✓
✓
27.60
417.70
UniScene [ 31 ]
✗
✓
✓
✓
6.45
71.94
DiST-4D [ 18 ]
✗
✓
✓
✓
7.40
25.55
DriveGAN [ 26 ]
✓
✗
✓
✓
73.40
502.30
Table 2: Quantitative comparison of multi-view video generation on the nuScenes validation set. Methods are grouped according to whether they (i) avoid heavy volumetric intermediate representations, (ii) support multi-view generation, (iii) generate video sequences, and (iv) operate without first-frame conditioning. For AR generation we use 20 sampling steps per frame, while single-shot generation uses 30 sampling steps.
Figure 3: Ablation on the proposed blend-forcing on long-horizon generation. We evaluate FID and FVD (lower is better) at different rollout lengths from 16 to 229 frames.
Figure 4: Qualitative comparison of long-horizon autoregressive generation before and after Blend-Forcing training. We generate a 229-frame six-view sequence and visualize the first frame (left) and the last frame (right). More cases in supplementary material.
Sampling Steps
KV Cache
CFG- Free
Cond. Cache
FID ↓
FVD ↓
mAP ↑
mIoU ↑
NPU Time
GPU Time
1
✓
✓
✓
25.94
139.53
27.33
36.52
1.09 s/it
0.84 s/it
2
✓
✓
✓
18.44
99.95
27.54
36.42
1.35 s/it
0.98 s/it
3
✓
✓
✓
17.35
94.69
27.39
36.35
1.63 s/it
1.15 s/it
4
✓
✓
✓
16.44
91.11
27.38
36.31
1.82 s/it
1.28 s/it
5
✓
✓
✓
15.81
90.03
27.29
36.26
2.12 s/it
1.43 s/it
20
✓
✓
✓
11.92
82.78
26.96
36.25
5.75 s/it
3.36 s/it
Table 3: Ablation on inference configurations including diffusion sampling steps, classifier-free guidance (CFG), KV cache and condition cache (Cond. Cache). We report inference latency per iteration (s/it) measured on NPU and H200 GPU.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Our model architecture consists of 7 backbone units (each containing 4 backbone MMDiT blocks and 1 cross-view attention block) and 3 control units. We have provided a detailed description of a single backbone unit and its corresponding control unit. In this design, every four backbone blocks are alternately arranged with a cross-view attention block.
Figure 6: Qualitative comparison of long-horizon autoregressive generation before (part a) and after (part b) blend-forcing training. We generate a 229-frame six-view sequence and visualize the first frame (left of each part) and the last frame (right of each part).
Closed-loop driving simulation requires real-time interaction beyond short offline clips, pushing current driving world models toward autoregressive (AR) rollout. Existing AR distillation approaches typically rely on frame sinks or student-side degradation training. The former transfers poorly to driving due to fast ego-motion and rapid scene changes, while the latter remains bounded by the teacher's single-pass output length and thus provides only a limited supervision horizon. A natural question is: can the teacher itself be extended via AR rollout to provide unbounded-horizon supervision at bounded memory cost? The key difficulty is that a standard teacher drifts under its own predictions, contaminating the supervision it provides. Our key insight is to make the teacher rollout-capable, ensuring reliable supervision from its own AR rollouts. This is instantiated as HorizonDrive, an anti-drifting training-and-distillation framework for AR driving simulation. First, scheduled rollout recovery (SRR) trains the base model to reconstruct ground-truth future clips from prediction-corrupted histories, yielding a teacher that remains stable across long AR rollouts. Second, the rollout-capable teacher is extended via AR rollout, providing long-horizon distribution-matching supervision under bounded memory, while a short-window student aligns to it with teacher rollout DMD (TRD) for efficient real-time deployment. HorizonDrive natively supports minute-scale AR rollout under bounded memory; on nuScenes, HorizonDrive reduces FID by 52% and FVD by 37%, and lowers ARE and DTW by 21% and 9% relative to the strongest long-horizon streaming baselines, while remaining competitive with single-pass driving video generators.
Conglang Zhang, Yifan Zhan, Qingjie Wang +10
Wuhan University · The University of Tokyo · Horizon Robotics +3
Evaluating end-to-end autonomous driving (E2E-AD) remains challenging, as existing driving simulation methods often trade off closed-loop interactivity (e.g., CARLA) and real-world visual fidelity (e.g., nuScenes). We present Point as Skeleton, a generative sensor simulation framework for state-updated autoregressive driving video generation, in which an autoregressive generator synthesizes visual observations from step-wise updated ego states, actor states, scene maps, and point-cloud skeleton conditions. To support closed-loop rollout, we introduce Reset-and-Roll, which adapts rolling diffusion inference to simulation by preventing future-conditioned latent states from being committed across simulation steps. To stabilize error accumulation during step-wise autoregressive rollout, we introduce point-cloud skeletons that decouple foreground and background assets and project them into camera-view painted-point and template-depth conditions, providing appearance and geometric cues. We further implement a nuPlan-based renderer-level closed-loop generative interface for evaluating generation under ego deviations from the original log. Experiments on nuScenes and nuPlan show that Point as Skeleton improves autoregressive generation quality during closed-loop rollout, demonstrating its potential for visually faithful closed-loop driving simulation. The code is available at https://github.com/krauwu/point-as-skeleton.
Songbur Wong, Xiaosong Jia, Junqi You +12
1Shanghai Jiao Tong University · 3Fudan University · 4Shanghai Artificial Intelligence Laboratory +3
Recent years have seen remarkable progress in autonomous driving, yet generalization to long-tail and open-world scenarios remains a major bottleneck for large-scale deployment. To address this challenge, some works use LLMs and VLMs for vision-language understanding and reasoning, enabling vehicles to interpret rare and safety-critical situations when generating actions. Others study generative world models to capture the spatio-temporal evolution of driving scenes, allowing agents to imagine possible futures before acting. Inspired by human intelligence, which unifies understanding and imagination, we explore a unified model for autonomous driving. We present LMGenDrive, the first framework that combines LLM-based multimodal understanding with generative world models for end-to-end closed-loop driving. Given multi-view camera inputs and natural-language instructions, LMGenDrive generates both future driving videos and control signals. This design provides complementary benefits: video prediction improves spatio-temporal scene modeling, while the LLM contributes strong semantic priors and instruction grounding from large-scale pretraining. We further propose a progressive three-stage training strategy, from vision pretraining to multi-step long-horizon driving, to improve stability and performance. LMGenDrive supports both low-latency online planning and autoregressive offline video generation. Experiments show that it significantly outperforms prior methods on challenging closed-loop benchmarks, with clear gains in instruction following, spatio-temporal understanding, and robustness to rare scenarios. These results suggest that unifying multimodal understanding and generation is a promising direction for more generalizable and robust embodied decision-making systems.