FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Driving

Yaoru Li; Federico Landi; Marco Godi; Xin Jin; Ruiju Fu; Yufei Ma; Muyang Sun; Heyu Si

cs.CVMar 16, 2026

FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Driving

Authors: Yaoru Li, Federico Landi, Marco Godi, Xin Jin, Ruiju Fu, Yufei Ma, Muyang Sun, Heyu Si, +1 more

Organizations: Zhejiang University · Huawei

Abstract

Despite rapid progress in autonomous driving, reliable training and evaluation of driving systems remain fundamentally constrained by the lack of scalable and interactive simulation environments. Recent generative video models achieve remarkable visual fidelity, yet most operate in open-loop settings and fail to support fine-grained frame-level interaction between agent actions and environment evolution. Building a learning-based closed-loop simulator for autonomous driving poses three major challenges: maintaining long-horizon temporal and cross-view consistency, mitigating autoregressive degradation under iterative self-conditioning, and satisfying low-latency inference constraints. In this work, we propose FAR-Drive, a frame-level autoregressive video generation framework for autonomous driving. We introduce a multi-view diffusion transformer with fine-grained structured control, enabling geometrically consistent multi-camera generation. To address long-horizon consistency and iterative degradation, we design a two-stage training strategy consisting of adaptive reference horizon conditioning and blend-forcing autoregressive training, which progressively improves consistency and robustness under self-conditioning. To meet low-latency interaction requirements, we further integrate system-level efficiency optimizations for inference acceleration. Experiments on the nuScenes dataset demonstrate that our method achieves state-of-the-art performance among existing closed-loop autonomous driving simulation approaches, while maintaining sub-second latency on a single GPU.

Figures & tables

Figure 1: Overview of FAR-Drive framework for autonomous driving closed-loop simulation. Given driving agent actions, the simulator generates temporally and geometrically consistent multi-view video in an interactive rollout manner. Three core challenges of closed-loop generation are highlighted in different colors: long-horizon consistency, autoregressive degradation, and low-latency interaction. Adaptive reference-horizon conditioning, blend-forcing training, and system-level efficiency optimizations correspond to these challenges, with colors indicating the associated solutions.

Figure 2: Overview of the proposed multi-view MMDiT architecture. Reference frames and noise latents are fed into the backbone DiT while structured controls including projected 3D bounding boxes and BEV maps are first encoded by a convolutional encoder and then processed by a control DiT conditioned on the same latent inputs. All control signals are additionally aggregated into a unified scene prompt, which is injected into every layer of both the backbone and control DiTs. The outputs of each Control DiT block are injected into the corresponding backbone DiT blocks through zero-initialized projection layers (dashed arrows). Since the Control DiT has fewer layers than the backbone, control injection is applied only to the early backbone blocks.

Method	AR Unit	FVD $\downarrow$	mAP $\uparrow$	mIoU $\uparrow$
MagicDrive [ 15 ]	60	217.94	11.49	18.27
MagicDrive [ 15 ]	16	218.12	11.86	18.34
MagicDrive3D [ 13 ]	16	210.40	12.05	18.27
MagicDrive-V2 [ 14 ]	241	94.84	18.17	20.40
DreamForge (DiT) [ 36 ]	5	103.61	19.17	34.36
DreamForge (SD1.5) [ 36 ]	5	233.20	22.52	32.98

Table 1: Quantitative comparison on multi-view autonomous driving video generation under different autoregressive (AR) units. AR Unit denotes the number of frames generated per forward pass: AR Unit = 1 corresponds to strict frame-level autoregressive rollout, while larger values indicate chunk-wise AR generation. Our methods generate 16 frames so AR Unit = 16 corresponds to single-shot generation. For AR generation we use 3 sampling steps per frame, while single-shot generation uses 30 sampling steps.

Method	Volumetric Condition-Free	Multi-view	Video	1st-Frame Condition-Free	FID $\downarrow$	FVD $\downarrow$
X-Scene [ 50 ]	✗	✓	✗	✓	11.29	-
OccScene [ 32 ]	✗	✓	✓	✓	11.87	-
WoVoGen [ 35 ]	✗	✓	✓	✓	27.60	417.70
UniScene [ 31 ]	✗	✓	✓	✓	6.45	71.94
DiST-4D [ 18 ]	✗	✓	✓	✓	7.40	25.55
DriveGAN [ 26 ]	✓	✗	✓	✓	73.40	502.30

Table 2: Quantitative comparison of multi-view video generation on the nuScenes validation set. Methods are grouped according to whether they (i) avoid heavy volumetric intermediate representations, (ii) support multi-view generation, (iii) generate video sequences, and (iv) operate without first-frame conditioning. For AR generation we use 20 sampling steps per frame, while single-shot generation uses 30 sampling steps.

Figure 3: Ablation on the proposed blend-forcing on long-horizon generation. We evaluate FID and FVD (lower is better) at different rollout lengths from 16 to 229 frames.

Figure 4: Qualitative comparison of long-horizon autoregressive generation before and after Blend-Forcing training. We generate a 229-frame six-view sequence and visualize the first frame (left) and the last frame (right). More cases in supplementary material.

Sampling Steps	KV Cache	CFG- Free	Cond. Cache	FID $\downarrow$	FVD $\downarrow$	mAP $\uparrow$	mIoU $\uparrow$	NPU Time	GPU Time
1	✓	✓	✓	25.94	139.53	27.33	36.52	1.09 s/it	0.84 s/it
2	✓	✓	✓	18.44	99.95	27.54	36.42	1.35 s/it	0.98 s/it
3	✓	✓	✓	17.35	94.69	27.39	36.35	1.63 s/it	1.15 s/it
4	✓	✓	✓	16.44	91.11	27.38	36.31	1.82 s/it	1.28 s/it
5	✓	✓	✓	15.81	90.03	27.29	36.26	2.12 s/it	1.43 s/it
20	✓	✓	✓	11.92	82.78	26.96	36.25	5.75 s/it	3.36 s/it

Table 3: Ablation on inference configurations including diffusion sampling steps, classifier-free guidance (CFG), KV cache and condition cache (Cond. Cache). We report inference latency per iteration (s/it) measured on NPU and H200 GPU.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Our model architecture consists of 7 backbone units (each containing 4 backbone MMDiT blocks and 1 cross-view attention block) and 3 control units. We have provided a detailed description of a single backbone unit and its corresponding control unit. In this design, every four backbone blocks are alternately arranged with a cross-view attention block.

Figure 6: Qualitative comparison of long-horizon autoregressive generation before (part a) and after (part b) blend-forcing training. We generate a 229-frame six-view sequence and visualize the first frame (left of each part) and the last frame (right of each part).

Explore similar work

HorizonDrive: Self-Corrective Autoregressive World Model for Long-horizon Driving Simulation
May 12, 2026Conglang Zhang, Yifan Zhan, Qingjie Wang +10World Model Learning Long-Horizon Video Generation
Wuhan University · The University of Tokyo · Horizon Robotics +3
Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation
Jul 7, 2026Songbur Wong, Xiaosong Jia, Junqi You +12Autoregressive Video Generation Autonomous Driving Simulation
1Shanghai Jiao Tong University · 3Fudan University · 4Shanghai Artificial Intelligence Laboratory +3
LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving
Apr 9, 2026Hao Shao, Letian Wang, Yang Zhou +5Autonomous Driving End-to-End Autonomous Driving
CUHK MMLab · University of Toronto · UC Berkeley

May 12, 2026J/K move · Enter open · S save

HorizonDrive: Self-Corrective Autoregressive World Model for Long-horizon Driving Simulation

Conglang Zhang, Yifan Zhan, Qingjie Wang, Zhanpeng Ouyang+9

Wuhan University · The University of Tokyo · Horizon Robotics+3

World Model Learning Long-Horizon Video Generation Diffusion Model Distillation World Models for Autonomous Driving Autoregressive Video Generation Autonomous Driving Simulation

Open paper AskPDF arXiv

Closed-loop driving simulation requires real-time interaction beyond short offline clips, pushing current driving world models toward autoregressive (AR) rollout. Existing AR distillation approaches typically rely on frame sinks or student-side degradation training. The former transfers poorly to driving due to fast ego-motion and rapid scene changes, while the latter remains bounded by the teacher's single-pass output length and thus provides only a limited supervision horizon. A natural question is: can the teacher itself be extended via AR rollout to provide unbounded-horizon supervision at bounded memory cost? The key difficulty is that a standard teacher drifts under its own predictions, contaminating the supervision it provides. Our key insight is to make the teacher rollout-capable, ensuring reliable supervision from its own AR rollouts. This is instantiated as HorizonDrive, an anti-drifting training-and-distillation framework for AR driving simulation. First, scheduled rollout recovery (SRR) trains the base model to reconstruct ground-truth future clips from prediction-corrupted histories, yielding a teacher that remains stable across long AR rollouts. Second, the rollout-capable teacher is extended via AR rollout, providing long-horizon distribution-matching supervision under bounded memory, while a short-window student aligns to it with teacher rollout DMD (TRD) for efficient real-time deployment. HorizonDrive natively supports minute-scale AR rollout under bounded memory; on nuScenes, HorizonDrive reduces FID by 52% and FVD by 37%, and lowers ARE and DTW by 21% and 9% relative to the strongest long-horizon streaming baselines, while remaining competitive with single-pass driving video generators.

Abstract

Explore similar work

HorizonDrive: Self-Corrective Autoregressive World Model for Long-horizon Driving Simulation

Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation

LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving