ZYT-World: A Real-Time Controllable World Model for Closed-Loop Autonomous-Driving Simulation

Boni Hu; Xiong Wei; Haoming Huang; Yong Huang; Chenbo Wang; Yi Yang; Jiancheng Wang; Ruicheng Zhu

cs.CVSep 18, 2026

ZYT-World: A Real-Time Controllable World Model for Closed-Loop Autonomous-Driving Simulation

Authors: Boni Hu, Xiong Wei, Haoming Huang, Yong Huang, Chenbo Wang, Yi Yang, Jiancheng Wang, Ruicheng Zhu, +15 more

Organizations: ZYT AI Team

Abstract

Generative world models offer controllable and repeatable closed-loop simulation for end-to-end and vision-language-action driving policies, but production deployment exposes three unresolved requirements: faithfully reproducing a mixed fisheye-pinhole rig at native resolutions; reconciling causal, per-timestep interaction with long-horizon stability and low latency; and preserving scene identity when a location is revisited. We present ZYT-World, a single architecture that natively generates four fisheye views with field of view > 180° and three pinhole views. Projection-specific Plucker adapters encode camera geometry, ego-motion adaptive layer normalization provides global motion control, and a lightweight pixel-aligned layout conditions traffic participants and signals through instance-level boxes, headings and colors. Heterogeneous training combines full-rig geometric coverage with high-resolution detail. Teacher forcing, causal consistency distillation, self-rollout distribution matching distillation, and RigCritic transform a 40-step bidirectional teacher into a one-step, per-latent streaming generator, with RigCritic evaluating the seven-view rig jointly. A 19M-parameter variational autoencoder decoder (TinyVAE), W8A8 quantization, and our inference engine reduce decoding, backbone, and incremental-execution costs, respectively. Finally, cross-trajectory pairs derived from real captures train a plug-in implicit-memory module that preserves place-specific evidence. On the internal multi-view test set, the one-step model retains more than 90% of the teacher's PSNR and SSIM, while FID, FVD, and LPIPS stay within 11% of the teacher. Under the generator-only timing in Figure 2, it is 107.7 times faster than the 40-step bidirectional teacher. TinyVAE decodes 59.8 times faster than Wan. 30s rollouts and cross-trajectory revisits show the intended long-horizon and memory behavior.

Figures & tables

Figure 1: Native seven-view mixed fisheye–pinhole generation under per-latent, one-step, bounded-history rollout: 4 FPS for the full seven-view rig on two GPUs.

Figure 2: Decoder and generator efficiency: TinyVAE decoder versus Wan decoder (left), and one-step AR versus the 40-step bidirectional teacher (right). Per-metric trade-offs are in Sec. 7.2 .

Work	Camera rig	Resolution	Interaction granularity	Long horizon (minute-scale)	Few-step generation
Vista ( Gao et al., 2024 )	Front only	576 $\times$ 1024	Full clip	$\times$	$\times$
MagicDrive-V2 ( Gao et al., 2025 )	6 pinhole	Up to 848 $\times$ 1600	Full clip	$\times$	$\times$
GAIA-2 (Wayve) ( Russell et al., 2025 )	5 cameras	Unified 448 $\times$ 960	Full clip	$\times$	$\times$
Epona ( Zhang et al., 2025 )	Front only	512 $\times$ 1024	Frame-wise	$\checkmark$	$\times$
X-World ( Zheng et al., 2026 )	7 cameras (1 fisheye)	Unreported	Chunk-wise	$\times$	4 steps
FAR-Drive ( Li et al., 2026b )	Pinhole (nuScenes)	Unified 576 $\times$ 1024	Frame-wise	$\times$	$\times$

Table 1: Public driving world models versus ZYT-World on camera rig, resolution, interaction granularity, long-horizon rollout, and few-step generation.

Figure 3: ZYT-World pipeline: from heterogeneous multi-view data and bidirectional pretraining, through autoregressive distillation, to real-time closed-loop inference.

Figure 4: Data cleaning pipeline and the rebalanced training distribution.

Figure 5: Construction of cross-trajectory memory training pairs.

Figure 6: Heterogeneous multi-view controllable DiT with plug-in memory.

Figure 8: Memory-latent retrieval, two-level position embedding, and propagation to all views.

Figure 9: Teacher-forcing mask and condition-aware fixed-length pruning.

Figure 10: One-step AR generation without (top) and with (bottom) pixel-domain LPIPS regularization. Green boxes mark local differences.

Figure 12: Pretraining-curriculum quality. (a) FVMD of already-trained vs newly merged cameras; (b) FVMD normalized to the pretraining Stage-4 final checkpoint; (c) relative improvement from the Stage-3 mean to the Stage-4 final checkpoint.

Figure 13: FSDP No-Reshard: default per-forward reshard (top) versus keeping parameters unsharded (bottom).

Figure 14: Default reshard versus No-Reshard on the DMD generator rollout.

Figure 15: Training-system speedups for AR post-training. (a) Step time at a fixed parallel configuration for TF and CD; (b) DMD throughput after sequence parallelism

8\rightarrow 4

. Bars compare the AR configuration with and without the memory path. Dashed line: pre-optimization baseline.

Figure 16: Checkpoint-level diagnostics on seven-view checkpoints. (a) Quality deviation from the cohort mean for FVMD, Sampson, and Det-F1 (faint: raw; thick: smoothed; spread in the legend); (b) Det-F1 versus training step; (c) FVMD versus Sampson.

Figure 17: Pretraining Stage-4 trajectories of low-resolution loss, FVMD, FDD, PSNR, SSIM, and FID.

Model	FID $\downarrow$	FVD $\downarrow$	FVMD $\downarrow$	FDD $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	Det-F1 $\uparrow$	Sampson $\downarrow$
w/o Memory	11.64	116.58	116.49	68.79	0.241	22.30	0.725	0.712	2.991
w/ Memory	11.72	113.63	101.93	64.55	0.216	23.06	0.741	0.726	2.965

Table 2: Same-protocol evaluation of w/o Memory versus w/ Memory.

Generator	Rel. speedup $\uparrow$	FID $\downarrow$	FVD $\downarrow$	FDD $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	Det-F1 $\uparrow$	Sampson $\downarrow$
Bidirectional teacher	1.0 $\times$ (ref.)	11.64	116.58	68.79	0.2411	22.30	0.7254	0.712	2.991
4-step AR	15.8 $\times$	10.52	118.22	76.36	0.2677	21.27	0.6954	0.567	3.480
2-step AR	26.4 $\times$	9.97	102.50	77.43	0.2646	21.26	0.6911	0.574	3.556
1-step AR	39.6 $\times$	11.83	123.28	81.79	0.2655	21.07	0.6951	0.581	3.413

Table 3: Bidirectional teacher and separately trained few-step AR operating points. Boldface is the best AR student per column (teacher is a reference). Speedup is generator-side first-latent wall-clock versus the 40-step teacher on one latent timestep, excluding TinyVAE, W8A8, and our inference engine.

Model	FID $\downarrow$	FVD $\downarrow$	FDD $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	Det-F1 $\uparrow$	Sampson $\downarrow$
Wan	11.6415	116.5837	68.7932	0.2411	22.3026	0.7254	0.7116	2.9908
TAEHV super	12.8877	127.9301	86.8353	0.2487	22.4476	0.7280	0.6867	3.0697
TinyVAE	11.6306	115.9042	72.0696	0.2356	22.3884	0.7249	0.7053	3.0601

Table 4: Generation quality in the full generation chain. The same latents, produced by the bidirectional model, are decoded by each of the three decoders; only the decoder differs between rows.

Model	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	char_sim/box $\uparrow$	Speedup $\uparrow$	Mem. savings $\uparrow$
Wan	32.5432	0.9006	0.0556	0.4715	1.0 $\times$ (ref.)	1.0 $\times$ (ref.)
TAEHV super	29.7920	0.8572	0.1011	0.2442	59.8 $\times$	26.8 $\times$
TinyVAE	31.8750	0.8865	0.0520	0.3334	59.8 $\times$	26.8 $\times$

Table 5: Fixed-latent reconstruction on 248 clips of 81 frames: PSNR, SSIM, LPIPS, and char_sim/box are computed against the ground-truth clips. Decode speedup and model-memory savings are ratios relative to Wan.

Figure 19: Front-narrow bidirectional generations across weather and lighting ( $t=0$ – $6.0$ s, $1.2$ s intervals).

Figure 20: Synchronized seven-view bidirectional generations at seven timestamps ( $t=0$ – $7.2$ s, $1.2$ s intervals).

Figure 21: Trajectory intervention from a shared intersection (left, straight, right).

Figure 22: Seven-view layout following for vehicles and lane markings over time.

Figure 23: Traffic-signal intervention: green, yellow, and red phases with the ego vehicle stopping on red.

Figure 24: Cross-trajectory revisit with and without memory.

Figure 25: One-step autoregressive generation before and after RigCritic.

Figure 26: Two 30 s autoregressive rollouts, sampled non-uniformly at $t=0.0$ , $4.3$ , $8.6$ , $17.1$ , $25.7$ , and $30.0$ s.

Figure 27: Same-latent reconstruction (a) and bidirectional generation (b): Wan, TAEHV super, and TinyVAE.

Explore similar work

X-World: Controllable Ego-Centric Multi-Camera World Models for Scalable End-to-End Driving
Mar 20, 2026Chaoda Zheng, Sean Li, Jinhao Deng +9World Models for Autonomous Driving Action-Conditioned Video Generation
GWM Team, XPeng Inc.
M $^\text{4}$ World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming
Jul 15, 2026Ke Cheng, Hanqiao Ye, Lei Shi +8Multi-View Video Generation Long-Horizon Video Generation
1Meituan · Institute of Automation, Chinese Academy of Sciences · 3Beijing Institute of Technology
Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving
May 18, 2026Lijun Zhou, Hongcheng Luo, Zhenxin Zhu +34Dynamic Scene Reconstruction World Models for Autonomous Driving
Xiaomi Auto World Model Team

Mar 20, 2026J/K move · Enter open · S save

X-World: Controllable Ego-Centric Multi-Camera World Models for Scalable End-to-End Driving

Chaoda Zheng, Sean Li, Jinhao Deng, Zhennan Wang+8

GWM Team, XPeng Inc.

World Models for Autonomous Driving Action-Conditioned Video Generation Autonomous Driving Simulation Controllable Video Generation Multi-View Video Generation

Open paper AskPDF arXiv

Scalable and reliable evaluation is increasingly critical in the end-to-end era of autonomous driving, where vision--language--action (VLA) policies directly map raw sensor streams to driving actions. Yet, current evaluation pipelines still rely heavily on real-world road testing, which is costly, biased toward limited scenario coverage, and difficult to reproduce. These challenges motivate a real-world simulator that can generate realistic future observations under proposed actions, while remaining controllable and stable over long horizons. We present X-World, an action-conditioned multi-camera generative world model that simulates future observations directly in video space. Given synchronized multi-view camera history and a future action sequence, X-World generates future multi-camera video streams that follow the commanded actions. To ensure reproducible and editable scene rollouts, X-World further supports optional controls over dynamic traffic agents and static road elements, and retains a text-prompt interface for appearance-level control (e.g., weather and time of day). Beyond world simulation, X-World also enables video style transfer by conditioning on appearance prompts while preserving the underlying action and scene dynamics. At the core of X-World is a multi-view latent video generator designed to explicitly encourage cross-view geometric consistency and temporal coherence under diverse control signals. Experiments show that X-World achieves high-quality multi-view video generation with (i) strong view consistency across cameras, (ii) stable temporal dynamics over long rollouts, and (iii) high controllability with strict action following and faithful adherence to optional scene controls. These properties make X-World a practical foundation for scalable and reproducible evaluation.

Abstract

Explore similar work

X-World: Controllable Ego-Centric Multi-Camera World Models for Scalable End-to-End Driving

M4^\text{4}4World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming

Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving

M $^\text{4}$ World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming