Generative world models offer controllable and repeatable closed-loop simulation for end-to-end and vision-language-action driving policies, but production deployment exposes three unresolved requirements: faithfully reproducing a mixed fisheye-pinhole rig at native resolutions; reconciling causal, per-timestep interaction with long-horizon stability and low latency; and preserving scene identity when a location is revisited. We present ZYT-World, a single architecture that natively generates four fisheye views with field of view > 180° and three pinhole views. Projection-specific Plucker adapters encode camera geometry, ego-motion adaptive layer normalization provides global motion control, and a lightweight pixel-aligned layout conditions traffic participants and signals through instance-level boxes, headings and colors. Heterogeneous training combines full-rig geometric coverage with high-resolution detail. Teacher forcing, causal consistency distillation, self-rollout distribution matching distillation, and RigCritic transform a 40-step bidirectional teacher into a one-step, per-latent streaming generator, with RigCritic evaluating the seven-view rig jointly. A 19M-parameter variational autoencoder decoder (TinyVAE), W8A8 quantization, and our inference engine reduce decoding, backbone, and incremental-execution costs, respectively. Finally, cross-trajectory pairs derived from real captures train a plug-in implicit-memory module that preserves place-specific evidence. On the internal multi-view test set, the one-step model retains more than 90% of the teacher's PSNR and SSIM, while FID, FVD, and LPIPS stay within 11% of the teacher. Under the generator-only timing in Figure 2, it is 107.7 times faster than the 40-step bidirectional teacher. TinyVAE decodes 59.8 times faster than Wan. 30s rollouts and cross-trajectory revisits show the intended long-horizon and memory behavior.
Figures & tables
Figure 1: Native seven-view mixed fisheye–pinhole generation under per-latent, one-step, bounded-history rollout: 4 FPS for the full seven-view rig on two GPUs.
Figure 2: Decoder and generator efficiency: TinyVAE decoder versus Wan decoder (left), and one-step AR versus the 40-step bidirectional teacher (right). Per-metric trade-offs are in Sec. 7.2 .
Work
Camera rig
Resolution
Interaction granularity
Long horizon (minute-scale)
Few-step generation
Vista ( Gao et al., 2024 )
Front only
576 × 1024
Full clip
×
×
MagicDrive-V2 ( Gao et al., 2025 )
6 pinhole
Up to 848 × 1600
Full clip
×
×
GAIA-2 (Wayve) ( Russell et al., 2025 )
5 cameras
Unified 448 × 960
Full clip
×
×
Epona ( Zhang et al., 2025 )
Front only
512 × 1024
Frame-wise
✓
×
X-World ( Zheng et al., 2026 )
7 cameras (1 fisheye)
Unreported
Chunk-wise
×
4 steps
FAR-Drive ( Li et al., 2026b )
Pinhole (nuScenes)
Unified 576 × 1024
Frame-wise
×
×
Table 1: Public driving world models versus ZYT-World on camera rig, resolution, interaction granularity, long-horizon rollout, and few-step generation.
Figure 3: ZYT-World pipeline: from heterogeneous multi-view data and bidirectional pretraining, through autoregressive distillation, to real-time closed-loop inference.
Figure 4: Data cleaning pipeline and the rebalanced training distribution.
Figure 5: Construction of cross-trajectory memory training pairs.
Figure 6: Heterogeneous multi-view controllable DiT with plug-in memory.
Figure 8: Memory-latent retrieval, two-level position embedding, and propagation to all views.
Figure 9: Teacher-forcing mask and condition-aware fixed-length pruning.
Figure 10: One-step AR generation without (top) and with (bottom) pixel-domain LPIPS regularization. Green boxes mark local differences.
Figure 12: Pretraining-curriculum quality. (a) FVMD of already-trained vs newly merged cameras; (b) FVMD normalized to the pretraining Stage-4 final checkpoint; (c) relative improvement from the Stage-3 mean to the Stage-4 final checkpoint.
Figure 14: Default reshard versus No-Reshard on the DMD generator rollout.
Figure 15: Training-system speedups for AR post-training. (a) Step time at a fixed parallel configuration for TF and CD; (b) DMD throughput after sequence parallelism 8→4 . Bars compare the AR configuration with and without the memory path. Dashed line: pre-optimization baseline.
Figure 16: Checkpoint-level diagnostics on seven-view checkpoints. (a) Quality deviation from the cohort mean for FVMD, Sampson, and Det-F1 (faint: raw; thick: smoothed; spread in the legend); (b) Det-F1 versus training step; (c) FVMD versus Sampson.
Figure 17: Pretraining Stage-4 trajectories of low-resolution loss, FVMD, FDD, PSNR, SSIM, and FID.
Model
FID ↓
FVD ↓
FVMD ↓
FDD ↓
LPIPS ↓
PSNR ↑
SSIM ↑
Det-F1 ↑
Sampson ↓
w/o Memory
11.64
116.58
116.49
68.79
0.241
22.30
0.725
0.712
2.991
w/ Memory
11.72
113.63
101.93
64.55
0.216
23.06
0.741
0.726
2.965
Table 2: Same-protocol evaluation of w/o Memory versus w/ Memory.
Generator
Rel. speedup ↑
FID ↓
FVD ↓
FDD ↓
LPIPS ↓
PSNR ↑
SSIM ↑
Det-F1 ↑
Sampson ↓
Bidirectional teacher
1.0 × (ref.)
11.64
116.58
68.79
0.2411
22.30
0.7254
0.712
2.991
4-step AR
15.8 ×
10.52
118.22
76.36
0.2677
21.27
0.6954
0.567
3.480
2-step AR
26.4 ×
9.97
102.50
77.43
0.2646
21.26
0.6911
0.574
3.556
1-step AR
39.6 ×
11.83
123.28
81.79
0.2655
21.07
0.6951
0.581
3.413
Table 3: Bidirectional teacher and separately trained few-step AR operating points. Boldface is the best AR student per column (teacher is a reference). Speedup is generator-side first-latent wall-clock versus the 40-step teacher on one latent timestep, excluding TinyVAE, W8A8, and our inference engine.
Model
FID ↓
FVD ↓
FDD ↓
LPIPS ↓
PSNR ↑
SSIM ↑
Det-F1 ↑
Sampson ↓
Wan
11.6415
116.5837
68.7932
0.2411
22.3026
0.7254
0.7116
2.9908
TAEHV super
12.8877
127.9301
86.8353
0.2487
22.4476
0.7280
0.6867
3.0697
TinyVAE
11.6306
115.9042
72.0696
0.2356
22.3884
0.7249
0.7053
3.0601
Table 4: Generation quality in the full generation chain. The same latents, produced by the bidirectional model, are decoded by each of the three decoders; only the decoder differs between rows.
Model
PSNR ↑
SSIM ↑
LPIPS ↓
char_sim/box ↑
Speedup ↑
Mem. savings ↑
Wan
32.5432
0.9006
0.0556
0.4715
1.0 × (ref.)
1.0 × (ref.)
TAEHV super
29.7920
0.8572
0.1011
0.2442
59.8 ×
26.8 ×
TinyVAE
31.8750
0.8865
0.0520
0.3334
59.8 ×
26.8 ×
Table 5: Fixed-latent reconstruction on 248 clips of 81 frames: PSNR, SSIM, LPIPS, and char_sim/box are computed against the ground-truth clips. Decode speedup and model-memory savings are ratios relative to Wan.
Figure 19: Front-narrow bidirectional generations across weather and lighting ( t=0 – 6.0 s, 1.2 s intervals).
Figure 20: Synchronized seven-view bidirectional generations at seven timestamps ( t=0 – 7.2 s, 1.2 s intervals).
Figure 21: Trajectory intervention from a shared intersection (left, straight, right).
Figure 22: Seven-view layout following for vehicles and lane markings over time.
Figure 23: Traffic-signal intervention: green, yellow, and red phases with the ego vehicle stopping on red.
Figure 24: Cross-trajectory revisit with and without memory.
Figure 25: One-step autoregressive generation before and after RigCritic.
Figure 26: Two 30 s autoregressive rollouts, sampled non-uniformly at t=0.0 , 4.3 , 8.6 , 17.1 , 25.7 , and 30.0 s.
Figure 27: Same-latent reconstruction (a) and bidirectional generation (b): Wan, TAEHV super, and TinyVAE.
Scalable and reliable evaluation is increasingly critical in the end-to-end era of autonomous driving, where vision--language--action (VLA) policies directly map raw sensor streams to driving actions. Yet, current evaluation pipelines still rely heavily on real-world road testing, which is costly, biased toward limited scenario coverage, and difficult to reproduce. These challenges motivate a real-world simulator that can generate realistic future observations under proposed actions, while remaining controllable and stable over long horizons. We present X-World, an action-conditioned multi-camera generative world model that simulates future observations directly in video space. Given synchronized multi-view camera history and a future action sequence, X-World generates future multi-camera video streams that follow the commanded actions. To ensure reproducible and editable scene rollouts, X-World further supports optional controls over dynamic traffic agents and static road elements, and retains a text-prompt interface for appearance-level control (e.g., weather and time of day). Beyond world simulation, X-World also enables video style transfer by conditioning on appearance prompts while preserving the underlying action and scene dynamics. At the core of X-World is a multi-view latent video generator designed to explicitly encourage cross-view geometric consistency and temporal coherence under diverse control signals. Experiments show that X-World achieves high-quality multi-view video generation with (i) strong view consistency across cameras, (ii) stable temporal dynamics over long rollouts, and (iii) high controllability with strict action following and faithful adherence to optional scene controls. These properties make X-World a practical foundation for scalable and reproducible evaluation.
Driving-world generation has emerged as a core capability for scalable autonomous-driving simulation, yet existing methods remain limited in object-level controllability and long-horizon stability. We present M4World, a Multi-view and Multimodal generative driving world model that synthesizes future surround-view video streams and synchronized LiDAR scans while supporting interactive object Manipulation and stable Minute-long streaming. Fine-grained object manipulation is realized through a flexible conditioning interface that supports explicit control over both the spatial layout and visual appearance of individual objects. Stable minute-long streaming, on the other hand, is achieved through a multi-stage training framework that enables online causal generation in only four denoising steps while maintaining coherent world dynamics throughout extended rollouts. Building on these components, we introduce an efficient few-clip post-training as well as a suite of visual reference-conditioned generation models, preserving general generation ability while allowing rare-case customization for long-tail controllability. To assess controllability beyond realism, we further introduce an automated VLM-based judging pipeline that evaluates scene-level condition adherence, view-wise object controllability, and cross-view object consistency. Comprehensive experiments show that M4World consistently delivers high generation quality, precise controllability, and stable minute-long streaming. Together with downstream long-tail augmentation and scene editing, these results demonstrate the potential of M4World for controllable, scalable driving simulation.
Ke Cheng, Hanqiao Ye, Lei Shi +8
1Meituan · Institute of Automation, Chinese Academy of Sciences · 3Beijing Institute of Technology
This report presents a unified technical system addressing the two core capabilities of world models for autonomous driving: world representation and world generation. For world representation, we propose WorldRec, a feed-forward reconstruction architecture driven by sparse scene queries. WorldRec initializes structured queries in 3D space, leveraging them to aggregate cross-view, cross-temporal features, thereby naturally enforcing spatial consistency across frames and yielding compact yet high-fidelity 3D Gaussian scene representations. For world generation, we propose WorldGen, a two-stage training framework of bidirectional pretraining followed by causal fine-tuning through three progressive stages (Teacher Forcing, ODE distillation, and DMD), enabling high-quality online causal video generation in as few as 4 denoising steps. Building on both modules, we further introduce the JWM, which deeply integrates WorldRec and WorldGen to achieve synergistic gains in generation stability, cross-frame consistency, and visual fidelity, providing a solid foundation for closed-loop simulation, data synthesis, and end-to-end training in autonomous driving.