Driving world models usually generate videos from six pinhole cameras with separate optical centres and small overlaps, and must learn to keep these views consistent. We present PanDWM, which stitches them into one 360∘ panoramic band. Camera boundaries then fall inside one image, so keeping the views consistent means keeping the image continuous, and each pixel still records its camera ray. Conditioned on layouts, a causal student generates the band window by window in four steps; it is distilled from a causal teacher and trained on its own rollouts. A feed-forward decoder lifts each frame along these rays into 3D Gaussians, and static Gaussians from neighbouring frames complete them into 4D scenes. On nuScenes, scored in one stitched 360∘ view, PanDWM has lower FID and FVD than the multi-camera baselines at 8-46× their speed, and keeps its 50-step teacher's FID while lowering FVD by 22-25%.
Figures & tables
Figure 2: PanDWM overview. (a) Panoramic history is split into overlapping halves and encoded by a VAE. Conditioned on layouts, text and ray maps, a causal diffusion transformer predicts 16 frames from 3 history frames in four steps; its final 3 frames seed the next window. (b) A feed-forward decoder lifts frame latents into pixel-aligned Gaussians along calibrated source-camera rays, at ranges relabelled with Depth Anything 3. Static geometry from neighbouring frames supports 4D completion. Panoramas are displayed as cylindrical bands; ellipsoids are enlarged for display.
Figure 3: Gaussian geometry and completion. (a) Panoramic pixels retain their calibrated source-camera origins. (b) Range and offset locate Gaussians relative to these rays ( Eq. 3 ); both halves’ predictions are retained in the overlap. (c) Ego-motion-aligned neighbouring static Gaussians fill uncovered pixels, while dynamic content comes from the current frame.
Method
Views
Hist.
NFE
s/frame
FID ↓
FVD-1 ↓
FVD-2 ↓
Obj. recall ↑
Road IoU ↑
UniMLVG [ 5 ]
6
3
100
20.3
24.3
51.3
64.5
66.8
74.0
OpenDWM (SD3.5) [ 39 ]
6
3
80
22.9
30.5
60.3
96.6
64.8
75.7
DreamForge [ 33 ]
6
3
60
35.9
17.3
46.0
64.8
48.3
72.4
DiST-4D [ 19 ]
6
1
60
15.9
17.2
55.8
71.0
41.4
72.2
OmniNWM [ 26 ]
6
1
50
89.9
25.4
73.0
113.9
30.4
71.3
Causal teacher †
2
3
100
33.7
9.93
47.7
82.5
58.9
75.4
Table 1: Comparison with driving video models on Val-150, evaluated in the panoramic band. Hist.: number of recorded history frames. FVD-1 and FVD-2 cover the first and second 16-frame windows. NFE counts denoiser evaluations per window, including both guidance branches. Time is measured on one A40 including VAE decoding, per 0.1 s of video (one frame at 10 Hz; DreamForge, DiST-4D and OmniNWM output 12 Hz). PanDWM is averaged over three seeds and baselines use seed 42. Bold marks the best method; † the causal teacher of PanDWM is a reference and is not ranked.
Figure 4: Band FVD-1 against time per frame on Val-150 (FID in Tab. 1 ); lower left is better.
Figure 5: Front-view predictions at 2.5 s for four fixed validation clips. Dashed ellipses mark corresponding objects (white: ground truth; green: kept; red: lost or displaced). PanDWM and HorizonDrive use layouts, Epona uses ego trajectory, and Vista and GEM use image history alone.
Figure 6: Front-left, front and front-right panoramic sectors at 2.5 s for three fixed validation clips. Ground truth and PanDWM are native panoramas; UniMLVG and OmniNWM are mapped from six views. Crops cover about ±13∘ elevation; brightness steps at camera boundaries come from the recorded exposures, which PanDWM reproduces. All methods use seed 42; OmniNWM receives one recorded frame and the ego trajectory.
Variant
FID ↓
FVD-1 ↓
FVD-2 ↓
Flicker
Starting student
19.30
103.4
172.4
2.334
Full recipe (500 it.)
17.07
77.1
131.3
2.005
w/o adversarial loss
17.95
81.2
141.2
1.993
w/o temporal disc. heads
16.75
78.0
131.8
2.041
Recorded history
17.17
74.3
133.2
2.007
Our model
14.28
56.8
93.3
1.580
Table 2: Ablations on Dev-78 with two seeds. Each variant changes one component of the full recipe and, like it, is trained for 500 iterations from the starting student. Our model, trained longer in stages, is a reference. Best values are bold; paired differences with 95% intervals are in the text.
Figure 7: Per-window FID and FVD on Test-L (83 scenes, seed 42), each computed on the 83 windows of one position.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
AbsRel ↓
δ1↑
Resolution: 224×400
Omni-Scene [ 51 ]
21.00
0.597
0.363
0.333
0.734
MCSplat ‡ [ 48 ]
18.85
0.511
0.442
0.270
0.644
Ours (3D)
18.21
0.510
0.436
0.334
0.560
Ours (4D)
19.29
0.533
0.410
0.298
0.642
Resolution: 280×518
Table 3: Reconstruction from recorded images on Test-S, evaluated in the six cameras within the band. PSNR, SSIM [ 50 ] and LPIPS [ 74 ] use renders composited over gray; AbsRel and δ1 use LiDAR depth at static points within 80 m with accumulated opacity ≥0.5 . Best per resolution in bold; ‡ LiDAR input. Ours (4D) adds four recorded frames 0.5–2 s before the input to the single-frame Ours (3D).
Method
0 m
±1 m
±2 m
±4 m
AbsRel ↓
DiST-4D [ 19 ]
23.5
33.1
38.8
48.4
0.467
DiST-4D + ReconDrive
39.2
44.5
57.4
83.7
0.715
Ours (4D)
21.8
27.0
34.0
50.5
0.316
Ours (4D) + refinement
18.5
19.1
22.8
34.9
0.316
Table 4: FID ↓ of views shifted laterally by the distance in each column, and static-region AbsRel, on 83 Test-S clips. Refinement changes the rendered images but not the Gaussians, so our two rows share one AbsRel.
Iterations
Added
Clip frames
k
0–1150
DMD, spatial heads
51
2
1150–1600
Temporal heads
51
2
1600–4000
Lmm
51
2
4000–4250
Lsharp
179
2–10
Table S1 : Training stages of the student. Each stage adds the listed terms to the objective of the previous stage. Clip frames include the 3 recorded history frames; k is the number of self-generated windows before the training window.
Method
FID ↓
FVD-1 ↓
FVD-2 ↓
UniMLVG [ 5 ]
5.38
23.4
38.6
OpenDWM (SD3.5) [ 39 ]
6.46
23.8
47.3
DreamForge [ 33 ]
9.77
27.8
41.5
DiST-4D [ 19 ]
11.92
35.8
47.2
OmniNWM [ 26 ]
14.38
43.1
79.0
Causal teacher †
5.49
35.0
64.0
Table S2 : Six-camera evaluation on Val-150 at 256×448 with recorded history. PanDWM is averaged over three seeds; the other methods use seed 42. Best method values are bold; † the causal teacher is a reference and is not considered.
FID ↓
FVD ↓
Method
3–5
6–8
9–11
3–5
6–8
9–11
UniMLVG [ 5 ]
37.7
46.7
51.8
75.8
88.6
101.4
OpenDWM (SD3.5) [ 39 ]
50.7
58.2
61.4
136.9
141.8
155.6
PanDWM (ours)
26.1
38.9
49.0
86.6
113.4
126.6
Causal teacher †
27.8
43.0
45.9
116.7
136.9
157.9
Recorded views †
17.6
17.5
17.3
23.0
23.3
23.7
Table S3 : Long-horizon comparison on Test-L (83 scenes, seed 42), pooled over window groups. Recorded views are the 256×448 camera views stitched in the same way. Best method values are bold; † references, not considered.
Input
Size
Tokens
Denoise
Decode
Total
Relative
Two halves
2×256×2304
4608
0.343
0.152
0.495
1.00
Six views
6×256×448
2688
0.169
0.072
0.241
0.49
Six views, average resolution
6×304×736
5244
0.339
0.141
0.479
0.97
Six views, nowhere coarser
6×320×832
6240
0.410
0.170
0.579
1.17
Six views
6×448×800
8400
0.573
0.250
0.824
1.66
Table S4 : Denoising and decoding time of the same four-step student on one H100, in seconds per generated frame (mean of three clips, text encoder on the GPU). Relative: time divided by that of the two halves.
Panoramic band
Six cameras, 256×448
Method
Views
Hist.
NFE
FID ↓
FVD-1 ↓
FVD-2 ↓
FID ↓
FVD-1 ↓
FVD-2 ↓
MagicDrive-V2 [ 15 ]
6
0
60
45.9
141.3
134.1
21.93
86.0
78.6
UniMLVG [ 5 ]
6
0
100
42.6
109.7
104.2
16.03
73.7
74.6
PanDWM (ours)
2
0
4
49.3
242.1
206.0
27.25
195.7
171.4
Table S5 : Setting G (conditions only) on Val-150 in the two evaluation spaces of the main paper. PanDWM: mean over three seeds; baselines: seed 42. NFE counts denoiser evaluations per window with both guidance branches; the window is 19 frames for UniMLVG and PanDWM and the whole 49-frame clip for MagicDrive-V2.
Method
Ratio
Excess at cameras
Excess at rear joint
UniMLVG [ 5 ]
14.71
−0.24[−0.42,−0.08]
–
OpenDWM (SD3.5) [ 39 ]
16.27
+1.32[1.10,1.56]
–
DreamForge [ 33 ]
12.70
+2.09[1.82,2.37]
–
DiST-4D [ 19 ]
12.19
+1.57[1.20,1.92]
–
OmniNWM [ 26 ]
12.97
+2.35[2.11,2.60]
–
Causal teacher
10.22
−0.07[−0.18,0.05]
+1.99[1.85,2.13]
Table S6 : Seam ratios at the six camera boundaries of the stitched band (pooled) on Val-150, and the excess over recorded views at the same resolution, stitched in the same way (95% intervals). Rear joint: the column where the two halves of PanDWM meet without overlap; the recorded band reads 0.85 there.
In this work, we aim to address the challenge of long-range memory in panoramic world models by exploiting the rotation-equivariant property of omnidirectional representations, where rotation can be treated as an implicit geometric transformation.Building on this insight, we propose PanoWorld, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory through Dense Panoramic Ray-Conditioning (DPRC) and Geometry-aware Memory Augmentation (GMA).Then, a three-stage training pipeline is introduced to progressively optimize each component. To better evaluate physical consistency under large-scale spatial variations and diverse illumination conditions, where existing datasets are relatively stable, we construct World360, a large-scale dataset consisting of both real-world video clips collected via panoramic unmanned aerial vehicles and high-quality simulated clips generated by AirSim360.Extensive experiments on World360 demonstrate the effectiveness of PanoWorld, outperforming alternative methods by a large margin.Our models, training code, and dataset will be publicly available. More information can be found on our project page: https://lihaoy-ux.github.io/panoworld-page/.
Haoyuan Li, Dizhe Zhang, Yuemei Zhou +7
1Insta360 Research · Institute of Automation Chinese Academy of Sciences · 3Tsinghua University +2
We present PanoWorld, a panoramic video world model that generates geometry-consistent 360° video from a single image and a caption. Existing panoramic video methods optimize primarily for visual realism and do not explicitly constrain the underlying 3D scene state, producing outputs that appear plausible yet exhibit inconsistent depth, broken correspondences, and implausible motion across the spherical surface. We address this gap by framing panoramic video generation as a geometry- and dynamics-consistent latent state modeling problem rather than pure visual synthesis. Building on a pre-trained perspective video world model, we introduce two lightweight regularizers: a depth consistency loss against pseudo ground-truth panoramic depth, and a trajectory consistency loss that supervises the 3D world-frame positions of tracked points across time. We further apply spherical-geometry-aware adaptation to the conditioning and positional encoding. We additionally introduce PanoGeo, a unified geometry-aware panoramic video dataset with consistent depth, trajectory, and prompt annotations across diverse real and synthetic sources, used for both training and stratified evaluation. Experiments show that PanoWorld improves geometric consistency over prior panoramic generation methods while maintaining competitive visual realism, establishing that panoramic video generation must be treated as a geometric modeling problem to support the holistic spatial understanding requirements of embodied AI applications. Code is available at https://github.com/ostadabbas/PanoWorld.
Interactive panoramic video generation aims to synthesize immersive 360\textdegree{} videos that remain visually coherent while following user-specified camera trajectories during exploration. However, progress is limited by a coupled data-and-model gap: existing panoramic video datasets are often short, weakly annotated, or lack camera trajectories, while existing camera-controlled video generation models are designed for perspective videos and do not directly support panoramic geometry. In this paper, we introduce MUGEN and Wan360 to address these limitations. MUGEN is a large-scale real-world panoramic video dataset tailored to interactive 360-degree world exploration, comprising over 1,300 hours of at least 4K panoramic videos with rich semantic and geometric annotations. Built on MUGEN, we further present Wan360, a camera-controllable interactive panoramic video generation model. Panoramic videos are commonly represented by EquiRectangular Projection (ERP), which unfolds a spherical 360-degree view into a rectangular frame with cyclic longitude seams and pole distortions. To this end, Wan360 introduces three parameter-free ERP-aware components: periodic longitude RoPE for seam-consistent positional encoding, ERP-aware padding for reducing boundary artifacts, and random roll yaw for consistent learning. For camera control, Wan360 uses a panoramic Plücker embedding that represents camera motion with ERP rays rather than perspective pinhole rays. Experiments show that MUGEN serves as a data foundation for panoramic world exploration, and that Wan360 enables high-quality, temporally coherent, camera-controllable 360-degree video generation.
Jiaming Tan, Zhen Li, Shuwei Shi +6
Alaya Lab · Beijing Institute of Technology · Shanghai Innovation Institute