Video world models should maintain a world that keeps evolving as the viewpoint changes. Yet when the camera turns away and returns, current models often freeze the unobserved scene, lose objects, or change their appearance, because content outside the view has no representation that keeps changing. We present PanoEvo, which uses a Panoramic Evolver to evolve the world state in the latent space of a panoramic video diffusion model, where every viewing direction is present at every time step, so the scene evolves whether or not it is observed. Existing panoramic video models are not trained to evolve a scene from a single image or the objects within it, so we build PanoEvo-Data, an event-centric dataset from real 360° videos, and train the Panoramic Evolver coarse-to-fine as an image-to-panoramic-video model. Since each view covers only a small part of the low-resolution panorama, directly projected views are blurry and often show low-quality details or structural errors. A Perspective Renderer, built on a perspective video diffusion model, reads this state via latent-space projection and direction-restricted cross-attention to render sharp, high-quality views. PanoEvo achieves the best overall score among 23 publicly evaluated models on WRBench and the highest total score on STEVO-Bench. Code, models, and data will be released.
Figures & tables
Figure 2: Overview of PanoEvo. (a) E turns an image I0 and optional text y into an evolving panoramic latent W , the world state, from which R renders a video V along any trajectory C . (b) E is fine-tuned in two stages and writes observed cells back from zref . (c) R reads W via projected latent maps and frustum cross-attention, where each video token attends to panoramic tokens within ρ of its ray.
Figure 3: Construction of PanoEvo-Data. A shared quality gate feeds three branches, two for the stages of E and one for R .
Coarse
Event
Render
Clips
20,812
1,400
20,526 ( × 5 traj.)
Source videos
–
846
–
Resolution
ERP 512×1024
ERP 512×1024
native ( ≥ 3840 wide)
Text
label / VLM caption
detailed / brief / empty
label / empty
Annotation
rule-based
events, verified
trajectories
Used for
E , Completion
E , Evolution
R
Table 1: PanoEvo-Data. Three subsets from real 360 ∘ videos, split by source video and sharing no time windows.
Visible
Re-observed
Content
Spatial–
Model
Params
Cam Align.
Integ.
Spatial
State
Spatial
State
Avg.
Avg.
State Avg.
Source-video condition
InSpatio-World 14B [ 19 ]
14B
.980
.700
.769
.624
.729
.643
.741
.693
.691
ReCamMaster [ 3 ]
1.3B
.937
.623
.653
.493
.662
.609
.663
.608
.604
HyDRA [ 9 ]
1.3B
.855
.611
.587
.460
.468
.410
.565
.507
.481
Geometry-cache condition
Table 2: Results on WRBench. Baselines are grouped by viewpoint condition as in [ 29 ] . Params: backbone size (total for MoE); for PanoEvo, both DiTs with added modules. Content Avg. averages Integ. and the four spatial/state metrics, and Spatial–State Avg. only the latter. The other nine officially evaluated models are low-sample and listed in the supplement. Best and second best: red and blue .
Figure 4: Qualitative comparison. Views at t=20 (turning away), 38 (farthest), and 76 (returned), with the subject boxed.
Model
State Progress
Physical Plaus.
Coherence
Obs. Control
Action Control
Total
Video models
HunyuanVideo 1.5 [ 48 ]
4.1
42.1
59.1
31.2
81.0
217.5
Wan2.2 [ 40 ]
7.7
52.0
58.4
46.2
76.5
240.8
CogVideoX [ 51 ]
1.4
68.5
67.1
22.2
75.5
234.7
Camera-controlled models
Gen3C [ 34 ]
0.0
30.6
82.4
90.6
57.6
261.2
Table 3: Results on STEVO-Bench (%). Baseline scores are the official leaderboard values reported in ReMind [ 50 ] .
Cam
Visible
Re-observed
E
Align.
Integ.
Spat.
State
Spat.
State
Avg.
Δ Avg.
Zero-shot [ 20 ]
.924
.645
.677
.531
.687
.625
.682
–
+ Completion
.922
.661
.743
.599
.740
.654
.720
+.038
+ Evolution
.964
.704
.788
.663
.771
.683
.762
+.042
Table 4: Training stages of E on WRBench. All rows use direct projection to isolate E . Δ Avg.: change over the previous row.
Metric
w/o Renderer
PanoEvo
Δ
WRBench
Integ.
.704
.720
+.016
Visible spatial
.788
.799
+.011
Visible state
.663
.651
− .012
Re-observed state
.683
.646
− .037
Avg.
.762
.753
− .009
Quality
VBench (8-dim mean)
.818
.854
+.036
Table 5: Effect of the Perspective Renderer. Both rows use the same Panoramic Evolver output, and “w/o Renderer” projects the decoded panorama directly. Sharpness: Laplacian variance.
Figure 5: Qualitative ablation. Each row adds one component (columns: t=20 , 38, 76). Zero-shot, the woman walks past instead of setting her handbag down, the Completion stage has her squat as her bag vanishes, the Evolution stage completes the action out of view, and the Renderer sharpens it without altering the state.
Cam
Vis.
Re-observed
Variant
Align.
State
Spat.
State
Avg.
Imaging
Full
.950
.651
.754
.646
.753
.729
w/o read-out loss
.935
.647
.748
.643
.751
.719
w/o frustum attention
.931
.645
.742
.637
.745
.714
w/o projected latent
.902
.642
.732
.625
.729
.703
Table 6: Ablation of the Perspective Renderer components. Each variant removes one component, is trained for 13,500 steps, and is tested on the same E output. Removing frustum attention also removes the read-out loss. Imaging: VBench imaging quality.
Params
New trainable
TFLOPs
Peak memory
Panoramic Evolver E
2.40B
0.106B
7,752 (38,759)
10.5 GiB
Perspective Renderer R
1.80B
0.385B
27,820
8.8 GiB
PanoEvo
4.20B
0.491B
35,572 (66,579)
10.5 GiB
Table 7: Inference cost for one 77-frame view. Parameters and FLOPs are those of the diffusion transformers, excluding encoders and VAEs. FLOPs of E are amortized over the five camera settings rendered from one panorama in WRBench (unamortized in parentheses). Peak memory is that of diffusion sampling.
Interactive video world models must maintain broad scene context under camera motion while producing high-fidelity observations with low latency. Existing approaches face a representation trade-off: perspective models operate on local views and must preserve off-screen content over long rollouts, whereas broader spatial coverage is typically obtained by synthesizing full-sphere videos or constructing explicit 3D representations. Motivated by the complementary roles of global context and selective local acuity in visual perception, we present AlayaVista, a camera-controllable streaming video world model that decouples panoramic world evolution from perspective observation synthesis. Given a single perspective image, AlayaVista constructs a 360-degree scene prior using a pretrained panorama expansion model and then evolves the scene as a camera-conditioned panoramic latent state. A latent viewport renderer maps this state to the requested perspective video latents, while a perspective refiner restores details, suppresses artifacts, and performs super-resolution. To support efficient streaming, we adapt the panoramic generator to chunk-autoregressive generation and distill both panoramic generation and perspective refinement into few-step processes. To provide the supervision required by this design, we construct MUGEN, a large-scale real-world panoramic video dataset containing 1,318 hours of videos at resolutions of at least 4K, together with rich semantic and geometric annotations.
Jiaming Tan, Mingliang Zhai, Zhen Li +3
Alaya Lab · Beijing Institute of Technology · The University of Tokyo
We present PanoWorld, a panoramic video world model that generates geometry-consistent 360° video from a single image and a caption. Existing panoramic video methods optimize primarily for visual realism and do not explicitly constrain the underlying 3D scene state, producing outputs that appear plausible yet exhibit inconsistent depth, broken correspondences, and implausible motion across the spherical surface. We address this gap by framing panoramic video generation as a geometry- and dynamics-consistent latent state modeling problem rather than pure visual synthesis. Building on a pre-trained perspective video world model, we introduce two lightweight regularizers: a depth consistency loss against pseudo ground-truth panoramic depth, and a trajectory consistency loss that supervises the 3D world-frame positions of tracked points across time. We further apply spherical-geometry-aware adaptation to the conditioning and positional encoding. We additionally introduce PanoGeo, a unified geometry-aware panoramic video dataset with consistent depth, trajectory, and prompt annotations across diverse real and synthetic sources, used for both training and stratified evaluation. Experiments show that PanoWorld improves geometric consistency over prior panoramic generation methods while maintaining competitive visual realism, establishing that panoramic video generation must be treated as a geometric modeling problem to support the holistic spatial understanding requirements of embodied AI applications. Code is available at https://github.com/ostadabbas/PanoWorld.
In this work, we aim to address the challenge of long-range memory in panoramic world models by exploiting the rotation-equivariant property of omnidirectional representations, where rotation can be treated as an implicit geometric transformation.Building on this insight, we propose PanoWorld, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory through Dense Panoramic Ray-Conditioning (DPRC) and Geometry-aware Memory Augmentation (GMA).Then, a three-stage training pipeline is introduced to progressively optimize each component. To better evaluate physical consistency under large-scale spatial variations and diverse illumination conditions, where existing datasets are relatively stable, we construct World360, a large-scale dataset consisting of both real-world video clips collected via panoramic unmanned aerial vehicles and high-quality simulated clips generated by AirSim360.Extensive experiments on World360 demonstrate the effectiveness of PanoWorld, outperforming alternative methods by a large margin.Our models, training code, and dataset will be publicly available. More information can be found on our project page: https://lihaoy-ux.github.io/panoworld-page/.
Haoyuan Li, Dizhe Zhang, Yuemei Zhou +7
1Insta360 Research · Institute of Automation Chinese Academy of Sciences · 3Tsinghua University +2