We introduce HY-World 2.0, a multi-modal world model framework that advances our prior project HY-World 1.0. HY-World 2.0 accommodates diverse input modalities, including text prompts, single-view images, multi-view images, and videos, and produces 3D world representations. With text or single-view image inputs, the model performs world generation, synthesizing high-fidelity, navigable 3D Gaussian Splatting (3DGS) scenes. This is achieved through a four-stage method: a) Panorama Generation with HY-Pano 2.0, b) Trajectory Planning with WorldNav, c) World Expansion with WorldStereo 2.0, and d) World Composition with WorldMirror 2.0. Specifically, we introduce key innovations to enhance panorama fidelity, enable 3D scene understanding and planning, and upgrade WorldStereo, our keyframe-based view generation model with consistent memory. We also upgrade WorldMirror, a feed-forward model for universal 3D prediction, by refining model architecture and learning strategy, enabling world reconstruction from multi-view images or videos. Also, we introduce WorldLens, a high-performance 3DGS rendering platform featuring a flexible engine-agnostic architecture, automatic IBL lighting, efficient collision detection, and training-rendering co-design, enabling interactive exploration of 3D worlds with character support. Extensive experiments demonstrate that HY-World 2.0 achieves state-of-the-art performance on several benchmarks among open-source approaches, delivering results comparable to the closed-source model Marble. We release all model weights, code, and technical details to facilitate reproducibility and support further research on 3D world models.
Figures & tables
Figure 1
Figure 1: Versatile applications of HY-World 2.0. Our framework unifies world generation (synthesizing immersive, explorable 3D worlds from text or single-view images) and world reconstruction (recovering 3D representation from multi-view inputs). These capabilities empower diverse applications, including robotics simulation, game development, and environment mapping.
Figure 2: Architecture of HY-World 2.0. Our framework presents a four-stage process to transform multi-modal inputs into immersive 3D worlds: (1) initializing the world via Panorama Generation , (2) deriving exploration camera paths through Trajectory Planning , (3) expanding the world observations via memory-driven World Expansion , and (4) constructing the final 3DGS assets using World Composition . The core model/algorithm used in each stage is denoted at the bottom.
Figure 3: Overview of the panorama generation architecture of HY-Pano 2.0. The Left side shows the framework pipeline of panorama generation, while the right side illustrates the circle padding (latent space) and the pixel blending (pixel space) for seamless panorama generation.
Figure 4: The initial scene parsing for trajectory planning. We obtain panoramic point clouds, meshes, semantic masks, and NavMesh via several pioneering works [ 67 , 9 , 23 , 50 ] .
Figure 5: Illustration of five modes of trajectories planned in WorldNav. Some trajectories are omitted for a simplified visualization.
Regular
Surrounding
Recon-Aware
Wandering
Aerial
Total
Max Number
9
5
10
3
8
35
Attached to Object
×
✓
✓
×
–
–
Iterative
×
×
✓
×
×
–
Table 1: Trajectory details of WorldNav. The aerial category comprises both surrounding and wandering trajectories. Note that the maximum number for surrounding and reconstruct-aware trajectories is determined by the count of object segments detected within the panorama.
Figure 6: Three training stages of WorldStereo 2.0 , progressively enabling camera control, memory-based consistency, and fast inference.
Figure 7: Overall pipeline of WorldStereo 2.0. (a) The main Video Diffusion Transformer (DiT) branch is enhanced by the retrieval-based improved Spatial-Stereo Memory (SSM++) for fine-grained consistency. (b) The camera control branch is guided by the panoramic point cloud, serving as Global-Geometric Memory (GGM) to confirm precise camera trajectory following and geometry-aware consistency. Here, we omit the VAE encoding/decoding for simplicity.
Table 2: Different video generation schemes for 3D reconstruction. Native video diffusion models (VDMs) need to produce long trajectories in a single pass to cover diverse viewpoints as much as possible. Autoregressive (AR) models sequentially generate long videos. WorldStereo 2.0 achieves multiple consistent generations based on a high-fidelity keyframe latent space with complementary viewpoints and memory mechanisms for subsequent reconstruction.
Figure 8: Reconstruction and novel-view generation with different VAE variants. Keyframe-VAE preserves appearance consistency in reconstructions and substantially improves the fidelity of generated novel views, particularly under large viewpoint changes. Please zoom in for details.
Figure 9: Keyframe-VAE in WorldStereo 2.0 versus a standard Video-VAE [ 64 ] . Unlike (a) Video-VAE, which performs spatio-temporal compression, (b) Keyframe-VAE applies spatial-only compression to better preserve high-frequency details and reduce artifacts essentially caused by Video-VAE encoding ( e.g . , motion blur and geometric distortion). Specifically, Keyframe-VAE loops the causal padding-based image encoding over (1+Tkf) times with a sparse frame set ( Tkf≪Tvid ) that spans the same viewpoint changes by sampling at larger temporal intervals.
Figure 10: Data construction of WorldStereo 2.0 memory training. (a) The global point clouds built for the GGM training with one reference view and Tg=2 target views. (b) Trajectory retrieval strategies for SSM++ training tailored to dataset characteristics: temporally misaligned retrieval for existing multi-view data (top), and multi-trajectory retrieval for synthetic data (bottom).
Figure 11: Illustration of the RoPE [ 59 ] modification in SSM++. Target frames are spatially concatenated with their corresponding retrieved reference views along the horizontal axis (resulting in width 2W ). Crucially, each retrieved view inherits the temporal index of its paired target frame before being fed into the main DiT branch.
Figure 12: Model architecture of WorldMirror 2.0, which is a unified feed-forward model that takes multi-view images with optional geometric priors (camera poses, intrinsics, depth maps) as input, and simultaneously predicts dense point clouds, depth maps, surface normals, camera parameters, and 3DGS through a shared Transformer backbone with task-specific DPT decoder heads.
Component
WorldMirror 1.0
WorldMirror 2.0
Model Improvements ( Section 6.2 )
Position Encoding
Absolute RoPE
Normalized RoPE
Depth Supervision
GT depth only
GT depth + GT/Pseudo normal
Invalid Pixel Modeling
Confidence only
Confidence + Depth mask head
Acceleration
None
Token/Frame SP + BF16 + FSDP
Data Improvements ( Section 6.3 )
Table 3: Comparison between WorldMirror 1.0 and WorldMirror 2.0. Improvements are organized by model architecture ( Section 6.2 ), training data, and training strategy ( Section 6.5 ). The bottom section summarizes the resulting capability gains.
Figure 13: Analysis of normalized position encoding. (a) Average cosine similarity of center-point RoPE encodings to other resolutions. Normalized RoPE maintains high cross-resolution consistency ( >0.95 ), while standard RoPE degrades significantly. (b) and (c) show the mean and standard deviation of encoding values across resolutions, respectively. Normalized RoPE exhibits near-constant statistics, whereas standard RoPE shows systematic mean drift, confirming that normalization converts position extrapolation into interpolation.
Figure 14: The pipeline of depth alignment. We align the WorldMirror depth estimated from generated keyframes with the panoramic point cloud. The point cloud is rendered as geometric guidance, identifying reliable regions to supervise the linear alignment. The outlier detection process effectively eliminated and corrected the alignment coefficients based on global statistics.
Figure 15: Comparison of reconstructed point clouds. (a) Reference panoramic point cloud estimated by MoGe2 [ 67 ] . (b-d) Results from state-of-the-art feedforward reconstruction methods: Mapanything [ 32 ] , DepthAnything3 (DA3) [ 40 ] , and WorldMirror 2.0. (e) WorldMirror 2.0 with the proposed depth alignment. Note that for all feedforward methods (b-d), the 10% of points with the lowest confidence are filtered out, and sky points are manually cropped for better visualization.
Figure 16: Visual results of panorama generation by HY-Pano 2.0. Our model supports both text and images of various resolutions as inputs.
Figure 17: Qualitative comparison on the text-to-panorama (T2P) task. Our method outperforms previous approaches in terms of layout coherence, fine-grained details, and overall visual aesthetics.
Figure 18: Qualitative comparison on the image-to-panorama (I2P) task. Our method outperforms previous approaches in extension plausibility, content richness, and overall quality.
Text-to-Panorama (T2P)
Image-to-Panorama (I2P)
Metric
DiT360
Matrix3D
HY-World 1.0
HY-Pano 2.0
CubeDiff
GenEx
HY-World 1.0
HY-Pano 2.0
CLIP-T / CLIP-I ↑
0.248
0.238
0.250
0.258
0.828
0.831
0.831
0.844
Q-Align Qual (Persp) ↑
3.788
2.983
3.992
4.103
2.938
2.917
3.317
4.026
Q-Align Qual (Equi) ↑
4.436
4.258
4.493
4.403
3.814
3.868
4.076
4.277
Q-Align Aes (Persp) ↑
2.882
2.126
3.404
3.376
2.319
2.445
2.638
3.208
Q-Align Aes (Equi) ↑
4.072
3.880
4.186
4.247
3.645
3.646
3.767
4.056
Table 4: Quantitative comparisons on text-to-panorama (T2P) and image-to-panorama (I2P).
Figure 19: Qualitative ablation results of trajectory planning. Relying solely on panoramic views (b) results in severe artifacts and incomplete geometry. By sequentially integrating views generated from (c) regular, (d) surrounding & reconstruction, (e) wandering, and (f) aerial trajectories, our method progressively eliminates blind spots, refines complex object structures, and enhances overall scene completeness. Please zoom in for details.
Methods
Tanks-and-Temples [ 35 ]
MipNeRF360 [ 5 ]
Precision ↑
Recall ↑
F1-Score ↑
AUC ↑
Precision ↑
Recall ↑
F1-Score ↑
AUC ↑
SEVA [ 87 ]
33.59
35.34
36.73
51.03
22.38
55.63
28.75
46.81
Gen3C [ 54 ]
46.73
25.51
31.24
42.44
23.28
75.37
35.26
52.10
Lyra [ 4 ]
50.38
28.67
32.54
43.05
30.02
58.60
36.05
49.89
FlashWorld [ 39 ]
26.58
20.72
22.29
30.45
35.97
53.77
42.60
53.86
WorldStereo 2.0
43.62
41.02
41.43
58.19
43.19
65.32
51.27
65.79
Table 5: Point cloud results of one-view-generated 3D reconstruction. Precision measures the percentage of generated points that fall within a distance threshold of ground-truth points, while Recall measures the percentage of ground-truth points covered by generated points. The F1-score is their harmonic mean, and AUC denotes the area under the curve across varying distance thresholds.
Methods
Camera Metrics
Visual Quality
RotErr ↓
TransErr ↓
ATE ↓
Q-Align ↑
CLIP-IQA+ ↑
Laion-Aes ↑
CLIP-I ↑
SEVA [ 87 ]
1.690
1.578
2.879
3.232
0.479
4.623
77.16
Gen3C [ 54 ]
0.944
1.580
2.789
3.353
0.489
4.863
82.33
WorldPlay [ 60 ]
3.481
1.288
2.722
3.628
0.552
5.103
86.79
WorldCompass [ 70 ]
3.452
1.068
2.379
3.615
0.548
5.111
85.51
WorldStereo [ 62 ] ∗
0.762
1.245
2.141
4.149
0.547
5.257
89.05
Table 6: Quantitative results of camera control capability. We evaluate performance across camera metrics and visual quality. Methods labeled with ∗ indicate that the model is only trained with camera control (domain-adaption) without memory capabilities.
Frozen Parts
VAE Types
Camera Metrics
Visual Quality
User Study
RotErr ↓
TransErr ↓
ATE ↓
Q-Align ↑
CLIP-IQA+ ↑
Laion-Aes ↑
Camera ↑
Quality ↑
Main DiT
Video-VAE
0.762
1.245
2.141
4.149
0.547
5.257
84.85%
46.46%
Main DiT
Keyframe-VAE
0.768
1.149
2.027
4.060
0.520
5.210
–
–
None
Keyframe-VAE
0.578
1.115
2.245
4.237
0.554
5.278
93.81%
60.61%
Cross-Attn
Keyframe-VAE
0.684
1.243
2.111
4.181
0.538
5.235
93.13%
60.95%
Cross-Attn + FFN
Keyframe-VAE
0.492
0.968
1.768
4.205
0.544
5.266
92.44%
64.39%
Table 7: Camera control ablation studies of WorldStereo 2.0. We evaluate performance across camera metrics, visual quality, and user study. Given the misalignment between the commonly used metrics and human perception, we prioritize user study results for model selection. The red row indicates the baseline (camera control of WorldStereo [ 62 ] ), while the blue row denotes our final domain-adaption setting.
Configuration
Photometric Metrics
Consistency
Camera Metrics
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR m↑
SSIM m↑
RotErr ↓
TransErr ↓
ATE ↓
Camera control baseline
16.13
0.474
0.349
28.81
0.448
0.396
0.053
0.071
A
GGM and SSM++ (A)
20.94
0.640
0.170
30.27
0.623
0.407
0.047
0.046
B
Trainable FFN (A,B)
21.56
0.667
0.162
30.44
0.624
0.351
0.036
0.035
C
Pointcloud augmentation (A,B,C)
21.36
0.632
0.163
30.72
0.619
0.360
0.050
0.053
D
Reference augmentation (A,B,C,D)
20.86
0.639
0.165
30.66
0.636
0.322
0.049
0.067
Table 8: Memory and distillation ablation studies of WorldStereo 2.0. PSNR m and SSIM m are computed within valid warping mask regions to evaluate consistency. The configuration A ∗ denotes a variant where the SSM incorporates reference features via temporal concatenation rather than spatial concatenation. Settings of our final memory and distilled models are colorized in green and yellow .
Figure 20: Point clouds and 3DGS comparisons with video2world [ 21 ] . “Pcd.” denotes point cloud. For each scene, both methods are evaluated within 300 images generated by WorldStereo 2.0.
Voxel Downsample
Adaptive Densification
MaskGaussian
GS Number
PSNR ↑
SSIM ↑
LPIPS ↓
6.000M
25.176
0.751
0.209
✓
1.000M
24.504
0.720
0.276
✓
✓
5.254M
25.158
0.750
0.210
✓
✓
✓
1.383M
25.017
0.747
0.216
✓
✓ †
✓
1.381M
25.023
0.747
0.215
Table 9: 3DGS ablation studies averaged over 10 scenes. † denotes that the adaptive densification is restricted to non-sky regions.
Figure 21: Results of the overall world generation pipeline. Each scene is visualized across two rows. The top row displays, from left to right: the generated panorama , the aligned point clouds , a global overview of splattings , and the extracted coarse mesh . The bottom row showcases novel views rendered from various viewpoints.
Figure 22: Interactive exploration within the generated 3D worlds of HY-World 2.0. By controlling virtual agents, users can navigate complex geometric structures ( e.g . , stairs and indoor layouts) with real-time collision detection and physically plausible feedback, demonstrating the readiness of our results for interactive applications.
Figure 23: Qualitative comparison with Marble [ 72 ] using the same panoramic inputs. The input panoramas are displayed on the left. For each scene, we present novel view renderings from the generated 3DGS models of both methods. Compared to Marble, our approach achieves higher fidelity to the input conditions, sharper textures, and superior geometric consistency across diverse viewpoints. Please zoom in for details.
Figure 24: Qualitative comparison with Marble [ 72 ] using the same input image. (a) displays input perspective images. For each scene, we compare both (b) generated panoramas and (c) 3DGS renderings. Compared to Marble, our approach better adheres to the input views while achieving comparable quality and superior completeness in 3DGS. Please zoom in for details.
Stage
Panorama
Trajectory Plan
World Expansion
Recon and Align
3DGS
Total
Time (sec)
15s
182s
286s
102s
127s
712s
Table 10: Runtime of each component in HY-World 2.0 for one single world generation. “Recon” means reconstruction. All efficiency evaluations are performed using NVIDIA H20 GPUs.
Method
7-Scenes (scene)
NRGBD (scene)
DTU (object)
Acc. ↓
Comp. ↓
Acc. ↓
Comp. ↓
Acc. ↓
Comp. ↓
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
Fast3R [ 77 ]
0.096
0.065
0.145
0.093
0.135
0.091
0.163
0.104
3.340
1.919
2.929
1.125
CUT3R [ 66 ]
0.094
0.051
0.101
0.050
0.104
0.041
0.079
0.031
4.742
2.600
3.400
1.316
FLARE [ 86 ]
0.085
0.058
0.142
0.104
0.053
0.024
0.051
0.025
2.541
1.468
3.174
1.420
VGGT [ 65 ]
0.046
0.026
0.057
0.034
0.051
0.029
0.066
0.038
1.338
0.779
1.896
0.992
Table 11: Point map reconstruction on 7-Scenes, NRGBD, and DTU. Both Acc. ( ↓ ) and Comp. ( ↓ ) are lower-is-better. Baseline methods are evaluated at M resolution. WorldMirror 2.0 generalizes across multiple resolutions. “+ all priors” denotes additionally providing camera pose, intrinsics, and depth as input. “L/M/H”: low ( 182×252 ), medium ( 378×518 ), high ( 756×1036 ) resolution. Best results are highlighted.
Method
Camera Pose (Avg.)
Depth (Avg.)
AUC@30 ↑
RTA@30 ↑
AbsRel ↓
δ<1.25↑
Fast3R [ 77 ]
61.68
81.86
0.353
0.666
CUT3R [ 66 ]
81.47
95.10
0.260
0.704
FLARE [ 86 ]
80.01
95.23
0.445
0.551
VGGT [ 65 ]
77.62
93.13
0.256
0.789
π3 [ 69 ]
85.90
95.62
0.151
0.805
Table 12: Results of camera pose estimation and depth estimation (left); Results of novel view synthesis (right). Camera pose and depth are evaluated on RealEstate10K; novel view synthesis is averaged across RealEstate10K and DL3DV. ↑ / ↓ indicate higher-/lower-is-better. Baseline methods are evaluated at M resolution. WorldMirror 2.0 generalizes across multiple resolutions. “L/M/H” denote low / medium / high inference resolution. Best results are highlighted.
Method
ScanNet
NYUv2
iBims-1
mean ↓
med ↓
22.5∘↑
mean ↓
med ↓
22.5∘↑
mean ↓
med ↓
22.5∘↑
OASIS [ 11 ]
32.8
28.5
38.5
29.2
23.4
48.4
32.6
24.6
46.6
EESNU [ 2 ]
–
–
–
16.2
8.5
77.2
20.0
8.4
73.4
Omnidata v1 [ 16 ]
22.9
12.3
66.1
23.1
12.9
66.3
19.0
7.5
76.1
Omnidata v2 [ 31 ]
16.2
8.5
79.5
17.2
9.7
76.5
18.2
7.0
77.4
DSine [ 3 ]
16.2
8.3
78.7
16.4
8.4
77.7
17.1
6.1
79.0
Table 13: Surface normal estimation on ScanNet, NYUv2, and iBims-1. We compare with both regression-based and diffusion-based approaches. ↑ / ↓ indicate higher-/lower-is-better. Baseline methods are evaluated at M resolution. WorldMirror 2.0 generalizes across multiple resolutions. “L/M/H” denote low / medium / high inference resolution. Best results are highlighted.
Figure 25: Visual comparison of WorldMirror 1.0 and 2.0. We compare predicted surface normals and reconstructed point clouds. WorldMirror 2.0 produces more accurate normals with finer structural details and more consistent multi-view point clouds.
Figure 26: Multi-resolution point cloud comparison of WorldMirror 1.0 and 2.0. We evaluate under dense (32 views, top) and sparse (8 views, bottom) settings at different inference resolutions. WorldMirror 1.0 degrades severely at high resolution, while WorldMirror 2.0 maintains consistent reconstruction quality across all resolutions.
Figure 27: Comparison with Pow3R and MapAnything under different prior conditions. All methods are evaluated at high resolution ( 756×1036 ). Results are averaged on 7-Scenes, NRGBD, and DTU datasets. Pow3R (pro) refers to the original Pow3R with Procrustes alignment. WorldMirror 2.0 demonstrates stronger geometric reasoning and 3D prior integration capabilities at high resolution.
Configuration
#GPUs
32 views
64 views
128 views
256 views
Mem. (GB)
Time (s)
Mem. (GB)
Time (s)
Mem. (GB)
Time (s)
Mem. (GB)
Time (s)
Baseline (FP32, 1 GPU)
1
24.95
2.45
38.56
6.27
59.26
18.00
OOM
OOM
+ BF16
1
15.10
2.11
25.06
5.65
41.73
16.96
75.05
56.96
+ SP ( × 2 GPUs)
2
26.32
1.59
41.31
3.73
64.75
10.53
OOM
OOM
+ SP ( × 4 GPUs)
4
25.55
1.09
39.71
2.38
61.53
6.27
OOM
OOM
+ SP + BF16 ( × 4 GPUs)
4
15.81
0.96
26.44
2.21
44.47
5.65
80.54
17.69
Table 14: Inference efficiency of WorldMirror 2.0. We report GPU memory (GB) and wall-clock time (s) for different numbers of input views at 518×378 resolution. “Baseline” refers to single-GPU FP32 inference (WorldMirror 1.0 setting). SP denotes Token/Frame Sequence Parallelism. All measurements on NVIDIA H20 GPUs.
We present ABot-3DWorld 0, a universal multimodal 3D world model that turns text, image, and video inputs into high-fidelity, explorable 3D worlds. At the heart of our framework is a unified Spatial Generative Primitive (SGP), a compact tuple of a high-quality panorama and a spatial point cloud that delivers an efficient description of any 3D space. Multimodal inputs are first lifted into this primitive; a 3D-consistent panoramic video generator then explores the primitive along a planned trajectory; finally, our panoramic video reconstruction engine converts the generated video into a clean, photorealistic 3D Gaussian Splatting (3DGS) world. This pipeline covers two regimes: rich inputs (multi-view sets, casual video) are lifted into the SGP through a geometry-rigorous recovery that mirrors the observed scene, while a single image or sentence is completed generatively into a creative world. The result is one low-barrier engine for general 3D content creation that further anchors generated worlds to geographic points of interest, enabling map-native spatial exploration at consumer scale. Experiments show that ABot-3DWorld 0 sets the state of the art among open-source methods and demonstrates stronger scene fidelity than Marble under rich multimodal inputs.
We propose Puffin-World, a unified multimodal architecture that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules. To reliably construct and interact with 3D worlds, our framework jointly models three native world states: physics (gravity field and latitude), geometry (depth), and appearance (image), together with a unified Omni-Camera representation that supports diverse tasks and flexible motions. Beyond modeling these states, we introduce a strategy for propagating physical dynamics across future frames. By grounding absolute camera properties in the real world, Puffin-World enables physically consistent and visually stable world generation. We further couple appearance and geometry within a single generative process, jointly synthesizing each future view and reconstructing its underlying geometry. This unified paradigm enables interleaved closed-loop applications requiring synergy across multiple tasks, including mimic and self-calibrated world exploration. To scale Puffin-World to complex scenarios, we construct Puffin-16M, comprising 15 million vision-language-camera triplets and 1 million trajectories featuring various and challenging motions. To foster further research in this area, we released the code, models, and datasets.
Kang Liao, Yihang Luo, Xiao-Ming Wu +7
1S-Lab, Nanyang Technological University · University of Michigan · 3Beijing Jiaotong University +1
Given the remarkable ability of 2D foundation image models to generate high-fidelity outputs, we investigate a fundamental question: do 2D foundation image models inherently possess 3D world model capabilities? To answer this, we systematically evaluate multiple state-of-the-art image generation models and Vision-Language Models (VLMs) on the task of 3D world synthesis. To harness and benchmark their potential implicit 3D capability, we propose an agentic framing to facilitate 3D world generation. Our approach employs a multi-agent architecture: a VLM-based director that formulates prompts to guide image synthesis, a generator that synthesizes new image views, and a VLM-backed two-step verifier that evaluates and selectively curates generated frames from both 2D image and 3D reconstruction space. Crucially, we demonstrate that our agentic approach provides coherent and robust 3D reconstruction, producing output scenes that can be explored by rendering novel views. Through extensive experiments across various foundation models, we demonstrate that 2D models do indeed encapsulate a grasp of 3D worlds. By exploiting this understanding, our method successfully synthesizes expansive, realistic, and 3D-consistent worlds.