HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds

Team HY-World; Chenjie Cao; Xuhui Zuo; Zhenwei Wang; Yisu Zhang; Junta Wu; Zhenyang Liu; Yuning Gong

Figure 1

Figure 1: Versatile applications of HY-World 2.0. Our framework unifies world generation (synthesizing immersive, explorable 3D worlds from text or single-view images) and world reconstruction (recovering 3D representation from multi-view inputs). These capabilities empower diverse applications, including robotics simulation, game development, and environment mapping.

Figure 2: Architecture of HY-World 2.0. Our framework presents a four-stage process to transform multi-modal inputs into immersive 3D worlds: (1) initializing the world via Panorama Generation , (2) deriving exploration camera paths through Trajectory Planning , (3) expanding the world observations via memory-driven World Expansion , and (4) constructing the final 3DGS assets using World Composition . The core model/algorithm used in each stage is denoted at the bottom.

Figure 3: Overview of the panorama generation architecture of HY-Pano 2.0. The Left side shows the framework pipeline of panorama generation, while the right side illustrates the circle padding (latent space) and the pixel blending (pixel space) for seamless panorama generation.

Figure 4: The initial scene parsing for trajectory planning. We obtain panoramic point clouds, meshes, semantic masks, and NavMesh via several pioneering works [ 67 , 9 , 23 , 50 ] .

Figure 5: Illustration of five modes of trajectories planned in WorldNav. Some trajectories are omitted for a simplified visualization.

	Regular	Surrounding	Recon-Aware	Wandering	Aerial	Total
Max Number	9	5	10	3	8	35
Attached to Object	$\times$	$\checkmark$	$\checkmark$	$\times$	–	–
Iterative	$\times$	$\times$	$\checkmark$	$\times$	$\times$	–

Table 1: Trajectory details of WorldNav. The aerial category comprises both surrounding and wandering trajectories. Note that the maximum number for surrounding and reconstruct-aware trajectories is determined by the count of object segments detected within the panorama.

Figure 6: Three training stages of WorldStereo 2.0 , progressively enabling camera control, memory-based consistency, and fast inference.

Figure 7: Overall pipeline of WorldStereo 2.0. (a) The main Video Diffusion Transformer (DiT) branch is enhanced by the retrieval-based improved Spatial-Stereo Memory (SSM++) for fine-grained consistency. (b) The camera control branch is guided by the panoramic point cloud, serving as Global-Geometric Memory (GGM) to confirm precise camera trajectory following and geometry-aware consistency. Here, we omit the VAE encoding/decoding for simplicity.

Table 2: Different video generation schemes for 3D reconstruction. Native video diffusion models (VDMs) need to produce long trajectories in a single pass to cover diverse viewpoints as much as possible. Autoregressive (AR) models sequentially generate long videos. WorldStereo 2.0 achieves multiple consistent generations based on a high-fidelity keyframe latent space with complementary viewpoints and memory mechanisms for subsequent reconstruction.

Figure 8: Reconstruction and novel-view generation with different VAE variants. Keyframe-VAE preserves appearance consistency in reconstructions and substantially improves the fidelity of generated novel views, particularly under large viewpoint changes. Please zoom in for details.

Figure 9: Keyframe-VAE in WorldStereo 2.0 versus a standard Video-VAE [ 64 ] . Unlike (a) Video-VAE, which performs spatio-temporal compression, (b) Keyframe-VAE applies spatial-only compression to better preserve high-frequency details and reduce artifacts essentially caused by Video-VAE encoding ( e.g . , motion blur and geometric distortion). Specifically, Keyframe-VAE loops the causal padding-based image encoding over

(1+T_{kf})

times with a sparse frame set (

T_{kf}\ll T_{vid}

) that spans the same viewpoint changes by sampling at larger temporal intervals.

Figure 10: Data construction of WorldStereo 2.0 memory training. (a) The global point clouds built for the GGM training with one reference view and

T_{g}=2

target views. (b) Trajectory retrieval strategies for SSM++ training tailored to dataset characteristics: temporally misaligned retrieval for existing multi-view data (top), and multi-trajectory retrieval for synthetic data (bottom).

Figure 11: Illustration of the RoPE [ 59 ] modification in SSM++. Target frames are spatially concatenated with their corresponding retrieved reference views along the horizontal axis (resulting in width

2W

). Crucially, each retrieved view inherits the temporal index of its paired target frame before being fed into the main DiT branch.

Figure 12: Model architecture of WorldMirror 2.0, which is a unified feed-forward model that takes multi-view images with optional geometric priors (camera poses, intrinsics, depth maps) as input, and simultaneously predicts dense point clouds, depth maps, surface normals, camera parameters, and 3DGS through a shared Transformer backbone with task-specific DPT decoder heads.

Model Improvements ( Section 6.2 )
Component	WorldMirror 1.0	WorldMirror 2.0
Position Encoding	Absolute RoPE	Normalized RoPE
Depth Supervision	GT depth only	GT depth + GT/Pseudo normal
Invalid Pixel Modeling	Confidence only	Confidence + Depth mask head
Acceleration	None	Token/Frame SP + BF16 + FSDP
Data Improvements ( Section 6.3 )

Table 3: Comparison between WorldMirror 1.0 and WorldMirror 2.0. Improvements are organized by model architecture ( Section 6.2 ), training data, and training strategy ( Section 6.5 ). The bottom section summarizes the resulting capability gains.

Figure 13: Analysis of normalized position encoding. (a) Average cosine similarity of center-point RoPE encodings to other resolutions. Normalized RoPE maintains high cross-resolution consistency (

>0.95

), while standard RoPE degrades significantly. (b) and (c) show the mean and standard deviation of encoding values across resolutions, respectively. Normalized RoPE exhibits near-constant statistics, whereas standard RoPE shows systematic mean drift, confirming that normalization converts position extrapolation into interpolation.

Figure 14: The pipeline of depth alignment. We align the WorldMirror depth estimated from generated keyframes with the panoramic point cloud. The point cloud is rendered as geometric guidance, identifying reliable regions to supervise the linear alignment. The outlier detection process effectively eliminated and corrected the alignment coefficients based on global statistics.

Figure 15: Comparison of reconstructed point clouds. (a) Reference panoramic point cloud estimated by MoGe2 [ 67 ] . (b-d) Results from state-of-the-art feedforward reconstruction methods: Mapanything [ 32 ] , DepthAnything3 (DA3) [ 40 ] , and WorldMirror 2.0. (e) WorldMirror 2.0 with the proposed depth alignment. Note that for all feedforward methods (b-d), the 10% of points with the lowest confidence are filtered out, and sky points are manually cropped for better visualization.

Figure 16: Visual results of panorama generation by HY-Pano 2.0. Our model supports both text and images of various resolutions as inputs.

Figure 17: Qualitative comparison on the text-to-panorama (T2P) task. Our method outperforms previous approaches in terms of layout coherence, fine-grained details, and overall visual aesthetics.

Figure 18: Qualitative comparison on the image-to-panorama (I2P) task. Our method outperforms previous approaches in extension plausibility, content richness, and overall quality.

	Text-to-Panorama (T2P)				Image-to-Panorama (I2P)
Metric	DiT360	Matrix3D	HY-World 1.0	HY-Pano 2.0	CubeDiff	GenEx	HY-World 1.0	HY-Pano 2.0
CLIP-T / CLIP-I $\uparrow$	0.248	0.238	0.250	0.258	0.828	0.831	0.831	0.844
Q-Align Qual (Persp) $\uparrow$	3.788	2.983	3.992	4.103	2.938	2.917	3.317	4.026
Q-Align Qual (Equi) $\uparrow$	4.436	4.258	4.493	4.403	3.814	3.868	4.076	4.277
Q-Align Aes (Persp) $\uparrow$	2.882	2.126	3.404	3.376	2.319	2.445	2.638	3.208
Q-Align Aes (Equi) $\uparrow$	4.072	3.880	4.186	4.247	3.645	3.646	3.767	4.056

Table 4: Quantitative comparisons on text-to-panorama (T2P) and image-to-panorama (I2P).

Figure 19: Qualitative ablation results of trajectory planning. Relying solely on panoramic views (b) results in severe artifacts and incomplete geometry. By sequentially integrating views generated from (c) regular, (d) surrounding & reconstruction, (e) wandering, and (f) aerial trajectories, our method progressively eliminates blind spots, refines complex object structures, and enhances overall scene completeness. Please zoom in for details.

Methods	Tanks-and-Temples [ 35 ]				MipNeRF360 [ 5 ]
Methods	Precision $\uparrow$	Recall $\uparrow$	F1-Score $\uparrow$	AUC $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1-Score $\uparrow$	AUC $\uparrow$
SEVA [ 87 ]	33.59	35.34	36.73	51.03	22.38	55.63	28.75	46.81
Gen3C [ 54 ]	46.73	25.51	31.24	42.44	23.28	75.37	35.26	52.10
Lyra [ 4 ]	50.38	28.67	32.54	43.05	30.02	58.60	36.05	49.89
FlashWorld [ 39 ]	26.58	20.72	22.29	30.45	35.97	53.77	42.60	53.86
WorldStereo 2.0	43.62	41.02	41.43	58.19	43.19	65.32	51.27	65.79

Table 5: Point cloud results of one-view-generated 3D reconstruction. Precision measures the percentage of generated points that fall within a distance threshold of ground-truth points, while Recall measures the percentage of ground-truth points covered by generated points. The F1-score is their harmonic mean, and AUC denotes the area under the curve across varying distance thresholds.

Methods	Camera Metrics			Visual Quality
Methods	RotErr $\downarrow$	TransErr $\downarrow$	ATE $\downarrow$	Q-Align $\uparrow$	CLIP-IQA+ $\uparrow$	Laion-Aes $\uparrow$	CLIP-I $\uparrow$
SEVA [ 87 ]	1.690	1.578	2.879	3.232	0.479	4.623	77.16
Gen3C [ 54 ]	0.944	1.580	2.789	3.353	0.489	4.863	82.33
WorldPlay [ 60 ]	3.481	1.288	2.722	3.628	0.552	5.103	86.79
WorldCompass [ 70 ]	3.452	1.068	2.379	3.615	0.548	5.111	85.51
WorldStereo [ 62 ] ∗	0.762	1.245	2.141	4.149	0.547	5.257	89.05

Table 6: Quantitative results of camera control capability. We evaluate performance across camera metrics and visual quality. Methods labeled with ∗ indicate that the model is only trained with camera control (domain-adaption) without memory capabilities.

Frozen Parts	VAE Types	Camera Metrics			Visual Quality			User Study
Frozen Parts	VAE Types	RotErr $\downarrow$	TransErr $\downarrow$	ATE $\downarrow$	Q-Align $\uparrow$	CLIP-IQA+ $\uparrow$	Laion-Aes $\uparrow$	Camera $\uparrow$	Quality $\uparrow$
Main DiT	Video-VAE	0.762	1.245	2.141	4.149	0.547	5.257	84.85%	46.46%
Main DiT	Keyframe-VAE	0.768	1.149	2.027	4.060	0.520	5.210	–	–
None	Keyframe-VAE	0.578	1.115	2.245	4.237	0.554	5.278	93.81%	60.61%
Cross-Attn	Keyframe-VAE	0.684	1.243	2.111	4.181	0.538	5.235	93.13%	60.95%
Cross-Attn + FFN	Keyframe-VAE	0.492	0.968	1.768	4.205	0.544	5.266	92.44%	64.39%

Table 7: Camera control ablation studies of WorldStereo 2.0. We evaluate performance across camera metrics, visual quality, and user study. Given the misalignment between the commonly used metrics and human perception, we prioritize user study results for model selection. The red row indicates the baseline (camera control of WorldStereo [ 62 ] ), while the blue row denotes our final domain-adaption setting.

	Configuration	Photometric Metrics			Consistency		Camera Metrics
	Configuration	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR ${}_{m}\uparrow$	SSIM ${}_{m}\uparrow$	RotErr $\downarrow$	TransErr $\downarrow$	ATE $\downarrow$
	Camera control baseline	16.13	0.474	0.349	28.81	0.448	0.396	0.053	0.071
A	GGM and SSM++ (A)	20.94	0.640	0.170	30.27	0.623	0.407	0.047	0.046
B	Trainable FFN (A,B)	21.56	0.667	0.162	30.44	0.624	0.351	0.036	0.035
C	Pointcloud augmentation (A,B,C)	21.36	0.632	0.163	30.72	0.619	0.360	0.050	0.053
D	Reference augmentation (A,B,C,D)	20.86	0.639	0.165	30.66	0.636	0.322	0.049	0.067

Table 8: Memory and distillation ablation studies of WorldStereo 2.0. PSNR m and SSIM m are computed within valid warping mask regions to evaluate consistency. The configuration A ∗ denotes a variant where the SSM incorporates reference features via temporal concatenation rather than spatial concatenation. Settings of our final memory and distilled models are colorized in green and yellow .

Figure 20: Point clouds and 3DGS comparisons with video2world [ 21 ] . “Pcd.” denotes point cloud. For each scene, both methods are evaluated within 300 images generated by WorldStereo 2.0.

Voxel Downsample	Adaptive Densification	MaskGaussian	GS Number	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
			6.000M	25.176	0.751	0.209
✓			1.000M	24.504	0.720	0.276
✓	✓		5.254M	25.158	0.750	0.210
✓	✓	✓	1.383M	25.017	0.747	0.216
✓	✓ †	✓	1.381M	25.023	0.747	0.215

Table 9: 3DGS ablation studies averaged over 10 scenes.

\dagger

denotes that the adaptive densification is restricted to non-sky regions.

Figure 21: Results of the overall world generation pipeline. Each scene is visualized across two rows. The top row displays, from left to right: the generated panorama , the aligned point clouds , a global overview of splattings , and the extracted coarse mesh . The bottom row showcases novel views rendered from various viewpoints.

Figure 22: Interactive exploration within the generated 3D worlds of HY-World 2.0. By controlling virtual agents, users can navigate complex geometric structures ( e.g . , stairs and indoor layouts) with real-time collision detection and physically plausible feedback, demonstrating the readiness of our results for interactive applications.

Figure 23: Qualitative comparison with Marble [ 72 ] using the same panoramic inputs. The input panoramas are displayed on the left. For each scene, we present novel view renderings from the generated 3DGS models of both methods. Compared to Marble, our approach achieves higher fidelity to the input conditions, sharper textures, and superior geometric consistency across diverse viewpoints. Please zoom in for details.

Figure 24: Qualitative comparison with Marble [ 72 ] using the same input image. (a) displays input perspective images. For each scene, we compare both (b) generated panoramas and (c) 3DGS renderings. Compared to Marble, our approach better adheres to the input views while achieving comparable quality and superior completeness in 3DGS. Please zoom in for details.

Stage	Panorama	Trajectory Plan	World Expansion	Recon and Align	3DGS	Total
Time (sec)	15s	182s	286s	102s	127s	712s

Table 10: Runtime of each component in HY-World 2.0 for one single world generation. “Recon” means reconstruction. All efficiency evaluations are performed using NVIDIA H20 GPUs.

Method	7-Scenes (scene)				NRGBD (scene)				DTU (object)
	Acc. $\downarrow$		Comp. $\downarrow$		Acc. $\downarrow$		Comp. $\downarrow$		Acc. $\downarrow$		Comp. $\downarrow$
	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.
Fast3R [ 77 ]	0.096	0.065	0.145	0.093	0.135	0.091	0.163	0.104	3.340	1.919	2.929	1.125
CUT3R [ 66 ]	0.094	0.051	0.101	0.050	0.104	0.041	0.079	0.031	4.742	2.600	3.400	1.316
FLARE [ 86 ]	0.085	0.058	0.142	0.104	0.053	0.024	0.051	0.025	2.541	1.468	3.174	1.420
VGGT [ 65 ]	0.046	0.026	0.057	0.034	0.051	0.029	0.066	0.038	1.338	0.779	1.896	0.992

Table 11: Point map reconstruction on 7-Scenes, NRGBD, and DTU. Both Acc. (

\downarrow

) and Comp. (

\downarrow

) are lower-is-better. Baseline methods are evaluated at M resolution. WorldMirror 2.0 generalizes across multiple resolutions. “+ all priors” denotes additionally providing camera pose, intrinsics, and depth as input. “L/M/H”: low (

182{\times}252

), medium (

378{\times}518

), high (

756{\times}1036

) resolution. Best results are highlighted.

Method	Camera Pose (Avg.)		Depth (Avg.)
Method	AUC@30 $\uparrow$	RTA@30 $\uparrow$	AbsRel $\downarrow$	$\delta{<}1.25\uparrow$
Fast3R [ 77 ]	61.68	81.86	0.353	0.666
CUT3R [ 66 ]	81.47	95.10	0.260	0.704
FLARE [ 86 ]	80.01	95.23	0.445	0.551
VGGT [ 65 ]	77.62	93.13	0.256	0.789
$\pi^{3}$ [ 69 ]	85.90	95.62	0.151	0.805

Table 12: Results of camera pose estimation and depth estimation (left); Results of novel view synthesis (right). Camera pose and depth are evaluated on RealEstate10K; novel view synthesis is averaged across RealEstate10K and DL3DV.

\uparrow

/

\downarrow

indicate higher-/lower-is-better. Baseline methods are evaluated at M resolution. WorldMirror 2.0 generalizes across multiple resolutions. “L/M/H” denote low / medium / high inference resolution. Best results are highlighted.

Method	ScanNet			NYUv2			iBims-1
Method	mean $\downarrow$	med $\downarrow$	$22.5^{\circ}\uparrow$	mean $\downarrow$	med $\downarrow$	$22.5^{\circ}\uparrow$	mean $\downarrow$	med $\downarrow$	$22.5^{\circ}\uparrow$
OASIS [ 11 ]	32.8	28.5	38.5	29.2	23.4	48.4	32.6	24.6	46.6
EESNU [ 2 ]	–	–	–	16.2	8.5	77.2	20.0	8.4	73.4
Omnidata v1 [ 16 ]	22.9	12.3	66.1	23.1	12.9	66.3	19.0	7.5	76.1
Omnidata v2 [ 31 ]	16.2	8.5	79.5	17.2	9.7	76.5	18.2	7.0	77.4
DSine [ 3 ]	16.2	8.3	78.7	16.4	8.4	77.7	17.1	6.1	79.0

Table 13: Surface normal estimation on ScanNet, NYUv2, and iBims-1. We compare with both regression-based and diffusion-based approaches.

\uparrow

/

\downarrow

indicate higher-/lower-is-better. Baseline methods are evaluated at M resolution. WorldMirror 2.0 generalizes across multiple resolutions. “L/M/H” denote low / medium / high inference resolution. Best results are highlighted.

Figure 25: Visual comparison of WorldMirror 1.0 and 2.0. We compare predicted surface normals and reconstructed point clouds. WorldMirror 2.0 produces more accurate normals with finer structural details and more consistent multi-view point clouds.

Figure 26: Multi-resolution point cloud comparison of WorldMirror 1.0 and 2.0. We evaluate under dense (32 views, top) and sparse (8 views, bottom) settings at different inference resolutions. WorldMirror 1.0 degrades severely at high resolution, while WorldMirror 2.0 maintains consistent reconstruction quality across all resolutions.

Figure 27: Comparison with Pow3R and MapAnything under different prior conditions. All methods are evaluated at high resolution (

756{\times}1036

). Results are averaged on 7-Scenes, NRGBD, and DTU datasets. Pow3R (pro) refers to the original Pow3R with Procrustes alignment. WorldMirror 2.0 demonstrates stronger geometric reasoning and 3D prior integration capabilities at high resolution.

Configuration	#GPUs	32 views		64 views		128 views		256 views
Configuration	#GPUs	Mem. (GB)	Time (s)	Mem. (GB)	Time (s)	Mem. (GB)	Time (s)	Mem. (GB)	Time (s)
Baseline (FP32, 1 GPU)	1	24.95	2.45	38.56	6.27	59.26	18.00	OOM	OOM
+ BF16	1	15.10	2.11	25.06	5.65	41.73	16.96	75.05	56.96
+ SP ( $\times$ 2 GPUs)	2	26.32	1.59	41.31	3.73	64.75	10.53	OOM	OOM
+ SP ( $\times$ 4 GPUs)	4	25.55	1.09	39.71	2.38	61.53	6.27	OOM	OOM
+ SP + BF16 ( $\times$ 4 GPUs)	4	15.81	0.96	26.44	2.21	44.47	5.65	80.54	17.69

Table 14: Inference efficiency of WorldMirror 2.0. We report GPU memory (GB) and wall-clock time (s) for different numbers of input views at

518\!\times\!378

resolution. “Baseline” refers to single-GPU FP32 inference (WorldMirror 1.0 setting). SP denotes Token/Frame Sequence Parallelism. All measurements on NVIDIA H20 GPUs.

HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds

Abstract

Figures & tables

World Models

ABot-3DWorld 0: A Universal World Model to Explore Any 3D Space

Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

WorldAgents: Can Foundation Image Models be Agents for 3D World Models?

Abstract

World Models

Explore similar work

ABot-3DWorld 0: A Universal World Model to Explore Any 3D Space

Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

WorldAgents: Can Foundation Image Models be Agents for 3D World Models?