OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder

Sensen Gao; Zhaoqing Wang; Qihang Cao; Dongdong Yu; Changhu Wang; Tongliang Liu; Mingming Gong; Jiawang Bian

cs.CVMar 17, 2026

OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder

Authors: Sensen Gao, Zhaoqing Wang, Qihang Cao, Dongdong Yu, Changhu Wang, Tongliang Liu, Mingming Gong, Jiawang Bian

Organizations: Mohamed bin Zayed University of Artificial Intelligence · AISphere · Shanghai Jiao Tong University · University of Syndey ·

Abstract

Existing diffusion-based 3D scene generation methods primarily operate in 2D image/video latent spaces, which makes maintaining cross-view appearance and geometric consistency inherently challenging. To bridge this gap, we present OneWorld, a framework that performs diffusion directly within a coherent 3D representation space. Central to our approach is the 3D Unified Representation Autoencoder (3D-URAE); it leverages pretrained 3D foundation models and augments their geometry-centric nature by injecting appearance and distilling semantics into a unified 3D latent space. Furthermore, we introduce token-level Cross-View-Correspondence (CVC) consistency loss to explicitly enforce structural alignment across views, and propose Manifold-Drift Forcing (MDF) to mitigate train-inference exposure bias and shape a robust 3D manifold by mixing drifted and original representations. Comprehensive experiments demonstrate that OneWorld generates high-quality 3D scenes with superior cross-view consistency compared to state-of-the-art 2D-based methods. Our code will be available at https://github.com/SensenGao/OneWorld.

Figures & tables

Figure 1: (a) OneWorld generates 3DGS from a single view and renders novel views. (b) Architecture: FlashWorld [ 32 ] diffuses in compressed video latents; Gen3R [ 20 ] compresses 3D features to align a 3D foundation encoder with video latents but can only generate geometry and appearance separately, rather than jointly. OneWorld generates directly in a unified 3D representation space, without compression or separate generation. (c) Performance comparison on WorldScore [ 10 ] and DL3DV [ 35 ] .

Figure 2: Overview of the proposed OneWorld framework. (a) We construct a unified 3D representation space by introducing appearance injection and semantic distillation . (b) During DiT [ 42 ] training, we incorporate cross-view correspondence , preserving cross-view geometric token correspondences from the target view to the conditioned view. (c) Manifold-drift forcing: we augment the original 3D manifold by mixing ground-truth 3D features with sampled 3D features, enabling a more robust 3D decoder.

Table 1: Ablation analysis of appearance injection and semantic distillation . (a) We analyze the effect of appearance injection from a reconstruction perspective. (b) We examine the benefit of semantic distillation for training the generative model.

Figure 3: Visualization of the effects of appearance injection and semantic distillation. We visualize the reconstruction results with and without appearance injection and present feature visualizations with and without semantic distillation .

Figure 4: Qualitative visual results on one-view-based novel view generation.

Figure 5: Visualization of 3DGS and rendered novel views.

Figure 6: Comparison of 3D scenes generated by different methods: Gen3R [ 20 ] uses point clouds, while FlashWorld [ 32 ] and OneWorld use 3DGS.

Method	RealEstate10K						DL3DV-10K
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	I2V Subj. $\uparrow$	I2V BG $\uparrow$	I.Q. $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	I2V Subj. $\uparrow$	I2V BG $\uparrow$	I.Q. $\uparrow$
LVSM [ 24 ]	18.54	0.694	0.336	0.993	0.991	0.487	15.18	0.516	0.482	0.962	0.966	0.421
Gen3C [ 45 ]	19.88	0.697	0.271	0.991	0.990	0.514	15.79	0.531	0.497	0.941	0.953	0.414
GF [ 62 ]	15.97	0.518	0.424	0.986	0.976	0.553	11.62	0.322	0.621	0.932	0.929	0.533
Aether [ 76 ]	16.13	0.611	0.413	0.991	0.989	0.536	13.37	0.492	0.566	0.957	0.963	0.451
FlashWorld [ 32 ]	20.18	0.724	0.256	0.993	0.994	0.584	16.02	0.566	0.451	0.964	0.969	0.538

Table 2: Experimental results on 1-view-based novel view generation. We report the performance of different methods on the RealEstate10K and DL3DV datasets.

Method	WorldScore-Indoor						DL3DV
Method	3D Consist.	Photo. Consist.	Obj. Cont.	Cont. Align.	Style Consist.	Subj. Quality	3D Consist.	Photo. Consist.	Obj. Cont.	Cont. Align.	Style Consist.	Subj. Quality
LVSM [ 24 ]	74.32	63.94	47.21	23.79	66.02	36.38	64.37	55.58	40.21	18.88	57.65	29.59
Gen3C [ 45 ]	74.11	78.74	51.36	27.12	71.18	40.05	69.41	69.96	44.18	21.92	62.23	33.07
GF [ 62 ]	78.05	69.21	46.95	24.62	68.71	38.66	67.12	58.44	41.67	20.35	59.92	31.74
Aether [ 76 ]	77.82	66.66	49.28	25.04	69.06	38.17	67.47	57.98	42.02	19.83	60.37	31.11
FlashWorld [ 32 ]	83.57	80.19	49.61	49.27	75.32	54.09	76.74	72.76	43.25	42.32	69.08	46.62

Table 3: WorldScore-style reference-free evaluation results. We report results on WorldScore-Indoor and on DL3DV as an outdoor benchmark.

Figure 7: Qualitative visualization of the ablation study under the 1-view-based NVS setting. We compare the full model with variants without Cross-View Correspondence and without Manifold-Drift Forcing.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	I2V Subj. $\uparrow$	I2V BG $\uparrow$	I.Q. $\uparrow$
OneWorld (full)	21.57	0.735	0.231	0.993	0.995	0.604
w/o CVC	19.10	0.682	0.284	0.989	0.990	0.566
w/o MDF	20.59	0.714	0.256	0.992	0.994	0.589

Table 4: Ablation on generation components evaluated on RealEstate10K under the 1-view NVS setting.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Margins ( $m_{1}{=}m_{2}{=}m$ , fix $\lambda_{\mathrm{sem}}{=}0.1$ )					Semantic weight (fix $m_{1}{=}m_{2}{=}0.05$ )
$m$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Sem. Sim. $\uparrow$	$\lambda_{\mathrm{sem}}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Sem. Sim. $\uparrow$
0.00	26.67	0.918	0.122	0.986	0.05	28.31	0.933	0.101	0.947
0.05	28.19	0.932	0.102	0.984	0.10	28.19	0.932	0.102	0.984
0.10	28.19	0.930	0.105	0.917	0.20	26.58	0.916	0.125	0.987

Appendix

Table 5: Detailed ablations on semantic distillation hyperparameters. Left: margins with

m_{1}{=}m_{2}

(fix

\lambda_{\mathrm{sem}}{=}0.1

). Right: semantic weight

\lambda_{\mathrm{sem}}

(fix

m_{1}{=}m_{2}{=}0.05

). We report PSNR/SSIM/LPIPS and semantic similarity (Sem. Sim.).

Setting	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	I2V Subj. $\uparrow$	I2V BG $\uparrow$	I.Q. $\uparrow$
Ablation on threshold $\tau$ (fix $\lambda_{\mathrm{cvc}}{=}0.2$ )
$\tau{=}0.80$	17.62	0.645	0.316	0.976	0.981	0.541
$\tau{=}0.90$	19.10	0.682	0.284	0.989	0.990	0.566
$\tau{=}0.95$	18.21	0.661	0.301	0.982	0.986	0.553
Ablation on CVC weight $\lambda_{\mathrm{cvc}}$ (fix $\tau{=}0.9$ )
$\lambda_{\mathrm{cvc}}{=}0.10$	18.05	0.657	0.305	0.981	0.986	0.552

Appendix

Table 6: Ablation on CVC hyperparameters evaluated on RealEstate10K under the 1-view NVS setting. Top: threshold

\tau

(fix

\lambda_{\mathrm{cvc}}{=}0.2

). Bottom: CVC weight

\lambda_{\mathrm{cvc}}

(fix

\tau{=}0.9

Figure 8: Prediction space comparison on single-scene overfitting. We overfit a single training scene for 2000 iterations and visualize the fitting results at different training steps. Compared to $v$ -prediction, $\mathbf{x}_{0}$ -prediction converges faster and reaches higher-fidelity structure and appearance under the same optimization budget.

Method	RealEstate10K			DL3DV
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
PixelSplat [ 3 ]	26.08	0.871	0.169	22.26	0.753	0.179
MVSplat [ 5 ]	26.02	0.872	0.177	22.61	0.759	0.174
DepthSplat [ 64 ]	27.02	0.879	0.174	23.39	0.796	0.135
AnySplat [ 23 ]	25.97	0.873	0.181	21.12	0.736	0.204
3D-URAE	28.19	0.932	0.102	24.68	0.816	0.140

Appendix

Table 7: Feed-forward 3D Gaussian Splatting comparison. We report PSNR, SSIM, and LPIPS on RealEstate10K and DL3DV.

Figure 9: Additional visualizations (I). 3DGS structure visualizations and the corresponding novel-view RGB renderings and depth map visualizations for OneWorld-generated 3D scenes.

Figure 10: Additional visualizations (II). More 3DGS structure visualizations with novel-view RGB renderings and depth map visualizations for OneWorld-generated 3D scenes.

Figure 11: More examples. Additional 3DGS-rendered novel-view RGB images and the corresponding depth map visualizations for OneWorld-generated 3D scenes.

3D Scene Generation

3D scene generation creates structured, explorable environments from text, images, maps, or other inputs.

See all 185 papers

Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale
Apr 13, 2026Dongxu Wei, Qi Xu, Zhiqi Li +63D Scene Generation 3D Diffusion Models
Westlake University · Afari Intelligent Drive · Zhejiang University
GTA: Advancing Image-to-3D World Generation via Geometry Then Appearance Video Diffusion
May 13, 2026Hanxin Zhu, Cong Wang, Peiyan Tu +4Multi-View Consistency Video Diffusion Models
School of Information Science and Technology, University of Science and Technology of China, Hefei, China. · Zhongguancun Academy, Beijing, China. · the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. +3
PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space
Jul 6, 2026Sensen Gao, Zhaoqing Wang, Qihang Cao +33D Reconstruction Pixel-Space Diffusion Models
Nanyang Technological University · AISphere

Apr 13, 2026J/K move · Enter open · S save

Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

Dongxu Wei, Qi Xu, Zhiqi Li, Hangning Zhou+5

Westlake University · Afari Intelligent Drive · Zhejiang University

3D Scene Generation 3D Diffusion Models 3D Scene Representation 3D Representation Learning Diffusion Transformer

Open paper AskPDF arXiv

Scene-level 3D generation has long been dominated by 2D multi-view or video diffusion models. Typically, these 2D-based approaches perform scene generation in a 2D latent space, which introduces two fundamental issues: (i) representing 3D scenes via 2D views leads to significant representation redundancy, and (ii) latent space rooted in 2D inherently limits the spatial consistency of the generated scenes. In this paper, we propose to perform 3D scene generation directly within a high-dimensional implicit 3D latent space derived from powerful 2D foundation models (e.g., DINOv2, SigLIP2, DA3). Then we tame diffusion transformers to perform diffusion modeling directly in the 3D latent space, enabling 3D-grounded scene generation. In our experiments, we not only demonstrate the advantages of our method over 2D-based approaches in terms of efficiency and spatial consistency, but also comprehensively analyze the impact of 3D latent representation on 3D scene generation performance. Furthermore, we validate the favorable scalability of our method with respect to representation and model capacity.

Abstract

3D Scene Generation

Explore similar work

Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

GTA: Advancing Image-to-3D World Generation via Geometry Then Appearance Video Diffusion

PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space