3D scene generation has long been dominated by 2D multi-view or video diffusion models. This is due not only to the lack of scene-level 3D latent representation, but also to the fact that most scene-level 3D visual data exists in the form of multi-view images or videos, which are naturally compatible with 2D diffusion architectures. Typically, these 2D-based approaches degrade 3D spatial extrapolation to 2D temporal extension, which introduces two fundamental issues: (i) representing 3D scenes via 2D views leads to significant representation redundancy, and (ii) latent space rooted in 2D inherently limits the spatial consistency of the generated 3D scenes. In this paper, we propose, for the first time, to perform 3D scene generation directly within an implicit 3D latent space to address these limitations. First, we repurpose frozen 2D representation encoders to construct our 3D Representation Autoencoder (3DRAE), which grounds view-coupled 2D semantic representations into a view-decoupled 3D latent representation. This enables representing 3D scenes observed from arbitrary numbers of views--at any resolution and aspect ratio--with fixed complexity and rich semantics. Then we introduce 3D Diffusion Transformer (3DDiT), which performs diffusion modeling in this 3D latent space, achieving remarkably efficient and spatially consistent 3D scene generation while supporting diverse conditioning configurations. Moreover, since our approach directly generates a 3D scene representation, it can be decoded to images and optional point maps along arbitrary camera trajectories without requiring per-trajectory diffusion sampling pass, which is common in 2D-based approaches.
Figures & tables
Figure 1: 3D-Grounded Scene Generation. (a) Our 3DRAE repurposes frozen 2D representation encoders to ground any number of views into fixed-length 3D latent tokens, which can be queried to decode images and point maps of arbitrary views. (b) Previous 2D diffusion-based methods perform diffusion modeling in the view-coupled 2D latent space, resulting in computational redundancy and limited spatial consistency. (c) Our 3DDiT performs diffusion modeling in the view-decoupled 3D latent space, enabling 3D-grounded generation with superior efficiency and spatial consistency.
Figure 2: We empirically find that overlapping regions across multiple views correspond to the same set of tokens. Therefore, we employ fixed-length 3D latent tokens to eliminate such multi-view redundancy.
Figure 3: The Architecture of Our 3DRAE.
Figure 4: Qualitative Comparison. Top : Comparison with state-of-the-art methods. Bottom : Comparison between our 3DRAE and 3DDiT to demonstrate the effectiveness of diffusion modeling. The red bounding boxes highlight failure cases in baselines.
Figure 5: Qualitative Ablations on 2D Encoders.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
(a) Network Architecture
Frozen 2D Encoder
DepthAnything3 (DA3) [ 41 ]
Model Config
ViT-Base
Patch Size
14
Hidden Dim
768
DINOv2 [ 50 ]
Model Config
ViT-Base
Appendix
Table 5: Details of 3DRAE architecture and hyperparameters.
Figure 6: Details of View Masking.
Figure 7: Impact of Adversarial Loss.
(a) Network Architecture
LightningDiT Backbone [ 83 ]
3DDiT-S
Depth
12
Hidden Dim
384
Num Heads
6
3DDiT-B
Depth
12
Appendix
Table 6: Details of 3DDiT architecture and hyperparameters.
Name
Domain
#Scenes
#Frames
Weight
CO3D [ 54 ]
Indoor/Object
27K
2.6M
7.1%
WildRGBD [ 76 ]
Indoor/Object
19k
6.8M
5.1%
ARKitScenes [ 5 ]
Indoor/Scene
3.4K
0.9M
13.2%
ScanNet [ 17 ]
Indoor/Scene
1.2K
1.9M
6.3%
ScanNet ++ [ 87 ]
Indoor/Scene
855
0.7M
3.4%
Taskonomy [ 91 ]
Indoor/Scene
537
4.6M
4.2%
Appendix
Table 7: Training Datasets.
Figure 8: Generated 3D Scenes under Single-View Conditioning. The red bounding boxes denote the conditional views. Generated point clouds are shown on the right.
Figure 9: Generated 3D Scenes under Sparse-View Conditioning. The red bounding boxes denote the conditional views. Generated point clouds are shown on the right.
Figure 10: Generated 3D Scenes without Appearance Conditioning. Using the same camera trajectory condition with different random seeds, we generate 3D scenes with varied appearances.
Figure 11: Zero-Shot Generated 3D Scenes under Single-View Conditioning. Rows 1 ∼ 2: results on Tanks and Temples dataset [ 33 ] . Rows 3 ∼ 4: results on Mip-NeRF 360 dataset [ 4 ] . Rows 5 ∼ 6 : results on DTU dataset [ 1 ] . The red bounding boxes denote the conditional views. Generated point clouds are shown on the right.
Figure 12: Representing Densely Observed 3D Scenes using 3DRAE. Our 3DRAE can represent the same 3D scenes but observed by different numbers of views (16 vs. 128 views) using the same set of 1K tokens. The red arrows highlight conditioning camera trajectories that specify the spatial extents of the generated scenes.
Existing diffusion-based 3D scene generation methods primarily operate in 2D image/video latent spaces, which makes maintaining cross-view appearance and geometric consistency inherently challenging. To bridge this gap, we present OneWorld, a framework that performs diffusion directly within a coherent 3D representation space. Central to our approach is the 3D Unified Representation Autoencoder (3D-URAE); it leverages pretrained 3D foundation models and augments their geometry-centric nature by injecting appearance and distilling semantics into a unified 3D latent space. Furthermore, we introduce token-level Cross-View-Correspondence (CVC) consistency loss to explicitly enforce structural alignment across views, and propose Manifold-Drift Forcing (MDF) to mitigate train-inference exposure bias and shape a robust 3D manifold by mixing drifted and original representations. Comprehensive experiments demonstrate that OneWorld generates high-quality 3D scenes with superior cross-view consistency compared to state-of-the-art 2D-based methods. Our code will be available at https://github.com/SensenGao/OneWorld.
Sensen Gao, Zhaoqing Wang, Qihang Cao +5
Mohamed bin Zayed University of Artificial Intelligence · AISphere · Shanghai Jiao Tong University +3
Recent video world models have increasingly adopted 3D memory mechanisms to integrate historical observations into a lightweight and persistent scene representation. This naturally raises a complementary question: if 3D representations can serve as an effective mechanism for memorizing observed scenes, can they also be directly used as the latent space for reconstructing and generating scenes? Motivated by this idea, we represent a scene as a compact and unstructured 3D-anchored latent space (SceneAE). Each latent combines an explicit 3D position with an implicit content feature and is spatially grounded to describe its local scene region, forming a sparse latent point cloud. Unlike conventional video-VAE latents that describe how a scene appears from particular viewpoints, our representation directly describes the scene itself, making it naturally applicable to both reconstruction and generation. For reconstruction, once the scene has been generated, our persistent 3D representation can be projected and decoded into RGB-D observations by different target cameras, without the need to regenerate a video for each new camera trajectory. For generation, we leverage the geometry-content separation to first generate camera-controlled 3D geometry and then synthesize content conditioned on it, reducing camera-appearance entanglement and improving camera controllability. Experiments show that this 3D latent achieves superior generation quality, computational cost, and camera controllability.
Shuofeng Sun, Wenzhao Zheng, Yunpeng Zhang +3
Beijing University of Posts and Telecommunications · University of California, Berkeley · PhiGent Robotics +1
We present SceneTok, a novel tokenizer for encoding view sets of scenes into a compressed and diffusable set of unstructured tokens. Existing approaches for 3D scene representation and generation commonly use 3D data structures or view-aligned fields. In contrast, we introduce the first method that encodes scene information into a small set of permutation-invariant tokens that is disentangled from the spatial grid. The scene tokens are predicted by a multi-view tokenizer given many context views and rendered into novel views by employing a light-weight rectified flow decoder. We show that the compression is 1-3 orders of magnitude stronger than for other representations while still reaching state-of-the-art reconstruction quality. Further, our representation can be rendered from novel trajectories, including ones deviating from the input trajectory, and we show that the decoder gracefully handles uncertainty. Finally, the highly-compressed set of unstructured latent scene tokens enables simple and efficient scene generation in 5 seconds, achieving a much better quality-speed trade-off than previous paradigms.
Mohammad Asim, Christopher Wewer, Jan Eric Lenssen
Max Planck Institute for Informatics, Saarland Informatics Campus