Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

Dongxu Wei; Qi Xu; Zhiqi Li; Hangning Zhou; Cong Qiu; Hailong Qin; Mu Yang; Zhaopeng Cui

cs.CVApr 13, 2026

Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

Authors: Dongxu Wei, Qi Xu, Zhiqi Li, Hangning Zhou, Cong Qiu, Hailong Qin, Mu Yang, Zhaopeng Cui,

Organizations: Westlake University · Afari Intelligent Drive · Zhejiang University

Abstract

3D scene generation has long been dominated by 2D multi-view or video diffusion models. This is due not only to the lack of scene-level 3D latent representation, but also to the fact that most scene-level 3D visual data exists in the form of multi-view images or videos, which are naturally compatible with 2D diffusion architectures. Typically, these 2D-based approaches degrade 3D spatial extrapolation to 2D temporal extension, which introduces two fundamental issues: (i) representing 3D scenes via 2D views leads to significant representation redundancy, and (ii) latent space rooted in 2D inherently limits the spatial consistency of the generated 3D scenes. In this paper, we propose, for the first time, to perform 3D scene generation directly within an implicit 3D latent space to address these limitations. First, we repurpose frozen 2D representation encoders to construct our 3D Representation Autoencoder (3DRAE), which grounds view-coupled 2D semantic representations into a view-decoupled 3D latent representation. This enables representing 3D scenes observed from arbitrary numbers of views--at any resolution and aspect ratio--with fixed complexity and rich semantics. Then we introduce 3D Diffusion Transformer (3DDiT), which performs diffusion modeling in this 3D latent space, achieving remarkably efficient and spatially consistent 3D scene generation while supporting diverse conditioning configurations. Moreover, since our approach directly generates a 3D scene representation, it can be decoded to images and optional point maps along arbitrary camera trajectories without requiring per-trajectory diffusion sampling pass, which is common in 2D-based approaches.

Figures & tables

Figure 1: 3D-Grounded Scene Generation. (a) Our 3DRAE repurposes frozen 2D representation encoders to ground any number of views into fixed-length 3D latent tokens, which can be queried to decode images and point maps of arbitrary views. (b) Previous 2D diffusion-based methods perform diffusion modeling in the view-coupled 2D latent space, resulting in computational redundancy and limited spatial consistency. (c) Our 3DDiT performs diffusion modeling in the view-decoupled 3D latent space, enabling 3D-grounded generation with superior efficiency and spatial consistency.

Figure 2: We empirically find that overlapping regions across multiple views correspond to the same set of tokens. Therefore, we employ fixed-length 3D latent tokens to eliminate such multi-view redundancy.

Figure 3: The Architecture of Our 3DRAE.

Figure 4: Qualitative Comparison. Top : Comparison with state-of-the-art methods. Bottom : Comparison between our 3DRAE and 3DDiT to demonstrate the effectiveness of diffusion modeling. The red bounding boxes highlight failure cases in baselines.

Panel of Figure 5 — Figure 5: Qualitative Ablations on 2D Encoders.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Network Architecture
Frozen 2D Encoder	DepthAnything3 (DA3) [ 41 ]
	Model Config	ViT-Base
	Patch Size	14
	Hidden Dim	768
	DINOv2 [ 50 ]
	Model Config	ViT-Base

Appendix

Table 5: Details of 3DRAE architecture and hyperparameters.

(a) Network Architecture
LightningDiT Backbone [ 83 ]	3DDiT-S
	Depth	12
	Hidden Dim	384
	Num Heads	6
	3DDiT-B
	Depth	12

Appendix

Table 6: Details of 3DDiT architecture and hyperparameters.

Name	Domain	#Scenes	#Frames	Weight
CO3D [ 54 ]	Indoor/Object	27K	2.6M	7.1%
WildRGBD [ 76 ]	Indoor/Object	19k	6.8M	5.1%
ARKitScenes [ 5 ]	Indoor/Scene	3.4K	0.9M	13.2%
ScanNet [ 17 ]	Indoor/Scene	1.2K	1.9M	6.3%
ScanNet ++ [ 87 ]	Indoor/Scene	855	0.7M	3.4%
Taskonomy [ 91 ]	Indoor/Scene	537	4.6M	4.2%

Appendix

Table 7: Training Datasets.

Figure 8: Generated 3D Scenes under Single-View Conditioning. The red bounding boxes denote the conditional views. Generated point clouds are shown on the right.

Figure 9: Generated 3D Scenes under Sparse-View Conditioning. The red bounding boxes denote the conditional views. Generated point clouds are shown on the right.

Figure 10: Generated 3D Scenes without Appearance Conditioning. Using the same camera trajectory condition with different random seeds, we generate 3D scenes with varied appearances.

Figure 11: Zero-Shot Generated 3D Scenes under Single-View Conditioning. Rows 1 $\sim$ 2: results on Tanks and Temples dataset [ 33 ] . Rows 3 $\sim$ 4: results on Mip-NeRF 360 dataset [ 4 ] . Rows 5 $\sim$ 6 : results on DTU dataset [ 1 ] . The red bounding boxes denote the conditional views. Generated point clouds are shown on the right.

Figure 12: Representing Densely Observed 3D Scenes using 3DRAE. Our 3DRAE can represent the same 3D scenes but observed by different numbers of views (16 vs. 128 views) using the same set of 1K tokens. The red arrows highlight conditioning camera trajectories that specify the spatial extents of the generated scenes.

3D Scene Generation

3D scene generation creates structured, explorable environments from text, images, maps, or other inputs.

See all 183 papers

OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder
Mar 17, 2026Sensen Gao, Zhaoqing Wang, Qihang Cao +53D Scene Generation 3D Diffusion Models
Mohamed bin Zayed University of Artificial Intelligence · AISphere · Shanghai Jiao Tong University +3
SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space
Oct 14, 2026Shuofeng Sun, Wenzhao Zheng, Yunpeng Zhang +33D Scene Generation 3D Scene Representation
Beijing University of Posts and Telecommunications · University of California, Berkeley · PhiGent Robotics +1
SceneTok: A Compressed, Diffusable Token Space for 3D Scenes
Feb 21, 2026Mohammad Asim, Christopher Wewer, Jan Eric Lenssen3D Scene Representation 3D Scene Generation
Max Planck Institute for Informatics, Saarland Informatics Campus

Mar 17, 2026J/K move · Enter open · S save

OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder

Sensen Gao, Zhaoqing Wang, Qihang Cao, Dongdong Yu+4

Mohamed bin Zayed University of Artificial Intelligence · AISphere · Shanghai Jiao Tong University+3

3D Scene Generation 3D Diffusion Models 3D Scene Representation 3D Representation Learning Multi-View Consistency

Open paper AskPDF arXiv

Existing diffusion-based 3D scene generation methods primarily operate in 2D image/video latent spaces, which makes maintaining cross-view appearance and geometric consistency inherently challenging. To bridge this gap, we present OneWorld, a framework that performs diffusion directly within a coherent 3D representation space. Central to our approach is the 3D Unified Representation Autoencoder (3D-URAE); it leverages pretrained 3D foundation models and augments their geometry-centric nature by injecting appearance and distilling semantics into a unified 3D latent space. Furthermore, we introduce token-level Cross-View-Correspondence (CVC) consistency loss to explicitly enforce structural alignment across views, and propose Manifold-Drift Forcing (MDF) to mitigate train-inference exposure bias and shape a robust 3D manifold by mixing drifted and original representations. Comprehensive experiments demonstrate that OneWorld generates high-quality 3D scenes with superior cross-view consistency compared to state-of-the-art 2D-based methods. Our code will be available at https://github.com/SensenGao/OneWorld.

Abstract

3D Scene Generation

Explore similar work

OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder

SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space

SceneTok: A Compressed, Diffusable Token Space for 3D Scenes