SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space

Shuofeng Sun; Wenzhao Zheng; Yunpeng Zhang; Yi Wu; Linqing Zhao; Haibin Yan

cs.CVOct 14, 2026

SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space

Authors: Shuofeng Sun, Wenzhao Zheng, Yunpeng Zhang, Yi Wu, Linqing Zhao, Haibin Yan

Organizations: Beijing University of Posts and Telecommunications · University of California, Berkeley · PhiGent Robotics · SynapX

Abstract

Recent video world models have increasingly adopted 3D memory mechanisms to integrate historical observations into a lightweight and persistent scene representation. This naturally raises a complementary question: if 3D representations can serve as an effective mechanism for memorizing observed scenes, can they also be directly used as the latent space for reconstructing and generating scenes? Motivated by this idea, we represent a scene as a compact and unstructured 3D-anchored latent space (SceneAE). Each latent combines an explicit 3D position with an implicit content feature and is spatially grounded to describe its local scene region, forming a sparse latent point cloud. Unlike conventional video-VAE latents that describe how a scene appears from particular viewpoints, our representation directly describes the scene itself, making it naturally applicable to both reconstruction and generation. For reconstruction, once the scene has been generated, our persistent 3D representation can be projected and decoded into RGB-D observations by different target cameras, without the need to regenerate a video for each new camera trajectory. For generation, we leverage the geometry-content separation to first generate camera-controlled 3D geometry and then synthesize content conditioned on it, reducing camera-appearance entanglement and improving camera controllability. Experiments show that this 3D latent achieves superior generation quality, computational cost, and camera controllability.

Figures & tables

Figure 1: (a) Previous VAEs construct view-bound grid latent representations, which can only reconstruct fixed observations. (b) We construct a compact 1D latent representation anchored to explicit 3D positions, which can achieve efficient scene reconstruction and scene generation tasks.

Figure 2: Overall Framework of the RGB-D Decoder. By leveraging the 3D properties of the latents, we can efficiently achieve RGBD rendering from arbitrary novel viewpoints.

Figure 3: Overall Framework of the Geometry-Content decoupled DiT. We first control the generation of the future geometric layout via the camera, and then control the generation of texture content via the geometric layout, thereby achieving more accurate camera control.

Cond.	Method	DL3DV-10K					RealEstate10K					GFLOPs	ms
Cond.	Method	PSNR $\uparrow$	SSIM $\uparrow$	Sub $\uparrow$	Bg $\uparrow$	MVC $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	Sub $\uparrow$	Bg $\uparrow$	MVC $\uparrow$	GFLOPs	ms
1-view	Aether	14.1	0.36	0.81	0.84	0.19	15.4	0.47	0.82	0.89	0.21	48874	364
	SceneTok	-	-	-	-	-	15.3	0.48	0.84	0.91	0.24	-	-
	Gen3R	14.7	0.39	0.85	0.88	0.21	15.8	0.53	0.86	0.92	0.24	18105	174
	GLD	15.6	0.45	0.87	0.91	0.22	16.9	0.58	0.89	0.94	0.28	23203	285
	Ours	15.4	0.45	0.88	0.93	0.28	17.1	0.60	0.90	0.94	0.33	8348	134

Table 1: Quantitative Comparison of Appearance Generation. We report the PSNR, SSIM, Subject Consistency, Background Consistency, Multi-view Consistency from VBench. We also report the GFLOPs and ms per step to evaluate the efficiency of the models.

Cond.	Method	DL3DV-10K			RealEstate10K
Cond.	Method	Accuracy $\downarrow$	Completeness $\downarrow$	CD $\downarrow$	Accuracy $\downarrow$	Completeness $\downarrow$	CD $\downarrow$
1view	Gen3R	1.74	1.92	2.01	0.75	0.91	0.96
	GLD	0.87	1.18	1.04	0.36	0.49	0.46
	Ours	0.34	0.55	0.44	0.13	0.22	0.18
2view	Gen3R	1.23	1.46	1.38	0.59	0.75	0.74
	GLD	0.63	0.91	0.85	0.27	0.44	0.39

Table 2: Quantitative Comparison of Geometry Generation. We report the average accuracy, completeness, and Chamfer Distance. The evaluation GT for Gen3R is provided by VGGT ( Wang et al., 2025 ) , while the evaluation GT for ours and GLD is provided by DA3 ( Lin et al., 2025 ) . This avoids the influence of performance differences among geometry foundation models.

Table 6

Figure 4: Qualitative Comparison of Different Methods under the 1-view setting.

Method	Input Reconstruction				Novel View Synthesis
	DL3DV		RE10K		DL3DV		RE10K
	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$
Wan-VAE	26.1	0.79	30.9	0.88	–	–	–	–
GLD	26.7	0.84	31.1	0.91	–	–	–	–
DA3-Giant	–	–	–	–	13.4	0.35	14.4	0.52
LVSM	–	–	–	–	17.8	0.53	21.0	0.71

Table 5: Quantitative comparison of RGB reconstruction. We report PSNR and SSIM for reconstructing the input views (Input Reconstruction) and for rendering 20 other viewpoints of the scene from 9 context frames (Novel View Synthesis). “–” denotes not supported or not reported.

Figure 5: Visualization of the Latent Encoder-Decoder Reconstruction.

Table 10

Figure 6: Qualitative comparison of geometry reconstruction. Red circles mark distorted regions.

Table 12

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visualization of the Decoder Reconstruction.

Table 14

Figure 8: Visualization in the 1-view setting with camera condition.

Figure 9: Visualization in the 1-view setting without camera condition.

Method	TartanAir				ScanNet++
Method	PSNR	SSIM	Accuracy	Completeness	PSNR	SSIM	Accuracy	Completeness
Aether	12.1	0.43	0.55	0.81	13.4	0.45	0.50	0.71
Gen3R	13.9	0.47	0.52	0.74	14.2	0.49	0.46	0.70
Ours	15.2	0.53	0.22	0.49	15.8	0.55	0.16	0.37

Appendix

Table 12: Quantitative Comparison of Appearance Generation on Out-of-Distribution Datasets with 1-view setting. We report 2D metrics (PSNR, SSIM) and 3D metrics (Accuracy, Completeness).

Table 18

Figure 10: Visualization in the 2-view setting with camera condition.

Figure 11: Visualization in the 2-view setting without camera condition.

3D Scene Generation

3D scene generation creates structured, explorable environments from text, images, maps, or other inputs.

See all 183 papers

Lyra 2.0: Explorable Generative 3D Worlds
Apr 14, 2026Tianchang Shen, Sherwin Bahmani, Kai He +123D Scene Generation 3D-Aware Video Generation
NVIDIA
Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale
Apr 13, 2026Dongxu Wei, Qi Xu, Zhiqi Li +63D Scene Generation 3D Diffusion Models
Westlake University · Afari Intelligent Drive · Zhejiang University
GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation
Sep 21, 2026Jiahao Lu, Minghao Yin, Wenbo Hu +53D Scene Generation Geometric Consistency in Video Generation
The Hong Kong University of Science and Technology · ARC Lab, Tencent IEG · The University of Hong Kong +1

Apr 14, 2026J/K move · Enter open · S save

Lyra 2.0: Explorable Generative 3D Worlds

Tianchang Shen, Sherwin Bahmani, Kai He, Sangeetha Grama Srinivasan+11

NVIDIA

3D Scene Generation 3D-Aware Video Generation Long-Horizon Video Generation Temporal Consistency in Video Generation Camera-Controlled Video Generation Memory-Augmented Video Generation

Open paper AskPDF arXiv

Recent advances in video generation enable a new paradigm for 3D scene creation: generating camera-controlled videos that simulate scene walkthroughs, then lifting them to 3D via feed-forward reconstruction techniques. This generative reconstruction approach combines the visual fidelity and creative capacity of video models with 3D outputs ready for real-time rendering and simulation. Scaling to large, complex environments requires 3D-consistent video generation over long camera trajectories with large viewpoint changes and location revisits, a setting where current video models degrade quickly. Existing methods for long-horizon generation are fundamentally limited by two forms of degradation: spatial forgetting and temporal drifting. As exploration proceeds, previously observed regions fall outside the model's temporal context, forcing the model to hallucinate structures when revisited. Meanwhile, autoregressive generation accumulates small synthesis errors over time, gradually distorting scene appearance and geometry. We present Lyra 2.0, a framework for generating persistent, explorable 3D worlds at scale. To address spatial forgetting, we maintain per-frame 3D geometry and use it solely for information routing -- retrieving relevant past frames and establishing dense correspondences with the target viewpoints -- while relying on the generative prior for appearance synthesis. To address temporal drifting, we train with self-augmented histories that expose the model to its own degraded outputs, teaching it to correct drift rather than propagate it. Together, these enable substantially longer and 3D-consistent video trajectories, which we leverage to fine-tune feed-forward reconstruction models that reliably recover high-quality 3D scenes.

Abstract

3D Scene Generation

Explore similar work

Lyra 2.0: Explorable Generative 3D Worlds

Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation