SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generation
Authors: Ling Wang, Hao-Xiang Guo, Xinzhou Wang, Fuchun Sun, Kai Sun, Pengkun Liu, Hang Xiao, Zhong Wang, Guangyuan Fu, Eric Li, Yang Liu, Yikai Wang
Organizations: Xi’an Research Institute of Hi-Tech · Tsinghua University · SkyWork AI · Tongji University · Fudan University · Beijing Normal University
We introduce SceneTransporter, an end-to-end framework for structured 3D scene generation from a single image. While existing methods generate part-level 3D objects, they often fail to organize these parts into distinct instances in open-world scenes. Through a debiased clustering probe, we reveal a critical insight: this failure stems from the lack of structural constraints within the model's internal assignment mechanism. Based on this finding, we reframe the task of structured 3D scene generation as a global correlation assignment problem. To solve this, SceneTransporter formulates and solves an entropic Optimal Transport (OT) objective within the denoising loop of the compositional DiT model. This formulation imposes two powerful structural constraints. First, the resulting transport plan gates cross-attention to enforce an exclusive, one-to-one routing of image patches to part-level 3D latents, preventing entanglement. Second, the competitive nature of the transport encourages the grouping of similar patches, a process that is further regularized by an edge-based cost, to form coherent objects and prevent fragmentation. Extensive experiments show that SceneTransporter outperforms existing methods on open-world scene generation, significantly improving instance-level coherence and geometric fidelity. Code and models will be publicly available at https://2019epwl.github.io/SceneTransporter/.
Figures & tables
Figure 1: Comparison between our end-to-end scene generation pipeline in (c) with compositional 3D latent diffusion and existing “divide and conquer” methods.
Figure 2: Qualitative Results on Vecset-based Latent Probing. Cluster and Cluster with CCA are our probes that perform in the compositional latent space of PartPacker; VAE clusters the latent obtained by encoding the fused geometry produced by PartPacker into the VAE. Colors denote part assignments.
Figure 3: Overview of the SceneTransporter pipeline. At each denoising step t , our Optimal-Transport–Guided Correlation Assignment framework formulates a global OT problem between image patches and part-level tokens within the compositional latent DiT. We compute a part-patch cost from Q/K similarity, regularized by image edges, and solve for an optimal transport plan using Sinkhorn iteration. The OT plan gates the cross attention to enforce an explicit patch-to-part routing, and the resulting gated attention map updates the latent zt . Attention maps transport over time, showing assignments becoming sharper and more instance-consistent.
Figure 4: Qualitative Comparison on Structured 3D Scene Generation across Methods. Different colors indicate different parts in the generated 3D scene.
Method
Instance Mask
Geometry Fidelity
Part Disentanglement
Inference Time (s)
ULIP ↑
ULIP-2 ↑
Uni3D ↑
IoU max↓
IoU mean↓
MIDI Huang et al. (2025)
✓
0.1397
0.2763
0.2518
0.0458
0.1642
149.68
PartCrafter Lin et al. (2025b)
✗
0.1177
0.3096
0.2635
0.0042
0.0539
157.97
PartPacker Tang et al. (2025)
✗
0.1417
0.3083
0.2887
0.0319
0.2142
47.41
Ours
✗
0.1466
0.3220
0.3021
0.0101
0.0926
54.99
Table 1: Quantitative Comparison on Structured 3D Scene Generation across Methods. Bold values indicate the best scores, while underlined values indicate the second-best scores among the fair comparison.
Figure 6
Figure 6: Qualitative Ablation Studies on the OT Plane-gated Cross Attention. Here, A_attn. and B_attn. denote the dual-volume soft attention probability maps, reshaped to the image patch grid (brighter means higher affinity). Hard affinity visualizes the argmax(A,B) patch assignments overlaid on the input image (blue → A, red → B). A_geo. and B_geo. are the geometries decoded from dual volumes, respectively, and Uni_geo. is their fused scene mesh. Row (a) shows our OT plan–gated cross-attention; row (b) shows the standard cross-attention.
Figure 7: Qualitative Ablation Studies on the OT Plan Progression over Denoising Steps. Each map visualizes the hard OT plan at a given denoising step: every cell is an image patch assigned to one volume (dark blue = A, light cyan = B). Left→right shows the OT plan’s evolution; later steps mostly stabilize with only local refinements.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Description
Value
εt
Entropic regularization weight
0.10
λedge
Edge regularization strength
0.8
γedge
Edge sensitivity
8.0
λt
Guidance strength
2.5
εg
Floor term
0.02
KOT
Number of Sinkhorn iterations
40
Appendix
Table 3: Hyperparameters of OT plan–gated cross-attention used in all experiments.
Figure 8: Qualitative Results on Structured 3D Scene Generation from Real-World Images. We use a GPT-5–based image editing model to transfer the style of real-world images, making them look like images rendered from a graphics engine.
Figure 9: Convergence of the entropic OT solver across OT-gated cross-attention layers. We plot the residuals of the dual variables and transport plan, as well as the marginal-constraint violation, for three representative OT-gated cross-attention layers.
Setting
Geometry Fidelity
Part Disentanglement
Inference Time (s)
Inference Memory (M)
ULIP ↑
ULIP-2 ↑
Uni3D ↑
IoU max↓
IoU mean↓
(a.1) w/o OT Plan–Gated Cross–Attention
0.1417
0.3083
0.2887
0.0319
0.2142
47.41
9030
(a.2) w/o Edge–Regularized Assignment Cost
0.1452
0.3164
0.2916
0.0241
0.1136
54.61
9068
(a.3) Full model
0.1466
0.3220
0.3021
0.0101
0.0926
54.99
9070
(b.1) εt=0.08
0.1460
0.3184
0.3003
0.0117
0.0914
55.30
9070
(b.2) εt=0.10∗
0.1466
0.3220
0.3021
0.0101
0.0926
54.99
9070
Appendix
Table 4: Comparison of metrics for ablation. Bold values indicate the best scores, while underlined values indicate the second-best scores among the fair comparison. Asterisk ( ∗ ) indicates the default settings in our method.
Figure 10: Effect of adding a residual vanilla cross-attention branch. Left to right: input image, PartPacker baseline, our OT-guided routing, and our OT-guided routing with an additional residual cross-attention branch (Ours_residual), which recovers the small crowded instances (e.g., boats) while preserving the improved global layout and part separation.
We present DistScene, a framework for single-image compositional 3D scene generation by jointly modeling the environment and individual objects. Unlike existing methods that represent scenes primarily as collections of objects, we model the environment as an explicit scene component to provide geometric context for object placement. Specifically, we introduce Scene-Frame Generation, which jointly generates separate environment and object components in a shared coordinate frame, allowing their geometry and relative placement to be learned together. Then we introduce Object-Centric Refinement to refine each object in a local frame with scene context. Finally, we develop Object-to-Scene Distillation to transfer pretrained object-generation priors to scene generation through automatically composed and rendered synthetic scenes. Evaluations on indoor and outdoor benchmarks demonstrate improved scene-level spatial coherence over the evaluated baselines. Project page: https://coolbeam.github.io/DistScene/
Kunming Luo, Hongyu Yan, Ken Deng +6
The Hong Kong University of Science and Technology · TeleAI, China Telecom
In this paper, we propose Extend3D, a training-free pipeline for 3D scene generation from a single image, built upon an object-centric 3D generative model. To overcome the limitations of fixed-size latent spaces in object-centric models for representing wide scenes, we extend the latent space in the x and y directions. Then, by dividing the extended latent space into overlapping patches, we apply the object-centric 3D generative model to each patch and couple them at each time step. Since patch-wise 3D generation with image conditioning requires strict spatial alignment between image and latent patches, we initialize the scene using a point cloud prior from a monocular depth estimator and iteratively refine occluded regions through SDEdit. We discovered that treating the incompleteness of 3D structure as noise during 3D refinement enables 3D completion via a concept, which we term under-noising. Furthermore, to address the sub-optimality of object-centric models for sub-scene generation, we optimize the extended latent during denoising, ensuring that the denoising trajectories remain consistent with the sub-scene dynamics. To this end, we introduce 3D-aware optimization objectives for improved geometric structure and texture fidelity. We demonstrate that our method yields better results than prior methods, as evidenced by human preference and quantitative experiments.
Existing diffusion-based 3D scene generation methods primarily operate in 2D image/video latent spaces, which makes maintaining cross-view appearance and geometric consistency inherently challenging. To bridge this gap, we present OneWorld, a framework that performs diffusion directly within a coherent 3D representation space. Central to our approach is the 3D Unified Representation Autoencoder (3D-URAE); it leverages pretrained 3D foundation models and augments their geometry-centric nature by injecting appearance and distilling semantics into a unified 3D latent space. Furthermore, we introduce token-level Cross-View-Correspondence (CVC) consistency loss to explicitly enforce structural alignment across views, and propose Manifold-Drift Forcing (MDF) to mitigate train-inference exposure bias and shape a robust 3D manifold by mixing drifted and original representations. Comprehensive experiments demonstrate that OneWorld generates high-quality 3D scenes with superior cross-view consistency compared to state-of-the-art 2D-based methods. Our code will be available at https://github.com/SensenGao/OneWorld.
Sensen Gao, Zhaoqing Wang, Qihang Cao +5
Mohamed bin Zayed University of Artificial Intelligence · AISphere · Shanghai Jiao Tong University +3