3D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Image
Authors: Ze-Xin Yin, Liu Liu, Xinjie Wang, Wei Sui, Zhizhong Su, Jian Yang, Jin Xie
Organizations: College of Computer Science, Nankai University · Horizon Robotics · D-Robotics · School of Intelligence Science and Technology, Nanjing University
Compositional 3D scene generation from a single view requires the simultaneous recovery of scene layout and 3D assets. Existing approaches mainly fall into two categories: feed-forward generation methods and per-instance generation methods. The former directly predict 3D assets with explicit 6DoF poses through efficient network inference, but they generalize poorly to complex scenes. The latter improve generalization through a divide-and-conquer strategy, but suffer from time-consuming pose optimization. To bridge this gap, we introduce 3D-Fixer, a novel in-place completion paradigm. Specifically, 3D-Fixer extends 3D object generative priors to generate complete 3D assets conditioned on the partially visible point cloud at the original locations, which are cropped from the fragmented geometry obtained from the geometry estimation methods. Unlike prior works that require explicit pose alignment, 3D-Fixer uses fragmented geometry as a spatial anchor to preserve layout fidelity. At its core, we propose a coarse-to-fine generation scheme to resolve boundary ambiguity under occlusion, supported by a dual-branch conditioning network and an Occlusion-Robust Feature Alignment (ORFA) strategy for stable training. Furthermore, to address the data scarcity bottleneck, we present ARSG-110K, the largest scene-level dataset to date, comprising over 110K diverse scenes and 3M annotated images with high-fidelity 3D ground truth. Extensive experiments show that 3D-Fixer achieves state-of-the-art geometric accuracy, which significantly outperforms baselines such as MIDI and Gen3DSR, while maintaining the efficiency of the diffusion process. Code and data will be publicly available at https://zx-yin.github.io/3dfixer.
Figures & tables
Figure 2: Architecture of the 3D-Fixer pipeline and dataset. (Top) Scene Decomposition extracts instance-level partial geometry from the input. (Bottom-left) Progressive Completion generates the full asset via three stages: 1) The Coarse Structure Completer hallucinates topology within a loose bound; 2) The Fine Shape Refiner sharpens geometry within a fine boundary; and 3) The Occlusion-Aware 3D Texturer applies observation-aligned textures. (Bottom-right) Our ARSG-110K Dataset provides high-quality assets and rich scene compositions for training.
Figure 3: 3D-Fixer extends the diffusion transformer from prior [ 57 ] (orange) to a dual-stream architecture (blue), where a trainable branch encoding scene-specific geometric cues interacts with a frozen generative branch to implement ORFA and enforce structural constraints.
Dataset
Source
Objects
Scenes
Images
Scan2CAD [ 2 ]
ShapeNet [ 4 ]
14.2K
706
2.5M
OpenRooms [ 29 ]
ShapeNet [ 4 ]
16.0K
706
2.5M
METASCENES [ 63 ]
ScanNet [ 10 ] , Objaverse [ 11 ]
15.4K
706
2.5M
R3DS [ 55 ]
ShapeNet [ 4 ] , Wayfair [ 43 ]
19.1K
370
194K
CAD-Estate [ 35 ]
ShapeNet [ 4 ]
100K
19.5K
-
BVS [ 16 ]
BEHAVIOR-1K [ 27 ]
6.7K
1000
-
Table 1: We compare ARSG-110K with existing 3D scene datasets. Our dataset significantly scales up scene and object diversity through procedural generation, offering over 3 million rendered views.
Table 2: Quantitative comparisons on synthetic datasets [ 13 , 1 , 5 ] and real-world dataset [ 10 , 63 ] . We report Scene ( S ) and Object ( O ) level Chamfer Distance (CD) and F-Score (FS), along with Bounding Box IoU and inference time.
#K
C2F
AL
CD S ↓
FS S ↑
CD O ↓
FS O ↑
IoU ↑
12
✗
✗
0.276
53.87
0.271
52.28
44.00
12
✓
✗
0.266
54.44
0.329
50.28
44.52
12
✓
✓
0.264
54.55
0.283
51.63
45.73
6
✓
✗
0.267
53.52
0.315
48.51
42.91
12
✓
✗
0.266
54.44
0.329
50.28
44.52
18
✓
✗
0.252
56.40
0.273
52.29
47.32
Table 3: Ablation studies. We evaluate the number of layers ( #K ), the coarse-to-fine strategy (C2F), the use of alignment loss (AL), the use of depth ratio embeddings (Dpt.), the inclusion of global features (S.), and the mixture of estimated geometry source from multiple geometry estimation methods (S. Dpt. v.s. M. Dpt.).
Figure 4: Visualization of the results on the Gen3DSR testset and our testset. The results on the Gen3DSR testset demonstrate the robustness of our scheme across different scenes, while the results on our test set show the great potential of our scheme in handling complex scenes.
Figure 5: Visualization of ablation studies. Experiments (a)-(d) are designed to evaluate the coarse-to-fine (C2F) strategy and the network layers (K), which are as follows: (a) w/o C2F, K=12; (b) w/ C2F, K=6; (c) w/ C2F, K=12; w/ C2F, K=18. Experiments (e)-(h) are designed to evaluate the Alignment Loss (AL), depth ratio embedding (Dpt.), and the global feature input (Glob.), which are as follows: (e) w/ C2F, K=12, AL, Dpt., and Glob.; (f) w/o AL and Dpt.; (g) w/o AL and Glob.; (h) w/o AL, Dpt. and Glob.
Figure 6: Visualization of our scheme handling initial geometry distortions and mask errors.
Figure 7: Visualization of our method handling complex occlusion patterns.
Table 4: Quantitative comparisons on our testset. We report Object ( O ) level Chamfer Distance (CD) and F-Score (FS), along with Bounding Box IoU, to quantify the robustness of our scheme on geometry distortion. We also report the object-level rendering metrics on our testset to quantify the texture quality compared to Gen3DSR (G3D).
Figure 8: Visual comparisons on our testset.
Figure 9: Visual comparisons on ScanNet.
Figure 10: Visual comparisons on real world captured images.
In this paper, we propose Extend3D, a training-free pipeline for 3D scene generation from a single image, built upon an object-centric 3D generative model. To overcome the limitations of fixed-size latent spaces in object-centric models for representing wide scenes, we extend the latent space in the x and y directions. Then, by dividing the extended latent space into overlapping patches, we apply the object-centric 3D generative model to each patch and couple them at each time step. Since patch-wise 3D generation with image conditioning requires strict spatial alignment between image and latent patches, we initialize the scene using a point cloud prior from a monocular depth estimator and iteratively refine occluded regions through SDEdit. We discovered that treating the incompleteness of 3D structure as noise during 3D refinement enables 3D completion via a concept, which we term under-noising. Furthermore, to address the sub-optimality of object-centric models for sub-scene generation, we optimize the extended latent during denoising, ensuring that the denoising trajectories remain consistent with the sub-scene dynamics. To this end, we introduce 3D-aware optimization objectives for improved geometric structure and texture fidelity. We demonstrate that our method yields better results than prior methods, as evidenced by human preference and quantitative experiments.
High-quality 3D scene assets are critical for embodied applications such as robotic manipulation, navigation, and simulation. Despite their strong object priors, recent single-image 3D generation models such as SAM3D remain insufficient for real-world scenes, where severe occlusions, redundant observations, and cross-view inconsistencies make reliable scene generation challenging. We introduce Scene-SAM3D, a training-free framework that extends SAM3D from single-view object generation to calibrated multi-view scene asset generation. Scene-SAM3D selects a compact set of complementary views, reducing observation redundancy while providing additional evidence for regions occluded in individual views. Based on the selected views, it performs step-efficient latent velocity fusion to integrate multi-view evidence and suppress cross-view conflicts in canonical space. Finally, a lightweight rigid-object Gaussian optimization refines the scene layout within 200 iterations while preserving the generated object geometry. Experiments on Replica and ScanNet++ demonstrate consistent improvements at both instance and scene levels, with our method reducing scene-level CD by 43.8% on Replica and 30.9% on ScanNet++, while cutting flow-model sampling FLOPs and wall-time latency by nearly 20% under the same multi-view setting. Code will be released at https://github.com/xibi777/Scene-SAM3D.
Yuqi Zhang, Yadan Luo, Xiangyu Sun +3
The University of Queensland · Shanghai AI Laboratory · East China Normal University
Generating pose-aligned 3D objects is challenging due to the spatial mismatches and transformation ambiguities inherent in decoupled canonical-then-rotate paradigms. To this end, we introduce Pose-Aware Diffusion (PAD), a novel end-to-end diffusion framework that synthesizes 3D geometry directly within the observation space. By unprojecting monocular depth into a partial point cloud and explicitly injecting it as a 3D geometric anchor, PAD abandons canonical assumptions to enforce rigorous spatial supervision. This native generation intrinsically resolves pose ambiguity, producing high-fidelity pose-aligned assets. Extensive experiments demonstrate that PAD achieves superior geometric alignment and image-to-3D correspondence compared to state-of-the-art methods. Additionally, PAD naturally extends to compositional 3D scene reconstruction via a simple union of independently generated objects, highlighting its robust ability to preserve precise spatial layouts.
Zihan Zhou, Luxi Chen, Jingzhi Zhou +4
Gaoling School of AI, Renmin University of China · VCIP, School of Computer Science, Nankai University · THU-Bosch MLCenter, Tsinghua University +1