Current compositional image-to-3D scene generation approaches construct 3D scenes by time-consuming iterative layout optimization or inflexible joint object-layout generation. Moreover, most methods rely on limited field-of-view perspective images, hindering the creation of complete 360-degree environments. To address these limitations, we design Pano3DComposer, an efficient feed-forward framework for panoramic images. To decouple object generation from layout estimation, we propose a plug-and-play Object-World Transformation Predictor. This module converts the 3D objects generated by off-the-shelf image-to-3D models from local to world coordinates. To achieve this, we adapt the VGGT architecture to Alignment-VGGT by using target object crop, multi-view object renderings and camera parameters to predict the transformation. The predictor is trained using pseudo-geometric supervision to address the shape discrepancy between generated and ground-truth objects. For input images from unseen domains, we further introduce a Coarse-to-Fine (C2F) alignment mechanism for Pano3DComposer that iteratively refines geometric consistency with feedback of scene rendering. Our method achieves superior geometric accuracy for image/text-to-3D tasks on synthetic and real-world datasets. It can generate a high-fidelity 3D scene in approximately 20 seconds on an RTX 4090 GPU. Project page: https://qiuzidian.github.io/pano3dcomposer-page/.
Figures & tables
Figure 1: Paradigms of compositional 3D scene generation.
Figure 2: Overview of Pano3DComposer. The framework takes a panoramic image I as input and generates a 3D scene Gscene through four stages: ( i ) Preprocessing, ( ii ) Object Generation & Alignment, ( iii ) Background Modeling, and ( iv ) Composition.
Figure 3: Illustration of the proposed Coarse-to-Fine (C2F) alignment mechanism.
Method
CD-S ↓
CD-O ↓
F-Score-S ↑
F-Score-O ↑
IoU-B ↑
Training Resources
Inference Time (s)
OPT
0.1059
0.1128
0.5535
0.5640
0.4010
–
120
ICP [ 18 ]
0.2483
0.2305
0.4524
0.4896
0.2830
–
1
DeepPanoContext [ 44 ]
0.7851
0.1657
0.3101
0.3822
0.0021
–
14
SceneGen [ 27 ]
0.1765
0.0914
0.4575
0.4827
0.1124
56 GPU days
63
Pano3DComposer (Ours)
0.0787
0.0765
0.6923
0.6926
0.5679
2 GPU days
20
Pano3DComposer-C2F (Ours)
0.0784
0.0762
0.6930
0.6937
0.5699
4 GPU days
24
Table 1: Comparison of major and alignment results on the 3D-FRONT test set. The best performance for each metric is highlighted in bold . OPT represents differentiable optimization-based alignment, and ICP denotes Iterative Closest Point alignment. “Pseudo Geometry” serves as a reference upper bound obtained via offline differentiable optimization of the transformation parameters introduced in Sec. 3.2.3 . Training resources are reported in 4090 GPU days. Inference time is tested on one 4090 GPU.
Figure 4: Visualization of panorama-to-3D scene composition results without background. Row 1: 3D-FRONT test set; Row 2: Structured3D test set; Row 3: real-world panoramas.
Figure 5: Visualization of panorama-to-3D scene composition results with background. The figure presents multi-view renderings of composed 3D scenes generated by our method. Row 1: 3D-FRONT test set; Row 2: Structured3D test set; Row 3: real-world panoramas.
Figure 6: Visualization of Text-to-3D scene generation results.
Stage
Time (s)
Background Inpainting
0.02
Background GS Reconstruction
0.16
Object Generation (per object)
∼ 4
Object Alignment (per object)
0.36
Object Refinement (per step)
0.18
Table 2: Runtime analysis of different processing stages.
Method
CD-S ↓
CD-O ↓
F-Score-S ↑
F-Score-O ↑
IoU-B ↑
Only LCD
0.8688
0.9027
0.1980
0.1888
0.0906
+ LPGD
0.1266
0.1219
0.5675
0.5670
0.4670
+ LMASK
0.1120
0.1063
0.5788
0.5850
0.4818
w/o Cam info
0.1850
0.1705
0.4673
0.4691
0.3830
Table 3: Ablation study on loss functions and training strategies.
Figure 7: Example inputs of SceneGen.
Method
CD-S ↓
CD-O ↓
F-Score-S ↑
F-Score-O ↑
IoU-B ↑
Full
0.1883
0.1946
0.4992
0.4907
0.3855
–D
0.1236
0.1177
0.5565
0.5550
0.4360
–D-F
0.0787
0.0765
0.6923
0.6926
0.5679
–D-F-G
0.1120
0.1063
0.5788
0.5850
0.4818
Table 4: Ablation of fine-tuning strategies. “–D”, “–D-F”, and “–D-F-G” indicate progressively freezing DINO, frame, and global attention modules.
Figure 8: Visualization of panorama-to-3D scene composition results without background.