Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Taobao & Tmall Group of Alibaba · The Hong Kong University of Science and Technology
Generating compositional 3D scenes from a single image requires recovering detailed object geometry and coherent spatial arrangements. Existing object-to-scene methods, including SAM3D and MIDI, extend object generation through image-context reasoning or inter-instance modeling. Coordinating objects remains challenging with limited scene-level supervision. Directly extending object-centric generators to entire scenes can degrade geometry and fail to preserve instance scale, orientation, and pose. We propose Bri3D, a framework bridging scene-level structural reasoning and object-level generative priors through instance-aware voxel scene anchors. Bri3D predicts a color-coded scene scaffold encoding coarse object geometry and spatial arrangements. Each instance is extracted as a scene anchor and refined into a pose-preserving, prior-compatible voxel support, enabling a frozen object generator to synthesize detailed textured geometry in the anchor-defined spatial reference. This coarse-to-fine design separates scene organization from object-scale detail synthesis, focusing scene-level training on spatial organization and anchor refinement while reusing pretrained object priors. Experiments demonstrate improvements in scene layout, object geometry, visual quality, and generalization. Bri3D predicts scene layouts efficiently; full scene generation is faster than SceneGen and SAM3D.
Figures & tables
Figure 1: Bridging the object–scene generation gap. Object-to-scene methods, such as MIDI and SAM3D, extend object generation to scene composition, where coordination errors can still produce collisions and implausible arrangements. Object-centric generators produce high-fidelity individual shapes, but their naive extension to full scenes causes structural collapse and distorted geometry. In contrast, Bri3D first generates a coherent instance-aware scene scaffold and then performs scene-anchored prior refinement, producing pose-preserving voxel supports for detail synthesis.
Figure 2: Overview of Bri3D. The Scene Scaffold Generator and Anchored Voxel Refiner replace the voxel-generation stage of a two-stage object generator, while its detail-synthesis stage remains frozen. SC-VAE denotes the Structure-Color VAE, and snowflakes indicate frozen modules. The scaffold is generated in latent space and decoded at 1283 ; each object is refined to a 643 voxel support. The anchor initialization follows Eq. 5 , including object-wise latent normalization.
Method
Geometry
Visual Quality
CD-S ↓
CD-O ↓
F-S ↑
F-O ↑
BBox IoU ↑
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
CLIP-S ↑
DINO-S ↑
DepR
9.07
18.81
78.91
56.15
0.3943
—
—
—
—
—
—
Gen3DSR
8.21
17.18
79.07
60.41
0.4456
16.30
0.8639
0.1621
19.96
0.9009
0.8139
PartCrafter
7.05
—
85.66
—
—
—
—
—
—
—
—
SceneGen
6.11
7.07
87.72
87.73
0.5382
16.50
0.8723
0.1552
19.80
0.9005
0.8201
MIDI
2.14
2.81
92.20
89.57
0.6238
16.97
0.8756
0.1522
19.78
0.9315
0.8247
Table 1: Quantitative comparisons on the 3D-FRONT [ 12 ] test set. Geometric metrics: scene/object-level Chamfer Distance (CD-S/CD-O, ×100 ), F-Score (F-S/F-O, %), and BBox IoU ( [0,1] ). Visual metrics: PSNR, SSIM, LPIPS, FID, CLIP-S, DINO-S.
Method
CD-S ↓
F-S ↑
CD-O ↓
F-O ↑
BBox IoU ↑
LPIPS ↓
DINO-S ↑
DepR
14.20
64.80
13.85
66.10
0.2420
-
-
Gen3DSR
12.70
68.90
12.35
69.70
0.2785
0.3328
0.7218
PartCrafter
11.30
74.60
—
—
—
-
-
SceneGen
12.10
71.20
11.56
72.80
0.2864
0.3187
0.7382
MIDI
6.20
80.50
5.74
82.60
0.4315
0.2635
0.7756
SAM3D
3.85
87.10
3.48
89.30
0.5628
0.2218
0.8021
Table 2: Cross-dataset generalization results. We evaluate scene- and object-level geometry, layout accuracy, and visual similarity on 300 scenes paired with rendered images and ground-truth 3D scenes. CD-S and CD-O are scaled by 100 ; F-S and F-O are F-Score percentages; BBox IoU is on the [0,1] scale. The best results are in bold; the second-best results (SAM3D in all reported columns) are underlined.
Figure 3: Qualitative results on the 3D-FRONT [ 12 ] test set. Our method generates more plausible layouts and higher-fidelity geometry, consistent with the quantitative improvements. Red boxes indicate inferior baseline results.
Figure 4: Generalization to diverse room scenes. The upper three rows compare reconstructions of a living room, a garage, and a classroom, with the input image followed by the results of DepR, PartCrafter, MIDI, SceneGen, SAM3D, and Bri3D. These examples cover different object categories, scales, and degrees of occlusion. The lower row shows a multi-room overview and reconstructions of its bathroom, bedroom, living room, and kitchen, each generated from a separate room-level input. Together, the examples illustrate object reconstruction across varied scene configurations and room types.
Model
1–10 Objects
11–20 Objects
Collision Rate (%) ↓
BBox IoU
Test Time
BBox IoU
Test Time
DepR
0.4032
15.3s
0.3008
20.8s
22
MIDI
0.6349
40.4s
0.4193
72.9s
14
SceneGen
0.5892
135.0s
0.4462
230.2s
38
SAM3D
0.6568
64.1s
0.4609
179.6s
34
Ours ∗
0.7980
5.1s (42.6s)
0.6732
5.2s (78.4s)
4
Table 3: Comparison of layout accuracy, inference time, and observed object collisions. BBox IoU is in [0,1] ; collision rate is the percentage of overlapping object pairs. ∗ Ours reports scaffold-stage anchor BBox IoU and collisions after separately voxelizing extracted instances. Times outside parentheses are for scaffold prediction; parentheses and baseline times report full generation.
Figure 5: Comparison of 3D scene layouts and failure cases. (a) Columns show the RGB input, ground-truth layout, and predictions from MIDI [ 18 ] , SceneGen [ 28 ] , SAM3D [ 3 ] , and Bri3D. Red wireframe boxes mark the ground-truth object extents, making displacement, missing instances, and shape deviations visible. Bri3D better follows the reference arrangements in these examples. (b) Red rectangles highlight our failures: nearby objects may merge in the cluttered seating area (Case 1), and some instances may be omitted in the densely populated classroom (Case 2).
Variant
CD-S/F-S ↓/↑
CD-O/F-O ↓/↑
BBox IoU ↑
Baseline
(a)
TRELLIS-FT ( 643 )
10.80/52.91
–
–
Scene Scaffold Generator
(b)
One-hot representation + CE loss ( 643 )
13.00/43.24
13.90/41.23
0.3144
(c)
Color representation + ℓ1 loss ( 643 )
7.24/55.57
8.61/53.55
0.3902
(d)
+ Rendering-based label loss
5.47/69.88
7.32/60.82
0.4653
Table 4: Ablation studies of Bri3D. Rows (b)–(f) progressively introduce the components of the Scene Scaffold Generator. Rows (g)/(h) compare interpolation and latent refinement from (f); (i)/(j) apply detail synthesis after (h). All retain the anchor boxes and BBox IoU of (f). CD-S/F-S and CD-O/F-O report scene/object metrics. CD is scaled by 100 , F-Score is in percent, and BBox IoU is in [0,1] . The best results within each component block are highlighted in bold.
Figure 6: Color-coded scene scaffold vs. connected component analysis. From left to right: RGB input, instance label map, geometry-only CCA result, and the predicted color-coded voxel grid. In the upper example, CCA groups the refrigerator, microwave, and table into one connected region; in the lower example, it merges the touching bed components and teddy bear. The predicted instance colors provide additional separation cues at these contacts, allowing joint position–color clustering to recover distinct object anchors. Colors indicate instance identity rather than surface texture.
Figure 7: Preliminary attempt on structural elements. From left to right: RGB input, label map including structural regions, predicted color-coded voxel grid, and reconstructed scene mesh. The dining-area and bedroom examples include room surfaces together with foreground furniture, illustrating how structural labels can extend the scaffold beyond object instances. This extension depends on reliable labels for those regions and remains preliminary; floors, walls, and ceilings are excluded from the main Bri3D experiments and quantitative results.
Figure 8: Additional qualitative comparisons and robustness to imperfect segmentation. (a) Geometry and layout comparisons with MIDI and SAM3D on indoor and outdoor scenes; MIDI is shown with normal shading. Bri3D produces coherent layouts and faithful object geometry, while MIDI suffers from distorted shapes and implausible arrangements. (b) With an erroneous instance label map (Wrong Seg), Bri3D exhibits only local cracks, whereas SAM3D produces incorrect object partitions and collisions, including interpenetration between the teddy bear and the bed.
Figure 9: Robustness to instance segmentation granularity, label colors, and implausible partitions. From left to right: input image, input instance segmentation, predicted semantic voxel grid, and reconstructed textured meshes with bounding boxes. The first two rows show the same tabletop scene containing a bottle, a mug, a large box, a ball, and two small stacks of blocks. The segmentation changes only the grouping of these small stacks: each stack receives one label in the first row, while its individual blocks receive separate labels in the second row. The last two rows show deliberately implausible partitions of a chair, together with changes in instance label colors. Despite these inconsistent instance boundaries, Bri3D preserves the overall object geometry and spatial arrangement.
We present DistScene, a framework for single-image compositional 3D scene generation by jointly modeling the environment and individual objects. Unlike existing methods that represent scenes primarily as collections of objects, we model the environment as an explicit scene component to provide geometric context for object placement. Specifically, we introduce Scene-Frame Generation, which jointly generates separate environment and object components in a shared coordinate frame, allowing their geometry and relative placement to be learned together. Then we introduce Object-Centric Refinement to refine each object in a local frame with scene context. Finally, we develop Object-to-Scene Distillation to transfer pretrained object-generation priors to scene generation through automatically composed and rendered synthetic scenes. Evaluations on indoor and outdoor benchmarks demonstrate improved scene-level spatial coherence over the evaluated baselines. Project page: https://coolbeam.github.io/DistScene/
Kunming Luo, Hongyu Yan, Ken Deng +6
The Hong Kong University of Science and Technology · TeleAI, China Telecom
Single-image 3D object generation can now produce high-fidelity assets, yet accurately placing them into a coherent scene layout remains an open challenge. A central difficulty lies in how object layout is represented. Holistic methods absorb placement into a scene-level generation process, sacrificing object-level detail. Compositional methods preserve object fidelity by decoupling geometry from layout, but typically parameterize layout as sparse, unbounded pose variables that are difficult to learn and generalize poorly under scarce scene-level supervision. We present Mira-Scene, a compositional 3D scene reconstruction framework that replaces sparse pose regression with dense, bounded correspondence recovery. At its core is the Canonical Coordinate Map (CCM), a pixel-aligned field that maps each visible object pixel to a surface coordinate in the object's bounded canonical space. When paired with a scene-space Point Cloud Map (PCM) from monocular geometry estimation, CCM induces dense canonical-to-scene correspondences from which object transformations are recovered through robust geometric alignment. Because CCM operates in bounded canonical space, it provides a stable prediction target that can be trained from scalable object-level 3D data without requiring scene-level layout annotations. Mira-Scene further introduces a multimodal diffusion transformer that jointly generates object geometry and CCMs, using modality-specific expert streams with shared attention and positional encoding to promote geometry-layout consistency. Experiments on indoor, outdoor, synthetic, and in-the-wild scenes show that Mira-Scene substantially outperforms strong baselines in layout accuracy, achieving relative gains of 39.8% in 3D-IoU and 16.5% in 2D-IoU over SAM3D, using limited open-source training data.
Generating complete 3D scenes from a single image requires inferring globally consistent geometry, object relationships, and environmental context from inherently ambiguous visual evidence. Despite recent progress in joint layout-and-mesh generation, existing methods often rely on holistic or weakly decomposed pipelines that entangle many factors at once and demand extensive scene-level supervision, limiting their generalization to complex real-world environments. We propose a multi-agent orchestration framework that decomposes single-image 3D scene generation into three structured stages: scene initialization, environment construction, and multi-agent refinement. The initialization stage extracts image-derived object masks, builds object-level 3D representations, and predicts an initial spatial layout to form a coarse 3D scene. The environment-construction stage then leverages this initialization together with point-map geometry to build an environmental scaffold of supporting surfaces, room boundaries, materials, and illumination. Finally, in the refinement stage, a planner agent identifies structural and visual inconsistencies, applies simple corrections directly, and dispatches specialist agents for complex localized revisions that are reintegrated into the global scene. To provide reliable structural initialization while reducing reliance on scene-level annotations, we further introduce a geometry-aware layout predictor supervised by sparse geometric priors derived from point maps. Unlike fully supervised layout generators, the predictor can be trained from segmentation-level data and generalizes robustly to diverse real-world scenes. Extensive experiments on benchmark datasets show that our method consistently outperforms prior approaches in geometric accuracy, spatial consistency, and perceptual realism.
Jeonghwan Kim, Yushi Lan, Yongwei Chen +3
Nanyang Technological University · University of Oxford · Meshy AI