ScenePlacer: Reformulating Scene-Level Object Pose Estimation as Visibility Field Learning

Ken Deng; Jiarui Liu; Guibiao Liao; Kunming Luo; Mochu Xiang; Hongyu Yan; Yebin Liu; Shenghua Gao

cs.CVOct 11, 2026

ScenePlacer: Reformulating Scene-Level Object Pose Estimation as Visibility Field Learning

Authors: Ken Deng, Jiarui Liu, Guibiao Liao, Kunming Luo, Mochu Xiang, Hongyu Yan, Yebin Liu, Shenghua Gao,

Organizations: The Hong Kong University of Science and Technology · Shenzhen Loop Area Institute · The University of Hong Kong · Tsinghua University

Abstract

We present ScenePlacer, a framework for scene-level pose estimation of objects generated by state-of-the-art native shape generation models, placing generated 3D objects into scenes given a partial geometric observation without modifying the underlying generative pipeline. Existing scene generation approaches either generate objects directly in scene coordinates, compromising their canonical-space priors, or regress poses from complete generated shapes in the shape generation process - the latter being a poorly conditioned problem that is sensitive to scale ambiguity and partial observations. ScenePlacer decouples object generation from pose estimation by reformulating alignment as conditional visibility-field learning. Given an observed partial point cloud from reconstruction and a generated mesh, it predicts a visibility field defined in object space, then estimates the rigid transformation by aligning the partial observation to visible parts on the mesh. We propose a pretrained 3D shape VAE as our backbone, preserving its mesh prior while conditioning it on a separately encoded observation. We further introduce visibility-aware, radius-constrained supervision and synthesize training pairs using only object-level meshes and rendered views, eliminating the need for scene-level pose annotations. On 3D-FUTURE, ScenePlacer consistently outperforms existing placement and pose-refinement baselines under both ground-truth and monocular depth estimation. With generated objects, it achieves a Chamfer distance of 0.0077 and an F-score of 92.40 under estimated depth and improves to 0.0023 and 98.27, respectively, under ground-truth geometry.

Figures & tables

Figure 2: Overview. Pipeline: The input image is first processed by a geometry estimator to obtain a point cloud and a mesh generated by a generative model; ScenePlacer is then used to place the object back into its corresponding spatial location, resulting in a compositional scene. Data Curation: Depth maps rendered from the ground-truth mesh are back-projected into oriented partial points, augmented by random translation, rotation and scales, and vis/invis samples with an exclusion radius provide supervision. Visibility Field Predictor: A frozen Shape Encoder and a trainable Partial Point Encoder produce mesh and observation latents, respectively; the Visibility Field Decoder conditions the mesh latents on the observation, and decodes visibility for query points sampled on the mesh surface, yielding the visible points used for alignment.

Figure 3: Qualitative comparison of scene-level layout predictions (The rightmost column uses MoGe Depth, while the others use GT-Depth.). Each row shows a scene generated by different methods, with the ground-truth layout shown for reference (red outline). Our method produces layouts that closely align with the ground truth, while baselines exhibit noticeable errors.

Depth	Metric	Any6D	3DFixer	SAM3D	SAM3D+PO	Ours (SAM3D obj.)	Ours (GT obj.)
GT	CD $\downarrow$	1.04	23.35	0.98	1.51	0.36	0.23
	[email protected] $\uparrow$	85.18	85.65	75.01	74.26	90.38	95.60
	[email protected] $\uparrow$	92.81	89.63	90.73	87.05	97.01	98.27
MoGe	CD $\downarrow$	2.87	23.43	1.31	1.92	0.77	0.70
	[email protected] $\uparrow$	74.14	76.51	71.40	65.79	78.53	81.94
	[email protected] $\uparrow$	86.80	86.48	88.77	83.48	92.40	93.38

Table 1: Scene-level quantitative comparison under different depth inputs. All values are multiplied by 100. Best in bold, second best underlined.

Depth	Metric	Shared-Encoder	Attn-Swap (w/o Aug.)	Ours (w/o Vis. RCS)	Ours (full)
GT	CD $\downarrow$	0.57	1.90	0.52	0.34
GT	[email protected] $\uparrow$	93.86	81.75	96.65	96.69
MoGe	CD $\downarrow$	0.99	2.07	0.87	0.74
MoGe	[email protected] $\uparrow$	88.53	76.75	91.84	92.14

Table 2: Ablation study. All values are multiplied by 100.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Visualization of positive (visible, blue) and negative (invisible, gray) point sampling under different radius.

Figure 5: Visibility field visualization.

Figure 6: Visibility positive prediction. Given an observed partial point cloud and a generated mesh, the network predicts per-point visibility on the mesh surface and we visualize all positive points in blue.

Depth	Metric	Only Pose Solver	Ours
GT	CD $\downarrow$	1.32	0.34
GT	[email protected] $\uparrow$	86.02	96.69
MoGe	CD $\downarrow$	1.71	0.74
MoGe	[email protected] $\uparrow$	81.34	92.14

Appendix

Table 3: Ablation study. All values are multiplied by 100.

Figure 9: Generalize to multi-view input.

Figure 10: Generalize to multi-view input.

Object Pose Estimation

Object pose estimation recovers an object's 3D position and orientation from images, depth, point clouds, or contact sensing.

See all 187 papers

Mira-Scene: Pixel-Aligned Layouts for Generative 3D Scene Reconstruction
Sep 20, 2026Yang-Tian Sun, Tianjia Liu, Zehuan Huang +73D Reconstruction 3D Scene Layout Generation
The University of Hong Kong · VAST
SceneReGen: Generative Reconstruction of 3D Scenes from a Single Image
Aug 25, 2026Zefan Tian, Yuteng Ye, Yiheng Zhang +53D Reconstruction Single-View 3D Reconstruction
Huawei · Northwestern Polytechnical University
DistScene: Object-to-Scene Distillation for 3D Scene Generation
Oct 3, 2026Kunming Luo, Hongyu Yan, Ken Deng +63D Scene Generation Image-to-3D Scene Generation
The Hong Kong University of Science and Technology · TeleAI, China Telecom

Sep 20, 2026J/K move · Enter open · S save

Mira-Scene: Pixel-Aligned Layouts for Generative 3D Scene Reconstruction

Yang-Tian Sun, Tianjia Liu, Zehuan Huang, Yi-Hua Huang+6

The University of Hong Kong · VAST

3D Reconstruction 3D Scene Layout Generation Single-View 3D Reconstruction Image-to-3D Scene Generation

Open paper AskPDF arXiv

Single-image 3D object generation can now produce high-fidelity assets, yet accurately placing them into a coherent scene layout remains an open challenge. A central difficulty lies in how object layout is represented. Holistic methods absorb placement into a scene-level generation process, sacrificing object-level detail. Compositional methods preserve object fidelity by decoupling geometry from layout, but typically parameterize layout as sparse, unbounded pose variables that are difficult to learn and generalize poorly under scarce scene-level supervision. We present Mira-Scene, a compositional 3D scene reconstruction framework that replaces sparse pose regression with dense, bounded correspondence recovery. At its core is the Canonical Coordinate Map (CCM), a pixel-aligned field that maps each visible object pixel to a surface coordinate in the object's bounded canonical space. When paired with a scene-space Point Cloud Map (PCM) from monocular geometry estimation, CCM induces dense canonical-to-scene correspondences from which object transformations are recovered through robust geometric alignment. Because CCM operates in bounded canonical space, it provides a stable prediction target that can be trained from scalable object-level 3D data without requiring scene-level layout annotations. Mira-Scene further introduces a multimodal diffusion transformer that jointly generates object geometry and CCMs, using modality-specific expert streams with shared attention and positional encoding to promote geometry-layout consistency. Experiments on indoor, outdoor, synthetic, and in-the-wild scenes show that Mira-Scene substantially outperforms strong baselines in layout accuracy, achieving relative gains of 39.8% in 3D-IoU and 16.5% in 2D-IoU over SAM3D, using limited open-source training data.

Abstract

Object Pose Estimation

Explore similar work

Mira-Scene: Pixel-Aligned Layouts for Generative 3D Scene Reconstruction

SceneReGen: Generative Reconstruction of 3D Scenes from a Single Image

DistScene: Object-to-Scene Distillation for 3D Scene Generation