Learning Spherical Occupancy Profiles for Multi-View 3D Reconstruction and Generation

YiHsuan Tsai

cs.CVAug 24, 2026

Learning Spherical Occupancy Profiles for Multi-View 3D Reconstruction and Generation

Authors: YiHsuan Tsai

Organizations: Nanjing University

Abstract

We study spherical occupancy profiles-the ray-wise occupancy probability profiles P(r) = T(r) o(r) distilled from multi-view 3D Gaussian reconstructions-as a unified intermediate representation for both discriminative and generative 3D reconstruction from images. On a 999-object subset of Google Scanned Objects with 48 turntable views each, we train (i) a discriminative per-ray decoder that injects global view-averaged and ray-specific image evidence into a FiLM-conditioned profile head, reaching median soft depth error 0.035 (normalized) on an independent 90-object test split, and (ii) a generative pipeline built on a profile VAE and a latent diffusion model, which supports unconditional sampling that matches the reconstruction manifold and image-conditioned multi-solution reconstruction whose per-object solution spread is quantifiable and tunable via classifier-free guidance. We further analyze the morphology of predicted profiles: post-hoc power sharpening and a learned sharpening target both recover ground-truth profile width without degrading depth, exposing a monotonic width-peak frontier in the L1-per-ray loss family and motivating a principled redefinition of morphology gates. Real-photo validation on two DTU scenes confirms the pipeline transfers to non-synthetic input. Our results suggest that ray-wise occupancy profiles offer a compact, learned, and uncertainty-aware interface between multi-view reconstruction and generative priors.

Figures & tables

Figure 1: Pipeline overview. (a) Volume distillation produces ground-truth spherical occupancy profiles from multi-view captures without mesh supervision; (b) a discriminative per-ray decoder maps images to profile fields; (c) a profile VAE with latent diffusion maps profile fields to a sampleable latent, enabling multi-solution reconstruction. The two branches share the same representation.

method	$\mathrm{dmed_{s}}$		$\mathrm{ch_{s}}$
	val	test	val	test
mean profile	0.0842	—	—	—
d8 (global cond.)	0.0338	0.0384	0.228	0.213
m3b v1 (per-ray L1)	0.0364	0.0356	0.236	0.211
m3b v2 $\gamma{=}1.5$	0.0354	0.0351	0.237	0.214
m3b v2 $\gamma{=}2$	0.0348	0.0356	0.243	0.217

Table 1: Discriminative reconstruction (median over objects; lower is better except peak).

\mathrm{dmed}

= soft-depth error,

\mathrm{ch}

= Chamfer to GT cloud.

Figure 2: Example ray profiles for a mid- and a hard object. GT profiles (black) are narrow with peak $\approx 1$ ; d8 (blue) is over-wide and low; the per-ray model (orange) is sharper; the sharpened-target model (green, $\gamma{=}2$ ) matches GT width on covered rays. Three covered rays (peak bins at the 25/50/75 percentiles) are shown per object.

$\Delta x$ [px]	shift / obj. radius	$\mathrm{dmed_{s}}$	rel. to baseline
0	0%	0.0338	1.00
2	4.2%	0.0338	1.00
4	8.3%	0.0351	1.04
9	18.7%	0.0373	1.10

Table 2: Center-robustness of d8 (90-object validation split). Shifting all 8 views by

\Delta x

px simulates an error in the object-center estimate of the spherical parameterization.

method	forward [ms]	dec [ms]	total [ms]	peak mem [MB]
d8 (1 object)	1.5	0.7	2.2	165
d8 (batch 8)	—	—	1.6/obj	525
m3b v2 (1 object)	3.6	1.2	4.8 a	474
VAE recon (M1)	0.6	0.9	1.5	241
latent diff. sample (M2)	18.5	1.0	19.5	239

Table 3: Median single-object inference latency and peak GPU memory (NVIDIA L20, fp32). forward = view encoding ( d8 / m3b ), latent encoding (M1), or 50-step DDIM (M2); dec = per-ray decoding of the

8192

-ray profile field.

model	views $K$	val	test
mean profile (no image cond.)	—	0.0842	—
d8 (profile)	8	0.0338	0.0384
d8 (profile)	2	0.0430	0.0419
d8 (profile)	1	0.0380	0.0442
RW (RayDF-style)	8	0.0396	0.0369
RW (RayDF-style)	1	0.0455	0.0422

Table 4: External baseline and view scaling on the same split and protocol. RW = ray-wise distance field re-implemented in the spirit of RayDF [ 18 ] , sharing the encoder, fusion, optimizer, and metric with d8 .

\mathrm{dmed_{s}}

= median absolute soft-depth error over covered rays (objects); lower is better.

views $K$	$\mathrm{dmed_{s}}$	$\sigma$	$\mathrm{dmed_{s}}$	roll	$\mathrm{dmed_{s}}$
1	0.0412	$6^{\circ}$	0.0384	$0^{\circ}$	0.0384
2	0.0445	$10^{\circ}$	0.0384	$2^{\circ}$	0.0383
8	0.0384	$15^{\circ}$	0.0385	$4^{\circ}$	0.0382
16	0.0374	$20^{\circ}$	0.0385	$6^{\circ}$	0.0386
		$30^{\circ}$	0.0384	$8^{\circ}$	0.0378

Table 5: Deployment sensitivity of the frozen

K{=}8

d8 model (mean-pool fusion), test split,

\mathrm{dmed_{s}}

(lower is better). Left: view-count scaling at inference. Center: Gaussian pose-encoding noise of standard deviation

\sigma

(five seeds). Right: image-plane roll of each input view.

method	FWHM	peak	$\mathrm{dmed_{s}}$
GT profiles	1.00	1.00	—
d8 (global cond.)	2.00	0.59	0.0338
m3b v1 (per-ray L1)	2.00	0.62	0.0364
m3b v2 $\gamma{=}1.5$	1.33	0.49	0.0354
m3b v2 $\gamma{=}2$	1.00	0.39	0.0348
v1 + post-hoc $p^{\gamma=2}$	1.00	0.38	0.0343

Table 6: Profile morphology (val). FWHM = width ratio pred/GT (GT

=1.0

); peak

=\max_{r}P(r)

(GT

\approx 1

Figure 3: Morphology frontier on the validation split. Both learned sharpening ( $\hat{s}^{\gamma}$ , blue) and post-hoc $p^{\gamma}$ (orange) trace the same monotonic width–peak trade-off; soft-depth error improves along the frontier. The original gate (grey box, FWHM $\leq 1.5$ $\wedge$ peak $\geq 0.5$ ) is unattainable.

model	$\mathrm{dmed_{s}}$	width	pairwise Chamfer
VAE reconstruction (M1)	0.0419	2.25	—
latent diffusion, unconditioned (M2)	—	= recon (2.25)	0.097 (GT 0.150)
cond. diffusion, single sample (M3a)	0.0489	—	—
cond. diffusion, best-of-8	0.0386	—	—
cond. diffusion, best-of-8, CFG $w{=}8$	0.0337	—	—

Table 7: Generative branch (val). Width

=

FWHM ratio to GT (GT

=1.0

); on-manifold per the criterion of Sec. 4.4 . Pairwise Chamfer is intra-object between samples; inter-object GT reference

=0.150

CFG weight $w$	spread	rel. to inter-object
0 (unconditioned)	0.0820	69%
1	0.0217 $[0.014,0.040]$	18%
8 (strong)	0.0445	37%

Table 8: Multi-solution spread vs. classifier-free guidance (M3a, val). Spread = intra-object pairwise Chamfer of

N{=}8

samples; inter-object reference 0.1194.

metric	scan1	scan6
silhouette overlap	1.000	1.000
cross-view depth, $\mathrm{p50}$	0.0063	0.011
consistency (10%)	0.90	0.75
floaters	0	0
GT Chamfer, $\mathrm{p50}$ [mm]	3.98	4.62
gt2cloud [mm]	1.27	1.64

Table 9: Real-photo front-end audit on DTU (GOF 20,000 iters).

Multi-View 3D Reconstruction

Multi-View 3D Reconstruction recovers scene geometry and appearance from images captured at multiple viewpoints.

See all 278 papers

GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction
May 22, 2026Katharina Schmid, Nicolas von Lützow, Jozef Hladký +23D Reconstruction Multi-View 3D Reconstruction
Technical University of Munich · Computing Systems Lab, Huawei Technologies, Switzerland
Free-Range Gaussians: Non-Grid-Aligned Generative 3D Gaussian Reconstruction
Apr 6, 2026Ahan Shabanov, Peter Hedman, Ethan Weber +103D Reconstruction Multi-View 3D Reconstruction
Meta Reality Labs · Simon Fraser University · University of Oxford +1
ReconPlusGen: Injecting Reconstruction Prior into Multi-view 3D Generation through Noise Inversion and Modulation
Sep 10, 2026Jiarui Liu, Heng Li, Weiyu Li +73D Shape Generation 3D Diffusion Models
HKUST · LightIllusions · BYD +1

May 22, 2026J/K move · Enter open · S save

GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction

Katharina Schmid, Nicolas von Lützow, Jozef Hladký, Angela Dai+1

Technical University of Munich · Computing Systems Lab, Huawei Technologies, Switzerland

3D Reconstruction Multi-View 3D Reconstruction 3D Mesh Reconstruction Indoor Scene Reconstruction Conditional Generative Modeling

Open paper AskPDF arXiv

We introduce a new approach to high-fidelity 3D scene reconstruction from multi-view RGB images that tightly couples reconstruction with a strong generative 3D prior. We cast scene reconstruction as conditional 3D generation over a set of spatially-localized, overlapping chunks that together tile the scene, scaling generation to large scene extents. Crucially, we inherit the fidelity and completeness of state-of-the-art generative shape models -- we use Trellis.2 as an example -- which we generalize to the scene level. To this end, we propose a projection-based conditioning mechanism that lifts posed multi-view image features into a coherent 3D representation aligned with the generative model, independent of view ordering and spatially anchored to the scene, yielding high-fidelity, multi-view consistent generated geometry. This enables lifting the strong object-level prior of Trellis.2 to multi-view, scene-scale generation, producing faithful, editable PBR mesh reconstructions of indoor environments. As a result, we obtain high-fidelity results that outperform cutting-edge reconstruction methods by 16%.

Abstract

Multi-View 3D Reconstruction

Explore similar work

GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction

Free-Range Gaussians: Non-Grid-Aligned Generative 3D Gaussian Reconstruction

ReconPlusGen: Injecting Reconstruction Prior into Multi-view 3D Generation through Noise Inversion and Modulation