Learning Spherical Occupancy Profiles for Multi-View 3D Reconstruction and Generation

YiHsuan Tsai

cs.CVAug 24, 2026

Learning Spherical Occupancy Profiles for Multi-View 3D Reconstruction and Generation

Authors: YiHsuan Tsai

Organizations: Nanjing University

Abstract

We study spherical occupancy profiles-the ray-wise occupancy probability profiles P(r) = T(r) o(r) distilled from multi-view 3D Gaussian reconstructions-as a unified intermediate representation for both discriminative and generative 3D reconstruction from images. On a 999-object subset of Google Scanned Objects with 48 turntable views each, we train (i) a discriminative per-ray decoder that injects global view-averaged and ray-specific image evidence into a FiLM-conditioned profile head, reaching median soft depth error 0.035 (normalized) on an independent 90-object test split, and (ii) a generative pipeline built on a profile VAE and a latent diffusion model, which supports unconditional sampling that matches the reconstruction manifold and image-conditioned multi-solution reconstruction whose per-object solution spread is quantifiable and tunable via classifier-free guidance. We further analyze the morphology of predicted profiles: post-hoc power sharpening and a learned sharpening target both recover ground-truth profile width without degrading depth, exposing a monotonic width-peak frontier in the L1-per-ray loss family and motivating a principled redefinition of morphology gates. Real-photo validation on two DTU scenes confirms the pipeline transfers to non-synthetic input. Our results suggest that ray-wise occupancy profiles offer a compact, learned, and uncertainty-aware interface between multi-view reconstruction and generative priors.

Figures & tables

Figure 1: Pipeline overview. (a) Volume distillation produces ground-truth spherical occupancy profiles from multi-view captures without mesh supervision; (b) a discriminative per-ray decoder maps images to profile fields; (c) a profile VAE with latent diffusion maps profile fields to a sampleable latent, enabling multi-solution reconstruction. The two branches share the same representation.

method	$\mathrm{dmed_{s}}$		$\mathrm{ch_{s}}$
	val	test	val	test
mean profile	0.0842	—	—	—
d8 (global cond.)	0.0338	0.0384	0.228	0.213
m3b v1 (per-ray L1)	0.0364	0.0356	0.236	0.211
m3b v2 $\gamma{=}1.5$	0.0354	0.0351	0.237	0.214
m3b v2 $\gamma{=}2$	0.0348	0.0356	0.243	0.217

Table 1: Discriminative reconstruction (median over objects; lower is better except peak).

\mathrm{dmed}

= soft-depth error,

\mathrm{ch}

= Chamfer to GT cloud.

Figure 2: Example ray profiles for a mid- and a hard object. GT profiles (black) are narrow with peak $\approx 1$ ; d8 (blue) is over-wide and low; the per-ray model (orange) is sharper; the sharpened-target model (green, $\gamma{=}2$ ) matches GT width on covered rays. Three covered rays (peak bins at the 25/50/75 percentiles) are shown per object.

$\Delta x$ [px]	shift / obj. radius	$\mathrm{dmed_{s}}$	rel. to baseline
0	0%	0.0338	1.00
2	4.2%	0.0338	1.00
4	8.3%	0.0351	1.04
9	18.7%	0.0373	1.10

Table 2: Center-robustness of d8 (90-object validation split). Shifting all 8 views by

\Delta x

px simulates an error in the object-center estimate of the spherical parameterization.

method	forward [ms]	dec [ms]	total [ms]	peak mem [MB]
d8 (1 object)	1.5	0.7	2.2	165
d8 (batch 8)	—	—	1.6/obj	525
m3b v2 (1 object)	3.6	1.2	4.8 a	474
VAE recon (M1)	0.6	0.9	1.5	241
latent diff. sample (M2)	18.5	1.0	19.5	239

Table 3: Median single-object inference latency and peak GPU memory (NVIDIA L20, fp32). forward = view encoding ( d8 / m3b ), latent encoding (M1), or 50-step DDIM (M2); dec = per-ray decoding of the

8192

-ray profile field.

model	views $K$	val	test
mean profile (no image cond.)	—	0.0842	—
d8 (profile)	8	0.0338	0.0384
d8 (profile)	2	0.0430	0.0419
d8 (profile)	1	0.0380	0.0442
RW (RayDF-style)	8	0.0396	0.0369
RW (RayDF-style)	1	0.0455	0.0422

Table 4: External baseline and view scaling on the same split and protocol. RW = ray-wise distance field re-implemented in the spirit of RayDF [ 18 ] , sharing the encoder, fusion, optimizer, and metric with d8 .

\mathrm{dmed_{s}}

= median absolute soft-depth error over covered rays (objects); lower is better.

views $K$	$\mathrm{dmed_{s}}$	$\sigma$	$\mathrm{dmed_{s}}$	roll	$\mathrm{dmed_{s}}$
1	0.0412	$6^{\circ}$	0.0384	$0^{\circ}$	0.0384
2	0.0445	$10^{\circ}$	0.0384	$2^{\circ}$	0.0383
8	0.0384	$15^{\circ}$	0.0385	$4^{\circ}$	0.0382
16	0.0374	$20^{\circ}$	0.0385	$6^{\circ}$	0.0386
		$30^{\circ}$	0.0384	$8^{\circ}$	0.0378

Table 5: Deployment sensitivity of the frozen

K{=}8

d8 model (mean-pool fusion), test split,

\mathrm{dmed_{s}}

(lower is better). Left: view-count scaling at inference. Center: Gaussian pose-encoding noise of standard deviation

\sigma

(five seeds). Right: image-plane roll of each input view.

method	FWHM	peak	$\mathrm{dmed_{s}}$
GT profiles	1.00	1.00	—
d8 (global cond.)	2.00	0.59	0.0338
m3b v1 (per-ray L1)	2.00	0.62	0.0364
m3b v2 $\gamma{=}1.5$	1.33	0.49	0.0354
m3b v2 $\gamma{=}2$	1.00	0.39	0.0348
v1 + post-hoc $p^{\gamma=2}$	1.00	0.38	0.0343

Table 6: Profile morphology (val). FWHM = width ratio pred/GT (GT

=1.0

); peak

=\max_{r}P(r)

(GT

\approx 1

Figure 3: Morphology frontier on the validation split. Both learned sharpening ( $\hat{s}^{\gamma}$ , blue) and post-hoc $p^{\gamma}$ (orange) trace the same monotonic width–peak trade-off; soft-depth error improves along the frontier. The original gate (grey box, FWHM $\leq 1.5$ $\wedge$ peak $\geq 0.5$ ) is unattainable.

model	$\mathrm{dmed_{s}}$	width	pairwise Chamfer
VAE reconstruction (M1)	0.0419	2.25	—
latent diffusion, unconditioned (M2)	—	= recon (2.25)	0.097 (GT 0.150)
cond. diffusion, single sample (M3a)	0.0489	—	—
cond. diffusion, best-of-8	0.0386	—	—
cond. diffusion, best-of-8, CFG $w{=}8$	0.0337	—	—

Table 7: Generative branch (val). Width

=

FWHM ratio to GT (GT

=1.0

); on-manifold per the criterion of Sec. 4.4 . Pairwise Chamfer is intra-object between samples; inter-object GT reference

=0.150

CFG weight $w$	spread	rel. to inter-object
0 (unconditioned)	0.0820	69%
1	0.0217 $[0.014,0.040]$	18%
8 (strong)	0.0445	37%

Table 8: Multi-solution spread vs. classifier-free guidance (M3a, val). Spread = intra-object pairwise Chamfer of

N{=}8

samples; inter-object reference 0.1194.

metric	scan1	scan6
silhouette overlap	1.000	1.000
cross-view depth, $\mathrm{p50}$	0.0063	0.011
consistency (10%)	0.90	0.75
floaters	0	0
GT Chamfer, $\mathrm{p50}$ [mm]	3.98	4.62
gt2cloud [mm]	1.27	1.64

Table 9: Real-photo front-end audit on DTU (GOF 20,000 iters).

Multi-View 3D Reconstruction

Multi-View 3D Reconstruction recovers scene geometry and appearance from images captured at multiple viewpoints.

See all 278 papers

May 22, 2026

GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction

We introduce a new approach to high-fidelity 3D scene reconstruction from multi-view RGB images that tightly couples reconstruction with a strong generative 3D prior. We cast scene reconstruction as conditional 3D generation over a set of spatially-localized, overlapping chunks that together tile the scene, scaling generation to large scene extents. Crucially, we inherit the fidelity and completeness of state-of-the-art generative shape models -- we use Trellis.2 as an example -- which we generalize to the scene level. To this end, we propose a projection-based conditioning mechanism that lifts posed multi-view image features into a coherent 3D representation aligned with the generative model, independent of view ordering and spatially anchored to the scene, yielding high-fidelity, multi-view consistent generated geometry. This enables lifting the strong object-level prior of Trellis.2 to multi-view, scene-scale generation, producing faithful, editable PBR mesh reconstructions of indoor environments. As a result, we obtain high-fidelity results that outperform cutting-edge reconstruction methods by 16%.

Katharina Schmid, Nicolas von Lützow, Jozef Hladký +2

Technical University of Munich · Computing Systems Lab, Huawei Technologies, Switzerland

3D Reconstruction Multi-View 3D Reconstruction 3D Mesh Reconstruction Indoor Scene Reconstruction Conditional Generative Modeling

Ask PDF

Apr 6, 2026

Free-Range Gaussians: Non-Grid-Aligned Generative 3D Gaussian Reconstruction

We present Free-Range Gaussians, a multi-view reconstruction method that predicts non-pixel, non-voxel-aligned 3D Gaussians from as few as four images. This is done through flow matching over Gaussian parameters. Our generative formulation of reconstruction allows the model to be supervised with non-grid-aligned 3D data, and enables it to synthesize plausible content in unobserved regions. Thus, it improves on prior methods that produce highly redundant grid-aligned Gaussians, and suffer from holes or blurry conditional means in unobserved regions. To handle the number of Gaussians needed for high-quality results, we introduce a hierarchical patching scheme to group spatially related Gaussians into joint transformer tokens, halving the sequence length while preserving structure. We further propose a timestep-weighted rendering loss during training, and photometric gradient guidance and classifier-free guidance at inference to improve fidelity. Experiments on Objaverse and Google Scanned Objects show consistent improvements over pixel and voxel-aligned methods while using significantly fewer Gaussians, with large gains when input views leave parts of the object unobserved.

Ahan Shabanov, Peter Hedman, Ethan Weber +10

Meta Reality Labs · Simon Fraser University · University of Oxford +1

3D Reconstruction Multi-View 3D Reconstruction Sparse-View 3D Reconstruction 3D Gaussian Splatting

Ask PDF

Sep 10, 2026

ReconPlusGen: Injecting Reconstruction Prior into Multi-view 3D Generation through Noise Inversion and Modulation

Qualitative results and an illustration of our core idea. Top left: reconstruction results on benchmark images. Top right: reconstruction results on real-world images. Bottom: illustration of reconstruction-guided noise initialization and modulation. Given multiple input images, we predict a point cloud in canonical space, deterministically inject the predicted geometry into the diffusion process through noise inversion, and modulate the resulting noise to preserve the generative flexibility required to complete unobserved regions and refine visible geometry.

Jiarui Liu, Heng Li, Weiyu Li +7

HKUST · LightIllusions · BYD +1

3D Shape Generation 3D Diffusion Models

Ask PDF

Abstract

Multi-View 3D Reconstruction

Explore similar work

GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction

Free-Range Gaussians: Non-Grid-Aligned Generative 3D Gaussian Reconstruction

ReconPlusGen: Injecting Reconstruction Prior into Multi-view 3D Generation through Noise Inversion and Modulation