Humans can understand the 3D structure of scenes even from highly unfamiliar or extreme viewpoints. Cognitive science suggests this generalization stems from a top-down generative inference strategy. In contrast, state-of-the-art 3D reconstruction neural networks typically rely on amortized, feedforward inference. While fast, this pattern-matching approach degrades when faced with out-of-distribution (OOD) scene conditions unseen during training. Inspired by the generative nature of human vision, we propose Infer3D, an inverse generative modeling framework that recovers the latent variables of a generative model to best match the monocular input. To prevent degenerate flat outputs, we couple the generative image prior with a frozen 3D lifting network that constrains the optimization space to a valid, multi-view-consistent 3D manifold. Across a variety of controlled synthetic and real-world tasks, Infer3D demonstrates superior OOD reconstruction accuracy compared to feedforward networks trained on equal or larger datasets. Furthermore, mirroring how human perception requires more time to process confusing stimuli, we introduce an adaptive inference strategy that expends more test-time compute on ambiguous OOD examples while efficiently routing in-distribution inputs. Project website: https://amanikiruga.github.io/infer3d-site/
Figures & tables
Figure 1
Figure 3: Analysis-by-synthesis inference paradigm. To handle OOD inputs, we employ a test-time optimization loop (red arrow). We freeze the Image Generator and 2D to 3D Lifter (indicated by lock icons) to preserve the learned prior. Instead, we optimize the Latent Code and Rendering Parameters (indicated by flame icons) to minimize the reconstruction loss L between the rendered reconstruction and the OOD input. This ensures the input to the lifter remains a valid predicted ID image while matching the OOD observation.
Figure 4: Robustness to geometric and sensor domain shifts. (a) Viewpoint shift. Columns show the OOD input, Splatter Image baseline, Infer3D (Ours), and ground truth. Unlike the baseline, our method recovers view-consistent geometry without collapsing or hallucinating structure. (b) Sensor shift (fisheye distortion). Left to right: (1) OOD fisheye input, (2) our learned displacement field, (3) CATSplat baseline (incorrectly modeling distortion as curved geometry), and (4) our disentangled 3D geometry.
Figure 5: Robustness to Unseen Texture and Lighting. Despite training exclusively on SRN Cars ( Sitzmann et al., 2019 ) , our method ( Infer3D ) integrates semantic features (DINOv2) to infer plausible 3D structures unlike feedforward baseline Splatter Image (2nd column) or generative baseline LGM (3rd column) trained on equal or larger (Objaverse) synthetic datasets.
Table 7Table 8Table 9
Figure 6: Reconstruction from in-the-wild real input images. Each row shows: the original photo; the binary mask fed to the model; the Splatter Image feed-forward prediction; and our analysis-by-synthesis reconstruction. Our method is robust to inputs of varying quality, including low-resolution photos, cluttered backgrounds, and masks that partially occlude the object.
Figure 11
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Generative Backbone Architectures. (a) StyleGAN2: Optimization occurs in the W latent space for semantic consistency. (b) Diffusion Autoencoder (DiffAE): Optimization leverages both a semantic latent zsem and a stochastic latent xT . Its architecture allows for an encoder-based warm-start.
Figure 9: Dataset splits for viewpoint robustness evaluation. Left: In-distribution training images from CO3D (Hydrants, Vases) and ShapeNet. These observations are captured from standard, constrained viewpoints, such as fixed elevation rings. Right: Out-of-distribution (OOD) test images featuring severe camera pose shifts. These include random SE(3) translations and SO(3) rotations over the full viewing sphere, resulting in unconventional angles, extreme elevations, and partial occlusions that challenge standard feedforward priors.
Parameter
Value or schedule
Meaning
R
30×20=600
Initial rotation–latent hypotheses
N
783
Total iterations
B
10
Maximum simultaneous hypotheses
kt
600→32→10→5
Pruning at t=3,122,302
λMSE
1→10→2
Changes at t=302,732
λLPIPS
0.5→2
Change at t=302
Appendix
Table 11: Optimization schedule. Population sizes, reconstruction-loss weights, and learning rates used in the reported inference procedure.
Configuration
PSNR ↑
Δ PSNR
Full Infer3D
20.29
–
Without pose optimization
17.19
−3.10
Without latent optimization
18.02
−2.27
Without both
16.21
−4.08
Without MSE
19.09
−1.20
Without LPIPS
19.76
−0.53
Appendix
Table 12: Component ablations on OOD ShapeNet Cars. Changes are relative to full Infer3D. The full configuration uses 30 pose and 20 latent initializations.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Splatter Image
15.73
0.665
0.266
Test-Time Training
12.41
0.569
0.382
Infer3D (DAE)
18.46
0.714
0.220
Appendix
Table 13: Comparison with direct test-time training on OOD CO3D Hydrants. Best results are shown in bold.
Category
Direct lifter CD ↓
Infer3D-EqM CD ↓
Reduction
Teapot
0.214
0.157
27%
Mug
0.225
0.171
24%
Vase
0.278
0.180
35%
Pineapple
0.147
0.096
35%
Wine bottle
0.145
0.109
25%
Helmet
0.214
0.199
7%
Appendix
Table 14: Reconstruction with an Equilibrium Matching prior. Chamfer Distance (CD) and its relative reduction compared with applying the frozen lifter directly.
Figure 10: Qualitative reconstruction with an EqM prior. Left: partially occluded inputs. Middle: direct reconstructions from the frozen Splatter Image lifter. Right: Infer3D-EqM reconstructions. Two views are shown per reconstruction.
Setting
Method
PSNR ↑
SSIM ↑
LPIPS ↓
In-distribution
SAM 3D
16.00
0.79
0.181
Splatter Image
24.27
0.911
0.107
Infer3D (SG)
21.61
0.878
0.133
OOD SO(3)
SAM 3D
16.18
0.79
0.194
Splatter Image
18.12
0.822
0.208
Infer3D (SG)
20.67
0.865
0.148
Appendix
Table 15: Comparison with SAM 3D on ShapeNet (11 categories). Best results within each setting are shown in bold.
Figure 11: SAM 3D failure cases under OOD input poses. Each row shows an input image (left), two views of the ground-truth shape (middle), and two views of the SAM 3D reconstruction (right).
Figure 12: Reconstruction pipeline for a robotic manipulation scene. A wrist-camera observation undergoes depth estimation and object segmentation, followed by Infer3D optimization.