Integrating Learned Priors through Prior-Space Reparameterization for Physically Based Inverse Rendering

Hun Hwangbo; Minjun Park; Yongsung Kim; Wooseok Song; Jungmin Ko; Junghyun Hwang; Sungroh Yoon

cs.CVNov 5, 2026

Integrating Learned Priors through Prior-Space Reparameterization for Physically Based Inverse Rendering

Authors: Hun Hwangbo, Minjun Park, Yongsung Kim, Wooseok Song, Jungmin Ko, Junghyun Hwang, Sungroh Yoon

Organizations: IPAI, Seoul National University · ECE, Seoul National University · FAM, Seoul National University

Abstract

Physically based inverse rendering decomposes appearance into materials and illumination, an ambiguous problem for which learned material predictions provide useful prior information. Most existing methods incorporate these cues through auxiliary regularization losses, which alter the optimization objective and require balancing their influence against photometric fitting. We propose a prior-guided material reparameterization that uses the predicted feature relationships to restrict the optimization space while leaving the objective unchanged. We fuse multi-view material predictions onto a mesh and construct a permutohedral lattice in the resulting prior feature space. Surface points with similar predictions share material variables, which are optimized jointly with geometry and illumination using a purely photometric objective. Experiments with two learned predictors on three synthetic benchmarks and Stanford-ORB demonstrate improved material decomposition and relighting. Our method improves relighting PSNR by 1.5-2.5 dB over the strongest evaluated mesh-based baseline on Synthetic4Relight and TensoIR-Synthetic, and outperforms a regularization loss using the same prior in controlled comparisons. We further show that our reparameterization pools noisy photometric gradients, producing updates that are more consistent across sampled views. This consistency enables a shortened joint optimization schedule that takes less than a minute while retaining relighting quality above all evaluated baselines on TensoIR-Synthetic.

Figures & tables

Figure 2: Pipeline overview and prior-space material updates. Multi-view images supervise surface reconstruction and provide material predictions. Input RGB refines predicted albedo before fusion onto the initial mesh. The prior defines a fixed permutohedral lattice over albedo RGB, roughness and metallic, with spacing $\sigma_{\mathrm{prior}}$ . Per-vertex photometric gradients are splatted onto the lattice, and the resulting updates are sliced back onto the mesh vertices. For visualization, the lattice is drawn as a regular 3-D grid. In the 5-D permutohedral lattice, each vertex uses the six corners of its enclosing simplex.

	Novel view synthesis		Albedo		Roughness	Normal	Relighting		Time (min)
Method	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$	MAE ( ∘ ) $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	Opt. $\downarrow$	Total $\downarrow$
Synthetic4Relight [ 44 ]
Neural-PBIR [ 36 ]	32.06	.940	29.54	.933	.133	–	30.76	.958	7.2	18
MaterialFusion [ 29 ]	32.66	.958	28.31	.941	.110	–	29.83	.949	122.2	122
DiffReg-PBIR [ 28 ]	32.54	.949	31.31	.958	.098	–	29.94	.958	23.1	128
Ours (MVInverse)	34.26	.954	33.29	.968	.073	–	32.23	.966	6.4	29

Table 1: Synthetic benchmarks. Means over four scenes each. Opt. denotes joint optimization time. Total additionally includes setup, prior inference and mesh initialization.

Panel of Figure 3 — Figure 3: Qualitative comparison. Novel views and relighting on Synthetic4Relight chair and Stanford-ORB gnome, and relighting on DTC-Synthetic teapot. Numbers are full-image PSNR (PSNR-H on Stanford-ORB).

	Novel view synthesis			Relighting
Method	PSNR-H	PSNR-L	SSIM	PSNR-H	PSNR-L	SSIM
Neural-PBIR	30.45	37.49	.9882	26.61	34.07	.9822
MaterialFusion	28.42	36.07	.9820	25.76	33.45	.9761
DiffReg-PBIR	30.45	38.92	.9887	27.80	36.16	.9843
Ours (MVI)	30.72	38.63	.9902	27.82	35.85	.9849
Ours (DR)	30.79	38.70	.9903	27.93	35.86	.9851

Table 2: Stanford-ORB [ 22 ] . The seven scenes of the light scene split, scored by the official benchmark (PSNR-H on HDR images, PSNR-L on tone-mapped images). MVI: MVInverse; DR: DiffusionRenderer.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
DiffReg-PBIR [ 28 ]	42.87	.9961	.0063
Ours (MVI)	44.27	.9962	.0060
Ours (DR)	44.49	.9969	.0054

Table 3: DTC-Synthetic. Relighting metrics using the released evaluation protocol: seven-scene means with three novel illuminations and 20 views per illumination.

	Synthetic4Relight				TensoIR-Synthetic				ORB light
Variant	NVS $\uparrow$	Relit $\uparrow$	Albedo $\uparrow$	Rough. $\downarrow$	NVS $\uparrow$	Relit $\uparrow$	Albedo $\uparrow$	Rough. $\downarrow$	NV $\uparrow$	NL $\uparrow$
Per-vertex, no prior	34.10	31.35	29.73	.129	31.50	27.46	26.23	.276	30.85	26.87
Per-vertex, prior as a loss	33.84	31.95	31.48	.094	31.19	27.93	27.43	.288	30.28	27.80
Prior as a reparameterization (ours)	34.26	32.23	33.29	.073	31.29	28.21	28.51	.233	30.72	27.82

Table 4: How the prior enters the material optimization. The variants share refined MVInverse predictions, scene initialization, illumination parameterization and the joint optimization schedule.

Reference
Method	NVS $\uparrow$	Relit $\uparrow$	Albedo $\uparrow$	Rough. $\downarrow$	Opt. (min) $\downarrow$
DiffReg-PBIR [ 28 ]	26.28	24.67	26.03	.340	23.51
Ours	31.29	28.21	28.51	.233	6.44
Ours (DR)	31.40	28.41	28.72	.234	5.91
Speedup
No prior	30.19	26.90	26.02	.268	0.55

Table 5: Optimization in less than a minute. TensoIR-Synthetic, four-scene means. Speedup uses 100 steps. The three-arm comparison uses the MVInverse version. (DR) additionally reports our DiffusionRenderer version.

Figure 4: Update consistency across sampled views. Material-logit update energy at step 100 for the three variants of Tab. 4 . Their energy is split into noise ( ) and a consistent component attributed to photometric ( ) and the prior loss ( ), as defined in Sec. 6.1 . Mean over 15 scenes.

Figure 5: Refining the prior prediction. Top: input RGB and fused MVInverse-predicted albedo on Stanford-ORB Curry before and after refinement. Bottom: ground truth and 1000-step relight render with and without refinement.

Dataset/prior	Mesh	Lattice	Oracle	Optim.	Oracle	Optim.
	Point counts		Per-vertex		Prior lattice
S4R/MVI	296.5K	26.4K	36.33	29.73	35.23	33.29
TIR/MVI	311.3K	40.7K	33.60	26.23	32.86	28.51
S4R/DR	287.5K	1.7K	36.38	29.91	35.12	33.08
TIR/DR	317.1K	1.8K	33.87	26.37	32.86	28.72

Table 6: Representational capacity of the lattice reparameterization. Four-scene albedo PSNR means for Synthetic4Relight and TensoIR-Synthetic, with MVInverse or DiffusionRenderer prior. Optimized results are 1000-step runs. Oracle is from a single seed.

		Synthetic4Relight				TensoIR-Synthetic				Stanford-ORB
Initialization	Material LS	NVS	Relit	Albedo	Rough. $\downarrow$	NVS	Relit	Albedo	Rough. $\downarrow$	NV	NL
MVI-MVI	w/o ( $\lambda=0$ )	34.33	32.22	32.75	.085	31.45	28.06	27.70	.255	30.99	27.57
MVI-MVI	with ( $\lambda=20$ )	34.28	32.25	33.21	.076	31.23	28.22	28.53	.233	30.74	27.86
DR-DR	w/o ( $\lambda=0$ )	34.27	32.32	32.87	.079	31.44	28.44	28.32	.250	30.98	27.88
DR-DR	with ( $\lambda=20$ )	34.20	32.28	33.06	.074	31.31	28.37	28.70	.237	30.77	27.95

Table S1: Effect of material Large Steps on a single lattice. Each pair shares the same refined predictions, initial scene, lattice spacing

\sigma_{\mathrm{prior}}

, illumination and 1000-step schedule. One seed per scene, means over four Synthetic4Relight, four TensoIR-Synthetic and seven Stanford-ORB scenes. ORB uses official HDR PSNR. Geometry and illumination retain their Large Steps parameterizations in both variants. MVI-MVI and DR-DR use MVInverse and DiffusionRenderer predictions, respectively, for both normal guidance and material priors.

Variant	NVS	Relit	Albedo	Rough. $\downarrow$
No prior	30.19	26.90	26.02	.268
Prior as a loss	30.08	27.40	26.97	.263
Mesh Large Steps	30.63	27.15	26.53	.254
STAdam (adaptation)	30.30	26.99	26.23	.264
Ours	30.60	27.50	27.51	.256

Table S3: Material smoothing under the 100-step schedule. TensoIR-Synthetic means over four scenes.

Dataset	Prior	Mesh vertices	Lattice points	Vertex	Linear	Logit	Per-vertex opt.	Lattice opt.
S4R	MVI	296.5K	26.4K	36.33	35.40	35.23	29.73	33.29
S4R	DR	287.5K	1.7K	36.38	35.26	35.12	29.91	33.08
TIR	MVI	311.3K	40.7K	33.60	32.92	32.86	26.23	28.51
TIR	DR	317.1K	1.8K	33.87	32.89	32.86	26.37	28.72

Table S4: Albedo oracle fits with both prior models. Four-scene means of 200-view albedo PSNR (dB), using one retained seed for oracle fits and point counts, and three seeds for both optimized representations. Mesh vertex and lattice point counts are scene means; K denotes thousands. Vertex denotes fused GT on the final mesh of ours; linear and logit denote lattice fits in the respective spaces.

Figure S1 : Reconstructing a fig tart. RGB texture reconstruction from gray using prior-space reparameterization. Numbers report full-image PSNR and occupied lattice vertices.

Figure S2 : Printed detail before joint optimization. Input RGB and, for each predictor, per-view albedo, unrefined fused albedo and refined fused albedo on two Stanford-ORB labels. Each row uses the same training camera and crop. Fused priors are interpolated on the same initial mesh.

Inverse Rendering

Inverse rendering estimates scene geometry, materials, lighting, and other causes of images from observed visual or sensor data.

See all 70 papers

Diffusion-Based Material Regularization for Physics-Based Inverse Rendering
Jun 30, 2026Jingwang Ling, Lifan Wu, Feng Xu +1Inverse Rendering Diffusion-Based Inverse Problems
University of Illinois Urbana-Champaign · NVIDIA · BNRist and School of Software, Tsinghua University
Dynamic Inverse Rendering for Enhanced Material-Lighting Decomposition
Jul 10, 2026Raza Yunus, Benjamin Ummenhofer, Jan Eric Lenssen +1Inverse Rendering Intrinsic Image Decomposition
University of Technology Nuremberg · Max Planck Institute for Informatics, Saarland Informatics Campus · Intel +1
GAINS: Gaussian-based Inverse Rendering from Sparse Multi-View Captures
Dec 10, 2025Patrick Noras, Jun Myeong Choi, Didier Stricker +2Inverse Rendering 3D Gaussian Splatting
University of Kaiserslautern-Landau, Kaiserslautern, Germany · German Research Center for Artificial Intelligence, Kaiserslautern, Germany · University of North Carolina at Chapel Hill, NC, USA +1

Jun 30, 2026J/K move · Enter open · S save

Diffusion-Based Material Regularization for Physics-Based Inverse Rendering

Jingwang Ling, Lifan Wu, Feng Xu, Shuang Zhao

University of Illinois Urbana-Champaign · NVIDIA · BNRist and School of Software, Tsinghua University

Inverse Rendering Diffusion-Based Inverse Problems Physically Based Rendering Inverse Problems

Open paper AskPDF arXiv

Reconstructing physics-based 3D assets -- geometry, materials, and illumination -- from multi-view images is a core problem in computer graphics and vision, and a prerequisite for realistic relighting and editing. Physics-based inverse rendering offers an accurate image-formation model, but is severely underconstrained: without strong priors, illumination is baked into materials, and reconstructions generalize poorly to novel views and lighting. Data-driven diffusion models, in contrast, predict visually plausible materials, yet their predictions rarely satisfy the rendering equation and are not directly usable for physics-based rendering. We bridge these two paradigms rather than replacing either. Our key idea is to treat the predictions of a state-of-the-art diffusion model not as target material values but as a similarity kernel for optimization: we introduce a regularization loss that penalizes deviations in the optimized material over surface regions where the diffusion predictions are near-constant, while leaving the optimization free to match the input images. Built on this regularizer, our end-to-end pipeline jointly reconstructs geometry, materials, and illumination, yielding high-quality assets that drop into standard rendering pipelines and relight faithfully. On the Synthetic4Relight, Stanford-ORB, and DTC-Synthetic datasets, our method significantly outperforms state-of-the-art baselines in both reconstruction accuracy and relighting quality.

Abstract

Inverse Rendering

Explore similar work

Diffusion-Based Material Regularization for Physics-Based Inverse Rendering

Dynamic Inverse Rendering for Enhanced Material-Lighting Decomposition

GAINS: Gaussian-based Inverse Rendering from Sparse Multi-View Captures