ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illumination

Jan-Niklas Dihlmann; Mark Boss; Simon Donne; Andreas Engelhardt; Hendrik P. A. Lensch; Varun Jampani

cs.CVMar 20, 2026

ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illumination

Authors: Jan-Niklas Dihlmann, Mark Boss, Simon Donne, Andreas Engelhardt, Hendrik P. A. Lensch, Varun Jampani

Organizations: University of Tübingen · Stability AI

Abstract

Reconstructing 3D assets from images has long required separate pipelines for geometry reconstruction, material estimation, and illumination recovery, each with distinct limitations and computational overhead. We present ReLi3D, the first unified end-to-end pipeline that simultaneously reconstructs complete 3D geometry, spatially-varying physically-based materials, and environment illumination from sparse multi-view images in under one second. Our key insight is that multi-view constraints can dramatically improve material and illumination disentanglement, a problem that remains fundamentally ill-posed for single-image methods. Key to our approach is the fusion of the multi-view input via a transformer cross-conditioning architecture, followed by a novel unified two-path prediction strategy. The first path predicts the object's structure and appearance, while the second path predicts the environment illumination from image background or object reflections. This, combined with a differentiable Monte Carlo multiple importance sampling renderer, creates an optimal illumination disentanglement training pipeline. In addition, with our mixed domain training protocol, which combines synthetic PBR datasets with real-world RGB captures, we establish generalizable results in geometry, material accuracy, and illumination quality. By unifying previously separate reconstruction tasks into a single feed-forward pass, we enable near-instantaneous generation of complete, relightable 3D assets. Project Page: https://reli3d.jdihlmann.com/

Figures & tables

Figure 1: Fast, illumination disentangled reconstructions. ReLi3D reconstructs high-quality 3D meshes with physically based materials from sparse input images, while disentangling illumination effects; all in just 0.3s. It is robustly trained on cross-domain datasets and excels in both single- and multi-view cases, on synthetic data as well as on real-world examples.

Figure 2: ReLi3D Overview. Multi-view input images are fused by a shared cross-conditioning transformer into two parallel paths: a Geometry & Appearance Path (blue) using a Triplane Transformer to predict mesh geometry and PBR materials, and an Illumination Path (green) using a Multi-View Illumination Transformer to estimate HDR environments. Both paths are unified through a differentiable Monte Carlo Multiple Importance Sampling rendering to learn to wproduce complete relightable 3D assets.

Figure 3: PBR & Relighting Results. We show that our spatially varying PBR prediction is faithful to the ground truth and therefore produces highly detailed and realistic relightings.

		Polyhaven + Blender Shinny
		Relighting			Image			Basecolor			Roughness			Metallic
Method	Time (s)	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LSSIMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	RMSE $\downarrow$
SF3D	0.26	15.79	0.843	0.150	18.03	0.875	0.120	18.42	0.831	0.220	19.60	0.876	0.127	28.37	0.888	0.116
SPAR3D	$0.36$	15.23	0.836	0.154	17.02	0.862	0.132	17.70	0.822	0.251	19.53	0.874	0.121	30.52	0.895	0.088
3DTopia-XL	$31.38$	14.20	0.869	0.140	14.60	0.853	0.168	19.52	0.818	0.330	15.16	0.847	0.191	27.60	0.861	0.071
Hunyuan3D	$69.40$	14.81	0.845	0.151	17.41	0.875	0.118	21.25	0.837	0.265	—	—	—	—	—	—

Table 1: Relighting & Image & PBR Metrics Comparison. (Left) Relighting performance. (Middle) Image reconstruction performance. (Right) PBR material reconstruction performance. While most methods produce only global PBR parameters, ours produce spatially varying material maps which increase in quality with more views.

Figure 4: Illumination Comparison. (Left) Single view, illumination prediction results compared to ground truth and SPAR3D, which also predicts RENI++ latents, indicating our severely improved method. (Right) Influence of increasing numbers of views and background information. Notice how well we can predict the illumination in the top rows with background information locate light sources correctly, whereas the bottom row is more spread out as it is inferred from diffuse surface reflections only.

		Gso + Standford Orb							Uco3D
		3D				Image			3D				Image
Method	Time (s)	CD $\downarrow$	FS $@$ 0.1 $\uparrow$	FS $@$ 0.2 $\uparrow$	FS $@$ 0.5 $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CD $\downarrow$	FS $@$ 0.1 $\uparrow$	FS $@$ 0.2 $\uparrow$	FS $@$ 0.5 $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
SF3D	0.28	0.132	0.543	0.810	0.974	17.64	0.856	0.131	0.248	0.297	0.564	0.867	12.79	0.748	0.288
SPAR3D	0.39	0.152	0.507	0.766	0.959	16.34	0.837	0.151	0.232	0.368	0.634	0.871	12.39	0.723	0.285
TripoSG	$8.54$	0.232	0.357	0.619	0.881	14.47	0.832	0.211	0.274	0.297	0.520	0.842	11.85	0.752	0.330
3DTopia-XL	$45.03$	0.217	0.341	0.636	0.907	14.40	0.831	0.183	0.250	0.262	0.512	0.888	12.00	0.727	0.304

Table 2: 3D and Image Metrics. ReLi3D clearly achieves SOTA in single and sparse multi-view reconstruction while also achieving great speeds. It is worth noting that that TripoSG and Hunyuan3D also produce signficantly higher vertex counts (100k+ vs 4.5k for ours).

	Stanford ORB
	3D				Image			Basecolor
Method	CD $\downarrow$	FS $@$ 0.1 $\uparrow$	FS $@$ 0.2 $\uparrow$	FS $@$ 0.5 $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
SF3D	0.152	0.512	0.769	0.954	17.75	0.891	0.111	18.52	0.865
SPAR3D	0.165	0.488	0.751	0.940	17.10	0.886	0.113	17.80	0.857
Trellis	0.152	0.561	0.782	0.948	17.13	0.888	0.112	—	—
Hunyuan3D	0.141	0.571	0.801	0.960	16.96	0.877	0.110	21.37	0.872

Table 3: Real-world Evaluation on Stanford ORB. Quantitative evaluation on Stanford ORB dataset showing 3D reconstruction, image quality, and basecolor material prediction performance. Our method outperforms baselines across all metrics and improves with more input views.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: PBR Decomposition Results. Our method is capable of producing highly detailed textures and geometries even from a single view. It is also the only method capable of reproducing accurate spatially varying PBR parameters, which are essential for relighting.

Figure 6: Reconstruction Results (Synthetic). Our method performs well across synthetic data and shows accurate reconstructions from a single view. Other methods show collaps with bend or flat predictions.

Figure 7: Reconstruction Results (Real World). Our method produces accurate reconstructions for real-world data, although challenging. Incorporating multiple views improves the performance further by clearing up uncertainties in unseen areas.

Figure 8: Real-world material prediction. Material maps (albedo, roughness, metallic, normal) for real-world objects from UCO3D dataset on very challenging settings, strong reflections and blur. Our method is still able to make a rough prediction and faithfully separates metallic and non-metallic materials.

Figure 9: Varying materials and complex objects. Results on the Blender Shiny dataset showing spatially varying PBR material prediction on complex multi-material objects. The figure shows predicted basecolor, roughness, metallic, and normal maps, along with relit renderings in novel environments.

Figure 10: Illumination Comparison. Comparison of illumination prediction results between DiffusionLight, SPAR3D, and our method (ReLi3D). Predicted environmens vary vastly while ours mimics the ground truth shape and color, DiffusionLight hallucinates a completely different environment, SPAR3D fails.

Time (s) $\downarrow$			PSNR $\uparrow$
Diffusion Light	ReLi3D	SPAR3D	Diffusion Light	ReLi3D	SPAR3D
21.46	0.34	0.33	20.93	20.88	17.10

Appendix

Table 4: Quantitative evaluation of illumination disentanglement. Comparison of environment map prediction and relighting quality on Polyhaven+HDRI dataset.

	3D				Image
Method	CD $\downarrow$	FS $@$ 0.1 $\uparrow$	FS $@$ 0.2 $\uparrow$	FS $@$ 0.5 $\uparrow$	PSNR $\uparrow$
ReLi3D	0.110	0.676	0.870	0.975	19.92
- MC-Render	0.114	0.668	0.865	0.971	17.54

Appendix

Table 5: Ablation study. Impact of removing the differentiable Monte-Carlo renderer (- MC-Render).

Method	3D Coverage Share (%)	Image Quality Share (%)	Basecolor Share (%)	Roughness Share (%)	Metallic Share (%)
Phase 2 (256) vs Phase 1 (128)	20.2	6.8	22.6	6.3	7.7
Phase 3 (512) vs Phase 2 (256)	70.3	50.0	23.9	31.3	41.0
MC (Full) vs Phase 3 (512)	9.5	43.2	53.5	62.4	51.3

Appendix

Table 6: Training stage contribution analysis. Average share of the total improvement from Phase 1 (128 Gaussians) to the full Monte Carlo stage that is attributable to each intermediate stage. Columns aggregate the metrics shown in Table 1: (1) 3D coverage (CD and [email protected]–0.5), (2) image quality (PSNR, SSIM, LPIPS), (3) basecolor (PSNR, SSIM, LSSIMSE), (4) roughness (PSNR, SSIM, RMSE), and (5) metallic (PSNR, SSIM, RMSE). Early Gaussian stages mainly expand 3D coverage, while the Monte Carlo refinement sharpens PBR material disentanglement.

	Polyhaven Subset
	3D				Image
Method	CD $\downarrow$	FS $@$ 0.1 $\uparrow$	FS $@$ 0.2 $\uparrow$	FS $@$ 0.5 $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
ReLi3D (Random Hero)	0.102	0.697	0.883	0.982	20.25	0.919	0.083
ReLi3D (Frontal Hero)	0.123	0.641	0.840	0.965	19.06	0.909	0.095

Appendix

Table 7: Hero view selection sensitivity. Comparison of metrics using random hero view selection versus always selecting the most frontal view on the Polyhaven dataset.

Figure 11: Failure cases. Failure cases showing challenges with baked in lighting for objects with strong self shadowing (fin of shark) and basecolor prediction difficulties in dark scenes (Rhino). Comparison includes results from Hunyuan3D and our method.

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 930 papers

Sep 7, 2026

RelightFormer: Feed-forward Generative Transformer for Multiview Object Relighting

Image relighting is traditionally tackled via complex inverse rendering pipelines, which suffer from ill-posed optimization, or single-image generative models that ignore crucial multi-view cues necessary for understanding 3D geometry and material interactions. To address these limitations, we introduce a feed-forward generative Transformer for direct single- and multi-view image relighting that entirely bypasses explicit intrinsic property estimation. Adapted from a video foundation model, our architecture features a latent illumination module that dynamically injects target environment maps into spatial features via cross-attention. Furthermore, we employ permutation-invariant positional encodings to symmetrically process unordered multi-view inputs without sequential bias. To train this robust data-driven model, we construct the massive Laval Objaverse Dataset (LOD), comprising 90K objects and 39K unique illuminations. Extensive experiments demonstrate state-of-the-art visual quality, photorealistic relighting quality, and strong zero-shot generalization across single-view, multi-view, and novel-view relighting tasks.

Hejun Wang, Jinxi Li, Junwei Jiang +4

Shenzhen Research Institute, The Hong Kong Polytechnic University, Hong Kong

Image Relighting

Ask PDF

Apr 22, 2026

GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers

Relighting a person from a single photo is an attractive but ill-posed task, as a 2D image ambiguously entangles 3D geometry, intrinsic appearance, and illumination. Current methods either use sequential pipelines that suffer from error accumulation, or they do not explicitly leverage 3D geometry during relighting, which limits physical consistency. Since relighting and estimation of 3D geometry are mutually beneficial tasks, we propose a unified Multi-Modal Diffusion Transformer (DiT) that jointly solves for both: GeoRelight. We make this possible through two key technical contributions: isotropic NDC-Orthographic Depth (iNOD), a distortion-free 3D representation compatible with latent diffusion models; and a strategic mixed-data training method that combines synthetic and auto-labeled real data. By solving geometry and relighting jointly, GeoRelight achieves better performance than both sequential models and previous systems that ignored geometry.

Yuxuan Xue, Ruofan Liang, Egor Zakharov +6

Codec Avatars Lab, Meta · University of Tübingen · Max Planck Institute for Informatics, Saarland Informatics Campus

Image-to-Image Translation Diffusion Transformer Monocular 3D Reconstruction Image Relighting Multimodal Diffusion Models

Ask PDF

Jun 30, 2026

Diffusion-Based Material Regularization for Physics-Based Inverse Rendering

Reconstructing physics-based 3D assets -- geometry, materials, and illumination -- from multi-view images is a core problem in computer graphics and vision, and a prerequisite for realistic relighting and editing. Physics-based inverse rendering offers an accurate image-formation model, but is severely underconstrained: without strong priors, illumination is baked into materials, and reconstructions generalize poorly to novel views and lighting. Data-driven diffusion models, in contrast, predict visually plausible materials, yet their predictions rarely satisfy the rendering equation and are not directly usable for physics-based rendering. We bridge these two paradigms rather than replacing either. Our key idea is to treat the predictions of a state-of-the-art diffusion model not as target material values but as a similarity kernel for optimization: we introduce a regularization loss that penalizes deviations in the optimized material over surface regions where the diffusion predictions are near-constant, while leaving the optimization free to match the input images. Built on this regularizer, our end-to-end pipeline jointly reconstructs geometry, materials, and illumination, yielding high-quality assets that drop into standard rendering pipelines and relight faithfully. On the Synthetic4Relight, Stanford-ORB, and DTC-Synthetic datasets, our method significantly outperforms state-of-the-art baselines in both reconstruction accuracy and relighting quality.

Jingwang Ling, Lifan Wu, Feng Xu +1

University of Illinois Urbana-Champaign · NVIDIA · BNRist and School of Software, Tsinghua University

Inverse Rendering Diffusion-Based Inverse Problems Physically Based Rendering Inverse Problems

Ask PDF

Abstract

3D Reconstruction

Explore similar work

RelightFormer: Feed-forward Generative Transformer for Multiview Object Relighting

GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers

Diffusion-Based Material Regularization for Physics-Based Inverse Rendering