Physically Based Rendering in the Latent Space

Vuk Radovanovic; Vishesh Gupta; Adrien Gruson; Binh-Son Hua

cs.GRSep 17, 2026

Physically Based Rendering in the Latent Space

Authors: Vuk Radovanovic, Vishesh Gupta, Adrien Gruson, Binh-Son Hua

Organizations: Trinity College Dublin

Abstract

Image diffusion models have shown impressive image generation capabilities but are often hard to control, in contrast to classical computer graphics pipelines such as physically based rendering. However, we observe that there is a bridge between light transport phenomena and the distribution of latent space values produced by such models. Thus, we introduce physically based rendering in the feature space learned by the variational autoencoders in generative models, enabling light transport simulation in the latent space. This allows us to leverage physically based rendering techniques to output latent maps for physically guided content generation. We propose modifications to the rendering equation, which, when paired with a differentiable renderer, can yield an optimal set of scene parameters that require only minimal refinement to accurately render into the pretrained latent space. We train our method on a single rendered image, and then demonstrate the generalization of the method to scene geometry changes, lighting changes, and camera view changes.

Figures & tables

Figure 1: Latent maps of the Cornell Box rendering. The zoom-in patches demonstrate visual effects unique to latent representations, including (A) the signed emitter, (B) highlighted geometry edges, and (C) signed shadows, which cannot be rendered using traditional physically based rendering.

Figure 2: We train a differentiable renderer to output to the latent space defined by a variational autoencoder. We assume a 3D scene is available for our rendering task. Our training pipeline consists of taking a ground truth RGB rendering of the scene and encoding it into the latent space. We then optimize the 3D scene parameters to make its rendering reproduce this latent representation using our modified rendering equation ( Eq. 7 ). The rendered latents are then refined using a per-scene refiner network. The final latents are decoded to reconstruct the RGB rendering. Once the pipeline is trained, it can be applied to render latents for variants of the same scene with edited scene settings. The blue dotted line represents backpropagation for scene parameter optimization. The yellow dotted line represents backpropagation for the refiner parameter training.

Figure 3: We compare the performance of the training view on rendered latent and refined latent with the ground truth. Representative latents for each case are shown on the left of each color rendering. Metrics shown are LPIPS/MSEx100.

Figure 4: Comparison between the baseline and our method on the Lamp scene. We observe that our latent modeling and rendering formula provide significant quality improvement compared to the baseline that only relies on latent normalization and traditional rendering. Metrics shown are LPIPS/MSEx100.

Figure 5: The error of the Lamp scene across 60 frames of scene editing variants. Frame 0 corresponds to the training view.

Figure 6: Qualitative results on the Veach-Bidir scene. Metrics shown are LPIPS/MSEx100.

	Var.	$\tilde{I}_{i}^{e}$ ( Eq. 4 )	$+\,\tilde{I}_{i}^{a}$ ( Eq. 5 )	$+\,\tilde{I}_{i}^{c}$ ( Eq. 6 )	+ Refiner
Cornell Box	Cam.	0.502 / 2.264	0.228 / 0.335	0.213 / 0.222	0.075 / 0.069
	Obj.	0.526 / 2.355	0.213 / 0.338	0.202 / 0.261	0.032 / 0.061
	Light	0.519 / 2.361	0.209 / 0.330	0.191 / 0.220	0.013 / 0.024
Lamp	Cam.	0.394 / 3.091	0.206 / 0.844	0.178 / 0.564	0.068 / 0.297
	Obj.	0.439 / 3.274	0.236 / 0.791	0.202 / 0.608	0.020 / 0.048
	Light	0.504 / 5.702	0.292 / 3.000	0.253 / 2.385	0.086 / 1.867

Table 1: Ablation results reporting the mean LPIPS / MSE

\times 100

achieved over the decoded RGBs. “Var.” denotes the scene-edit variant: camera, object, and light motion.

Figure 7: Empirical convergence vs. samples per pixel (spp) for a novel view (top) and a training view (bottom) of the Veach-Bidir scene. LPIPS on the decoded RGB (left) and DSSIM on the latents (right).

Figure 8: Equal-error runtime comparison between RGB path tracing and our latent rendering method (with refiner), for RGB space (left) and latent space (right). The plot is based on frame 30 of the move_camera test case of the Lamp scene.

Panel of Figure 9 — Figure 9: Application using score distillation sampling (SDS) [ 38 ] . We use the pretrained diffusion model to iteratively update the reflectance of the teapot following guidance from the text prompt “A blue teapot and a lamp on a table.”

Figure 10: We compare the performance of the scene editing on rendered latent and refined latent with the ground truth. Representative latents for each case are shown on the left of each color rendering. Metrics shown are LPIPS/MSEx100.

Figure 11: We compare the performance of novel view scene editing on rendered latents and refined latents with the ground truth. We showcase complex scenes with texture support. Metrics shown are LPIPS/MSEx100.

Physically Based Rendering

Physically based rendering models how light interacts with geometry and materials to produce images for synthesis, relighting, and reconstruction.

See all 41 papers

GeRM: A Generative Rendering Model From Physically Realistic to Photorealistic
Apr 10, 2026Jiayuan Lu, Rengan Xie, Xuancheng Jin +5Controllable Image Generation Neural Rendering
State Key Lab of CAD&CG, Zhejiang University and Zhejiang Lab, China · State Key Laboratory of Industrial Control Technology, Zhejiang University, China · Zhejiang University, China
Diffusion-Based Material Regularization for Physics-Based Inverse Rendering
Jun 30, 2026Jingwang Ling, Lifan Wu, Feng Xu +1Inverse Rendering Diffusion-Based Inverse Problems
University of Illinois Urbana-Champaign · NVIDIA · BNRist and School of Software, Tsinghua University
Learning Latent Proxies for Controllable Single-Image Relighting
Mar 16, 2026Haoze Zheng, Zihao Wang, Xianfeng Wu +5Physics-Informed Diffusion Models Controllable Image Generation
HKUST · HKPolyU · CUHK

Apr 10, 2026J/K move · Enter open · S save

GeRM: A Generative Rendering Model From Physically Realistic to Photorealistic

Jiayuan Lu, Rengan Xie, Xuancheng Jin, Zhizhen Wu+4

State Key Lab of CAD&CG, Zhejiang University and Zhejiang Lab, China · State Key Laboratory of Industrial Control Technology, Zhejiang University, China · Zhejiang University, China

Controllable Image Generation Neural Rendering Multimodal Diffusion Models

Open paper AskPDF arXiv

While physically-based rendering (PBR) simulates light transport that guarantees physical realism, achieving true photorealistic rendering (PRR) demands prohibitive time and labor, and still struggles to capture the intractable richness of the real world. We propose GeRM, the first multimodal generative rendering model to bridge the gap from PBR to PRR (P2P). We formulate this P2P transition by learning a distribution transfer vector (DTV) field to direct the generative process. To achieve this, we introduce a multi-condition ControlNet that synthesizes PBR images and progressively transitions them into PRR images, guided by G-buffers, text prompts, and cues for enhanced regions. To improve the model's grasp of the image distribution shift driven by text prompts, we propose a residual perceptual transfer mechanism to associate text prompts with corresponding targeted modification regions, which more clearly defines the incremental component updates. To supervise this transfer process, we introduce a multi-agent visual language model framework to construct an expert-guided pairwise transfer dataset, named P2P-50K, where each paired sample corresponds to a specific transfer vector in the DTV field. Extensive experiments demonstrate that GeRM synthesizes high-quality controllable images and outperforms state-of-the-art baselines across diverse applications, including PBR and PRR image synthesis and editing.

Abstract

Physically Based Rendering

Explore similar work

GeRM: A Generative Rendering Model From Physically Realistic to Photorealistic

Diffusion-Based Material Regularization for Physics-Based Inverse Rendering

Learning Latent Proxies for Controllable Single-Image Relighting