GeRM: A Generative Rendering Model From Physically Realistic to Photorealistic
Authors: Jiayuan Lu, Rengan Xie, Xuancheng Jin, Zhizhen Wu, Qi Ye, Tian Xie, Hujun Bao, Rui Wang. Yuchi Huo
Organizations: State Key Lab of CAD&CG, Zhejiang University and Zhejiang Lab, China · State Key Laboratory of Industrial Control Technology, Zhejiang University, China · Zhejiang University, China
While physically-based rendering (PBR) simulates light transport that guarantees physical realism, achieving true photorealistic rendering (PRR) demands prohibitive time and labor, and still struggles to capture the intractable richness of the real world. We propose GeRM, the first multimodal generative rendering model to bridge the gap from PBR to PRR (P2P). We formulate this P2P transition by learning a distribution transfer vector (DTV) field to direct the generative process. To achieve this, we introduce a multi-condition ControlNet that synthesizes PBR images and progressively transitions them into PRR images, guided by G-buffers, text prompts, and cues for enhanced regions. To improve the model's grasp of the image distribution shift driven by text prompts, we propose a residual perceptual transfer mechanism to associate text prompts with corresponding targeted modification regions, which more clearly defines the incremental component updates. To supervise this transfer process, we introduce a multi-agent visual language model framework to construct an expert-guided pairwise transfer dataset, named P2P-50K, where each paired sample corresponds to a specific transfer vector in the DTV field. Extensive experiments demonstrate that GeRM synthesizes high-quality controllable images and outperforms state-of-the-art baselines across diverse applications, including PBR and PRR image synthesis and editing.
Figures & tables
Figure 1. Given the physical attributes as input, the physically realistic ( ϕ ) results produced by GeRM are shown on the left side of the figure, with greater photorealism ( ρ ) as they move closer to the right. The sequence 1–4 illustrates this progressive realistic evolution: the model iteratively enriches the scene with realistic lighting dynamics (e.g., the warm sunlight and ground highlight), fine-grained material textures , and context-aware details (e.g., natural fabric wrinkles on the sofa), resulting in a high-fidelity photorealisitic image that consistent with the original image.
Figure 2. A P2P Quad characterizing the correlation between physical realism and photorealsm. Iϕ , Iρ , Eϕ and Eρ are physically realistic image, photorealistic image, digital existence, and real-world existence, respectively.
Figure 3. Overview of our GeRM framework. Our pipeline operates on a multi-condition framework that integrates physical G-buffers with task-adaptive spatial constraints—ranging from zero masks for PBR synthesis to user-specified regions or VLM-driven automatic maps for editing and progressive realism. These combined inputs are fed into the multi-condition ControlNet, which predicts a distribution transfer vector (DTV) field. This DTV field guides the iterative evolution of the scene, transforming the initial physical state ( ϕ ) into the final photorealistic target ( ρ6 ) via iteratively progressive generation.
Figure 4. Pipeline for constructing the progressive pairwise P2P dataset. We employ a multi-agent VLM framework—comprising the Global Auditor , Contextual Enricher , and Local Refiner —to collaboratively generate realistic refinement prompts based on the current state. These prompts are fed into CLIPSeg to produce spatial semantic masks, accurately localizing the regions for realistic modification. Guided by these spatially grounded instructions, the generative backbone (Kontext) executes localized updates, progressively evolving the sterile Engine Render into a photorealistic state (Kontext- ρt ).
Figure 5. Why SOTA Editing Models Are Insufficient for Photorealistic Generation. Under the progressive prompts shortened like “physically realistic appearance,” Nano Banana retains a synthetic appearance, while Kontext exhibits severe spatial instability. In contrast, our method achieves superior photorealism and spatial alignment.
Figure 6. Visual comparison of our iterative PRR generation against standard rendering engines. From left to right: input G-buffers, the baseline render from various engines (Mitsuba3, Blender, Falcor, and UE5), and the PRR results generated by our method. As shown in the four right columns, our approach gradually enhances the realism of the scene, generating details and lighting effects that surpass the original engine renders.
Figure 7. Comparisons of PBR and PRR synthesis on indoor and outdoor scenes. Given input G-buffers (leftmost), the region left of the dashed line validates physical accuracy against PBR Reference and RGB ↔ X. The region to the right evaluates our final PRR generation against IntrinsicControlNet.
Figure 8. Demonstration of progressive editing on different subsets of G-buffer channels. We visualize an editing sequence where modifications are applied sequentially from top to bottom. In each row, the generation builds upon the result from the previous step, controlled by the specific edited G-buffer channel or text prompt shown in the “Edited Conditions” column. We compare our method against RGB ↔ X and IntrinsicEdit, alongside the ablation variants without random dropout (Ours w/o RD) or G-buffer guidance (Ours w/o G-buffers). Additional editing cases are provided in the Appendix A.4 .
Figure 9. Progressive semantic-aware P2P generation. We visualize masks, prompts, and generation results across progressive steps. While existing baselines struggle to balance strict localized mask control with true photorealism, our approach successfully confines high-fidelity enhancements to designated areas, preserving unmasked regions.
US ↑
KID ↓
CLIP (Global-T) ↑
IR ↑
Ours- ρ 3
3.52
0.05834
0.2883
-0.3197
Ours- ρ 6
3.99
0.04572
0.2920
-0.2084
ICNet
1.40
0.06804
0.2806
-0.5022
FLUX.1-Kontext
2.50
0.05975
0.2830
-0.3017
Nano Banana
2.48
0.09024
0.2783
-0.4468
Mitsuba3
1.13
0.08010
0.2857
-0.5444
Table 1. Quantitative evaluation of photorealism. US denotes user study. To measure the capability of generating photorealistic images and details, we employ highly detailed realistic prompts generated by GPT-4o. Our method achieves state-of-the-art performance in terms of realism and text-image alignment. Red , Orange , and Yellow indicate top 3 results.
PSNR ↑
SSIM ↑
LPIPS ↓
CLIP (Global-I) ↑
Ours- ϕ
30.17
0.9520
0.0469
0.9907
RGB ↔ X
9.6972
0.3915
0.4986
0.8861
RGB ↔ X*
21.61
0.9129
0.0796
0.9497
Table 2. Quantitative comparsion on PBR synthesis. " ∗ " means masked images on single object scenes.
BG Accuracy
Semantic Alignment
Overall Quality
Method
PSNR ↑
SSIM ↑
DINO ↑
Global-T ↑
Local-T ↑
Dir-T ↑
BLIP ↑
IR ↑
Aes ↑
Δ CLIP-T ↑
Q-Score ↑
Ours- ρ
25.46
0.8641
0.9428
0.2875
0.2458
0.0887
0.6762
-0.2795
3.9603
0.0381
0.3807
FLUX.1-Fill-dev
31.62
0.9452
0.9883
0.1919
0.2055
0.0740
0.3798
-1.3512
3.9582
0.0264
0.3366
FLUX.1-Kontext-dev
15.39
0.4575
0.6764
0.2830
0.2214
0.0748
0.7486
-0.3017
3.9577
0.0424
0.2877
DiffEdit
24.13
0.8397
0.9276
0.1664
0.1805
0.0002
0.0443
-1.9112
3.9596
0.0009
0.2952
ICEdit
25.12
0.8440
0.8418
0.2806
0.2194
0.0773
0.3635
-0.5022
3.9601
0.0392
0.3470
Table 3. Quantitative comparison of semantic-aware editing. We evaluate our method against various editing methods. FLUX.1-Kontext-dev serves as the backbone for our dataset construction, while Nano Banana represents a state-of-the-art commercial solution. Our method consistently ranks high across metrics, demonstrating superior overall quality while maintaining a robust balance between background preservation accuracy and semantic alignment. Red , Orange , and Yellow indicate 1st, 2nd, and 3rd best results.
Figure 10. Visual ablation study of our PRR generation method. The generation proceeds progressively from top to bottom, where each row represents a generation stage building upon the result of the previous step and the corresponding G-buffer feature (shown on the left). We compare Ours- ρ -Full against variants without residual perceptual transfer mechanism (RPT) during training/inference, without our P2P-50K dataset (P2P), and a direct P2P translation in the RGB domain without G-buffer guidance (w/o G-buffers). The "Mask" column visualizes the semantic injection regions at each stage.
Figure 11. Controllable Photorealism via G-buffer Relaxation. Enabled by our explicit G-buffer controls, users are granted the flexibility to dictate the level of realism by progressively relaxing g-buffer constraints. Notably, explicitly instructing the model to preserve geometry via text prompts without G-buffer guidance (Ours- ρ 6-w/o GB.-PreserveGeo.) fails to prevent severe structural alterations. While Ours- ρ6 -PreserveGeo. strictly maintains the original layout, Ours- ρ6 -NoAdditions enhances materials purely on existing structures without generating new objects. Ultimately, allowing the model to naturally introduce context-aware additions (Ours- ρ6 ) often yields the most authentic and photorealistic results.
Figure 12. Impact of G-buffer guidance scale ( c_scale ). We visualize the progressive photorealistic generation across varying levels of physical control. For a fair and direct comparison, the exact same textual prompts and regional masks are applied across all different scale settings. As generation progresses (left to right), strong guidance (top rows) enforces strict spatial alignment with the input G-buffers. Conversely, relaxing c_scale (bottom rows) grants the model greater freedom to inject richer textures and complex object variations. This demonstrates that regulating c_scale empowers users to flexibly balance strict structural preservation and generative creativity depending on their specific intents.
BG Acc.
SA
Photorealism
Method
PSNR ↑
SSIM ↑
CLIP(Loc-T) ↑
IR ↑
KID ↓
CLIP(Glob-T) ↑
Ours- ρ
25.46
0.8641
0.2458
-0.2795
0.0569
0.2875
w/o P2P
29.79
0.9332
0.2326
-0.5131
0.0936
0.2779
w/o RD
25.27
0.8599
0.2430
–
–
–
w/o RPT (tr)
23.02
0.8072
0.2453
–
–
–
w/o RPT (inf)
17.64
0.5365
0.2510
–
–
–
Table 4. Quantitative ablation study. BG Acc., SA, P2P, RD, RPT, and GB. stand for background accuracy, semantic alignment, our P2P dataset, random dropout, residual perceptual transfer mechanism, and G-buffers, respectively. Photorealism denotes step-average metrics. Results show P2P is vital for realism, while other components mainly remain semantic alignment and background preservation.
Figure 13. Convergence analysis: progressive and single realistic prompt. The top panel shows progressive generation guided by sequential VLM prompts and their corresponding Prompt Attention (PA) maps. The bottom panel shows generation with a single static prompt, displaying the target PA mask and pixel-wise residuals (Diff) between steps. Overall, our method robustly converges without structural collapse, achieving gradual stabilization under progressive prompts and immediate execution under a single prompt.
Figure 14. Quantitative convergence analysis under the progressive iterative realistic prompt setting. We monitor the trajectories of Semantic Attention Intensity (blue), CLIP Score (green), and KID Score (red) across 12 generation steps. The vertical dotted line marks all metrics reach convergence at around step 6 ∼ 9.
Figure 15. Robustness on inversed G-buffers from real photos. Our method demonstrates strong robustness when applied to G-buffers estimated directly from real-world images. Specifically, Ours- ϕ strictly adheres to the physical features of the inversed G-buffers, while Ours- ρ successfully recovers high-fidelity photorealism, achieving a visual quality comparable to the original real photos.
Image diffusion models have shown impressive image generation capabilities but are often hard to control, in contrast to classical computer graphics pipelines such as physically based rendering. However, we observe that there is a bridge between light transport phenomena and the distribution of latent space values produced by such models. Thus, we introduce physically based rendering in the feature space learned by the variational autoencoders in generative models, enabling light transport simulation in the latent space. This allows us to leverage physically based rendering techniques to output latent maps for physically guided content generation. We propose modifications to the rendering equation, which, when paired with a differentiable renderer, can yield an optimal set of scene parameters that require only minimal refinement to accurately render into the pretrained latent space. We train our method on a single rendered image, and then demonstrate the generalization of the method to scene geometry changes, lighting changes, and camera view changes.
We often aim to generate images that are both photorealistic and 3D-consistent, adhering to precise geometry, material, and viewpoint controls. Typically, this is achieved by fine-tuning an image generator, pre-trained on billions of real images, using renders of synthetic 3D assets, where annotations for control signals are available. While this approach can learn the desired controls, it often compromises the realism of the images due to domain gap between photographs and renders. We observe that this issue largely arises from the model learning an unintended association between the presence of control signals and the synthetic appearance of the images. To address this, we introduce Realiz3D, a lightweight framework for training diffusion models, that decouples controls and visual domain. The key idea is to explicitly learn visual domain, real or synthetic, separately from other control signals by introducing a co-variate that, fed into small residual adapters, shifts the domain. Then, the generator can be trained to gain controllability, without fitting to specific visual domain. In this way, the model can be guided to produce realistic images even when controls are applied. We enhance control transferability to the real domain by leveraging insights about roles of different layers and denoising steps in diffusion-based generators, informing new training and inference strategies that further mitigate the gap. We demonstrate the advantages of Realiz3D in tasks as text-to-multiview generation and texturing from 3D inputs, producing outputs that are 3D-consistent and photorealistic.
We present PIXLRelight, a feed-forward approach for physically controllable single-image relighting. Existing methods either provide limited lighting control (e.g. through text or environment maps), accumulate errors when chaining inverse and forward rendering, or require costly per-image optimization. Our key idea is to bridge physically based rendering (PBR) and learned image synthesis through a shared intrinsic conditioning that can be obtained from either real photographs or PBR renders. At training time, paired multi-illumination photographs are decomposed into albedo, diffuse shading, and non-diffuse residuals, which condition the model. At inference time, the same conditioning is computed from a path-traced render of a coarse 3D reconstruction of the input under user-specified PBR lights. A transformer-based neural renderer then applies the target illumination to the source photograph, preserving fine image detail through a per-pixel affine modulation. PIXLRelight enables arbitrary PBR-style lighting control, achieves state-of-the-art relighting quality, and runs in under a tenth of a second per image. Code and models are available at https://mlfarinha.github.io/pixl-relight/.
Miguel Farinha, Ronald Clark
Department of Computer Science University of Oxford