cs.CVApr 10, 2026
SaveGeRM: A Generative Rendering Model From Physically Realistic to Photorealistic
Organizations: State Key Lab of CAD&CG, Zhejiang University and Zhejiang Lab, China · State Key Laboratory of Industrial Control Technology, Zhejiang University, China · Zhejiang University, China
Abstract
While physically-based rendering (PBR) simulates light transport that guarantees physical realism, achieving true photorealistic rendering (PRR) demands prohibitive time and labor, and still struggles to capture the intractable richness of the real world. We propose GeRM, the first multimodal generative rendering model to bridge the gap from PBR to PRR (P2P). We formulate this P2P transition by learning a distribution transfer vector (DTV) field to direct the generative process. To achieve this, we introduce a multi-condition ControlNet that synthesizes PBR images and progressively transitions them into PRR images, guided by G-buffers, text prompts, and cues for enhanced regions. To improve the model's grasp of the image distribution shift driven by text prompts, we propose a residual perceptual transfer mechanism to associate text prompts with corresponding targeted modification regions, which more clearly defines the incremental component updates. To supervise this transfer process, we introduce a multi-agent visual language model framework to construct an expert-guided pairwise transfer dataset, named P2P-50K, where each paired sample corresponds to a specific transfer vector in the DTV field. Extensive experiments demonstrate that GeRM synthesizes high-quality controllable images and outperforms state-of-the-art baselines across diverse applications, including PBR and PRR image synthesis and editing.
Figures & tables
Figure 1. Given the physical attributes as input, the physically realistic ( ) results produced by GeRM are shown on the left side of the figure, with greater photorealism ( ) as they move closer to the right. The sequence 1–4 illustrates this progressive realistic evolution: the model iteratively enriches the scene with realistic lighting dynamics (e.g., the warm sunlight and ground highlight), fine-grained material textures , and context-aware details (e.g., natural fabric wrinkles on the sofa), resulting in a high-fidelity photorealisitic image that consistent with the original image.
Figure 2. A P2P Quad characterizing the correlation between physical realism and photorealsm. , , and are physically realistic image, photorealistic image, digital existence, and real-world existence, respectively.
Figure 3. Overview of our GeRM framework. Our pipeline operates on a multi-condition framework that integrates physical G-buffers with task-adaptive spatial constraints—ranging from zero masks for PBR synthesis to user-specified regions or VLM-driven automatic maps for editing and progressive realism. These combined inputs are fed into the multi-condition ControlNet, which predicts a distribution transfer vector (DTV) field. This DTV field guides the iterative evolution of the scene, transforming the initial physical state ( ) into the final photorealistic target ( ) via iteratively progressive generation.
Figure 4. Pipeline for constructing the progressive pairwise P2P dataset. We employ a multi-agent VLM framework—comprising the Global Auditor , Contextual Enricher , and Local Refiner —to collaboratively generate realistic refinement prompts based on the current state. These prompts are fed into CLIPSeg to produce spatial semantic masks, accurately localizing the regions for realistic modification. Guided by these spatially grounded instructions, the generative backbone (Kontext) executes localized updates, progressively evolving the sterile Engine Render into a photorealistic state (Kontext- ).
Figure 5. Why SOTA Editing Models Are Insufficient for Photorealistic Generation. Under the progressive prompts shortened like “physically realistic appearance,” Nano Banana retains a synthetic appearance, while Kontext exhibits severe spatial instability. In contrast, our method achieves superior photorealism and spatial alignment.
Figure 6. Visual comparison of our iterative PRR generation against standard rendering engines. From left to right: input G-buffers, the baseline render from various engines (Mitsuba3, Blender, Falcor, and UE5), and the PRR results generated by our method. As shown in the four right columns, our approach gradually enhances the realism of the scene, generating details and lighting effects that surpass the original engine renders.
Figure 7. Comparisons of PBR and PRR synthesis on indoor and outdoor scenes. Given input G-buffers (leftmost), the region left of the dashed line validates physical accuracy against PBR Reference and RGB X. The region to the right evaluates our final PRR generation against IntrinsicControlNet.
Figure 8. Demonstration of progressive editing on different subsets of G-buffer channels. We visualize an editing sequence where modifications are applied sequentially from top to bottom. In each row, the generation builds upon the result from the previous step, controlled by the specific edited G-buffer channel or text prompt shown in the “Edited Conditions” column. We compare our method against RGB X and IntrinsicEdit, alongside the ablation variants without random dropout (Ours w/o RD) or G-buffer guidance (Ours w/o G-buffers). Additional editing cases are provided in the Appendix A.4 .
Figure 9. Progressive semantic-aware P2P generation. We visualize masks, prompts, and generation results across progressive steps. While existing baselines struggle to balance strict localized mask control with true photorealism, our approach successfully confines high-fidelity enhancements to designated areas, preserving unmasked regions.
| US | KID | CLIP (Global-T) | IR | |
| Ours- 3 | 3.52 | 0.05834 | 0.2883 | -0.3197 |
| Ours- 6 | 3.99 | 0.04572 | 0.2920 | -0.2084 |
| ICNet | 1.40 | 0.06804 | 0.2806 | -0.5022 |
| FLUX.1-Kontext | 2.50 | 0.05975 | 0.2830 | -0.3017 |
| Nano Banana | 2.48 | 0.09024 | 0.2783 | -0.4468 |
| Mitsuba3 | 1.13 | 0.08010 | 0.2857 | -0.5444 |
Table 1. Quantitative evaluation of photorealism. US denotes user study. To measure the capability of generating photorealistic images and details, we employ highly detailed realistic prompts generated by GPT-4o. Our method achieves state-of-the-art performance in terms of realism and text-image alignment. Red , Orange , and Yellow indicate top 3 results.
| PSNR | SSIM | LPIPS | CLIP (Global-I) | |
| Ours- | 30.17 | 0.9520 | 0.0469 | 0.9907 |
| RGB X | 9.6972 | 0.3915 | 0.4986 | 0.8861 |
| RGB X* | 21.61 | 0.9129 | 0.0796 | 0.9497 |
Table 2. Quantitative comparsion on PBR synthesis. " " means masked images on single object scenes.
| BG Accuracy | Semantic Alignment | Overall Quality | |||||||||
| Method | PSNR | SSIM | DINO | Global-T | Local-T | Dir-T | BLIP | IR | Aes | CLIP-T | Q-Score |
| Ours- | 25.46 | 0.8641 | 0.9428 | 0.2875 | 0.2458 | 0.0887 | 0.6762 | -0.2795 | 3.9603 | 0.0381 | 0.3807 |
| FLUX.1-Fill-dev | 31.62 | 0.9452 | 0.9883 | 0.1919 | 0.2055 | 0.0740 | 0.3798 | -1.3512 | 3.9582 | 0.0264 | 0.3366 |
| FLUX.1-Kontext-dev | 15.39 | 0.4575 | 0.6764 | 0.2830 | 0.2214 | 0.0748 | 0.7486 | -0.3017 | 3.9577 | 0.0424 | 0.2877 |
| DiffEdit | 24.13 | 0.8397 | 0.9276 | 0.1664 | 0.1805 | 0.0002 | 0.0443 | -1.9112 | 3.9596 | 0.0009 | 0.2952 |
| ICEdit | 25.12 | 0.8440 | 0.8418 | 0.2806 | 0.2194 | 0.0773 | 0.3635 | -0.5022 | 3.9601 | 0.0392 | 0.3470 |
Table 3. Quantitative comparison of semantic-aware editing. We evaluate our method against various editing methods. FLUX.1-Kontext-dev serves as the backbone for our dataset construction, while Nano Banana represents a state-of-the-art commercial solution. Our method consistently ranks high across metrics, demonstrating superior overall quality while maintaining a robust balance between background preservation accuracy and semantic alignment. Red , Orange , and Yellow indicate 1st, 2nd, and 3rd best results.
Figure 10. Visual ablation study of our PRR generation method. The generation proceeds progressively from top to bottom, where each row represents a generation stage building upon the result of the previous step and the corresponding G-buffer feature (shown on the left). We compare Ours- -Full against variants without residual perceptual transfer mechanism (RPT) during training/inference, without our P2P-50K dataset (P2P), and a direct P2P translation in the RGB domain without G-buffer guidance (w/o G-buffers). The "Mask" column visualizes the semantic injection regions at each stage.
Figure 11. Controllable Photorealism via G-buffer Relaxation. Enabled by our explicit G-buffer controls, users are granted the flexibility to dictate the level of realism by progressively relaxing g-buffer constraints. Notably, explicitly instructing the model to preserve geometry via text prompts without G-buffer guidance (Ours- 6-w/o GB.-PreserveGeo.) fails to prevent severe structural alterations. While Ours- -PreserveGeo. strictly maintains the original layout, Ours- -NoAdditions enhances materials purely on existing structures without generating new objects. Ultimately, allowing the model to naturally introduce context-aware additions (Ours- ) often yields the most authentic and photorealistic results.
Figure 12. Impact of G-buffer guidance scale ( ). We visualize the progressive photorealistic generation across varying levels of physical control. For a fair and direct comparison, the exact same textual prompts and regional masks are applied across all different scale settings. As generation progresses (left to right), strong guidance (top rows) enforces strict spatial alignment with the input G-buffers. Conversely, relaxing (bottom rows) grants the model greater freedom to inject richer textures and complex object variations. This demonstrates that regulating empowers users to flexibly balance strict structural preservation and generative creativity depending on their specific intents.
| BG Acc. | SA | Photorealism | ||||
| Method | PSNR | SSIM | CLIP(Loc-T) | IR | KID | CLIP(Glob-T) |
| Ours- | 25.46 | 0.8641 | 0.2458 | -0.2795 | 0.0569 | 0.2875 |
| w/o P2P | 29.79 | 0.9332 | 0.2326 | -0.5131 | 0.0936 | 0.2779 |
| w/o RD | 25.27 | 0.8599 | 0.2430 | – | – | – |
| w/o RPT (tr) | 23.02 | 0.8072 | 0.2453 | – | – | – |
| w/o RPT (inf) | 17.64 | 0.5365 | 0.2510 | – | – | – |
Table 4. Quantitative ablation study. BG Acc., SA, P2P, RD, RPT, and GB. stand for background accuracy, semantic alignment, our P2P dataset, random dropout, residual perceptual transfer mechanism, and G-buffers, respectively. Photorealism denotes step-average metrics. Results show P2P is vital for realism, while other components mainly remain semantic alignment and background preservation.
Figure 13. Convergence analysis: progressive and single realistic prompt. The top panel shows progressive generation guided by sequential VLM prompts and their corresponding Prompt Attention (PA) maps. The bottom panel shows generation with a single static prompt, displaying the target PA mask and pixel-wise residuals (Diff) between steps. Overall, our method robustly converges without structural collapse, achieving gradual stabilization under progressive prompts and immediate execution under a single prompt.
Figure 14. Quantitative convergence analysis under the progressive iterative realistic prompt setting. We monitor the trajectories of Semantic Attention Intensity (blue), CLIP Score (green), and KID Score (red) across 12 generation steps. The vertical dotted line marks all metrics reach convergence at around step 6 9.
Figure 15. Robustness on inversed G-buffers from real photos. Our method demonstrates strong robustness when applied to G-buffers estimated directly from real-world images. Specifically, Ours- strictly adheres to the physical features of the inversed G-buffers, while Ours- successfully recovers high-fidelity photorealism, achieving a visual quality comparable to the original real photos.
More in this topic
Controllable Image Generation
Controllable image generation steers images using spatial, visual, semantic, stylistic, or physical constraints beyond ordinary text prompts.
See all 215 papers