cs.CVAug 14, 2026
SaveRGBX-Next: Towards Realistic Generative Rendering from G-Buffers
Organizations: University of California, Santa Barbara, NVIDIA, USA · NVIDIA, USA · NVIDIA, Czech Republic · NVIDIA, University of California, San Diego, USA · Mohamed bin Zayed University of Artificial Intelligence, UAE
Abstract
Diffusion models have achieved impressive results in image, video, and streaming generation. However, compared to traditional 3D rendering, they still lack precise control over the generated output. We believe a viable path forward is to use generative models as learned renderers conditioned on traditionally rendered G-buffers. We introduce RGBX-Next, a unified generative framework for forward and inverse rendering, which allows estimating G-buffers from images, videos, and streams, and rendering realistic images, videos, and streams from G-buffers. Our key contribution is a general recipe for finetuning diffusion transformer (DiT) models into generative forward and inverse renderers. We show that the resulting models achieve high quality in both realistic generative rendering and intrinsic decomposition. We will make all our models publicly available. We believe that the design principles presented in this paper will benefit future research on controllable generative forward and inverse rendering.
Figures & tables
Figure 1. We present RGBX-Next , a unified generative framework for forward and inverse rendering, which allows estimating G-buffers from images, videos, and streams, and rendering realistic images, videos, and streams from G-buffers. Left: Our RGB X model estimates clean G-buffers from 17-frame real video segments. Middle: Various X RGB models trained on real RGB data produce highly realistic results from synthetically rendered or estimated G-buffers. Right: Streaming models finetuned from our video models produce coherent results over hundreds of frames, for RGB X as well as X RGB .
Figure 2. High-level overview of our paper. We first describe a simple version of image and video RGB X models in section 4.1 , followed by an improved version using our QK type embedding and clean input tokens techniques in section 4.2 . We estimate G-buffers from real video data with our RGB X model in section 5.1 , and train X RGB models with X-patchify in section 5.2 . We discuss conditioning dropout and lighting control in sections 5.3 and 5.4 . We introduce streaming extensions of our models in section 6 .
| Symbol | Explanation | def. |
|---|---|---|
| , | Modality index and modality set. | eq. 1 , eq. 7 |
| , | Input and output modality sets. | eq. 7 |
| , , , , , , | RGB and G-buffer modalities. | eq. 7 |
| , , | Clean/noisy latent for modality , and Gaussian noise. | eq. 2 , eq. 4 |
| , | Predicted flow velocity and flow-matching loss. | eq. 3 , eq. 4 |
| , | Patchify and X -patchify maps. | eq. 5 , eq. 16 |
Table 1. Core notation used in the paper.
Figure 3. A typical variational auto-encoder (VAE) of recent video models offers compression. More precisely, it encodes key frames to latent key frames, and further groups of 4 frames into latent motion frames, thus mapping RGB frames into latent frames of lower spatial resolution and 16 latent channels.
Figure 4. A DiT video model operates on latent frames as inputs and outputs. It patchifies blocks of each latent frame into tokens , and operates on the set of tokens in transformer layers, which also receive text prompt embeddings and timestep information through cross-attention. Token order is arbitrary; embeddings are used to tell the model which token is which.
Figure 5. The preliminary RGB X model concatenates the output X and input RGB, distinguished by frame ID. Input tokens have no noise added and no loss computed on them. The text prompt is repurposed for output modality selection, e.g. "albedo", "normal", etc.
Figure 6. (a) Our QK type embedding encodes modality and input/output by adding learned offsets to the query and key vectors. (b) Our clean token technique modifies the timestep embedding mechanism to always feed timestep 0 (i.e., no noise) to the input tokens, resulting in faster and higher-quality finetuning.
Figure 7. Training X RGB on synthetic data leads to the model quickly adopting the synthetic look; real data with annotated G-buffers is key to realistic X RGB results.
Figure 8. Example from our (RGB, X) dataset used to train our X RGB models: a real video with estimated G-buffers, together with the VLM-produced caption.
Figure 9. Our X-patchify technique uses channel concatenation for a flexible number of input modalities, learning a patchify operator to map all of them into the same tokens, while still keeping outputs as separate tokens.
Figure 10. Our models can support full dropout of a G-buffer, partial dropout (where black means no guidance), and blurring. These techniques can be applied on a fraction of diffusion iterations, allowing for control in guidance strength.
Figure 11. The overall design of our streaming models. For each chunk, the model receives current input frames and one stitching frame from the previous chunk, whose input and output are already known, then produces the new output frames through a diffusion process. Optional long-context tokens provide a clean reference frame for long-term memory. For simplicity, the figure illustrates the RGB X case with albedo as X .
Figure 12. Our proof-of-concept unified model, RGB X, supports forward and inverse rendering within the same model weights. The input and output tokens are marked using embeddings, just like modalities, letting the model infer the desired task.
Figure 13. Qualitative RGB X comparison on (RGB, X) test set. Columns show the input RGB, RGB X ( Zeng et al., 2024 ) , DiffusionRenderer ( Liang et al., 2025 ) (specifically its newer version based on Cosmos ( Agarwal et al., 2025 ) ), and ours. Our model produces flatter albedo with less residual shading, supports both depth and irradiance, and avoids the sky-depth failure shown by DiffusionRenderer; DiffusionRenderer produces normals of similar visual quality.
Figure 14. Ablation of RGB X conditioning choices and our proposed techniques. We compare channel-wise concatenation and VACE ( Jiang et al., 2025 ) with the preliminary frame-wise model, then add QK type embeddings and clean input tokens. All variants are trained for the same duration.
| PSNR | Albedo | Normal | Irradiance | Depth |
|---|---|---|---|---|
| RGB X | 18.39 | 19.25 | 11.66 | N/A |
| DiffusionRenderer | 15.29 | 21.54 | N/A | 17.72 |
| Ours | 20.17 | 21.22 | 25.19 | 29.47 |
| SSIM | Albedo | Normal | Irradiance | Depth |
| RGB X | 0.7252 | 0.6901 | 0.7162 | N/A |
| DiffusionRenderer | 0.7425 | 0.7372 | N/A | 0.6570 |
Table 2. Quantitative comparison of our RGB X results to RGB X ( Zeng et al., 2024 ) and DiffusionRenderer ( Liang et al., 2025 ) on the test set of the Hypersim dataset. Our result is best across all three metrics for depth and albedo, and in two out of three metrics on normal and irradiance.
Figure 15. Results from various X RGB models, conditioned on albedo, normal, depth, and combinations of albedo with irradiance (blurred and dropped after the first 20% diffusion iterations), and albedo with albedo-free direct lighting. The first column uses synthetic G-buffers rendered from 3D scenes; the remaining columns use G-buffers estimated from real videos. All models support video; please see supplementary video for many more video results.
Figure 16. Realism control with text guidance. We show input albedo buffers (left), CFG with a positive prompt for realistic imagery and a negative prompt for synthetic imagery, pushing towards realism (middle), and the same prompts flipped, producing a more synthetic appearance (right).
Figure 17. Qualitative X RGB comparison with RGB X ( Zeng et al., 2024 ) . The first row uses synthetic G-buffers rendered from 3D scenes; the remaining rows use G-buffers estimated from real videos. Compared with RGB X , our results appear less synthetic. Since RGB X ( Zeng et al., 2024 ) is an image model, its outputs also exhibit temporal inconsistencies; please refer to the supplementary video.
Figure 18. Qualitative X RGB comparison with albedo conditioning against RGB X ( Zeng et al., 2024 ) , VACE ( Jiang et al., 2025 ) , and a channel-wise concatenation baseline, which follows the conditioning mechanism used by DiffusionRenderer ( Liang et al., 2025 ) . All methods except RGB X are trained on our (RGB, X) dataset from the same base model for the same training time. VACE and channel-wise models produce broadly realistic results but do not follow the guidance closely, while RGB X shows less realism due to being trained on exclusively indoor scene content.
| Method | FID |
|---|---|
| Channel-wise concat. | 88.6039 |
| VACE ( Jiang et al., 2025 ) | 62.2883 |
| RGB X ( Zeng et al., 2024 ) | 62.3884 |
| Ours | 45.3871 |
Table 3. FID comparison for X RGB on the held-out (RGB, X) test set. We compute FID between generated frames and real video frames from the test set. Lower is better.
Figure 19. Additional text prompts can control lighting and material appearance without additional G-buffer inputs. With the default prompt (“Photograph, high-resolution, 4K, high-quality”), the result follows the albedo tone but misses the warm ambient light from the yellow strip lights. Appending “yellow strip lights create warm ambient light, metal dancing robot at center” better matches the path-traced reference and shifts the robot from diffuse to glossy.
Figure 20. Spatial conditioning dropout for sparse albedo guidance. We show sparse input albedo, where zero-valued regions mean “no guidance” rather than black albedo (left), the baseline X RGB model without segment-dropout training, which produces flat visuals in these regions (middle), and our segment-dropout model, which preserves visual richness in unguided regions (right).
Figure 21. Ablation for streaming RGB X . Teacher forcing trains on ground-truth stitching frames but uses previous predictions at inference, leading to blur and error accumulation after roughly 100–200 frames (a). Stitching-frame augmentations help only slightly (b), and long-context tokens improve color preservation but do not eliminate drift (c). Combining these components with hybrid self forcing training yields stable estimates over hundreds of frames (d).
Figure 22. Streaming X RGB results with self forcing training, applied to synthetic albedo sequences (top) and estimated ones from real videos (bottom). The generated videos maintain temporal coherence and lighting consistency across frames; full sequences are provided in the supplementary material.