cs.CVAug 14, 2026

RGBX-Next: Towards Realistic Generative Rendering from G-Buffers

Authors: Zheng Zeng, Marco Salvi, Lifan Wu, Jan Novák, Daqi Lin, Saeed Hadadan, Yichen Sheng, Robert Pottorff, +4 more

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Diffusion models have achieved impressive results in image, video, and streaming generation. However, compared to traditional 3D rendering, they still lack precise control over the generated output. We believe a viable path forward is to use generative models as learned renderers conditioned on traditionally rendered G-buffers. We introduce RGBX-Next, a unified generative framework for forward and inverse rendering, which allows estimating G-buffers from images, videos, and streams, and rendering realistic images, videos, and streams from G-buffers. Our key contribution is a general recipe for finetuning diffusion transformer (DiT) models into generative forward and inverse renderers. We show that the resulting models achieve high quality in both realistic generative rendering and intrinsic decomposition. We will make all our models publicly available. We believe that the design principles presented in this paper will benefit future research on controllable generative forward and inverse rendering.

Figures & tables

Figure 1. We present RGBX-Next , a unified generative framework for forward and inverse rendering, which allows estimating G-buffers from images, videos, and streams, and rendering realistic images, videos, and streams from G-buffers. Left: Our RGB →\rightarrow X model estimates clean G-buffers from 17-frame real video segments. Middle: Various X →\rightarrow RGB models trained on real RGB data produce highly realistic results from synthetically rendered or estimated G-buffers. Right: Streaming models finetuned from our video models produce coherent results over hundreds of frames, for RGB →\rightarrow X as well as X →\rightarrow RGB .
Figure 2. High-level overview of our paper. We first describe a simple version of image and video RGB ↔\leftrightarrow X models in section 4.1 , followed by an improved version using our QK type embedding and clean input tokens techniques in section 4.2 . We estimate G-buffers from real video data with our RGB →\rightarrow X model in section 5.1 , and train X →\rightarrow RGB models with X-patchify in section 5.2 . We discuss conditioning dropout and lighting control in sections 5.3 and 5.4 . We introduce streaming extensions of our models in section 6 .
SymbolExplanationdef.
mm , M\mathcal{M}Modality index and modality set.eq. 1 , eq. 7
Min\mathcal{M}_{\mathrm{in}} , Mout\mathcal{M}_{\mathrm{out}}Input and output modality sets.eq. 7
I\mathbf{I} , a\mathbf{a} , n\mathbf{n} , 0pt0pt , m\mathbf{m} , E\mathbf{E} , d\mathbf{d}RGB and G-buffer modalities.eq. 7
xm\mathbf{x}^{m} , xtm\mathbf{x}^{m}_{t} , ϵ\boldsymbol{\epsilon}Clean/noisy latent for modality mm , and Gaussian noise.eq. 2 , eq. 4
vθ\mathbf{v}_{\theta} , LFM\mathcal{L}_{\mathrm{FM}}Predicted flow velocity and flow-matching loss.eq. 3 , eq. 4
P\mathcal{P} , PX\mathcal{P}_{\textsf{X}}Patchify and X -patchify maps.eq. 5 , eq. 16
Table 1. Core notation used in the paper.
Figure 3. A typical variational auto-encoder (VAE) of recent video models offers 8×8×48\times 8\times 4 compression. More precisely, it encodes key frames to latent key frames, and further groups of 4 frames into latent motion frames, thus mapping 4k+14k+1 RGB frames into k+1k+1 latent frames of 8×8\times lower spatial resolution and 16 latent channels.
Figure 4. A DiT video model operates on latent frames as inputs and outputs. It patchifies 2×22\times 2 blocks of each latent frame into tokens , and operates on the set of tokens in transformer layers, which also receive text prompt embeddings and timestep information through cross-attention. Token order is arbitrary; embeddings are used to tell the model which token is which.
Figure 5. The preliminary RGB →\rightarrow X model concatenates the output X and input RGB, distinguished by frame ID. Input tokens have no noise added and no loss computed on them. The text prompt is repurposed for output modality selection, e.g. "albedo", "normal", etc.
Figure 6. (a) Our QK type embedding encodes modality and input/output by adding learned offsets to the query and key vectors. (b) Our clean token technique modifies the timestep embedding mechanism to always feed timestep 0 (i.e., no noise) to the input tokens, resulting in faster and higher-quality finetuning.
Figure 7. Training X →\rightarrow RGB on synthetic data leads to the model quickly adopting the synthetic look; real data with annotated G-buffers is key to realistic X →\rightarrow RGB results.
Figure 8. Example from our (RGB, X) dataset used to train our X →\rightarrow RGB models: a real video with estimated G-buffers, together with the VLM-produced caption.
Figure 9. Our X-patchify technique uses channel concatenation for a flexible number of input modalities, learning a patchify operator to map all of them into the same tokens, while still keeping outputs as separate tokens.
Figure 10. Our models can support full dropout of a G-buffer, partial dropout (where black means no guidance), and blurring. These techniques can be applied on a fraction of diffusion iterations, allowing for control in guidance strength.
Figure 11. The overall design of our 1/N1/N streaming models. For each chunk, the model receives NN current input frames and one stitching frame from the previous chunk, whose input and output are already known, then produces the NN new output frames through a diffusion process. Optional long-context tokens provide a clean reference frame for long-term memory. For simplicity, the figure illustrates the RGB →\rightarrow X case with albedo as X .
Figure 12. Our proof-of-concept unified model, RGB ×\times X, supports forward and inverse rendering within the same model weights. The input and output tokens are marked using embeddings, just like modalities, letting the model infer the desired task.
Figure 13. Qualitative RGB →\rightarrow X comparison on (RGB, X) test set. Columns show the input RGB, RGB ↔\leftrightarrow X ( Zeng et al., 2024 ) , DiffusionRenderer ( Liang et al., 2025 ) (specifically its newer version based on Cosmos ( Agarwal et al., 2025 ) ), and ours. Our model produces flatter albedo with less residual shading, supports both depth and irradiance, and avoids the sky-depth failure shown by DiffusionRenderer; DiffusionRenderer produces normals of similar visual quality.
Figure 14. Ablation of RGB →\rightarrow X conditioning choices and our proposed techniques. We compare channel-wise concatenation and VACE ( Jiang et al., 2025 ) with the preliminary frame-wise model, then add QK type embeddings and clean input tokens. All variants are trained for the same duration.
PSNRAlbedoNormalIrradianceDepth
RGB ↔\leftrightarrow X18.3919.2511.66N/A
DiffusionRenderer15.2921.54N/A17.72
Ours20.1721.2225.1929.47
SSIMAlbedoNormalIrradianceDepth
RGB ↔\leftrightarrow X0.72520.69010.7162N/A
DiffusionRenderer0.74250.7372N/A0.6570
Table 2. Quantitative comparison of our RGB →\rightarrow X results to RGB ↔\leftrightarrow X ( Zeng et al., 2024 ) and DiffusionRenderer ( Liang et al., 2025 ) on the test set of the Hypersim dataset. Our result is best across all three metrics for depth and albedo, and in two out of three metrics on normal and irradiance.
Figure 15. Results from various X →\rightarrow RGB models, conditioned on albedo, normal, depth, and combinations of albedo with irradiance (blurred and dropped after the first 20% diffusion iterations), and albedo with albedo-free direct lighting. The first column uses synthetic G-buffers rendered from 3D scenes; the remaining columns use G-buffers estimated from real videos. All models support video; please see supplementary video for many more video results.
Figure 16. Realism control with text guidance. We show input albedo buffers (left), CFG with a positive prompt for realistic imagery and a negative prompt for synthetic imagery, pushing towards realism (middle), and the same prompts flipped, producing a more synthetic appearance (right).
Figure 17. Qualitative X →\rightarrow RGB comparison with RGB ↔\leftrightarrow X ( Zeng et al., 2024 ) . The first row uses synthetic G-buffers rendered from 3D scenes; the remaining rows use G-buffers estimated from real videos. Compared with RGB ↔\leftrightarrow X , our results appear less synthetic. Since RGB ↔\leftrightarrow X ( Zeng et al., 2024 ) is an image model, its outputs also exhibit temporal inconsistencies; please refer to the supplementary video.
Figure 18. Qualitative X →\rightarrow RGB comparison with albedo conditioning against RGB ↔\leftrightarrow X ( Zeng et al., 2024 ) , VACE ( Jiang et al., 2025 ) , and a channel-wise concatenation baseline, which follows the conditioning mechanism used by DiffusionRenderer ( Liang et al., 2025 ) . All methods except RGB ↔\leftrightarrow X are trained on our (RGB, X) dataset from the same base model for the same training time. VACE and channel-wise models produce broadly realistic results but do not follow the guidance closely, while RGB ↔\leftrightarrow X shows less realism due to being trained on exclusively indoor scene content.
MethodFID ↓\downarrow
Channel-wise concat.88.6039
VACE ( Jiang et al., 2025 )62.2883
RGB ↔\leftrightarrow X ( Zeng et al., 2024 )62.3884
Ours45.3871
Table 3. FID comparison for X →\rightarrow RGB on the held-out (RGB, X) test set. We compute FID between generated frames and real video frames from the test set. Lower is better.
Figure 19. Additional text prompts can control lighting and material appearance without additional G-buffer inputs. With the default prompt (“Photograph, high-resolution, 4K, high-quality”), the result follows the albedo tone but misses the warm ambient light from the yellow strip lights. Appending “yellow strip lights create warm ambient light, metal dancing robot at center” better matches the path-traced reference and shifts the robot from diffuse to glossy.
Figure 20. Spatial conditioning dropout for sparse albedo guidance. We show sparse input albedo, where zero-valued regions mean “no guidance” rather than black albedo (left), the baseline X →\rightarrow RGB model without segment-dropout training, which produces flat visuals in these regions (middle), and our segment-dropout model, which preserves visual richness in unguided regions (right).
Figure 21. Ablation for 1/161/16 streaming RGB →\rightarrow X . Teacher forcing trains on ground-truth stitching frames but uses previous predictions at inference, leading to blur and error accumulation after roughly 100–200 frames (a). Stitching-frame augmentations help only slightly (b), and long-context tokens improve color preservation but do not eliminate drift (c). Combining these components with hybrid self forcing training yields stable estimates over hundreds of frames (d).
Figure 22. Streaming X →\rightarrow RGB results with self forcing training, applied to synthetic albedo sequences (top) and estimated ones from real videos (bottom). The generated videos maintain temporal coherence and lighting consistency across frames; full sequences are provided in the supplementary material.

Explore similar work

CardsList
Oct 1, 2026cs.CV

DAGS: Disentangled Appearance-and-Geometry Steering of a Frozen Image DiT for Temporally Stabilized Generative Rendering

Diffusion transformers (DiTs) generate high-fidelity images from text and image conditions, but their outputs carry large variance and their faithfulness to a desired target depends heavily on how the condition is supplied. We present DAGS, a lightweight, attention-free, disentangled appearance and geometry conditioning scheme that steers a frozen image DiT to produce high-fidelity, highly faithful, and independently controllable renders. Two small convolutional encoders compute conditioning features once per frame and inject them as a learned, per-layer, element-wise residual into the image tokens, avoiding the quadratic cost of stacking conditions through attention. Because control and temporal handling live outside the frozen backbone, we retain its vast pretrained prior and eliminate backbone-overfitting risk. We further add a small recurrent lighting stabilizer and a training-free temporal guidance term that, coupled with our conditioning, elevate a per-frame image model into a streaming renderer. DAGS produces controllable, high-quality renders at a fraction of the compute of path tracing; it is not real-time, trading compute for controllability and quality. On a matched 1-spp + G-buffer input, per-frame DAGS reconstructs +8.6 dB / +10.1 dB PSNR over the real-time denoiser Intel OIDN and the diffusion renderer RGB<->X while being 2.5-8x more temporally stable perceptually (temporal-LPIPS flicker).

Karthik Mohan Kumar, Damian Andrysiak, Pedro Antonio Pena +2

Advanced Micro Devices, Inc. San Jose, USA · Advanced Micro Devices, Inc. Łódź, Poland · Advanced Micro Devices, Inc. Orlando, USA +1

Controllable Image GenerationNeural RenderingTemporal Consistency in Video GenerationDiffusion Model GuidanceDiffusion Transformer
AskPDF
Sep 25, 2026cs.CV

ControlGS: Conditioning Neural Gaussians for Downstream-Processing-Aware XR Rendering

Extended Reality (XR) users do not directly perceive the output of a rendering engine. Instead, rendered images pass through a post-processing pipeline and the physical display-optics path before reaching the eye. Critically, the exact downstream processing can vary significantly at run time, influenced by, for instance, camera pose and display power budget. Traditional 3DGS methods either implicitly assume that this downstream pipeline preserves image quality or cannot adapt to downstream processing changes. To bridge this gap, we present ControlGS, an XR Gaussian rendering pipeline that optimizes end-to-end visual quality. ControlGS models and integrates the entire downstream processing, between the rendering output and the human eye, into the optimization objective. To adapt to downstream processing at run time, ControlGS dynamically generates Gaussian primitives conditioned upon the downstream processing parameters. Experiments show that ControlGS consistently improves end-to-end post-optics XR quality across different neural Gaussian backbones and datasets, with minimal overhead. Code is available at https://horizon-lab.org/controlgs/.

Weikai Lin, Junjie Zhao, Carl Marshall +2

University of Rochester, Rochester, USA · Reality Labs, Meta, Redmond, USA

Neural Rendering3D Gaussian Splatting
AskPDF
Sep 4, 2026cs.CV

ReaDiT Guidance: Control for Image and Video Generation using Diffusion Transformer Features

We present DiT Readout (ReaDiT) Guidance, a lightweight framework for controlling generation with Diffusion Transformer (DiT) models via their internal feature representations. ReaDiT Guidance uses features from a single DiT block to steer the generative process according to spatial targets - like depth, pose, or edge maps - provided at test time. Furthermore, since modern text-to-video models are largely built on DiT backbones, ReaDiT Guidance naturally extends to video generation, enabling camera and motion control. Experimental results demonstrate that our approach achieves competitive or improved results compared to existing feature-based and off-the-shelf adapter-based approaches while requiring fewer parameters.

Jay Mahajan, Chang Liu, Rauf Makharov +3

University of Illinois Urbana-Champaign · Google

Diffusion Model GuidanceControllable Image GenerationControllable Video GenerationCamera-Controlled Video GenerationDiffusion Transformer
AskPDF