Reconstructing a renderable 3D model from images is a useful but challenging task. Recent feedforward 3D reconstruction methods have demonstrated remarkable success in efficiently recovering geometry, but still cannot accurately model the complex appearances of these 3D reconstructed models. Recent diffusion-based generative models can synthesize realistic images or videos of an object using reference images without explicitly modeling its appearance, which provides a promising direction for object rendering, but lacks accurate control over the viewpoints. In this paper, we propose GO-Renderer, a unified framework integrating the reconstructed 3D proxies to guide the video generative models to achieve high-quality object rendering on arbitrary viewpoints under arbitrary lighting conditions. Our method not only enjoys the accurate viewpoint control using the reconstructed 3D proxy but also enables high-quality rendering in different lighting environments using diffusion generative models without explicitly modeling complex materials and lighting. Extensive experiments demonstrate that GO-Renderer achieves state-of-the-art performance across the object rendering tasks, including synthesizing images on new viewpoints, rendering the objects in a novel lighting environment, and inserting an object into an existing video.
Figures & tables
Figure 1: Results of our GO-Renderer. Driven by multi-view references and 3D poses, GO-Renderer renders multi-view consistent object videos seamlessly integrated into novel environments along arbitrary camera trajectories.
Figure 2: Overview of GO-Renderer. Given multi-view reference images, we first establish a coarse 3D geometry via 3D reconstruction to render reference and target coordinate maps. These maps bridge explicit 3D geometry with the multi-view reference images via video diffusion model. Specifically, the reference images and their coordinate maps are paired via channel-wise concatenation, while the target coordinate maps are combined with optional appearance guidance. These spatial conditions are then jointly processed by a text-conditioned Video Diffusion Transformer to render high-fidelity object videos with precise viewpoint control.
Figure 3: Comparision of rendering quality and lighting consistency.
Method
PSNR ↑
SSIM ↑
Text Align ↑
Motion Smoothness ↑
Aesthetic Quality ↑
Imaging Quality ↑
Phantom [ 26 ]
11.400
0.514
24.620
0.984
0.568
0.709
UniLumos(GT) [ 27 ]
12.060
0.585
25.630
0.978
0.569
0.669
UniLumos(ReconViGen [ 5 ] )
11.010
0.338
25.870
0.975
0.571
0.666
UniLumos(AnySplat [ 16 ] )
11.160
0.362
25.930
0.975
0.557
0.624
Wan2.2Fun
14.300
0.641
25.250
0.984
0.530
0.601
Ours
18.260
0.684
27.720
0.988
0.573
0.712
Table 1: Quantitative comparison of Rendering Quality and Lighting Consistency. Bold number indicate the best performance.
Method
CLIP
DINO
Img/Avg. ↑
Img/Max. ↑
Avg. ↑
Max. ↑
AnySplat
0.861
0.934
0.191
0.323
ReconViaGen
0.796
0.927
0.448
0.727
Ours
0.888
0.956
0.725
0.860
Table 2: Quantitative comparison of Multi-view Consistency and Reconstruction. Bold numbers indicate the best performance.
Figure 4: Qualitative comparison of Multi-view Consistency.
Figure 7
Figure 7: Application of our GO-Renderer.
Figure 8: Failure cases. The top row (GT) displays the ground-truth sequence. In Case 1 and Case 2, the model is conditioned on the multi-view appearances of a cupcake, but provided with the explicit spatial coordinate maps of a bottle and a cube, respectively. This geometric inconsistency prevents the model from reconciling the conflicting spatial and appearance conditions, leading to structural artifacts and degraded rendering fidelity.
Diffusion models have achieved impressive results in image, video, and streaming generation. However, compared to traditional 3D rendering, they still lack precise control over the generated output. We believe a viable path forward is to use generative models as learned renderers conditioned on traditionally rendered G-buffers. We introduce RGBX-Next, a unified generative framework for forward and inverse rendering, which allows estimating G-buffers from images, videos, and streams, and rendering realistic images, videos, and streams from G-buffers. Our key contribution is a general recipe for finetuning diffusion transformer (DiT) models into generative forward and inverse renderers. We show that the resulting models achieve high quality in both realistic generative rendering and intrinsic decomposition. We will make all our models publicly available. We believe that the design principles presented in this paper will benefit future research on controllable generative forward and inverse rendering.
Zheng Zeng, Marco Salvi, Lifan Wu +9
University of California, Santa Barbara, NVIDIA, USA · NVIDIA, USA · NVIDIA, Czech Republic +2
Rendering views using 3D scene representations such as Gaussian Splatting (3DGS), Neural Radiance Fields (NeRF), meshes, or even point clouds produces artifacts when input views are sparse or target views lie far from the input. Recent work mitigates these artifacts using diffusion-based generative priors, but is specialized to individual representations and require custom architectures or extensive retraining. We present FixAnything, a single model for fixing a wide range of rendering artifacts. It does so by repurposing a pretrained video generative model, leveraging its implicit multi-view priors with only minimal modification and lightweight finetuning. Our key insight is that even noisily-rendered sequences preserve camera motion and coarse scene structure, allowing cleanup to be formulated as video-to-video translation. To control what scene structure should be preserved, we introduce a binary mask denoting the clean pixels, enabling the model to anchor its output to high-quality inputs (e.g. training views) while refining the rest. To encourage FixAnything to produce 3D-consistent renderings that support downstream reconstruction, we use camera pose accuracy (recovered via structure-from-motion) as a reward signal for direct preference optimization (DPO). Across four distinct 3D representations, FixAnything consistently improves rendering quality with lightweight finetuning, demonstrating that a single generalist video prior can replace multiple specialist refinement pipelines. The simplicity of the framework enables immediate adoption of stronger future video models without architectural redesign.
Modern image-and-text-to-video diffusion models can synthesize highly realistic videos by iteratively denoising an initial Gaussian noise tensor conditioned on reference image and text inputs. However, existing approaches still lack precise and unified controllability over both object motion and camera motion within a single generation process. We present UniCaMo, a unified framework that enables simultaneous control of object trajectories and camera viewpoints by directly constructing the input noise of the diffusion model. Specifically, UniCaMo builds a shared 3D-grounded motion-consistent noise space across latent video frames. Sparse 3D point tracks are used to warp the Gaussian noise of the reference frame along desired object trajectories, while a virtual spherical noise representation provides globally consistent noise values for newly revealed scene regions under camera motion. By combining local track-guided noise warping with global sphere-based noise sampling, UniCaMo maintains geometric and temporal consistency under both object movement and viewpoint changes. Because UniCaMo modifies only the input noise, it requires no auxiliary adapters, control branches, or architectural changes to the underlying video diffusion model. With lightweight LoRA fine-tuning on large pretrained video diffusion models, including Wan 2.1 (14B), UniCaMo achieves state-of-the-art results in both video quality and motion controllability on standard controllable video generation benchmarks.
Long Vu, Tan Ngo, Animesh Karnewar +5
Qualcomm AI Research · Trinity College Dublin, Ireland