cs.CVNov 21, 2024

Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation and Reconstruction

Authors: Yuanhao Cai, He Zhang, Kai Zhang, Yixun Liang, Mengwei Ren, Fujun Luan, Qing Liu, Soo Ye Kim, 

Organizations: Johns Hopkins University · Adobe Research · HKUST · Shanghai Jiao Tong University

Abstract

Existing feedforward image-to-3D methods mainly rely on 2D multi-view diffusion models that cannot guarantee 3D consistency. These methods easily collapse when changing the prompt view direction and mainly handle object-centric cases. In this paper, we propose a novel single-stage 3D diffusion model, DiffusionGS, for object generation and scene reconstruction from a single view. DiffusionGS directly outputs 3D Gaussian point clouds at each timestep to enforce view consistency and allow the model to generate robustly given prompt views of any directions, beyond object-centric inputs. Plus, to improve the capability and generality of DiffusionGS, we scale up 3D training data by developing a scene-object mixed training strategy. Experiments show that DiffusionGS yields improvements of 2.20 dB/23.25 and 1.34 dB/19.16 in PSNR/FID for objects and scenes than the state-of-the-art methods, without depth estimator. Plus, our method enjoys over 5×\times faster speed (∼\sim6s on an A100 GPU). Our Project page at https://caiyuanhao1998.github.io/project/DiffusionGS/ shows the video and interactive results. The code and models are publicly available at https://github.com/caiyuanhao1998/Open-DiffusionGS . An online Gradio demo with high-quality mesh exportation is available at https://huggingface.co/spaces/CaiYuanhao/DiffusionGS-Object

Figures & tables

Explore similar work

CardsList
  1. D3GS: Depth, DINO, and RGB Diffusion Co-Guided 3D Gaussian Splatting for Sparse-View Reconstruction

    Sep 19, 2026Yunqi Gao, Zhanfeng Liao, Hanzhang Tu +7Sparse-View 3D Reconstruction3D Gaussian Splatting

  2. ConFixGS: Learning to Fix Feedforward 3D Gaussian Splatting with Confidence-Aware Diffusion Priors in Driving Scenes

    May 10, 2026Rui Song, Tianhui Cai, Markus Gross +53D Gaussian SplattingDiffusion Guidance

  3. ReconSplat: Generalizable 3D Scene Reconstruction Beyond Observed Views

    Aug 28, 2026Giuseppe Stracquadanio, Kevin Raj, Julia Grabinski +1Feed-Forward 3D ReconstructionNovel View Synthesis