Paper ID: 2410.22070 • Published Oct 29, 2024
FreeGaussian: Annotation-free Controllable 3D Gaussian Splats with Flow Derivatives
Qizhi Chen, Delin Qu, Junli Liu, Yiwen Tang, Haoming Song, Dong Wang, Bin Zhao, Xuelong Li
TL;DR
Get AI-generated summaries with premium
Get AI-generated summaries with premium
Reconstructing controllable Gaussian splats from monocular video is a
challenging task due to its inherently insufficient constraints. Widely adopted
approaches supervise complex interactions with additional masks and control
signal annotations, limiting their real-world applications. In this paper, we
propose an annotation guidance-free method, dubbed FreeGaussian, that
mathematically derives dynamic Gaussian motion from optical flow and camera
motion using novel dynamic Gaussian constraints. By establishing a connection
between 2D flows and 3D Gaussian dynamic control, our method enables
self-supervised optimization and continuity of dynamic Gaussian motions from
flow priors. Furthermore, we introduce a 3D spherical vector controlling
scheme, which represents the state with a 3D Gaussian trajectory, thereby
eliminating the need for complex 1D control signal calculations and simplifying
controllable Gaussian modeling. Quantitative and qualitative evaluations on
extensive experiments demonstrate the state-of-the-art visual performance and
control capability of our method. Project page: this https URL