Learning Explicit Continuous Motion Representation for Dynamic Gaussian Splatting from Monocular Videos

Xuankai Zhang; Junjin Xiao; Shangwei Huang; Wei-shi Zheng; Qing Zhang

cs.CVMar 26, 2026

Learning Explicit Continuous Motion Representation for Dynamic Gaussian Splatting from Monocular Videos

Authors: Xuankai Zhang, Junjin Xiao, Shangwei Huang, Wei-shi Zheng, Qing Zhang

Organizations: School of Computer Science and Engineering, Sun Yat-sen University, China · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China

Abstract

We present an approach for high-quality dynamic Gaussian Splatting from monocular videos. To this end, we in this work go one step further beyond previous methods to explicitly model continuous position and orientation deformation of dynamic Gaussians, using an SE(3) B-spline motion bases with a compact set of control points. To improve computational efficiency while enhancing the ability to model complex motions, an adaptive control mechanism is devised to dynamically adjust the number of motion bases and control points. Besides, we develop a soft segment reconstruction strategy to mitigate long-interval motion interference, and employ a multi-view diffusion model to provide multi-view cues for avoiding overfitting to training views. Extensive experiments demonstrate that our method outperforms state-of-the-art methods in novel view synthesis. Our code is available at https://github.com/hhhddddddd/se3bsplinegs.

Figures & tables

Figure 2: Overview of our method. We first initialize static Gaussians via depth reprojection and dynamic Gaussians from tracking points, by modeling their transformations with learnable SE(3) B-spline Motion Bases. We then adjust the number of motion bases and control points based on an adaptive control mechanism. Next, we employ a soft segment reconstruction strategy to fuse dynamic Gaussians at different reference timestamps to the observation timestamp, and further supplement the monocular video with scene-level multi-view cues derived from a multi-view diffusion model.

Panel of Figure 3 — Figure 3: Visual comparison of novel view synthesis on the iPhone dataset [ 12 ] .

Panel of Figure 4 — Figure 4: Visual comparison of novel view synthesis on the NVIDIA dataset. Note, images are cropped to highlight dynamic regions.

Panel of Figure 5 — Figure 5: Visual ablation study on scenes from the iPhone and NVIDIA datasets.

Method	iPhone			NVIDIA			Param.	Training Time	FPS
Method	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Param.	Training Time	FPS
MarbleGS [ 37 ]	16.08	0.568	0.433	24.56	0.667	0.110	51.6M	13 hrs	8.367
SoM [ 40 ]	17.13	0.674	0.279	24.58	0.651	0.124	173.4M	1 hr	38.754
HiMoR [ 24 ]	16.02	0.558	0.325	24.89	0.694	0.118	185.2M	1 hr	34.156
MoDec-GS [ 21 ]	14.65	0.320	0.461	23.94	0.628	0.132	86.1M	1 hr	9.074
SplineGS [ 32 ]	15.52	0.483	0.371	27.12	0.872	0.052	158.4M	2 hrs	387.469

Table 1: Quantitative comparison of novel view synthesis on the iPhone [ 12 ] and NVIDIA [ 48 ] datasets. Note that all training times are measured on the skating scene using a single NVIDIA RTX 4090 GPU, and FPS is measured on the same GPU at 480 × 360 resolution.

Method	iPhone			NVIDIA
Method	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
w/o Adap.	18.84	0.685	0.350	26.87	0.803	0.128
w/o Soft.	19.02	0.628	0.328	27.06	0.832	0.085
w/o $\mathcal{L}_{sds}$	19.39	0.715	0.288	27.13	0.845	0.074
w/o $\mathcal{L}_{smo}$	19.18	0.713	0.295	27.15	0.861	0.076
Ours	20.17	0.729	0.274	27.81	0.871	0.049

Table 2: Ablation studies on the iPhone and NVIDIA dataset. “w/o Adap.” denotes removing the adaptive control in SE(3) B-spline motion bases, “w/o Soft.” means omitting soft segment reconstruction, and “w/o

\mathcal{L}_{sds}

” and “w/o

\mathcal{L}_{smo}

” refer to ignoring multi-view SDS loss and the camera smoothness loss, respectively.

Method	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$
w/ Pose trans.	18.17	0.675	0.328
w/ Motion scaff.	19.26	0.698	0.296
Ours	20.17	0.729	0.274

Table 3: Ablation study of SE(3) B-spline motion bases on the iPhone dataset. “w/ Pose trans.” means replacing our motion representation with the SE(3)-based pose transformation used in SoM [ 40 ] . “w/ Motion scaff.” indicates replacing the representation with the motion scaffolds used in MoSca [ 22 ] .

Panel of Figure 6 — Figure 6: Effect of SE(3) B-spline Motion Bases.

Panel of Figure 7 — Figure 7: Failure cases. Our method fails to handle dynamic scenes with large non-rigid motion.

Metric	MarbleGS [ 37 ]	HiMoR [ 24 ]	MoSca [ 22 ]	Ours
PCK-T $\uparrow$	0.803	0.809	0.821	0.833

Table 4: Comparison of correspondence on the iPhone dataset.

Method	iPhone			NVIDIA
Method	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
w/ Prior pert.	20.11	0.725	0.278	27.74	0.867	0.051
Ours	20.17	0.729	0.274	27.81	0.871	0.049

Table 5: Effect of 2D prior errors. “w/ Prior pert.” indicates that the 2D tracking prior is perturbed by adding random noise within the range of [-15, 15].

Panel of Figure 8 — Figure 8: Visual comparison of novel view synthesis.

Panel of Figure 9 — Figure 9: Effect of pruning strategy.

Method	iPhone			NVIDIA
Method	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
w/ Random	19.84	0.721	0.283	27.59	0.853	0.062
w/ All	19.37	0.716	0.293	27.26	0.848	0.065
Ours	20.17	0.729	0.274	27.81	0.871	0.049

Table 6: Ablation on the iPhone and NVIDIA datasets. “w/ Random” denotes randomly pruning a control point with error below

\epsilon_{prune}

, while “w/ All” prunes all control points with errors below

\epsilon_{prune}

Figure 10: Effect of soft segmentation reconstruction.

Panel of Figure 11 — Figure 11: Visual comparison of novel view synthesis on the scene “Apple” of the iPhone dataset [ 12 ] . The time interval between adjacent images is ten frames.

Panel of Figure 12 — Figure 12: Visual comparison of novel view synthesis on the scene “Paper-windmill” of the iPhone dataset [ 12 ] . The time interval between adjacent images is ten frames.

Panel of Figure 13 — Figure 13: Visual comparison of novel view synthesis on the scene “Balloon1” of the NVIDIA dataset [ 48 ] . The time interval between adjacent images is two frames.

Panel of Figure 14 — Figure 14: Visual comparison of novel view synthesis on the scene “Umbrella” of the NVIDIA dataset [ 48 ] . The time interval between adjacent images is two frames.

Dynamic Gaussian Splatting

See all 14 papers

SMG: Semantic Motion Graph for Monocular Dynamic Gaussian Splatting
Aug 31, 2026Haozheng Yu, Xinyu Yang, Rundong Luo +23D Gaussian Splatting Dynamic Gaussian Splatting
Cornell University
RiGS: Rigid-aware 4D Gaussian Splatting from a Single Monocular Video
May 22, 2026Chenyu Wu, Wanhua Li, Zhu-Tian Chen +13D Gaussian Splatting Monocular 3D Reconstruction
Harvard University · Zhejiang University · Nanyang Technological University +1
EvoGS: Modeling Deformation Evolution for Dynamic Gaussian Splatting
Sep 1, 2026Wei Dong, Shahram Shirani, Jun Chen +13D Gaussian Splatting Dynamic Gaussian Splatting
The Department of Electrical & Computer Engineering, McMaster University

Aug 31, 2026J/K move · Enter open · S save

SMG: Semantic Motion Graph for Monocular Dynamic Gaussian Splatting

Haozheng Yu, Xinyu Yang, Rundong Luo, Jennifer J. Sun+1

Cornell University

3D Gaussian Splatting Dynamic Gaussian Splatting Dynamic Scene Reconstruction Gaussian Splatting

Open paper AskPDF arXiv

We study dynamic Gaussian Splatting from monocular videos. While recent advancements in dynamic Gaussian splatting offer a promising foundation for modeling dynamic scenes, they often overfit to the training views and fail under occlusion or complex scene motion due to the lack of reliable regularization signals in under-constrained regions. We propose Semantic Motion Graph (SMG), a novel approach models the Gaussian motion as the low-rank semantic motion. Our key insight is that the real-world scene motion is often structured by semantic coherence: regions that are spatially close and semantically related tend to exhibit consistent dynamics. To leverage this prior, we construct SMG to model structured motion of the scene. The Gaussian motion is driven by the motion of SMG nodes. We further observe that the uncertainty of Gaussian motion arises from both unreliable off-the-shelf priors and weakly constrained regions during optimization. SMG addresses this by using reliable graph nodes to guide the motion of nearby unreliable nodes. To evaluate dynamic Gaussian splatting under challenging real-world scenarios, we introduce a new multiview dataset collected under an ego-exo setup. Extensive experiments demonstrate that SMG achieves state-of-the-art performance on monocular dynamic Gaussian splatting across challenging real-world benchmarks. Project page: https://smg-gaussian.github.io/.

Abstract

Dynamic Gaussian Splatting

Explore similar work

SMG: Semantic Motion Graph for Monocular Dynamic Gaussian Splatting

RiGS: Rigid-aware 4D Gaussian Splatting from a Single Monocular Video

EvoGS: Modeling Deformation Evolution for Dynamic Gaussian Splatting