Space-Time Forecasting of Dynamic Scenes with Motion-aware Gaussian Grouping

Junmyeong Lee; Hoseung Choi; Minsu Cho

Figure 2: Overall pipeline of MoGaF. Given a video, MoGaF generates future frames of the scene. To achieve realistic forecasting, our method builds on 4DGS representation and proceeds as follows: (1) Gaussian Grouping: Gaussians are clustered into motion-consistent object groups, with each group labeled as rigid or non-rigid using grounded 2D segmentation. (2) Group-wise Optimization: Grouped Gaussians are refined with rigidity-aware motion constraints: rigid groups are guided by a shared

SE(3)

transform, while non-rigid groups are regularized with local motion smoothness. (3) Group-wise Forecasting: For each group, a lightweight Transformer-based forecaster extrapolates Gaussian trajectories beyond the observed frames, enabling rendering at novel viewpoints for future timesteps.

Figure 3: Result of Gaussian grouping. Compared to a (a) simple extension of 3DGS grouping [ 26 ] and (b) single-frame mask–based region growing, our hybrid approach produces complete and reliable motion-aware Gaussian groups.

Obs. ratio	Scene	GSPred [ 50 ]			GSPred-SoM † [ 50 ]			ODE-GS-SoM † [ 36 ]			MoGaF (Ours)
Obs. ratio	Scene	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$
80%	apple	15.75	0.5589	0.4338	17.22	0.8122	0.4806	16.35	0.8135	0.4803	17.97	0.8199	0.4341
	block	12.61	0.5220	0.5363	12.20	0.5807	0.5352	12.07	0.5717	0.5583	12.62	0.5875	0.5352
	paper-windmill	14.76	0.2244	0.4329	19.16	0.5573	0.2142	19.13	0.5512	0.2154	19.34	0.5552	0.2052
	spin	14.54	0.4652	0.4002	15.12	0.6701	0.3166	15.69	0.6593	0.3354	15.96	0.6740	0.3236
	teddy	11.15	0.5790	0.5752	11.26	0.5823	0.6942	10.06	0.5821	0.7093	11.99	0.5607	0.6157

Table 1: Forecasting results on the iPhone dataset. We forecast frames beyond the observed time window and evaluate the predicted frames rendered from held-out test viewpoints. The leftmost column ( Obs. ratio ) indicates the fraction of input training frames used by each model; 80% and 60% correspond to forecasting the remaining 20% and 40% of frames, respectively. † denotes methods that are re-implemented for the SoM-based 4DGS representation and retain their original forecasting scheme.

Figure 4: Qualitative results on iPhone dataset. We present forecasted frames from test camera views. (a) and (b) correspond to settings where the first 80% and 60% of frames are used for training, and the remaining 20% and 40% are forecasted, respectively.

Scene	GSPred [ 50 ]			Ours
Scene	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Trex	20.63	0.9271	0.0627	20.80	0.9379	0.0551
Mutant	25.14	0.9375	0.0415	25.69	0.9246	0.0482
Jumpingjacks	19.84	0.9087	0.0924	19.86	0.9090	0.0920
Standup	21.99	0.9170	0.0796	22.32	0.9201	0.0756
Bouncingballs	23.68	0.9603	0.0540	25.03	0.9623	0.0436

Table 2: Forecasting results on D-NeRF dataset. We use 60% of the frames as training and predict the remaining 40%.

Figure 5: Forecasting results on D-NeRF dataset. We render extrapolated future frames. Note that in Obs. Timesteps , the first and second columns show renderings reconstructed from training views at the first and last observed timesteps, respectively.

Figure 6: Long-term forecasting results on iPhone dataset. (a) shows GT views from test cameras at the observed timesteps, and (b) presents the forecasted renderings for timesteps beyond the observations. Note that

t

denotes the timestep of the last observed frame.

Table 3: Ablation results. (Top) Tracking performance on iPhone dataset. (Bottom) Masking performance on D-NeRF dataset.

Figure 7: Effect of masking. Applying contiguous-span masking during training enhances the robustness of motion forecasting.

Figure 8: Gaussian grouping results. We present Gaussian grouping results of our method and Gaga-4D on iPhone dataset [ 10 ] , trained on the full sequence.

Figure 9: Overview of the forecaster. (a) Training stage: The forecaster is trained for each Gaussian group

{G}^{({k})}

by minimizing the loss

\mathcal{L}_{\text{group}}^{(k)}

between the predicted

\mathbf{\hat{x}}_{t}

and the observed

\mathbf{x}_{t}

at time

T

. We randomly apply contiguous masking to the inputs and gradually reduce the masking ratio later in training. (b) Inference stage: Future motion is generated via autoregressive rollout for each

{G}^{({k})}

, where the input sequence is iteratively updated with each newly predicted frame. Note that the input length is always

T\!-\!1

, the same as in training.

Obs. ratio	Scene	GSPred			GSPred-SoM			MoGaF (Ours)
Obs. ratio	Scene	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$	mPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$
70%	apple	13.21	0.5562	0.6942	16.58	0.7339	0.5007	16.45	0.7220	0.5329
	block	13.11	0.5333	0.5240	13.47	0.6192	0.4900	14.10	0.6110	0.5030
	paper-windmill	14.67	0.2168	0.4410	18.73	0.5221	0.2191	18.98	0.5197	0.2108
	spin	15.67	0.4738	0.3975	16.00	0.6797	0.3049	15.87	0.6618	0.3438
	teddy	11.46	0.5886	0.5708	10.84	0.5666	0.6555	11.87	0.5562	0.5578

Table 4: Forecasting results on the iPhone dataset. We forecast frames beyond the observed time window and evaluate the predicted frames rendered from held-out test viewpoints. The leftmost column ( Obs. ratio ) indicates the fraction of input training frames used by each model.

Scene	GSPred [ 50 ]			Ours
Scene	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Trex	21.09	0.9406	0.0461	20.60	0.9405	0.0499
Mutant	28.16	0.9560	0.0256	28.72	0.9552	0.0218
Jumpingjacks	20.51	0.9184	0.0760	20.12	0.9220	0.0734
Standup	25.96	0.9403	0.0481	30.38	0.9544	0.0335
Bouncingballs	26.63	0.9714	0.0361	26.82	0.9728	0.0367

Table 5: Forecasting results on D-NeRF dataset. We use 80% of the frames as input and predict the remaining 20%.

Methods	3D tracking			2D tracking			Photometric
Methods	EPE $\downarrow$	$\delta_{3D}^{.10}$ $\uparrow$	$\delta_{3D}^{.05}$ $\uparrow$	AJ $\uparrow$	$\delta_{\text{avg}}$ $\uparrow$	OA $\uparrow$	MPSNR $\uparrow$	mSSIM $\uparrow$	mLPIPS $\downarrow$
SoM [ 37 ]	0.1016	70.62	46.59	37.61	49.17	86.99	16.72	0.6462	0.3914
MoGaF (Ours)	0.0989	70.63	47.64	38.54	49.93	86.85	16.75	0.6462	0.3887

Table 6: Quantitative evaluation results on scene interpolation. We compare tracking and photometric performance against SoM [ 37 ] on iPhone dataset [ 10 ] .

Table 7: Ablation studies on group-wise design and region growing. We analyze the contribution of each component in our method across 3D/2D motion tracking and novel view synthesis.

Figure 10: Qualitative evaluation results on scene interpolation. We report NVS performance of SoM [ 37 ] and MoGaF on iPhone dataset [ 10 ] , evaluated on test camera viewpoints over the full training sequence.

Figure 11: Long-term forecasting results on iPhone dataset. (a) shows GT views from test cameras at the observed timesteps, and (b) presents the forecasted renderings for timesteps beyond the observations. Note that

t

denotes the timestep of the last observed frame.

Figure 12: Qualitative results on iPhone dataset. We present forecasted frames from test camera views. (a) and (b) correspond to settings where the first 80% and 60% of frames are used for training, and the remaining 20% and 40% are forecasted, respectively.

Figure 13: Qualitative results on D-NeRF dataset. We present forecasted frames from test camera views. (a) and (b) correspond to settings where the first 80% and 60% of frames are used for training, and the remaining 20% and 40% are forecasted, respectively.

Space-Time Forecasting of Dynamic Scenes with Motion-aware Gaussian Grouping

Abstract

Figures & tables

Spatiotemporal Forecasting

Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding

RiGS: Rigid-aware 4D Gaussian Splatting from a Single Monocular Video

4DGS-JEPA: Temporally Compositional Joint-Embedding Prediction for Dynamic Gaussian Splatting

Abstract

Spatiotemporal Forecasting

Explore similar work

Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding

RiGS: Rigid-aware 4D Gaussian Splatting from a Single Monocular Video

4DGS-JEPA: Temporally Compositional Joint-Embedding Prediction for Dynamic Gaussian Splatting