4DGS-JEPA: Temporally Compositional Joint-Embedding Prediction for Dynamic Gaussian Splatting

Yongchao Huang

cs.AIAug 24, 2026

4DGS-JEPA: Temporally Compositional Joint-Embedding Prediction for Dynamic Gaussian Splatting

Authors: Yongchao Huang

Abstract

Dynamic Gaussian Splatting provides an explicit representation of evolving 3D scenes, but existing approaches are primarily optimized for reconstruction, future-state generation, or rendering rather than for learning reusable predictive dynamics. We propose 4DGS-JEPA, a Gaussian-native joint-embedding predictive architecture for causal multi-horizon prediction over dynamic Gaussian scenes. The model uses a hierarchical scene-, motion-group-, and Gaussian-level representation together with a horizon-conditioned transition operator that supports both direct prediction and recursive rollout. Its central principle is temporal composition: different chronological transition paths reaching the same future endpoint should produce compatible predictive states. Endpoint and multi-horizon path supervision anchor these predictions to future target embeddings, while a selective geometry decoder and geometry-level composition ground the learned dynamics in consistent group motion and Gaussian geometry without requiring complete future appearance reconstruction. We further introduce a hybrid correspondence mechanism that combines persistent canonical identity with residual optimal-transport matching under reordering and topology change. We characterize zero-loss path agreement and finite-error rollout accumulation theoretically. Three controlled experiments provide mechanism-level evidence that temporal composition reduces latent path dependence while retaining predictive accuracy, geometry-level composition improves consistency of decoded motion, and hybrid correspondence preserves reliable identity while remaining robust when correspondence becomes ambiguous. Together, 4DGS-JEPA provides a predictive, temporally compositional formulation of dynamic Gaussian worlds.

Figures & tables

Method	Native state	Future target	JEPA	Full recon.	Path comp.
4D-GS ( Wu et al., 2024 )	Gaussians	time query	no	yes	no
GWM ( Lu et al., 2025 )	Gaussians	future Gaussians	no	yes	no
GAF ( Chai et al., 2025 )	Gaussians	frame/action	no	yes	no
MoGaF ( Lee et al., 2026 )	Gaussians	future Gaussians	no	yes	no
Structured 4D ( Li et al., 2026 )	3D latent	future 3D	no	yes	no
V-JEPA 2 ( Assran et al., 2025 )	video latent	future latent	yes	no	no

Table 1: Positioning relative to representative prior work. “Full reconstruction” indicates that decoding a complete future appearance or renderable state is central to the training objective. “Path comp.” denotes explicit consistency between alternative temporal prediction paths.

Panel of Figure 1 — Figure 1: Overview of 4DGS-JEPA. A causal Gaussian history is encoded into a hierarchical predictive state with scene-, motion-group-, and Gaussian-level tokens. A shared horizon-conditioned transition produces direct and recursively composed future predictions. The future target history is constructed by forward-only tracking from the causal Gaussian state and subsequent observations, and an EMA target encoder provides the corresponding target embedding. Target embeddings provide endpoint and path supervision, while direct–composed agreement enforces temporal composition. A selective geometry decoder maps the predicted latent dynamics to explicit Gaussian motion, deformation, and existence variables. The decoded states are constrained by target-relative geometry accuracy, geometry-level composition, and source-relative structural priors. Solid arrows denote computational flow and dashed arrows denote supervision or consistency constraints.

Method	Direct FDE $\downarrow$	Rollout FDE $\downarrow$	$\Delta_{\mathrm{roll}}$ $\downarrow$	CompErr $\downarrow$	GeoCompErr $\downarrow$	Unseen FDE $\downarrow$
Endpoint	$0.03013{\pm}0.00042$	$0.03061{\pm}0.00065$	$0.00048{\pm}0.00023$	$0.00244{\pm}0.00055$	$0.00739{\pm}0.00082$	$0.03621{\pm}0.00080$
Endpoint + Path	$0.03195{\pm}0.00243$	$0.03228{\pm}0.00256$	$0.00033{\pm}0.00014$	$0.00065{\pm}0.00007$	$\mathbf{0.00649{\pm}0.00069}$	$0.03838{\pm}0.00304$
Endpoint + Composition	$\mathbf{0.02905{\pm}0.00095}$	$\mathbf{0.02940{\pm}0.00101}$	$0.00035{\pm}0.00014$	$0.00096{\pm}0.00011$	$0.00667{\pm}0.00074$	$\mathbf{0.03466{\pm}0.00122}$
Endpoint + Path + Composition	$0.03206{\pm}0.00245$	$0.03236{\pm}0.00255$	$\mathbf{0.00029{\pm}0.00013}$	$\mathbf{0.00058{\pm}0.00003}$	$0.00653{\pm}0.00089$	$0.03837{\pm}0.00283$

Table 2: Experiment 1 results averaged over 3 random seeds. Direct FDE, rollout FDE, and

\Delta_{\mathrm{roll}}

are averaged over supervised horizons

\{1,2,4,8\}

; composition metrics are averaged over non-trivial supervised paths. Unseen FDE is rollout FDE over the held-out horizons

\{3,6\}

. Lower is better for all metrics.

Panel of Figure 2 — Figure 2: Prediction accuracy and temporal composition across horizons. Panel (a) measures target-relative direct FDE: the learned models strongly outperform static persistence, while Endpoint + Composition becomes more accurate than causal constant-velocity extrapolation at the longest supervised horizon. Panel (b) measures direct–composed latent discrepancy: endpoint supervision alone leaves substantial path dependence, whereas path supervision and explicit temporal composition reduce it, with the combined objective giving the lowest CompErr.

Method	Direct FDE $\downarrow$	Direct cov. err. $\downarrow$	$E_{T}^{\mathrm{target}}$ $\downarrow$	$E_{T}^{\mathrm{dc}}$ $\downarrow$	$D_{\mathrm{match}}$ $\downarrow$
Latent composition only	$\mathbf{0.06614{\pm}0.00271}$	$0.19897{\pm}0.00387$	$0.26124{\pm}0.01038$	$0.00724{\pm}0.00057$	$0.00192{\pm}0.00015$
+ $\mathrm{SE}(3)$ composition	$0.06675{\pm}0.00408$	$\mathbf{0.19758{\pm}0.00330}$	$\mathbf{0.25554{\pm}0.00119}$	$\mathbf{0.00202{\pm}0.00031}$	$0.00138{\pm}0.00036$
Full geometry composition	$0.06728{\pm}0.00388$	$0.19758{\pm}0.00290$	$0.25582{\pm}0.00130$	$0.00235{\pm}0.00062$	$\mathbf{0.00130{\pm}0.00012}$

Table 3: Geometry-level composition at the longest supervised horizon

\Delta=8

, averaged over 3 random seeds. Direct FDE, direct covariance error, and

E_{T}^{\mathrm{target}}

are target-relative;

E_{T}^{\mathrm{dc}}

and

D_{\mathrm{match}}

measure direct–composed disagreement. Lower is better for all metrics.

Panel of Figure 3 — Figure 3: Explicit geometry consistency across non-trivial horizons. Panel (a) reports the direct–composed group-transform error $E_{T}^{\mathrm{dc}}$ in eq. 71 ; explicit $\mathrm{SE}(3)$ composition substantially reduces the discrepancy at every evaluated horizon. Panel (b) reports the Gaussian-level discrepancy $D_{\mathrm{match}}$ in eq. 42 ; the full geometry-composition objective gives the strongest primitive-level consistency. Explicit geometry regularization therefore further reduces residual path dependence beyond latent composition and common target anchoring.

Method	Permutation lineage $\uparrow$	Split/merge lineage $\uparrow$	Birth/death F1 $\uparrow$	Geometry discrepancy $\downarrow$
Index matching	$0.04056{\pm}0.00507$	$0.15563{\pm}0.00907$	$0.61134{\pm}0.00381$	$0.74311{\pm}0.00449$
OT only	$0.99789{\pm}0.00024$	$0.98100{\pm}0.00099$	$\mathbf{1.00000{\pm}0.00000}$	$0.05010{\pm}0.00014$
Hybrid	$\mathbf{1.00000{\pm}0.00000}$	$\mathbf{0.98715{\pm}0.00103}$	$\mathbf{1.00000{\pm}0.00000}$	$\mathbf{0.04906{\pm}0.00013}$

Table 4: Controlled Gaussian-identity stress test, averaged over 3 independent repetitions. LineageMass and birth/death F1 are higher-is-better; geometry discrepancy is lower-is-better.

Panel of Figure 4 — Figure 4: Mixed correspondence stress test as perturbation severity increases. Hybrid maintains higher LineageMass and lower induced geometry discrepancy than OT only at every evaluated severity. As more persistent canonical identities become invalid, the margin decreases, consistent with Hybrid increasingly relying on residual OT.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Setting
Motion groups	$3$
Gaussians per group	$6$
Sequence length	$24$
Group-center range	$[-1,1]^{3}$
Within-group center std.	$0.16$
Initial Gaussian scale	$[0.04,0.10]$

Appendix

Table 5: Controlled Gaussian-world generator used in Experiments 1 and 2.

Setting	Value
Context length $L$	$4$
Train / validation / test scenes	$512/32/48$
Train / validation / test samples	$6656/416/624$
Embedding dimension	$48$
Decoder horizon embedding	$16$
Transformer layers	$1$

Appendix

Table 6: Common model and optimization settings for Experiments 1 and 2.

Method	Held-out, seen segments		Unseen segments
	CompErr $\downarrow$	GeoCompErr $\downarrow$	CompErr $\downarrow$	GeoCompErr $\downarrow$
Endpoint	$0.01134{\pm}0.00244$	$0.01428{\pm}0.00160$	$0.00235{\pm}0.00047$	$0.00901{\pm}0.00120$
Endpoint + Path	$0.00286{\pm}0.00013$	$0.01105{\pm}0.00146$	$0.00064{\pm}0.00006$	$\mathbf{0.00778{\pm}0.00119}$
Endpoint + Composition	$0.00454{\pm}0.00049$	$0.01204{\pm}0.00125$	$0.00096{\pm}0.00011$	$0.00803{\pm}0.00110$
Endpoint + Path + Composition	$\mathbf{0.00260{\pm}0.00009}$	$\mathbf{0.01096{\pm}0.00157}$	$\mathbf{0.00057{\pm}0.00002}$	$0.00782{\pm}0.00141$

Appendix

Table 7: Generalization to temporal decompositions not used during Experiment 1 training, averaged over 3 random seeds. Lower is better.

	$E_{T}^{\mathrm{dc}}$ $\downarrow$			$D_{\mathrm{match}}$ $\downarrow$
$\Delta$	Latent	+ $\mathrm{SE}(3)$	Full	Latent	+ $\mathrm{SE}(3)$	Full
$2$	$0.001685{\pm}0.000121$	$\mathbf{0.000466{\pm}0.000089}$	$0.000528{\pm}0.000163$	$0.000429{\pm}0.000056$	$0.000360{\pm}0.000051$	$\mathbf{0.000350{\pm}0.000056}$
$4$	$0.004297{\pm}0.000723$	$\mathbf{0.001018{\pm}0.000259}$	$0.001102{\pm}0.000355$	$0.001048{\pm}0.000167$	$0.000664{\pm}0.000153$	$\mathbf{0.000626{\pm}0.000082}$
$8$	$0.007238{\pm}0.000573$	$\mathbf{0.002015{\pm}0.000311}$	$0.002345{\pm}0.000619$	$0.001923{\pm}0.000152$	$0.001378{\pm}0.000362$	$\mathbf{0.001298{\pm}0.000124}$

Appendix

Table 8: Horizon-wise direct–composed geometry consistency in Experiment 2, averaged over 3 random seeds. Lower is better.

Setting	Value
Active / reserved primitives	$24/6$
Motion-group labels	$4$
Semantic dimension	$6$
Perturbation severities	$\{0.10,0.25,0.40\}$
Position-noise std.	$0.025$
Log-scale-noise std.	$0.020$

Appendix

Table 9: Experiment 3 configuration.

Severity	Method	LineageMass $\uparrow$	Geometry discrepancy $\downarrow$
$0.10$	OT only	$0.95694{\pm}0.00752$	$0.05693{\pm}0.00157$
$0.10$	Hybrid	$\mathbf{0.96735{\pm}0.00620}$	$\mathbf{0.05541{\pm}0.00108}$
$0.25$	OT only	$0.95148{\pm}0.00091$	$0.06502{\pm}0.00171$
$0.25$	Hybrid	$\mathbf{0.95988{\pm}0.00082}$	$\mathbf{0.06349{\pm}0.00147}$
$0.40$	OT only	$0.91962{\pm}0.00344$	$0.07718{\pm}0.00183$
$0.40$	Hybrid	$\mathbf{0.92605{\pm}0.00137}$	$\mathbf{0.07603{\pm}0.00158}$

Appendix

Table 10: Mixed permutation, split, merge, birth, and death perturbations as severity increases, averaged over 3 independent Monte Carlo repetitions.

Hidden IDs	OT lineage	Hybrid lineage	OT geometry	Hybrid geometry
$0$	$0.95148$	$\mathbf{0.95988}$	$0.06502$	$\mathbf{0.06349}$
$0.10$	$0.95148$	$\mathbf{0.95916}$	$0.06502$	$\mathbf{0.06361}$
$0.25$	$0.95148$	$\mathbf{0.95838}$	$0.06502$	$\mathbf{0.06376}$
$0.50$	$0.95148$	$\mathbf{0.95649}$	$0.06502$	$\mathbf{0.06409}$

Appendix

Table 11: Persistent-ID dropout under mixed perturbations at severity

0.25

Explore similar work

May 25, 2026

Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors

Reconstructing dynamic 3D scenes from multi-view videos is a foundational task for robotics, AR/VR, and digital twins. While 3D Gaussian Splatting (3DGS) provides state-of-the art scene representation, its dynamic adaptations (4DGS) face significant challenges. Early MLP-based deformation models excel at interpolating within training frames but fail at temporal extrapolation --- important feature for forecasting in predictive systems. To address this, physically grounded approaches have been proposed to enable robust future forecasting, while, concurrently, semantic 4DGS methods have been developed for scene analysis and language-driven prompting. However, there is currently no framework that unites both physical consistency and semantic understanding. In this paper, we propose the first, to the best of our knowledge, unified 4DGS approach that is both physically grounded and semantically aware. Furthermore, to optimize computational efficiency, we introduce a novel rigid-body constraint mechanism. Instead of predicting the physical dynamics for every Gaussian particle, our model predicts the motion of a single anchor Gaussian per semantic object, updating the remaining Gaussians via relative transformations. This architectural design significantly accelerates novel view synthesis compared to an unconstrained baseline, while enabling object-accurate extrapolation and intuitive semantic interaction. Our project page: https://be2rlab.github.io/r5dgs/.

Denis Gridusov, Maxim Popov, Sergey Kolyubin

ITMO University, St. Petersburg, Russia

3D Gaussian Splatting 4D Reconstruction Dynamic Scene Reconstruction Novel View Synthesis Spatiotemporal Forecasting

Ask PDF

Apr 3, 2026

GP-4DGS: Probabilistic 4D Gaussian Splatting from Monocular Video via Variational Gaussian Processes

We present GP-4DGS, a novel framework that integrates Gaussian Processes (GPs) into 4D Gaussian Splatting (4DGS) for principled probabilistic modeling of dynamic scenes. While existing 4DGS methods focus on deterministic reconstruction, they are inherently limited in capturing motion ambiguity and lack mechanisms to assess prediction reliability. By leveraging the kernel-based probabilistic nature of GPs, our approach introduces three key capabilities: (i) uncertainty quantification for motion predictions, (ii) motion estimation for unobserved or sparsely sampled regions, and (iii) temporal extrapolation beyond observed training frames. To scale GPs to the large number of Gaussian primitives in 4DGS, we design spatio-temporal kernels that capture the correlation structure of deformation fields and adopt variational Gaussian Processes with inducing points for tractable inference. Our experiments show that GP-4DGS enhances reconstruction quality while providing reliable uncertainty estimates that effectively identify regions of high motion ambiguity. By addressing these challenges, our work takes a meaningful step toward bridging probabilistic modeling and neural graphics.

Mijeong Kim, Jungtaek Kim, Bohyung Han

1ECE and 2IPAI, Seoul National University, Korea · University of Wisconsin–Madison, USA

3D Gaussian Splatting Dynamic Scene Reconstruction Gaussian Processes

Ask PDF

Sep 1, 2026

EvoGS: Modeling Deformation Evolution for Dynamic Gaussian Splatting

Recent extensions of 3D Gaussian Splatting (3DGS) enable real-time novel view synthesis in dynamic scenes by learning time-conditioned Gaussian deformations. However, existing MLP-based methods typically estimate deformations independently at each timestamp, making them less robust to large or abrupt motions. To address this issue, we propose \textbf{EvoGS}, a 3DGS-based dynamic reconstruction framework that models Gaussian deformation as a temporal evolution process. EvoGS maintains persistent deformation states for each Gaussian, extrapolates future states from historical deformation states, and corrects the predictions with MLP-derived observations. The correction is adaptively weighted using a temporal residual memory and evolution statistics such as deformation velocity and trajectory deviation. To further improve reconstruction quality, EvoGS introduces deformation-aware densification. Clone and split operations are performed along corrected deformation directions, while an uncertainty-aware strategy suppresses densification for Gaussians with unstable deformation histories. Experiments show that EvoGS improves dynamic novel view synthesis quality and achieves competitive performance across benchmarks.

Wei Dong, Shahram Shirani, Jun Chen +1

The Department of Electrical & Computer Engineering, McMaster University

3D Gaussian Splatting Dynamic Scene Reconstruction Novel View Synthesis Gaussian Splatting

Ask PDF