4DGS-JEPA: Temporally Compositional Joint-Embedding Prediction for Dynamic Gaussian Splatting

Yongchao Huang

cs.AIAug 24, 2026

4DGS-JEPA: Temporally Compositional Joint-Embedding Prediction for Dynamic Gaussian Splatting

Authors: Yongchao Huang

Abstract

Dynamic Gaussian Splatting provides an explicit representation of evolving 3D scenes, but existing approaches are primarily optimized for reconstruction, future-state generation, or rendering rather than for learning reusable predictive dynamics. We propose 4DGS-JEPA, a Gaussian-native joint-embedding predictive architecture for causal multi-horizon prediction over dynamic Gaussian scenes. The model uses a hierarchical scene-, motion-group-, and Gaussian-level representation together with a horizon-conditioned transition operator that supports both direct prediction and recursive rollout. Its central principle is temporal composition: different chronological transition paths reaching the same future endpoint should produce compatible predictive states. Endpoint and multi-horizon path supervision anchor these predictions to future target embeddings, while a selective geometry decoder and geometry-level composition ground the learned dynamics in consistent group motion and Gaussian geometry without requiring complete future appearance reconstruction. We further introduce a hybrid correspondence mechanism that combines persistent canonical identity with residual optimal-transport matching under reordering and topology change. We characterize zero-loss path agreement and finite-error rollout accumulation theoretically. Three controlled experiments provide mechanism-level evidence that temporal composition reduces latent path dependence while retaining predictive accuracy, geometry-level composition improves consistency of decoded motion, and hybrid correspondence preserves reliable identity while remaining robust when correspondence becomes ambiguous. Together, 4DGS-JEPA provides a predictive, temporally compositional formulation of dynamic Gaussian worlds.

Figures & tables

Method	Native state	Future target	JEPA	Full recon.	Path comp.
4D-GS ( Wu et al., 2024 )	Gaussians	time query	no	yes	no
GWM ( Lu et al., 2025 )	Gaussians	future Gaussians	no	yes	no
GAF ( Chai et al., 2025 )	Gaussians	frame/action	no	yes	no
MoGaF ( Lee et al., 2026 )	Gaussians	future Gaussians	no	yes	no
Structured 4D ( Li et al., 2026 )	3D latent	future 3D	no	yes	no
V-JEPA 2 ( Assran et al., 2025 )	video latent	future latent	yes	no	no

Table 1: Positioning relative to representative prior work. “Full reconstruction” indicates that decoding a complete future appearance or renderable state is central to the training objective. “Path comp.” denotes explicit consistency between alternative temporal prediction paths.

Panel of Figure 1 — Figure 1: Overview of 4DGS-JEPA. A causal Gaussian history is encoded into a hierarchical predictive state with scene-, motion-group-, and Gaussian-level tokens. A shared horizon-conditioned transition produces direct and recursively composed future predictions. The future target history is constructed by forward-only tracking from the causal Gaussian state and subsequent observations, and an EMA target encoder provides the corresponding target embedding. Target embeddings provide endpoint and path supervision, while direct–composed agreement enforces temporal composition. A selective geometry decoder maps the predicted latent dynamics to explicit Gaussian motion, deformation, and existence variables. The decoded states are constrained by target-relative geometry accuracy, geometry-level composition, and source-relative structural priors. Solid arrows denote computational flow and dashed arrows denote supervision or consistency constraints.

Method	Direct FDE $\downarrow$	Rollout FDE $\downarrow$	$\Delta_{\mathrm{roll}}$ $\downarrow$	CompErr $\downarrow$	GeoCompErr $\downarrow$	Unseen FDE $\downarrow$
Endpoint	$0.03013{\pm}0.00042$	$0.03061{\pm}0.00065$	$0.00048{\pm}0.00023$	$0.00244{\pm}0.00055$	$0.00739{\pm}0.00082$	$0.03621{\pm}0.00080$
Endpoint + Path	$0.03195{\pm}0.00243$	$0.03228{\pm}0.00256$	$0.00033{\pm}0.00014$	$0.00065{\pm}0.00007$	$\mathbf{0.00649{\pm}0.00069}$	$0.03838{\pm}0.00304$
Endpoint + Composition	$\mathbf{0.02905{\pm}0.00095}$	$\mathbf{0.02940{\pm}0.00101}$	$0.00035{\pm}0.00014$	$0.00096{\pm}0.00011$	$0.00667{\pm}0.00074$	$\mathbf{0.03466{\pm}0.00122}$
Endpoint + Path + Composition	$0.03206{\pm}0.00245$	$0.03236{\pm}0.00255$	$\mathbf{0.00029{\pm}0.00013}$	$\mathbf{0.00058{\pm}0.00003}$	$0.00653{\pm}0.00089$	$0.03837{\pm}0.00283$

Table 2: Experiment 1 results averaged over 3 random seeds. Direct FDE, rollout FDE, and

\Delta_{\mathrm{roll}}

are averaged over supervised horizons

\{1,2,4,8\}

; composition metrics are averaged over non-trivial supervised paths. Unseen FDE is rollout FDE over the held-out horizons

\{3,6\}

. Lower is better for all metrics.

Panel of Figure 2 — Figure 2: Prediction accuracy and temporal composition across horizons. Panel (a) measures target-relative direct FDE: the learned models strongly outperform static persistence, while Endpoint + Composition becomes more accurate than causal constant-velocity extrapolation at the longest supervised horizon. Panel (b) measures direct–composed latent discrepancy: endpoint supervision alone leaves substantial path dependence, whereas path supervision and explicit temporal composition reduce it, with the combined objective giving the lowest CompErr.

Method	Direct FDE $\downarrow$	Direct cov. err. $\downarrow$	$E_{T}^{\mathrm{target}}$ $\downarrow$	$E_{T}^{\mathrm{dc}}$ $\downarrow$	$D_{\mathrm{match}}$ $\downarrow$
Latent composition only	$\mathbf{0.06614{\pm}0.00271}$	$0.19897{\pm}0.00387$	$0.26124{\pm}0.01038$	$0.00724{\pm}0.00057$	$0.00192{\pm}0.00015$
+ $\mathrm{SE}(3)$ composition	$0.06675{\pm}0.00408$	$\mathbf{0.19758{\pm}0.00330}$	$\mathbf{0.25554{\pm}0.00119}$	$\mathbf{0.00202{\pm}0.00031}$	$0.00138{\pm}0.00036$
Full geometry composition	$0.06728{\pm}0.00388$	$0.19758{\pm}0.00290$	$0.25582{\pm}0.00130$	$0.00235{\pm}0.00062$	$\mathbf{0.00130{\pm}0.00012}$

Table 3: Geometry-level composition at the longest supervised horizon

\Delta=8

, averaged over 3 random seeds. Direct FDE, direct covariance error, and

E_{T}^{\mathrm{target}}

are target-relative;

E_{T}^{\mathrm{dc}}

and

D_{\mathrm{match}}

measure direct–composed disagreement. Lower is better for all metrics.

Panel of Figure 3 — Figure 3: Explicit geometry consistency across non-trivial horizons. Panel (a) reports the direct–composed group-transform error $E_{T}^{\mathrm{dc}}$ in eq. 71 ; explicit $\mathrm{SE}(3)$ composition substantially reduces the discrepancy at every evaluated horizon. Panel (b) reports the Gaussian-level discrepancy $D_{\mathrm{match}}$ in eq. 42 ; the full geometry-composition objective gives the strongest primitive-level consistency. Explicit geometry regularization therefore further reduces residual path dependence beyond latent composition and common target anchoring.

Method	Permutation lineage $\uparrow$	Split/merge lineage $\uparrow$	Birth/death F1 $\uparrow$	Geometry discrepancy $\downarrow$
Index matching	$0.04056{\pm}0.00507$	$0.15563{\pm}0.00907$	$0.61134{\pm}0.00381$	$0.74311{\pm}0.00449$
OT only	$0.99789{\pm}0.00024$	$0.98100{\pm}0.00099$	$\mathbf{1.00000{\pm}0.00000}$	$0.05010{\pm}0.00014$
Hybrid	$\mathbf{1.00000{\pm}0.00000}$	$\mathbf{0.98715{\pm}0.00103}$	$\mathbf{1.00000{\pm}0.00000}$	$\mathbf{0.04906{\pm}0.00013}$

Table 4: Controlled Gaussian-identity stress test, averaged over 3 independent repetitions. LineageMass and birth/death F1 are higher-is-better; geometry discrepancy is lower-is-better.

Panel of Figure 4 — Figure 4: Mixed correspondence stress test as perturbation severity increases. Hybrid maintains higher LineageMass and lower induced geometry discrepancy than OT only at every evaluated severity. As more persistent canonical identities become invalid, the margin decreases, consistent with Hybrid increasingly relying on residual OT.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Setting
Motion groups	$3$
Gaussians per group	$6$
Sequence length	$24$
Group-center range	$[-1,1]^{3}$
Within-group center std.	$0.16$
Initial Gaussian scale	$[0.04,0.10]$

Appendix

Table 5: Controlled Gaussian-world generator used in Experiments 1 and 2.

Setting	Value
Context length $L$	$4$
Train / validation / test scenes	$512/32/48$
Train / validation / test samples	$6656/416/624$
Embedding dimension	$48$
Decoder horizon embedding	$16$
Transformer layers	$1$

Appendix

Table 6: Common model and optimization settings for Experiments 1 and 2.

Method	Held-out, seen segments		Unseen segments
	CompErr $\downarrow$	GeoCompErr $\downarrow$	CompErr $\downarrow$	GeoCompErr $\downarrow$
Endpoint	$0.01134{\pm}0.00244$	$0.01428{\pm}0.00160$	$0.00235{\pm}0.00047$	$0.00901{\pm}0.00120$
Endpoint + Path	$0.00286{\pm}0.00013$	$0.01105{\pm}0.00146$	$0.00064{\pm}0.00006$	$\mathbf{0.00778{\pm}0.00119}$
Endpoint + Composition	$0.00454{\pm}0.00049$	$0.01204{\pm}0.00125$	$0.00096{\pm}0.00011$	$0.00803{\pm}0.00110$
Endpoint + Path + Composition	$\mathbf{0.00260{\pm}0.00009}$	$\mathbf{0.01096{\pm}0.00157}$	$\mathbf{0.00057{\pm}0.00002}$	$0.00782{\pm}0.00141$

Appendix

Table 7: Generalization to temporal decompositions not used during Experiment 1 training, averaged over 3 random seeds. Lower is better.

	$E_{T}^{\mathrm{dc}}$ $\downarrow$			$D_{\mathrm{match}}$ $\downarrow$
$\Delta$	Latent	+ $\mathrm{SE}(3)$	Full	Latent	+ $\mathrm{SE}(3)$	Full
$2$	$0.001685{\pm}0.000121$	$\mathbf{0.000466{\pm}0.000089}$	$0.000528{\pm}0.000163$	$0.000429{\pm}0.000056$	$0.000360{\pm}0.000051$	$\mathbf{0.000350{\pm}0.000056}$
$4$	$0.004297{\pm}0.000723$	$\mathbf{0.001018{\pm}0.000259}$	$0.001102{\pm}0.000355$	$0.001048{\pm}0.000167$	$0.000664{\pm}0.000153$	$\mathbf{0.000626{\pm}0.000082}$
$8$	$0.007238{\pm}0.000573$	$\mathbf{0.002015{\pm}0.000311}$	$0.002345{\pm}0.000619$	$0.001923{\pm}0.000152$	$0.001378{\pm}0.000362$	$\mathbf{0.001298{\pm}0.000124}$

Appendix

Table 8: Horizon-wise direct–composed geometry consistency in Experiment 2, averaged over 3 random seeds. Lower is better.

Setting	Value
Active / reserved primitives	$24/6$
Motion-group labels	$4$
Semantic dimension	$6$
Perturbation severities	$\{0.10,0.25,0.40\}$
Position-noise std.	$0.025$
Log-scale-noise std.	$0.020$

Appendix

Table 9: Experiment 3 configuration.

Severity	Method	LineageMass $\uparrow$	Geometry discrepancy $\downarrow$
$0.10$	OT only	$0.95694{\pm}0.00752$	$0.05693{\pm}0.00157$
$0.10$	Hybrid	$\mathbf{0.96735{\pm}0.00620}$	$\mathbf{0.05541{\pm}0.00108}$
$0.25$	OT only	$0.95148{\pm}0.00091$	$0.06502{\pm}0.00171$
$0.25$	Hybrid	$\mathbf{0.95988{\pm}0.00082}$	$\mathbf{0.06349{\pm}0.00147}$
$0.40$	OT only	$0.91962{\pm}0.00344$	$0.07718{\pm}0.00183$
$0.40$	Hybrid	$\mathbf{0.92605{\pm}0.00137}$	$\mathbf{0.07603{\pm}0.00158}$

Appendix

Table 10: Mixed permutation, split, merge, birth, and death perturbations as severity increases, averaged over 3 independent Monte Carlo repetitions.

Hidden IDs	OT lineage	Hybrid lineage	OT geometry	Hybrid geometry
$0$	$0.95148$	$\mathbf{0.95988}$	$0.06502$	$\mathbf{0.06349}$
$0.10$	$0.95148$	$\mathbf{0.95916}$	$0.06502$	$\mathbf{0.06361}$
$0.25$	$0.95148$	$\mathbf{0.95838}$	$0.06502$	$\mathbf{0.06376}$
$0.50$	$0.95148$	$\mathbf{0.95649}$	$0.06502$	$\mathbf{0.06409}$

Appendix

Table 11: Persistent-ID dropout under mixed perturbations at severity

0.25

Explore similar work

Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors
May 25, 2026Denis Gridusov, Maxim Popov, Sergey Kolyubin3D Gaussian Splatting 4D Reconstruction
ITMO University, St. Petersburg, Russia
GP-4DGS: Probabilistic 4D Gaussian Splatting from Monocular Video via Variational Gaussian Processes
Apr 3, 2026Mijeong Kim, Jungtaek Kim, Bohyung Han3D Gaussian Splatting Dynamic Scene Reconstruction
1ECE and 2IPAI, Seoul National University, Korea · University of Wisconsin–Madison, USA
EvoGS: Modeling Deformation Evolution for Dynamic Gaussian Splatting
Sep 1, 2026Wei Dong, Shahram Shirani, Jun Chen +13D Gaussian Splatting Dynamic Scene Reconstruction
The Department of Electrical & Computer Engineering, McMaster University

May 25, 2026J/K move · Enter open · S save

Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors

Denis Gridusov, Maxim Popov, Sergey Kolyubin

ITMO University, St. Petersburg, Russia

3D Gaussian Splatting 4D Reconstruction Dynamic Scene Reconstruction Novel View Synthesis Spatiotemporal Forecasting

Open paper AskPDF arXiv

Reconstructing dynamic 3D scenes from multi-view videos is a foundational task for robotics, AR/VR, and digital twins. While 3D Gaussian Splatting (3DGS) provides state-of-the art scene representation, its dynamic adaptations (4DGS) face significant challenges. Early MLP-based deformation models excel at interpolating within training frames but fail at temporal extrapolation --- important feature for forecasting in predictive systems. To address this, physically grounded approaches have been proposed to enable robust future forecasting, while, concurrently, semantic 4DGS methods have been developed for scene analysis and language-driven prompting. However, there is currently no framework that unites both physical consistency and semantic understanding. In this paper, we propose the first, to the best of our knowledge, unified 4DGS approach that is both physically grounded and semantically aware. Furthermore, to optimize computational efficiency, we introduce a novel rigid-body constraint mechanism. Instead of predicting the physical dynamics for every Gaussian particle, our model predicts the motion of a single anchor Gaussian per semantic object, updating the remaining Gaussians via relative transformations. This architectural design significantly accelerates novel view synthesis compared to an unconstrained baseline, while enabling object-accurate extrapolation and intuitive semantic interaction. Our project page: https://be2rlab.github.io/r5dgs/.

Abstract

Explore similar work

Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors

GP-4DGS: Probabilistic 4D Gaussian Splatting from Monocular Video via Variational Gaussian Processes

EvoGS: Modeling Deformation Evolution for Dynamic Gaussian Splatting