cs.CVMar 15, 2026

4D Synchronized Fields: Motion-Language Gaussian Splatting for Temporal Scene Understanding

Authors: Mohamed Rayan Barhdadi, Samir Abdaljalil, Rasul Khanbayov, Erchin Serpedin, Hasan Kurban

Organizations: Texas A&M University · Hamad Bin Khalifa University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Current 4D representations decouple geometry, motion, and semantics: reconstruction methods discard interpretable motion structure; language-grounded methods attach semantics after motion is learned, blind to how objects move; and motion-aware methods encode dynamics as opaque per-point residuals without object-level organization. We propose 4D Synchronized Fields, a 4D Gaussian representation that learns object-factored motion in-loop during reconstruction and synchronizes language to the resulting kinematics through a per-object conditioned field. Each Gaussian trajectory is decomposed into shared object motion plus an implicit residual, and a kinematic-conditioned ridge map predicts temporal semantic variation, yielding a single representation in which reconstruction, motion, and semantics are structurally coupled and enabling open-vocabulary temporal queries that retrieve both objects and moments. On HyperNeRF, 4D Synchronized Fields achieves 28.52 dB mean PSNR, the highest among all language-grounded and motion-aware baselines, within 1.5 dB of reconstruction-only methods. On targeted temporal-state retrieval, the kinematic-conditioned field attains 0.884 mean accuracy, 0.815 mean vIoU, and 0.733 mean tIoU, surpassing 4D LangSplat (0.620, 0.433, and 0.439 respectively) and LangSplat (0.415, 0.304, and 0.262). Ablation confirms that kinematic conditioning is the primary driver, accounting for +0.45 tIoU over a static-embedding-only baseline. 4D Synchronized Fields is the only method that jointly exposes interpretable motion primitives and temporally grounded language fields from a single trained representation. Code will be released.

Figures & tables

Figure 1: 4D Synchronized Fields learns a deformable 4D Gaussian scene whose per-Gaussian trajectories are decomposed in-loop into shared object motion plus implicit residuals. The resulting synchronized tracks and kinematics condition an object-time language field, trained from projected object crops and a per-object ridge map, enabling open-vocabulary temporal queries that retrieve both objects and moments , grounded in learned motion structure.
Figure 2: Method overview. A time-conditioned deformation MLP Dθ\mathcal{D}_{\theta} predicts per-Gaussian deltas yielding deformed positions xi(t)\mathbf{x}_{i}(t) . A shared object-motion model Mϕ\mathcal{M}_{\phi} produces per-object transforms yielding object-predicted positions x~i(t)\tilde{\mathbf{x}}_{i}(t) . The residual ri(t)=xi(t)−x~i(t)\mathbf{r}_{i}(t)=\mathbf{x}_{i}(t)-\tilde{\mathbf{x}}_{i}(t) is defined implicitly and used only in regularizers; rendering uses xi(t)\mathbf{x}_{i}(t) unchanged. After training, synchronized tracks and 28D kinematic features are extracted and used to fit per-object ridge maps from kinematics to semantic residuals, yielding an object-time language field for open-vocabulary temporal queries; the resulting structured scene description is then leveraged by a multimodal LLM for downstream reasoning.
Table 1: Novel-view reconstruction on HyperNeRF (held-out 4D LangSplat test split, 2×2\times resolution). Best per category in darker shade; our results bold . Categories are not directly comparable : reconstruction-only methods (orange) optimize purely for pixel fidelity with no motion or language objectives. Despite additional structural constraints, our method (blue) closes to within 1.5 dB of the recon-only upper bound while outperforming all language-grounded and motion-aware baselines .
Figure 3: Targeted temporal-state retrieval on americano (left: “glass in luminous-liquid phase”) and espresso (right: “glass cup with liquid above midpoint”). Rows: RGB input, 4D LangSplat activation, 4D Synchronized Fields activation, ground truth. 4D LangSplat activates broadly across the temporal extent; 4D Synchronized Fields produces tighter activations aligned with ground-truth intervals, reflecting kinematic conditioning’s sensitivity to motion-correlated state transitions.
Table 2: Targeted temporal-state retrieval on four HyperNeRF scenes with human-annotated state labels. Best per metric in darker shade; our results bold .
Table 6
VariantAccvIoUtIoU
Full (ridge + kinematics)0.8840.8150.733
Static embedding only0.6260.4460.279
Δ\Delta (drop)−- 0.258−- 0.369−- 0.453
Table 5: Kinematic conditioning ablation (4-scene mean). Removing the ridge map from kinematics to semantics.
Table S1: Neu3D per-scene results for 4D Synchronized Fields (20k iter). Phase 1: reconstruction-only backbone. Phase 2: with motion decomposition. ρˉ\bar{\rho} : rigid-share ratio.
More in this topic

4D Scene Understanding

4D scene understanding reconstructs and reasons about how objects, geometry, and actions change across space and time.

See all 57 papers

Explore similar work

CardsList
Apr 2, 2026

Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding

Volumetric video seeks to model dynamic scenes as temporally coherent 4D representations. While recent Gaussian-based approaches achieve impressive rendering fidelity, they primarily emphasize appearance but are largely agnostic to instance-level structure, limiting stable tracking and semantic reasoning in highly dynamic scenarios. In this paper, we present Director, a unified spatio-temporal Gaussian representation that jointly models human performance, high-fidelity rendering, and instance-level semantics. Our key insight is that embedding instance-consistent semantics naturally complements 4D modeling, enabling more accurate scene decomposition while supporting robust dynamic scene understanding. To this end, we leverage temporally aligned instance masks and sentence embeddings derived from Multimodal Large Language Models to supervise the learnable semantic features of each Gaussian via two MLP decoders, enabling language-aligned 4D representations and enforcing identity consistency over time. To enhance temporal stability, we bridge 2D optical flow with 4D Gaussians and finetune their motions, yielding reliable initialization and reducing drift. For the training, we further introduce a geometry-aware SDF constraints, along with regularization terms that enforces surface continuity, enhancing temporal coherence in dynamic foreground modeling. Experiments demonstrate that Director achieves temporally coherent 4D reconstructions while simultaneously enabling instance segmentation and open-vocabulary querying.

Yuheng Jiang, Yiwen Cai, Zihao Wang +5

ShanghaiTech University · ByteDance

Dynamic Scene Reconstruction3D Gaussian Splatting4D Scene UnderstandingLanguage-Embedded 3DGSInstance SegmentationOpen-Vocabulary 3D Segmentation
AskPDF
Oct 21, 2026

R4DGS: Referring Segmentation in 4D Gaussian Splatting

Dynamic representations based on 4D Gaussian Splatting (4DGS) have shown strong capabilities in reconstruction and rendering, yet current methods remain limited in supporting complex natural-language referring in dynamic scenes. Existing 4D Gaussian methods are primarily designed for rendering-oriented dynamic modeling, while current semantic extensions often rely on scene-specific semantic optimization or are evaluated on relatively simple query settings. In this paper, we study Referring 4D Gaussian Splatting (R4DGS), a task for grounding temporally varying, multi-target or reasoning-intensive, and zero-target or distractor expressions in dynamic 4D scenes. To support this task, we introduce R4D-Bench-QA, a benchmark with structured query annotations. We further present ReferGaussian, a unified framework built upon 4DGS that converts low-level dynamic Gaussians into persistent, language-addressable entities for training-free referring inference. ReferGaussian organizes mask-supported Gaussian entities in an EntityBank, uses Multimodal Large Language Models (MLLMs) to decompose complex queries into trackable object phrases and structured referring constraints, and performs constraint-aware grounding without scene-specific semantic optimization. Experiments on R4D-Bench-QA and the 4D LangSplat protocol demonstrate improved temporal target selection and competitive spatiotemporal grounding without scene-specific semantic training. Project resources are available at https://chenbangpu.github.io/R4DGS/.

Bangpu Chen, Yaxuan Li, Shirui Peng +5

China University of Geosciences Wuhan, China · Xinjiang University Urumqi, China · University of Nottingham Ningbo China Ningbo, China +1

Multimodal Grounding4D Scene UnderstandingDynamic Gaussian Splatting
AskPDF
Apr 7, 2026

PanopticQuery: Unified Query-Time Reasoning for 4D Scenes

Understanding dynamic 4D environments through natural language queries requires not only accurate scene reconstruction but also robust semantic grounding across space, time, and viewpoints. While recent methods using neural representations have advanced 4D reconstruction, they remain limited in contextual reasoning, especially for complex semantics such as interactions, temporal actions, and spatial relations. A key challenge lies in transforming noisy, view-dependent predictions into globally consistent 4D interpretations. We introduce PanopticQuery, a framework for unified query-time reasoning in 4D scenes. Our approach builds on 4D Gaussian Splatting for high-fidelity dynamic reconstruction and introduces a multi-view semantic consensus mechanism that grounds natural language queries by aggregating 2D semantic predictions across multiple views and time frames. This process filters inconsistent outputs, enforces geometric consistency, and lifts 2D semantics into structured 4D groundings via neural field optimization. To support evaluation, we present Panoptic-L4D, a new benchmark for language-based querying in dynamic scenes. Experiments demonstrate that PanopticQuery sets a new state of the art on complex language queries, effectively handling attributes, actions, spatial relationships, and multi-object interactions. A video demonstration is available in the supplementary materials.

Ruilin Tang, Yang Zhou, Zhong Ye +3

School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · School of Computing and Information Systems, Singapore Management University, Singapore 188065 · School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510006, China +1

4D Scene UnderstandingMultimodal GroundingMulti-View Consistency
AskPDF