cs.CVApr 2, 2026

Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding

Authors: Yuheng Jiang, Yiwen Cai, Zihao Wang, Yize Wu, Sicheng Li, Zhuo Su, Shaohui Jiao, Lan Xu

Organizations: ShanghaiTech University · ByteDance

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Volumetric video seeks to model dynamic scenes as temporally coherent 4D representations. While recent Gaussian-based approaches achieve impressive rendering fidelity, they primarily emphasize appearance but are largely agnostic to instance-level structure, limiting stable tracking and semantic reasoning in highly dynamic scenarios. In this paper, we present Director, a unified spatio-temporal Gaussian representation that jointly models human performance, high-fidelity rendering, and instance-level semantics. Our key insight is that embedding instance-consistent semantics naturally complements 4D modeling, enabling more accurate scene decomposition while supporting robust dynamic scene understanding. To this end, we leverage temporally aligned instance masks and sentence embeddings derived from Multimodal Large Language Models to supervise the learnable semantic features of each Gaussian via two MLP decoders, enabling language-aligned 4D representations and enforcing identity consistency over time. To enhance temporal stability, we bridge 2D optical flow with 4D Gaussians and finetune their motions, yielding reliable initialization and reducing drift. For the training, we further introduce a geometry-aware SDF constraints, along with regularization terms that enforces surface continuity, enhancing temporal coherence in dynamic foreground modeling. Experiments demonstrate that Director achieves temporally coherent 4D reconstructions while simultaneously enabling instance segmentation and open-vocabulary querying.

Figures & tables

Figure 2: Overview of Director. Using temporally consistent SAM3 masks and sentence embeddings, our method decomposes the scene into static background and dynamic foreground, learning language- and instance-aligned features for robust tracking, high-quality rendering, and accurate instance segmentation.
Figure 3: Gallery of our results. All images are rendered from novel views. The first four rows show high-fidelity rendering results under challenging scenarios, including fast motions and severe occlusions. The fifth row presents the corresponding instance-level 4D segmentation of the fourth row.
Figure 4: Qualitative comparison with 4DGS [ 71 ] , Spacetime Gaussian [ 37 ] , and TaoGS [ 21 ] . Ours shows the best quality. Note that the zoomed-in regions are extremely small in the original image, where even the ground truth is not very sharp.
Figure 5: Qualitative comparison with 4D segmentation methods, including SA4D [ 20 ] , SADG [ 36 ] , and 4-LEGS [ 9 ] .Ours achieves more accurate instance segmentation.
MethodST-NeRFMPEG GSC
PSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
4DGS34.7220.9560.062533.5420.9360.0912
Spacetime Gaussian37.9490.9650.052437.8070.9540.0526
TaoGS37.7190.9630.056436.1370.9520.0671
Ours38.9120.9670.046338.2410.9590.0537
Table 1: Rendering quality comparison on the corresponding dataset. Green and yellow cells indicate the best and second-best results, respectively.
MethodPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrowmIoU ↑\uparrowRecall ↑\uparrowF1F_{1} ↑\uparrow
SA4D33.9510.9420.09170.52350.56670.6464
SADG33.4210.9420.12590.49950.53300.6052
4-LEGS31.6550.9390.0784–––
Ours38.2980.9650.04700.83050.87640.8878
Table 2: Rendering quality and segmentation comparison on the basketball dataset.
Figure 6: Visualization of querying results. The top row shows the query–frame similarity scores across frames. The bottom row presents several reference views.
Figure 7: Qualitative ablation study on key training components.
Table 9
More in this topic

Dynamic Scene Reconstruction

Dynamic scene reconstruction builds time-varying 3D models of moving objects and environments from video or multi-view images.

See all 288 papers

Explore similar work

CardsList
May 25, 2026

Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors

Reconstructing dynamic 3D scenes from multi-view videos is a foundational task for robotics, AR/VR, and digital twins. While 3D Gaussian Splatting (3DGS) provides state-of-the art scene representation, its dynamic adaptations (4DGS) face significant challenges. Early MLP-based deformation models excel at interpolating within training frames but fail at temporal extrapolation --- important feature for forecasting in predictive systems. To address this, physically grounded approaches have been proposed to enable robust future forecasting, while, concurrently, semantic 4DGS methods have been developed for scene analysis and language-driven prompting. However, there is currently no framework that unites both physical consistency and semantic understanding. In this paper, we propose the first, to the best of our knowledge, unified 4DGS approach that is both physically grounded and semantically aware. Furthermore, to optimize computational efficiency, we introduce a novel rigid-body constraint mechanism. Instead of predicting the physical dynamics for every Gaussian particle, our model predicts the motion of a single anchor Gaussian per semantic object, updating the remaining Gaussians via relative transformations. This architectural design significantly accelerates novel view synthesis compared to an unconstrained baseline, while enabling object-accurate extrapolation and intuitive semantic interaction. Our project page: https://be2rlab.github.io/r5dgs/.

Denis Gridusov, Maxim Popov, Sergey Kolyubin

ITMO University, St. Petersburg, Russia

3D Gaussian Splatting4D ReconstructionDynamic Scene ReconstructionNovel View SynthesisSpatiotemporal Forecasting
AskPDF
Mar 31, 2026

MotionScale: Reconstructing Appearance, Geometry, and Motion of Dynamic Scenes with Scalable 4D Gaussian Splatting

Realistic reconstruction of dynamic 4D scenes from monocular videos is essential for understanding the physical world. Despite recent progress in neural rendering, existing methods often struggle to recover accurate 3D geometry and temporally consistent motion in complex environments. To address these challenges, we propose MotionScale, a 4D Gaussian Splatting framework that scales efficiently to large scenes and extended sequences while maintaining high-fidelity structural and motion coherence. At the core of our approach is a scalable motion field parameterized by cluster-centric basis transformations that adaptively expand to capture diverse and evolving motion patterns. To ensure robust reconstruction over long durations, we introduce a progressive optimization strategy comprising two decoupled propagation stages: 1) A background extension stage that adapts to newly visible regions, refines camera poses, and explicitly models transient shadows; 2) A foreground propagation stage that enforces motion consistency through a specialized three-stage refinement process. Extensive experiments on challenging real-world benchmarks demonstrate that MotionScale significantly outperforms state-of-the-art methods in both reconstruction quality and temporal stability. Project page: https://hrzhou2.github.io/motion-scale-web/.

Haoran Zhou, Gim Hee Lee

Department of Computer Science, National University of Singapore

4D ReconstructionDynamic Scene Reconstruction3D Gaussian SplattingMonocular 3D Reconstruction
AskPDF
May 29, 2026

Learning Global Motion with Compact Gaussians for Feed-Forward 4D Reconstruction

Dynamic scene reconstruction from monocular video remains a fundamental challenge in computer vision. Existing feed-forward methods predict 3D Gaussians pixel-wise for each frame, suffering from duplicated Gaussians and view-dependent biases that hinder effective learning of scene motion. We present C4G, a feed-forward 4D reconstruction framework built upon a compact set of timestamp-conditioned learnable Gaussian query tokens. Each token aggregates corresponding features across the full temporal context and decodes a 3D Gaussian whose position is modulated by the target timestamp, enabling globally coherent motion modeling without per-scene optimization. To capture fine-grained details, we further introduce a video diffusion model-based rendering enhancement module. Since our framework effectively aggregates features into Gaussians, we extend this capability to feature lifting, producing a 4D feature field that supports point tracking and dynamic scene understanding. C4G achieves strong novel-view synthesis performance using significantly fewer Gaussians and without requiring camera poses, while exhibiting stronger motion modeling and robustness to large temporal gaps.

Mungyeom Kim, Minkyeong Jeon, Honggyu An +10

KAIST AI · ETH Zürich, ETH AI Center · Sony AI +1

Point Tracking3D Gaussian Splatting4D ReconstructionDynamic Scene ReconstructionFeed-Forward 3D Reconstruction
AskPDF