Inst4DGS: Instance-Decomposed 4D Gaussian Splatting with Multi-Video Label Permutation Learning

Yonghan Lee; Dinesh Manocha

cs.CVMar 19, 2026

Inst4DGS: Instance-Decomposed 4D Gaussian Splatting with Multi-Video Label Permutation Learning

Authors: Yonghan Lee, Dinesh Manocha

Organizations: University of Maryland, College Park, USA

Abstract

We present Inst4DGS, an instance-decomposed 4D Gaussian Splatting (4DGS) approach with long-horizon per-Gaussian trajectories. While dynamic 4DGS has advanced rapidly, instance-decomposed 4DGS remains underexplored, largely due to the difficulty of associating inconsistent instance labels across independently segmented multi-view videos. We address this challenge by introducing per-video label-permutation latents that learn cross-video instance matches through a differentiable Sinkhorn layer, enabling direct multi-view supervision with consistent identity preservation. This explicit label alignment yields sharp decision boundaries and temporally stable identities without identity drift. To further improve efficiency, we propose instance-decomposed motion scaffolds that provide low-dimensional motion bases per object for long-horizon trajectory optimization. Experiments on Panoptic Studio and Neural3DV show that Inst4DGS jointly supports tracking and instance decomposition while achieving state-of-the-art rendering and segmentation quality. On the Panoptic Studio dataset, Inst4DGS improves PSNR from 26.10 to 28.36, and instance mIoU from 0.6310 to 0.9129, over the strongest baseline.

Figures & tables

Figure 2: Temporally consistent identity reconstruction. Our method maintains stable instance identities over time while achieving high-fidelity tracking and rendering. In contrast, TRASE [ 15 ] exhibits limited motion tracking and identity drift (red box).

Figure 3: Inst4DGS Pipeline: Given multi-view videos and per-video segmentation labels, our method reconstructs an instance-decomposed 4DGS scene with long-term per-instance trajectories. The pipeline has two stages: (1) instance-decomposed 3DGS initialization and (2) sequential 4D optimization. A per-video learnable latent differentiably aligns inconsistent instance labels across views, and per-instance motion bases provide a low-dimensional motion scaffold for efficient optimization.

Figure 4: Example of local and canonical instance maps produced by our method. Because each video stream is segmented independently, SAM3 annotations contain conflicting labels across views (blue circle). Our method resolves this by learning cross-view label permutations to produce consistent canonical labels (white circles). Our differentiable permutation module maps canonical labels ( $4^{th}$ row) to view-specific local labels ( $3^{rd}$ row), enabling direct supervision despite cross-view inconsistencies.

	Capability		Dataset
Method	Tracking	Instance	PanopticStudio					Neural3DV
			PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	Time $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mIoU $\uparrow$	Time $\downarrow$
SpacetimeGaussians [ 16 ]	$\times$	$\times$	26.99	0.9211	0.0744	$\times$	28.16	28.04	0.9364	0.0569	$\times$	50.04
Dynamic3DGaussians [ 18 ]	$\checkmark$	$\times$	26.64	0.8548	0.2832	$\times$	157.09	28.26	0.9029	0.1785	$\times$	-
SA4D [ 8 ]	$\triangle$	$\checkmark$	21.42	0.7674	0.3588	0.4618	23.36	28.24	0.9290	0.1454	0.7263	61.78
TRASE [ 15 ]	$\triangle$	$\checkmark$	26.10	0.8990	0.1674	0.6402	102.60	29.29	0.9278	0.1417	0.8978	244.80

Table 1: Photometric rendering comparison across datasets.

\checkmark

indicates support for the corresponding capability, where Tracking denotes long-term trajectory tracking and Instance denotes instance-decomposed reconstruction.

\triangle

denotes tracking via implicit deformation field. We also report training time in minutes.

Figure 5: Qualitative photometric rendering comparison. SA4D and TRASE fail under diverse motions in Panoptic Studio (basketball, football, softball), while our method preserves high-fidelity renderings. We also outperform STSG and Dynamic3DGS, which lack instance-decomposed rendering capability.

Figure 6: Qualitative instance segmentation comparison. SA4D and TRASE are not aligned with test-view labels since they lack cross-view label matching. Our method yields sharper boundaries via direct label supervision, while TRASE shows blurred boundaries and SA4D has incomplete coverage (blue). Although SAM3 may miss the fast-moving ball (white), our unseen-object masking prevents locally untracked objects from degrading our results.

Method	coffee_martini		cook_spinach		cut_roasted_beef		flame_steak		sear_steak
	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$
GaussianGrouping	0.8295	0.9890	0.8974	0.9941	0.9512	0.9972	0.8279	0.9900	0.9261	0.9959
SAGA	0.2201	0.8081	0.8125	0.9881	0.6982	0.9767	0.8439	0.9911	0.8959	0.9941
SA4D	0.8583	0.9910	0.8987	0.9941	0.8645	0.9914	0.8898	0.9940	0.9047	0.9948
DGD	0.7875	0.9865	0.8150	0.9883	0.8170	0.9877	0.6771	0.9776	0.7638	0.9854
CGC	0.8359	0.9895	0.9034	0.9945	0.9052	0.9943	0.8530	0.9920	0.8794	0.9934

Table 2: Dynamic segmentation performance on Neural3DV scenes. We report Dynamic mIoU and mAcc. Our method shows the best segmentation performance.

Method	basketball		football		tennis		softball		coffee_martini		flame_steak
	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$
SA4D	0.3789	0.9418	0.5029	0.9536	0.4892	0.9693	0.4762	0.9721	0.9392	0.9966	0.5134	0.9640
TRASE	0.5716	0.9577	0.6994	0.9804	0.6317	0.9766	0.6581	0.9813	0.9221	0.9931	0.8734	0.9920
Ours	0.9314	0.9936	0.9269	0.9928	0.8769	0.9841	0.9165	0.9919	0.9851	0.9979	0.8988	0.9931

Table 3: Instance segmentation performance across scenes. We report instance mIoU and mAcc. While TRASE performs competitively on Neural3DV scenes ( flame_steak , coffee_martini ), its performance drops on Panoptic Studio scenes. In contrast, our method consistently achieves high-fidelity instance segmentation across both datasets.

Motion Model Ablation
Setting	Rendering			Dynamic Seg.		Instance Seg.
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	mIoU ${}_{\text{dyn}}\uparrow$	mAcc ${}_{\text{dyn}}\uparrow$	mIoU $\uparrow$	mAcc $\uparrow$
w/o Motion Bases	19.2514	0.7778	0.5435	0.3299	0.9303	0.1057	0.9080
w/o Instance Grouping	28.1538	0.9170	0.0858	0.9253	0.9926	0.8802	0.9882
Instance Model Ablation
w/o Permutation (No Sinkhorn + No ST + No Track Masking)	24.4539	0.8578	0.2720	0.0989	0.1455	0.1076	0.502

Table 4: Ablation study : Our instance-decomposed motion bases and Sinkhorn-based label matching jointly contribute to improved rendering quality and instance segmentation performance.

3D Gaussian Splatting

3D Gaussian Splatting represents scenes as optimizable 3D Gaussian primitives for fast, photorealistic rendering from new viewpoints.

See all 1,019 papers

Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding
Apr 2, 2026Yuheng Jiang, Yiwen Cai, Zihao Wang +5Dynamic Scene Reconstruction 3D Gaussian Splatting
ShanghaiTech University · ByteDance
Implicit 4D Gaussian Splatting for Fast Motion with Large Inter-Frame Displacements
Jul 14, 2026Seung-gyeom Kim, Areum Kim, Yongjae Yoo +13D Gaussian Splatting 4D Reconstruction
Department of Applied Artificial Intelligence, Hanyang University · Hanyang University ERICA
TRiGS: Temporal Rigid-Body Motion for Scalable 4D Gaussian Splatting
Apr 1, 2026Suwoong Yeom, Joonsik Nam, Seunggyu Choi +74D Reconstruction Dynamic Scene Reconstruction
Sogang University · POSTECH · Seoul National University +2

Apr 2, 2026J/K move · Enter open · S save

Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding

Yuheng Jiang, Yiwen Cai, Zihao Wang, Yize Wu+4

ShanghaiTech University · ByteDance

Dynamic Scene Reconstruction 3D Gaussian Splatting 4D Scene Understanding Language-Embedded 3DGS Instance Segmentation Open-Vocabulary 3D Segmentation

Open paper AskPDF arXiv

Volumetric video seeks to model dynamic scenes as temporally coherent 4D representations. While recent Gaussian-based approaches achieve impressive rendering fidelity, they primarily emphasize appearance but are largely agnostic to instance-level structure, limiting stable tracking and semantic reasoning in highly dynamic scenarios. In this paper, we present Director, a unified spatio-temporal Gaussian representation that jointly models human performance, high-fidelity rendering, and instance-level semantics. Our key insight is that embedding instance-consistent semantics naturally complements 4D modeling, enabling more accurate scene decomposition while supporting robust dynamic scene understanding. To this end, we leverage temporally aligned instance masks and sentence embeddings derived from Multimodal Large Language Models to supervise the learnable semantic features of each Gaussian via two MLP decoders, enabling language-aligned 4D representations and enforcing identity consistency over time. To enhance temporal stability, we bridge 2D optical flow with 4D Gaussians and finetune their motions, yielding reliable initialization and reducing drift. For the training, we further introduce a geometry-aware SDF constraints, along with regularization terms that enforces surface continuity, enhancing temporal coherence in dynamic foreground modeling. Experiments demonstrate that Director achieves temporally coherent 4D reconstructions while simultaneously enabling instance segmentation and open-vocabulary querying.

Abstract

3D Gaussian Splatting

Explore similar work

Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding

Implicit 4D Gaussian Splatting for Fast Motion with Large Inter-Frame Displacements

TRiGS: Temporal Rigid-Body Motion for Scalable 4D Gaussian Splatting