R4DGS: Referring Segmentation in 4D Gaussian Splatting

Bangpu Chen; Yaxuan Li; Shirui Peng; Xiangtian Si; Liuxin Chu; Xitong Cao; Hongbo Jin; Jiayu Ding

cs.CVOct 21, 2026

R4DGS: Referring Segmentation in 4D Gaussian Splatting

Authors: Bangpu Chen, Yaxuan Li, Shirui Peng, Xiangtian Si, Liuxin Chu, Xitong Cao, Hongbo Jin, Jiayu Ding

Organizations: China University of Geosciences Wuhan, China · Xinjiang University Urumqi, China · University of Nottingham Ningbo China Ningbo, China · Peking University Shenzhen, China

Abstract

Dynamic representations based on 4D Gaussian Splatting (4DGS) have shown strong capabilities in reconstruction and rendering, yet current methods remain limited in supporting complex natural-language referring in dynamic scenes. Existing 4D Gaussian methods are primarily designed for rendering-oriented dynamic modeling, while current semantic extensions often rely on scene-specific semantic optimization or are evaluated on relatively simple query settings. In this paper, we study Referring 4D Gaussian Splatting (R4DGS), a task for grounding temporally varying, multi-target or reasoning-intensive, and zero-target or distractor expressions in dynamic 4D scenes. To support this task, we introduce R4D-Bench-QA, a benchmark with structured query annotations. We further present ReferGaussian, a unified framework built upon 4DGS that converts low-level dynamic Gaussians into persistent, language-addressable entities for training-free referring inference. ReferGaussian organizes mask-supported Gaussian entities in an EntityBank, uses Multimodal Large Language Models (MLLMs) to decompose complex queries into trackable object phrases and structured referring constraints, and performs constraint-aware grounding without scene-specific semantic optimization. Experiments on R4D-Bench-QA and the 4D LangSplat protocol demonstrate improved temporal target selection and competitive spatiotemporal grounding without scene-specific semantic training. Project resources are available at https://chenbangpu.github.io/R4DGS/.

Figures & tables

Figure 1. Example visualizations of ReferGaussian for dynamic and multi-target referring in 4D scenes. Qualitative referring segmentation results for a complete cookie, two actively typing hands, a glass cup containing light-colored liquid, and an empty glass cup across three frames.

Figure 2. Task comparison between Gaussian semantic queries and R4DGS. R4DGS resolves temporal, relational, exclusion, and cardinality constraints into zero-, one-, or multi-target mask sequences. Task-definition comparison figure for R4DGS, including zero-target, few-target, and multi-target outcomes in panel (d).

Dataset	Source	Scenes	Query Scale	Domain	Task Type	Temporal	Sentence-level Referring	Dynamic Semantics	Temp. Query	Spa. Query
LERF	LangSplat	4	208 words	2D	Open-vocab seg.	$\times$	$\times$	$\times$	$\times$	$\checkmark$
Ref-LERF	ReferSplat	4	63 inst.	2D	Ref. seg.	$\times$	$\times$	$\times$	$\times$	$\checkmark$
ScanNet	OpenGaussian	10	84 words	3D	Open-vocab seg.	$\times$	$\times$	$\times$	$\times$	$\times$
lerf_ovs_ langsplat	Segment then Splat	5	208 words	3D	Open-vocab seg.	$\times$	$\times$	$\times$	$\times$	$\checkmark$
HyperNeRF- Annotation	4D LangSplat	4	13 sentences	4D	Open-vocab seg.	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
R4D-Bench (Ours)	R4DGS	12	89 sentences	4D	Ref. seg.	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

Table 1. Comparison of representative Gaussian-language benchmarks, domains, and query support. Checkmarks denote supported capabilities.

Figure 5. Overview of ReferGaussian, from Gaussian entity construction to constraint-aware spatiotemporal reasoning. The three stages discover persistent entities, organize reusable evidence, and resolve structured queries. ReferGaussian framework with query-guided mask tracking, mask-supported Gaussian entity construction, EntityBank formation, and constraint-aware referent selection.

Method	R4D-Bench-QA
	Acc (%) $\uparrow$	vIoU (%) $\uparrow$
Segment then Splat	55.6	28.4
4D LangSplat	58.4	32.1
ReferGaussian (Ours)	76.5	34.4

Table 2. Quantitative comparison on R4D-Bench-QA. Acc is frame-wise temporal activation accuracy; vIoU combines temporal localization and mask overlap.

Method	americano		chickchicken		split-cookie		espresso		Average
	Acc	vIoU	Acc	vIoU	Acc	vIoU	Acc	vIoU	Acc	vIoU
LangSplat ( Qin et al., 2024 )	45.19	23.16	53.26	18.20	73.58	33.08	44.03	16.15	54.01	22.65
4D LangSplat ( Li et al., 2025 )	89.42	66.07	96.73	90.62	95.28	83.14	81.89	49.20	90.83	72.26
ReferGaussian (Ours)	99.43	68.13	82.89	60.87	90.91	55.79	98.80	67.01	93.01	62.95

Table 3. Quantitative comparison of time-sensitive querying on the 4D LangSplat HyperNeRF protocol, with per-scene and average Acc and vIoU.

Figure 6. Qualitative comparison on R4D-Bench-QA for temporal-state and exclusion queries. Columns show query types, and rows compare frame-aligned predictions. Representative qualitative comparison on R4D-Bench-QA with two query columns and method rows for RGB, ground truth, ReferGaussian, Segment then Splat, and 4D LangSplat.

Group	Setting	vIoU (%)
Entity	Direct GSAM mask reference	57.4
Entity	Framewise tracklet association	38.2
Entity	One-pass Gaussian voting	12.4
Entity	Iterative Gaussian lifting	34.4
Mask	Original masks	34.4
Mask	Erosion, radius $3/5$	$33.8/32.1$

Table 4. Ablations of entity construction and mask evidence. Rows report vIoU for lifting variants and controlled mask perturbations.

Multimodal Grounding

Multimodal grounding links language and other signals to the specific objects, regions, events, or actions that support an interpretation.

See all 185 papers

Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors
May 25, 2026Denis Gridusov, Maxim Popov, Sergey Kolyubin3D Gaussian Splatting 4D Reconstruction
ITMO University, St. Petersburg, Russia
PanopticQuery: Unified Query-Time Reasoning for 4D Scenes
Apr 7, 2026Ruilin Tang, Yang Zhou, Zhong Ye +34D Scene Understanding Multimodal Grounding
School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · School of Computing and Information Systems, Singapore Management University, Singapore 188065 · School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510006, China +1
Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding
Apr 2, 2026Yuheng Jiang, Yiwen Cai, Zihao Wang +5Dynamic Scene Reconstruction 3D Gaussian Splatting
ShanghaiTech University · ByteDance

May 25, 2026J/K move · Enter open · S save

Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors

Denis Gridusov, Maxim Popov, Sergey Kolyubin

ITMO University, St. Petersburg, Russia

3D Gaussian Splatting 4D Reconstruction Dynamic Scene Reconstruction Novel View Synthesis Spatiotemporal Forecasting

Open paper AskPDF arXiv

Reconstructing dynamic 3D scenes from multi-view videos is a foundational task for robotics, AR/VR, and digital twins. While 3D Gaussian Splatting (3DGS) provides state-of-the art scene representation, its dynamic adaptations (4DGS) face significant challenges. Early MLP-based deformation models excel at interpolating within training frames but fail at temporal extrapolation --- important feature for forecasting in predictive systems. To address this, physically grounded approaches have been proposed to enable robust future forecasting, while, concurrently, semantic 4DGS methods have been developed for scene analysis and language-driven prompting. However, there is currently no framework that unites both physical consistency and semantic understanding. In this paper, we propose the first, to the best of our knowledge, unified 4DGS approach that is both physically grounded and semantically aware. Furthermore, to optimize computational efficiency, we introduce a novel rigid-body constraint mechanism. Instead of predicting the physical dynamics for every Gaussian particle, our model predicts the motion of a single anchor Gaussian per semantic object, updating the remaining Gaussians via relative transformations. This architectural design significantly accelerates novel view synthesis compared to an unconstrained baseline, while enabling object-accurate extrapolation and intuitive semantic interaction. Our project page: https://be2rlab.github.io/r5dgs/.

Abstract

Multimodal Grounding

Explore similar work

Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors

PanopticQuery: Unified Query-Time Reasoning for 4D Scenes

Director: Instance-aware Gaussian Splatting for Dynamic Scene Modeling and Understanding