PanopticQuery: Unified Query-Time Reasoning for 4D Scenes

Ruilin Tang; Yang Zhou; Zhong Ye; Wenxi Liu; Yan Huang; Shengfeng He

cs.CVApr 7, 2026

PanopticQuery: Unified Query-Time Reasoning for 4D Scenes

Authors: Ruilin Tang, Yang Zhou, Zhong Ye, Wenxi Liu, Yan Huang, Shengfeng He

Organizations: School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · School of Computing and Information Systems, Singapore Management University, Singapore 188065 · School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510006, China · College of Computer and Data Science, Fuzhou University, Fuzhou 350108, China

Abstract

Understanding dynamic 4D environments through natural language queries requires not only accurate scene reconstruction but also robust semantic grounding across space, time, and viewpoints. While recent methods using neural representations have advanced 4D reconstruction, they remain limited in contextual reasoning, especially for complex semantics such as interactions, temporal actions, and spatial relations. A key challenge lies in transforming noisy, view-dependent predictions into globally consistent 4D interpretations. We introduce PanopticQuery, a framework for unified query-time reasoning in 4D scenes. Our approach builds on 4D Gaussian Splatting for high-fidelity dynamic reconstruction and introduces a multi-view semantic consensus mechanism that grounds natural language queries by aggregating 2D semantic predictions across multiple views and time frames. This process filters inconsistent outputs, enforces geometric consistency, and lifts 2D semantics into structured 4D groundings via neural field optimization. To support evaluation, we present Panoptic-L4D, a new benchmark for language-based querying in dynamic scenes. Experiments demonstrate that PanopticQuery sets a new state of the art on complex language queries, effectively handling attributes, actions, spatial relationships, and multi-object interactions. A video demonstration is available in the supplementary materials.

Figures & tables

Fig. 1: While state-of-the-art embedding-based methods, such as 4D LangSplat, perform well on static attribute queries, they struggle with actions and interactions, often collapsing multiple instances of the same category into a single response, for example selecting all cats (left). PanopticQuery introduces query-time reasoning that leverages spatio-temporal context to disambiguate entities based on their interaction patterns, enabling precise grounding of the target object, such as the cat actively playing with the toy, where prior methods fail (right). This example is from our newly proposed Panoptic-L4D dataset.

Fig. 2: The Panoptic-L4D Construction Pipeline. Our two-phase process ensures high-quality, unambiguous data. An LLM agent proposes diverse scenarios, which are then manually selected and modeled in Blender to create realistic 4D scenes. The candidate (prompt, mask) pairs consist of human-annotated masks and VLM-generated prompts. These pairs then undergo an annotation-verification loop, where human annotators must unanimously agree on the prompt’s referent, eliminating ambiguity before inclusion in the final benchmark.

Fig. 3: Examples from the Panoptic-L4D Benchmark. Our dataset spans diverse environments, including outdoor scenes, room-scale interactions, and complex tabletop arrangements. The queries challenge models to understand fine-grained actions, interactions, and spatial relations.

Dataset	Type	Scale	Scenes	Setup	Prompts	GT-Objs	Ambig.
D-NeRF	Syn.	Object	8	Mono.	–	–	$\times$
HyperNeRF	Real	Object	17	Mono.	30 ∗	30 ∗	$\times$
Neu3D	Real	Tabletop	6	Multi-view / Fwd.	30 ∗	30 ∗	$\times$
Ours	S/R	T / R / O	25	Multi-view / Fwd.	325	124	$\checkmark$

TABLE I: Comparison with existing 4D datasets. Ours is the only dataset covering full-scale scenarios (Tabletop, Room, Outdoor) with ambiguous (action / spatial relation / interaction) annotations. ( ∗ : Prompts and annotations adapted from 4D LangSplat [ 7 ] .)

Panel of Figure 5 — Fig. 4: Linguistic Diversity in Panoptic-L4D. Word clouds visualize the vocabulary of our benchmark. (a) Nouns describing common objects and scene types. (b) The emphasis on verbs ( e.g. , walking, holding, sleeping) and prepositions ( e.g. , on, inside, behind) highlights the dataset’s focus on relational and dynamic reasoning, moving beyond simple attribute matching.

Fig. 5: Overview of PanopticQuery. We render multi-view RGB/depth videos from an initial 4DGS and obtain prompt-conditioned masks with a multi-view semantic grounder. Our multi-view semantic consensus validates masks by back-projecting them to 3D and re-projecting to other views, discarding irrelevant views and retaining consistent ones (e.g., $\mathrm{IoU}\!>\!\delta$ and frame accuracy $>\!\epsilon$ ); masks supported by enough votes ( $\geq\tau$ ) are lifted to the 4DGS for grouping-based embedding, enabling spatio-temporal querying.

Method	Attr. Query		Act. Query		Spat. Query		Inter. Query
Method	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU
LangSplat	69.45	42.14	53.33	42.42	26.67	17.16	21.42	12.13
4D LangSplat	84.60	68.73	83.33	70.15	53.33	43.57	57.14	42.93
Ours	89.55	75.47	91.67	76.52	73.33	62.27	78.57	67.41

TABLE II: Quantitative comparison of different query types on Neu3D. Our method achieves superior performance across all categories. Cells highlighted in red and yellow denote the best and second-best performance within each query category.

Method	Attr. Query		Act. Query		Spat. Query		Inter. Query
Method	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU
LangSplat	47.79	28.81	18.27	12.72	10.12	6.61	16.03	11.47
4D LangSplat	71.78	54.30	34.72	29.00	21.26	15.13	28.56	22.84
Ours	76.72	67.11	83.66	75.14	81.19	72.97	69.83	63.73

TABLE III: Quantitative comparison of query types on Panoptic-L4D dataset. Our method significantly outperforms other methods in dynamic and complex scenarios. Cells highlighted in red and yellow denote the best and second-best performance within each query category.

Panel of Figure 9 — Fig. 6: Qualitative Comparison on Neu3D dataset.

Panel of Figure 10 — Fig. 8: Various query types’ results of our method on Panoptic-L4D dataset.

Dataset	Attr.&Act.		Attr.&Int.		Attr.&Spat.		Spat.&Act.		Spat.&Int.
Dataset	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU
Neu3D	91.67	76.47	83.33	71.26	88.89	74.06	100.00	80.61	66.67	57.93
Panoptic-L4D	82.32	72.91	79.17	70.98	80.25	70.05	83.58	69.18	77.21	67.95

TABLE IV: Quantitative results on multi-type combination queries. We evaluate methods on five complex composite queries across two datasets.

	Attr. Query		Act. Query		Spat. Query		Inter. Query
	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU
w/o MVSC	76.24	67.20	79.15	67.04	71.76	64.29	65.15	60.10
Ours	76.72	67.11	83.66	75.14	81.19	72.97	69.83	63.73

TABLE V: Ablation study on Multi-View Semantic Consensus (MVSC). It indicates that the proposed consensus mechanism plays a key role in producing 4D-consistent results. Cells highlighted in red denote the best performance within each query category.

# Cams	Attr. Query		Act. Query		Spat. Query		Inter. Query
# Cams	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU
1 cam	71.33	62.31	61.48	49.82	52.66	43.51	58.14	45.29
4 cams	78.62	66.13	67.48	53.30	62.66	47.44	61.18	46.89
6 cams	77.18	68.19	65.41	52.29	61.04	50.28	62.22	48.68
8 cams	76.72	67.11	83.66	75.14	81.19	72.97	69.83	63.73
10 cams	75.73	69.20	82.83	72.81	82.12	73.20	66.61	58.96

TABLE VI: Ablation study on the number of cameras. We evaluate the impact of camera density on different query types using mAcc(%) and mIoU(%). Cells highlighted in red and yellow denote the best and second-best performance within each query category.

VLM	Attr. Query		Act. Query		Spat. Query		Inter. Query
VLM	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU
w/o VLM	75.49	65.03	73.98	62.87	79.15	70.30	60.22	51.28
Gemini3-Pro	76.34	66.92	83.29	73.92	81.52	72.18	70.02	62.08
GPT5.2	77.04	67.47	82.26	73.51	79.19	73.29	69.13	61.36
Qwen2.5-VL (7B)	76.16	66.71	82.96	72.94	80.21	72.42	69.39	61.94
Qwen3-VL (8B)	76.72	67.11	83.66	75.14	81.19	72.97	69.83	63.73

TABLE VII: Ablation study on various Vision-Language Models (VLMs). We evaluate the impact of different VLM backbones on our framework’s reasoning capabilities across four query categories. Cells highlighted in red and yellow denote the best and second-best performance within each query category.

Scale	Attr.		Act.		Spat.		Inter.
Scale	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU
Outdoor	82.08	72.15	82.76	73.46	87.65	79.26	70.43	64.92
Tabletop	68.06	59.96	77.62	68.00	84.44	72.92	75.00	62.59
Room	76.00	68.75	90.48	80.58	71.43	64.94	73.91	66.00

TABLE VIII: Average quantitative results on Panoptic-L4D across different spatial scales. We report mAcc(%) and mIoU(%) for four query types.

4D Scene Understanding

4D scene understanding reconstructs and reasons about how objects, geometry, and actions change across space and time.

See all 57 papers

R4DGS: Referring Segmentation in 4D Gaussian Splatting
Oct 21, 2026Bangpu Chen, Yaxuan Li, Shirui Peng +5Multimodal Grounding 4D Scene Understanding
China University of Geosciences Wuhan, China · Xinjiang University Urumqi, China · University of Nottingham Ningbo China Ningbo, China +1
UniQuery4R: Unified 4D Scene Reconstruction from a Single Query
Aug 18, 2026Tiancheng Chen, Sheng Tang, Wenhua Jin +4Scene Flow Estimation Dynamic Scene Reconstruction
Kosmo Research · Automotive Engineering Department, Jilin University · School of Software, Tsinghua University
CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenes
Aug 19, 2026Kumal Hewagamage, Isuranga Senavirathne, Sasika Amarasinghe +4Vision-Language Models Vision-Language Alignment
University of Moratuwa, Sri Lanka · Singapore-MIT Alliance for Research & Technology (SMART) Centre, Singapore · Agency for Science, Technology and Research (A*STAR), Singapore

Oct 21, 2026J/K move · Enter open · S save

R4DGS: Referring Segmentation in 4D Gaussian Splatting

Bangpu Chen, Yaxuan Li, Shirui Peng, Xiangtian Si+4

China University of Geosciences Wuhan, China · Xinjiang University Urumqi, China · University of Nottingham Ningbo China Ningbo, China+1

Multimodal Grounding 4D Scene Understanding Dynamic Gaussian Splatting

Open paper AskPDF arXiv

Dynamic representations based on 4D Gaussian Splatting (4DGS) have shown strong capabilities in reconstruction and rendering, yet current methods remain limited in supporting complex natural-language referring in dynamic scenes. Existing 4D Gaussian methods are primarily designed for rendering-oriented dynamic modeling, while current semantic extensions often rely on scene-specific semantic optimization or are evaluated on relatively simple query settings. In this paper, we study Referring 4D Gaussian Splatting (R4DGS), a task for grounding temporally varying, multi-target or reasoning-intensive, and zero-target or distractor expressions in dynamic 4D scenes. To support this task, we introduce R4D-Bench-QA, a benchmark with structured query annotations. We further present ReferGaussian, a unified framework built upon 4DGS that converts low-level dynamic Gaussians into persistent, language-addressable entities for training-free referring inference. ReferGaussian organizes mask-supported Gaussian entities in an EntityBank, uses Multimodal Large Language Models (MLLMs) to decompose complex queries into trackable object phrases and structured referring constraints, and performs constraint-aware grounding without scene-specific semantic optimization. Experiments on R4D-Bench-QA and the 4D LangSplat protocol demonstrate improved temporal target selection and competitive spatiotemporal grounding without scene-specific semantic training. Project resources are available at https://chenbangpu.github.io/R4DGS/.

Abstract

4D Scene Understanding

Explore similar work

R4DGS: Referring Segmentation in 4D Gaussian Splatting

UniQuery4R: Unified 4D Scene Reconstruction from a Single Query

CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenes