3D-SceneQ: Empowering 3D LLM With Query-Guided Adaptive Pruning and Multi-Modal Feature Enhancement

Yongqi Shan; Lu Zhang; Jiazuo Yu; Yunzhi Zhuge; Huchuan Lu

cs.CVOct 27, 2026

3D-SceneQ: Empowering 3D LLM With Query-Guided Adaptive Pruning and Multi-Modal Feature Enhancement

Authors: Yongqi Shan, Lu Zhang, Jiazuo Yu, Yunzhi Zhuge, Huchuan Lu

Organizations: School of Artificial Intelligence, Dalian University of Technology, Dalian 116081, China. · School of Information and Communication Engineering, Dalian University of Technology, Dalian 116081, China.

Abstract

Current 3D scene understanding pipelines typically concatenate vast numbers of 3D object tokens with text tokens and feed the resulting sequence to a Large Language Model (LLM). However, existing 3D-LLMs for scene understanding encounter two major limitations: (i) excessive inclusion of task-irrelevant object data, introducing noise that reduces reasoning accuracy and increases hallucinations; and (ii) reliance solely on point cloud data, which inherently lacks the rich semantic information available in complementary 2D modalities, such as color, material properties, texture, and high-level contextual relationships. To address these challenges, we introduce 3D-SceneQ, a 3D LLM that unifies adaptive token pruning with multimodal semantic enrichment, markedly advancing scene understanding, reasoning, and grounding. Specifically, we propose a Query-Guided Adaptive Pruning (QGAP) module that adaptively selects task-relevant objects based on the intended meaning of user instructions, leveraging a learnable latent query to retain critical information while effectively suppressing irrelevant noise. In addition, we introduce a Multi-modal Object-level Feature Enhancement (MOFE) module, which integrates 2D feature embeddings from pre-trained models into 3D representation to enrich object-level semantic information. Evaluations show that 3D-SceneQ outperforms current state-of-the-art methods across multiple benchmarks (e.g., +5.7 CIDEr on Scan2Cap and +17.0 EM@1 on SQA3D), demonstrating its remarkable capabilities in various 3D language reasoning tasks.

Figures & tables

Fig. 1: (a) Conventional methods concatenate 3D and text tokens. (b) 3D-SceneQ uses a latent query to select task-relevant objects, then fuses 2D semantics for a compact, informative scene representation. represents the concatenation operation, denotes multimodal feature fusion, and denotes the projection of 2D features into 3D space.

Fig. 2: Overview of the 3D-SceneQ framework, which integrates multi-view 2D images, 3D point clouds, and textual instructions to generate informed responses. QGAP refers to the Query-Guided Adaptive Pruning module, and MOFE denotes the Multi-modal Object-level Feature Enhancement module.

Fig. 3: Overview of the Query-Guided Adaptive Pruning (QGAP) module, which retains task-relevant object tokens based on a global latent query.

Fig. 4: Overview of the Multi-modal Object-level Feature Enhancement (MOFE) module, which combines features across modalities. represents mapping multi-view 2D features to each point in 3D space, with aggregation and averaging across views. denotes pooling point-level features into object-level features using object masks.

Panel of Table 5 — TABLE I: Prompt structure used in 3D-SceneQ, combining role description, object context, and user instruction in a unified input format.

Component	Number of Parameters
QGAP	7.1M
MOFE	70.3M
Trainable Parameters (Total)	122.6M
3D-SceneQ (Overall)	7.0B

TABLE II: Parameter overview for 3D-SceneQ and its key components.

Method	Scan2Cap (val)					ScanQA (val)				SQA3D (test)
Method	CIDEr	BLEU-4	METEOR	ROUGE	Sim	CIDEr	BLEU-4	METEOR	ROUGE	EM@1
Task-specific models
Scan2Cap [ 19 ]	35.2	22.4	21.4	43.5	-	-	-	-	-	41.0 †
Vote2Cap-DETR [ 60 ]	61.8	34.5	26.2	54.4	-	-	-	-	-	-
ScanRefer+MCAN [ 61 ]	-	-	-	-	-	55.4	7.9	11.5	30.0	-
ScanQA [ 20 ]	-	-	-	-	-	64.9	10.1	13.1	33.3	47.2

TABLE III: Comparison of 3D-SceneQ with state-of-the-art models on Scan2Cap, ScanQA, and SQA3D benchmarks across various evaluation metrics. “EM@1" denotes top-1 exact match accuracy; Scan2Cap n-gram metrics use IoU

\geq

0.5. † is GPT-3 answers based on generated captions. “Sim” denotes sentence similarity.

Method	Scan2Cap (val)					ScanQA (val)				SQA3D (test)
Method	CIDEr	BLEU-4	METEOR	ROUGE	Sim	CIDEr	BLEU-4	METEOR	ROUGE	EM@1
Baseline	72.4	38.2	27.9	58.1	55.3	101.4	13.2	20.0	49.2	50.0
Baseline + RandFilter	52.3	38.0	27.8	62.3	55.2	90.1	13.4	18.1	43.7	68.9
Baseline + QGAP	78.4	44.1	30.4	65.6	64.2	101.6	14.1	19.9	48.7	70.1
Baseline + MOFE	77.8	44.4	30.5	65.1	64.1	102.0	14.6	20.0	48.8	66.7
3D-SceneQ	78.1	43.6	30.5	65.3	64.2	104.7	15.2	20.4	49.1	67.0

TABLE IV: Performance of 3D-SceneQ with different pruning and enhancement strategies. RandFilter refers to random pruning without adaptive mechanisms.

Method	Scan2Cap (val)					ScanQA (val)				SQA3D (test)
Method	CIDEr	BLEU-4	METEOR	ROUGE	Sim	CIDEr	BLEU-4	METEOR	ROUGE	EM@1
Baseline	72.4	38.2	27.9	58.1	55.3	101.4	13.2	20.0	49.2	50.0
Baseline +10% LowSimFilt	73.7	43.7	30.0	65.2	63.1	93.3	13.2	18.3	45.6	67.2
Baseline +20% LowSimFilt	74.5	43.5	30.2	65.3	63.8	97.6	13.2	18.8	46.7	69.0
Baseline +30% LowSimFilt	69.8	42.7	29.7	64.8	62.0	90.5	12.4	18.0	44.5	69.0
Baseline +40% LowSimFilt	67.2	42.4	29.6	64.3	61.9	89.4	11.5	16.5	42.4	61.3

TABLE V: Performance of 3D-SceneQ with different token pruning ratios. “LowSimFilt" refers to pruning a fixed proportion of object tokens with the lowest similarity to the query.

Method	Scan2Cap (val)					ScanQA (val)				SQA3D (test)
Method	CIDEr	BLEU-4	METEOR	ROUGE	Sim	CIDEr	BLEU-4	METEOR	ROUGE	EM@1
Baseline	72.4	38.2	27.9	58.1	55.3	101.4	13.2	20.0	49.2	50.0
Baseline +10% RandFilt	62.7	40.2	28.7	63.2	57.5	98.2	15.4	19.2	46.3	62.5
Baseline +20% RandFilt	55.4	38.6	28.1	62.8	56.1	93.2	14.3	18.3	44.8	68.4
Baseline +30% RandFilt	53.7	38.7	27.8	62.7	54.3	89.4	13.7	17.5	43.4	66.6
Baseline +40% RandFilt	52.3	38.0	27.8	62.3	55.2	90.1	13.4	18.1	43.7	68.9

TABLE VI: Performance of 3D-SceneQ with different token pruning ratios. “RandFilt" denotes randomly removing a fixed percentage of object tokens without adaptive selection.

Method	Scan2Cap (val)					ScanQA (val)				SQA3D (test)
Method	CIDEr	BLEU-4	METEOR	ROUGE	Sim	CIDEr	BLEU-4	METEOR	ROUGE	EM@1
Baseline	72.4	38.2	27.9	58.1	55.3	101.4	13.2	20.0	49.2	50.0
Mild Noise	78.0	43.4	30.4	65.4	63.9	100.4	13.8	19.5	47.6	68.7
Severe Noise	77.4	43.6	30.5	64.2	64.1	98.2	14.2	18.9	46.2	68.1
Baseline + MOFE	77.7	44.4	30.5	65.1	64.1	102.0	14.6	20.0	48.8	66.7

TABLE VII: Performance of 3D-SceneQ under simulated camera calibration noise. Mild noise perturbs focal length and SE(3) parameters with Gaussian noise sampled from

\mathcal{N}(0,(0.05x)^{2})

, and is applied with 10% probability. Severe noise increases the perturbation scale to

\mathcal{N}(0,(0.10x)^{2})

with 15% probability, where

x

denotes the original parameter value.

Method	ScanNet			3RScan
Method	Scan2Cap	ScanQA	SQA3D	3RQA	3RDialog	3RPlan
w/o Align	28.3	20.4	68.9	46.4	29.8	21.3
ScanNet-only	30.4	22.2	50.1	37.3	6.3	1.9
3RScan-only	15.8	14.0	25.9	50.2	29.9	21.4
3D-SceneQ	30.5	24.3	67.0	48.8	29.9	22.7

TABLE VIII: Performance of 3D-SceneQ under different training setups. w/o Align trains the model without the alignment stage; ScanNet-only and 3RScan-only are variants trained exclusively on scenes from their respective datasets. Exact-Match (EM) accuracy is reported for question-answering tasks, while METEOR is used for all other tasks. Underlined values denote zero-shot results on unseen scenes.

Fig. 5: Qualitative comparison between the baseline and our 3D-SceneQ model, with original scenes and their filtered counterparts shown before and after QGAP. (a) and (b) highlight the improved accuracy of 3D-SceneQ’s color perception, while (c) and (d) showcase its richer, more detailed answers. Red boxes indicate regions where objects have been pruned by QGAP, distinguishing them from the original scene.

Fig. 6: Adaptive object filtering with Query-Guided Adaptive Pruning (QGAP): (a)-(c) visualize progressively higher pruning ratios, from low to high, revealing how QGAP selectively removes irrelevant objects in response to the user’s query. Red boxes indicate regions where objects have been pruned by QGAP, distinguishing them from the original scene.

Fig. 7: Qualitative evidence of 3D-SceneQ’s hallucination mitigation: (a)–(c) reduce hallucinations of nonexistent or irrelevant objects, (d)–(f) lessen attribute hallucinations, and (g)–(h) correctly distinguish objects with similar visual features.

Fig. 8: Qualitative comparisons on out-of-distribution scenes from Matterport3D. (a) and (b) demonstrate 3D-SceneQ’s capability to identify and describe the most salient objects within the scene. (c) and (d) highlight the model’s ability to perceive material and shape attributes. (e) and (f) illustrate the accuracy of object grounding in visually complex environments. (g) and (h) showcase 3D-SceneQ’s sensitivity to object color.

3D Scene Understanding

3D scene understanding builds geometric and semantic representations of environments for spatial reasoning, perception, and interaction.

See all 295 papers

Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM
Mar 29, 2026Haifeng Huang, Yilun Chen, Zehan Wang +23D VLMs 3D Scene Understanding
School of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China. · Shanghai AI Laboratory, Shanghai 200032, China.
SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding
Sep 27, 2026Xiangqi Li, Libo Huang, Jiarui Zhao +43D Scene Representation 3D VQA
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
Sep 29, 2026Jaewoo Jung, Hyeonseo Yu, Honggyu An +103D Spatial Reasoning 3D Scene Representation
KAIST AI · ETH Zürich · Google +2

Mar 29, 2026J/K move · Enter open · S save

Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM

Haifeng Huang, Yilun Chen, Zehan Wang, Jiangmiao Pang+1

School of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China. · Shanghai AI Laboratory, Shanghai 200032, China.

3D VLMs 3D Scene Understanding 3D Scene Representation 3D Visual Grounding 3D VQA

Open paper AskPDF arXiv

Recent advancements in multi-modal large language models (MLLMs) have shown strong potential for 3D scene understanding. However, existing methods struggle with fine-grained object grounding and contextual reasoning, limiting their ability to interpret and interact with complex 3D environments. In this paper, we present Chat-Scene++, an MLLM framework that represents 3D scenes as context-rich object sequences. By structuring scenes as sequences of objects with contextual semantics, Chat-Scene++ enables object-centric representation and interaction. It decomposes a 3D scene into object representations paired with identifier tokens, allowing LLMs to follow instructions across diverse 3D vision-language tasks. To capture inter-object relationships and global semantics, Chat-Scene++ extracts context-rich object features using large-scale pre-trained 3D scene-level and 2D image-level encoders, unlike the isolated per-object features in Chat-Scene. Its flexible object-centric design also supports grounded chain-of-thought (G-CoT) reasoning, enabling the model to distinguish objects at both category and spatial levels during multi-step inference. Without the need for additional task-specific heads or fine-tuning, Chat-Scene++ achieves state-of-the-art performance on five major 3D vision-language benchmarks: ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D. These results highlight its effectiveness in scene comprehension, object grounding, and spatial reasoning. Additionally, without reconstructing 3D worlds through computationally expensive processes, we demonstrate its applicability to real-world scenarios using only 2D inputs.

Abstract

3D Scene Understanding

Explore similar work

Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM

SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering