Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM

Haifeng Huang; Yilun Chen; Zehan Wang; Jiangmiao Pang; Zhou Zhao

cs.CVMar 29, 2026

Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM

Authors: Haifeng Huang, Yilun Chen, Zehan Wang, Jiangmiao Pang, Zhou Zhao

Organizations: School of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China. · Shanghai AI Laboratory, Shanghai 200032, China.

Abstract

Recent advancements in multi-modal large language models (MLLMs) have shown strong potential for 3D scene understanding. However, existing methods struggle with fine-grained object grounding and contextual reasoning, limiting their ability to interpret and interact with complex 3D environments. In this paper, we present Chat-Scene++, an MLLM framework that represents 3D scenes as context-rich object sequences. By structuring scenes as sequences of objects with contextual semantics, Chat-Scene++ enables object-centric representation and interaction. It decomposes a 3D scene into object representations paired with identifier tokens, allowing LLMs to follow instructions across diverse 3D vision-language tasks. To capture inter-object relationships and global semantics, Chat-Scene++ extracts context-rich object features using large-scale pre-trained 3D scene-level and 2D image-level encoders, unlike the isolated per-object features in Chat-Scene. Its flexible object-centric design also supports grounded chain-of-thought (G-CoT) reasoning, enabling the model to distinguish objects at both category and spatial levels during multi-step inference. Without the need for additional task-specific heads or fine-tuning, Chat-Scene++ achieves state-of-the-art performance on five major 3D vision-language benchmarks: ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D. These results highlight its effectiveness in scene comprehension, object grounding, and spatial reasoning. Additionally, without reconstructing 3D worlds through computationally expensive processes, we demonstrate its applicability to real-world scenarios using only 2D inputs.

Figures & tables

Fig. 1: Chat-Scene++ surpasses previous SOTA methods on five benchmarks in the field of 3D scene understanding.

Fig. 2: Example of utilizing object identifiers in conversation.

Fig. 3: Overall model architecture of Chat-Scene++. The model structures a 3D scene as a context-rich object sequence, forming the scene embeddings for the LLM input. Specifically, it decomposes the 3D scene into a sequence of object representations paired with object ID tokens. Context-rich object features are extracted using large-scale pre-trained 3D and 2D models. These features are then mapped to the LLM’s embedding space, where scene embeddings are constructed by sequentially combining object IDs with their corresponding object-level embeddings. By leveraging flexible object IDs, Grounded CoT can be optionally enabled to enhance reasoning over object relationships.

Fig. 4: Illustration of multi-modal context-rich feature extraction. Chat-Scene [ 24 ] extracts object-centric features for separate object proposals, while Chat-Scene++ extracts context-rich object features using 3D scene-level encoders and 2D image-level encoders.

Panel of Table 5 — TABLE I: Prompt template for the large language model.

Panel of Table 6 — TABLE II: Templates for grounded chain-of-thought.

Fig. 5: Examples of various 3D scene-language understanding tasks . All the tasks are unified to single-turn question-answering pairs without extra task heads. Object identifiers are used to reference and ground the object during the conversation.

Training Data	Source	# Sample
ScanRefer	ScanRefer [ 14 ]	33K
ScanRefer_CoT	ScanRefer [ 14 ]	24K
Multi3DRefer	Multi3DRefer [ 15 ]	39K
Scan2Cap	Scan2Cap [ 17 ]	33K
ScanQA	ScanQA [ 18 ]	26K
ScanQA_CoT	ScanQA [ 18 ]	23K

TABLE III: Statistics of training data.

Panel of Table 9 — TABLE IV: Performance comparison. “Expert models” are tailored for specific tasks using task-oriented heads, while “LLM-based models” are designed for general instructions and responses. Entries in gray denote the incorporation of ground-truth objects into the model inputs.

Panel of Table 10 — TABLE V: Ablation studies on object identifiers. “Plain Text” employs plain text for object numbers, “Gaussian” uses fixed Gaussian embeddings, and “Learnable” learns new identifier tokens. “Random Order” means using random ID order during training. “Token Cost” denotes the total tokens for $N$ objects, including object identifiers.

Panel of Table 11 — TABLE VI: Ablation studies on object features. “Pos.” adds additional position embeddings to introduce spatial context.

Panel of Table 12 — TABLE VII: Ablation studies on grounded CoT. “QA CoT” uses CoT data for visual question answering task ScanQA. “Grounded CoT” uses CoT data for visual grounding task ScanRefer. “Two-stage” involves an additional training stage using CoT data. “Space-level” and “Category-level” denote two different templates for grounded CoT.

LLM	ScanRefer	Multi3dRefer	Scan2Cap	ScanQA	SQA3D
LLM	[email protected]	[email protected]	[email protected]	CIDEr	EM
OPT-1.3B [ 69 ]	50.5	54.1	81.3	88.7	54.5
Tiny-Vicuna-1B [ 1 ]	52.6	56.1	81.9	91.7	54.7
Vicuna-7B [ 1 ]	53.4	58.7	83.0	93.5	55.8
Vicuna-13B [ 1 ]	53.0	57.9	84.2	94.2	56.1
LLaMA3-8B [ 70 ]	55.7	60.4	84.7	91.0	56.8

TABLE VIII: Ablation studies on LLM backbones.

Method	What	Is	How	Can	Which	Others	Avg.
w/o G-CoT	46.9	67.9	53.8	71.2	50.0	56.2	55.8
w/ G-CoT	47.1	69.2	53.3	71.0	52.6	56.7	56.4
$\Delta$ (95% CI)	+0.2 $\pm$ 0.1	+1.3 $\pm$ 0.3	–0.5 $\pm$ 0.2	–0.2 $\pm$ 0.1	+2.5 $\pm$ 0.5	+0.5 $\pm$ 0.2	+0.6 $\pm$ 0.1

TABLE IX: Zero-shot grounded CoT results on SQA3D. We report per-type accuracy and the paired-bootstrap 95% CI of

\Delta

(w/ G-CoT

-

w/o G-CoT).

Method	3RQA	3RDialog	3RPlan
LEO (zero-shot)	35.8	25.5	23.4
Ours (zero-shot)	38.2	34.3	35.0
LEO (fine-tuned)	51.9	73.3	81.1
Ours (fine-tuned)	58.1	85.7	96.5

TABLE X: Open-vocabulary results on 3RScan [ 71 ]

Method	Exist	Count	Object	Status	Comparison	Accuracy
Finetuned
LiDAR-LLM [ 73 ]	74.5	15.0	37.8	45.9	57.8	48.6
OccLLaMA [ 74 ]	79.9	18.9	42.8	49.1	65.2	53.4
LSceneLLM [ 75 ]	83.6	19.6	44.8	53.8	68.7	56.4
BEVDet+BUTD [ 72 ]	83.7	20.9	48.8	52.0	67.7	57.0
CenterPoint+BUTD [ 72 ]	84.1	21.3	49.2	55.9	69.2	58.1

TABLE XI: Zero-shot evaluation on the NuScenes-QA [ 72 ] validation set. Methods are grouped by training setting (finetuned vs. zero-shot). Best zero-shot results are bolded.

Fig. 6: Visualization results of visual grounding using 2D-only inputs (RGB scans). “GT” denotes the 2D masks projected from the ground-truth 3D point cloud mask of the target object.

Panel of Figure 18 — Fig. 7: Scaling trend of inference time and GPU memory w.r.t. the number of object proposals. Chat-Scene++ uses 3 tokens per object and $N=100$ object proposals in main experiments. The inference time and GPU memory are measured during LLM forwarding on ScanQA.

Method	Visual Grounding		ScanQA	SQA3D
Method	[email protected]	[email protected]	CIDEr	EM
Random	1.4	0.5	–	–
Ours	22.8	10.8	85.6	52.9
Upperbound	54.9	22.2	–	–

TABLE XII: Evaluation results on ScanNet tasks using 2D-only input. “Random” denotes predictions made using randomly selected instances, while “Upperbound” represents the best-matched instances from the segmented results.

		Numerical Answer				Multiple-Choice Answer
Methods	Avg.	Obj. Count	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route Plan	Appr. Order
LongVILA-8B	21.6	29.1	9.1	16.7	0.0	29.6	30.7	32.5	25.5
VILA-1.5-8B	28.9	17.4	21.8	50.3	18.8	32.1	34.8	31.0	24.8
LongVA-7B	29.2	38.0	16.6	38.9	22.2	33.1	43.3	25.4	15.7
LLaVA-OV-7B	32.4	47.7	20.2	47.4	12.3	42.5	35.2	29.4	24.4
LLaVA-Video-7B	35.6	48.5	14.0	47.8	24.2	43.5	42.4	34.0	30.6

TABLE XIII: Performance comparison on VSI-Bench [ 33 ] using 2D-only input. “Avg.” denotes overall performance.

3D VLMs

Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.

See all 204 papers

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
Sep 29, 2026Jaewoo Jung, Hyeonseo Yu, Honggyu An +103D Spatial Reasoning 3D Scene Representation
KAIST AI · ETH Zürich · Google +2
SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding
Sep 27, 2026Xiangqi Li, Libo Huang, Jiarui Zhao +43D Scene Representation 3D VQA
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
MAG-3D: Multi-Agent Grounded Reasoning for 3D Understanding
Apr 10, 2026Henry Zheng, Chenyue Fang, Rui Huang +33D Spatial Reasoning 3D Scene Understanding
Tsinghua University · Pico, ByteDance

Sep 29, 2026J/K move · Enter open · S save

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Jaewoo Jung, Hyeonseo Yu, Honggyu An, Jisang Han+9

KAIST AI · ETH Zürich · Google+2

3D Spatial Reasoning 3D Scene Representation 3D Scene Understanding 3D VLMs Multimodal Spatial Reasoning

Open paper AskPDF arXiv

Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.

Abstract

3D VLMs

Explore similar work

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding

MAG-3D: Multi-Agent Grounded Reasoning for 3D Understanding