3D-IDE: 3D Implicit Depth Emergent

Chushan Zhang; Ruihan Lu; Jinguang Tong; Yikai Wang; Hongdong Li

cs.CVMar 28, 2026

3D-IDE: 3D Implicit Depth Emergent

Authors: Chushan Zhang, Ruihan Lu, Jinguang Tong, Yikai Wang, Hongdong Li

Organizations: School of Computing, Australian National University · FreiNexus · School of EECS, The University of Queensland · School of Artificial Intelligence, Beijing Normal University

Abstract

Leveraging 3D information within Multimodal Large Language Models (MLLMs) has recently shown significant advantages for indoor scene understanding. However, existing methods, including those using explicit ground-truth 3D positional encoding and those grafting external 3D foundation models for implicit geometry, struggle with the trade-off in 2D-3D representation fusion, leading to suboptimal deployment. To this end, we propose 3D-Implicit Depth Emergence, a method that reframes 3D perception as an emergent property derived from geometric self-supervision rather than explicit encoding. Our core insight is the Implicit Geometric Emergence Principle: by strategically leveraging privileged geometric supervision through mechanisms like a fine-grained geometry validator and global representation constraints, we construct an information bottleneck. This bottleneck forces the model to maximize the mutual information between visual features and 3D structures, allowing 3D awareness to emerge naturally within a unified visual representation. Unlike existing approaches, our method enables 3D perception to emerge implicitly, disentangling features in dense regions and, crucially, eliminating depth and pose dependencies during inference with zero latency overhead. This paradigm shift from external grafting to implicit emergence represents a fundamental rethinking of 3D knowledge integration in visual-language models. Extensive experiments demonstrate that our method surpasses SOTA on multiple 3D scene understanding benchmarks. Our approach achieves a 55% reduction in inference latency while maintaining strong performance across diverse downstream tasks, underscoring the effectiveness of meticulously designed auxiliary objectives for dependency-free 3D understanding. Source code can be found at github.com/ChushanZhang/3D-IDE.

Figures & tables

Figure 1: Comparison of 3D-aware designs for video-LLMs. (a) Explicit coordinate injection fuses 2D features with coarse 3D positional embeddings and requires 3D inputs at inference. (b) Dual encoders separately process RGB and geometry, then fuse their outputs, increasing complexity and latency. (c) 3D-IDE uses a single visual encoder trained so that 3D awareness emerges implicitly, enabling efficient RGB-only inference.

Method	External 3D inputs	Scan2Cap	ScanRefer	Multi3DRefer
Method	External 3D inputs	[email protected]	[email protected]	[email protected]
2D-MLLM (LLaVA-Next-Video) [ 50 ]	$\times$	31.0	-	-
3D-MLLM (Video-3D LLM) [ 55 ]	$\times$	31.5	53.7	46.0
3D-MLLM (Video-3D LLM) [ 55 ]	✓	83.8	58.1	58.0

Table 1: Performance gap between 3D-trained models with and without 3D inputs. On 3D VQA benchmarks, removing 3D signals reduces accuracy to the level of zero-shot 2D MLLMs.

Panel of Figure 2 — Figure 2: Illustration of the double information loss in explicit coordinate injection. (a) RGB frame with a 2D patch whose pixels are back-projected to point cloud. (b) Pooling collapses all patch points into one token, losing local structure. (c) Voxelization merges distinct 3D points into the same voxel, further degrading fine-grained geometry and harming downstream 3D reasoning.

Figure 3: The 3D-IDE framework . Our approach avoids the “Double Information Loss” (see Fig. 2 ) inherent in explicit coordinate injection methods. Instead of injecting coarse, lossy coordinates, we use a privileged training module (green box) that is detached at inference for zero latency . This module forces the model to learn a fine-grained 3D representation implicitly via two parallel gradient signals (green arrows): a geometric gradient from a weak depth validator and a global gradient from a frozen foundation model guidance.

Specialists
Method	ScanRefer		Multi3DRefer		Scan2Cap		ScanQA		SQA3D
Method	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	C	EM	EM
ScanRefer [ 5 ]	37.3	24.3	–	–	–	–	–	–	–
MVT [ 23 ]	40.8	33.3	–	–	–	–	–	–	–
3DVG-Trans [ 52 ]	45.9	34.5	–	–	–	–	–	–	–
ViL3DRel [ 8 ]	47.9	37.7	–	–	–	–	–	–	–

Table 2: Performance comparison on 3D scene understanding benchmarks. Specialists are single-task methods, while generalists are trained for multiple tasks. Bold indicates the best result. Our method belongs to the generalist group without 3D geometric.

Panel of Figure 4 — Figure 4: Qualitative results on three 3D vision-language tasks.

Components			ScanRefer		Multi3DRef
Global.	Geometric.	Cross-view.	[email protected]	[email protected]	[email protected]	[email protected]
$\times$	$\times$	$\times$	53.7	47.8	46.0	42.4
$\checkmark$	$\times$	$\times$	56.9	50.8	55.6	51.1
$\checkmark$	$\checkmark$ $\text{pretrain}_{\text{vggt}}$	$\times$	59.6	53.2	58.7	53.2
$\checkmark$	$\checkmark$ scratch	$\times$	59.8	53.3	59.7	54.3
$\checkmark$	$\checkmark$ $\text{pretrain}_{\text{vggt}}$	$\checkmark$	60.5	54.1	59.7	54.5

Table 3: Ablation Study. Effect of the components in our model.

\checkmark

denotes the component is enabled,

\times

denotes it is disabled.

Method	Spatial Correspondences		Normal Prediction
	Recall@2cm $\uparrow$	$\theta_{90}^{120}\uparrow$	RMSE $\downarrow$	mAcc $\uparrow$
Baseline [ 55 ]	40.15	21.38	32.26	52.13
Baseline + ours	42.27	23.06	31.36	53.49

Table 4: Geometric Representation Evaluation. We compare our method against the baseline on the tasks of Spatial Correspondences and Normal Prediction. Our method demonstrates superior performance by consistently outperforming the baseline across all four metrics. Detailed results are available in appendix.

Method	Params(B) $\downarrow$	Mean Time (s) $\downarrow$	Tokens/s $\uparrow$	Peak Mem(GB) $\downarrow$
VG LLM-8B [ 53 ]	9.25	3.60	4.32	21.10
Ours	8.06	1.61	10.72	18.35

Table 5: Inference Efficiency Comparison. Our model achieves over 2

\times

faster inference and higher generation throughput while using less GPU memory than VG-LLM under identical settings.

Panel of Figure 5 — Figure 5: More qualitative results on three 3D vision-language tasks: language-guided object localization (top), region-level captioning (middle), and spatial question answering (bottom). In the grounding examples, green 3D bounding boxes denote the ground-truth targets, red boxes the predictions of the baseline, and blue boxes the predictions of our model. Our method better aligns with the targets and produces more accurate captions and answers.

Components			ScanRefer		Multi3DRef
Global.	Geometric.	Cross-view.	[email protected]	[email protected]	[email protected]	[email protected]
$\times$	$\times$	$\times$	53.7	47.8	46.0	42.4
$\checkmark$	$\times$	$\times$	56.9	50.8	55.6	51.1
$\times$	$\checkmark$	$\checkmark$	58.6	51.9	57.8	52.5
$\checkmark$	$\checkmark$ scratch	$\times$	59.8	53.3	59.7	54.3
$\checkmark$	$\checkmark$ scratch	$\checkmark$	60.9	54.5	59.8	54.9

Table 6: Additional Ablation Study. Effect of

\mathcal{L}_{\text{global}}

to speed-up training.

\checkmark

denotes enabled,

\times

denotes disabled.

Figure 6: 3D-IDE retains performance w/o global supervision, at the cost of longer training and higher VRAM (OOM at 6-frame).

Components	ScanRefer	Multi3DRefer	Scan2Cap	ScanQA	SQA3D
	[email protected] $\uparrow$	[email protected] $\uparrow$	CIDEr $\uparrow$	CIDEr $\uparrow$	EM $\uparrow$
Baseline	53.7	46.0	31.5	99.7	58.6
$+\,\mathcal{L}_{\text{global}}$	56.9	55.6	77.7	100.0	57.8
$+\,\mathcal{L}_{\text{geometry}}$	59.8	59.7	78.7	101.9	59.0
$+\,\mathcal{L}_{\text{cross-view}}$ (Full)	60.9	59.8	79.0	102.1	59.2

Table 7: Extended ablation across all five benchmarks. Each row cumulatively adds one IGEP component. The full model consistently achieves the best results across all tasks.

Method	3D inputs	ScanRefer		Multi3DRefer
Method	3D inputs	[email protected]	[email protected]	[email protected]	[email protected]
3DRS [ 24 ]	✓	62.9	56.1	60.4	54.9
3DRS* [ 24 ]	$\times$	56.95	50.83	55.8	51.1
3D-IDE (Ours)	$\times$	60.9	54.5	59.8	54.9

Table 8: Effect of removing 3D geometric inputs from 3DRS at inference. 3DRS* denotes the RGB-only variant. Our method closes most of the gap to the full 3DRS while using no 3D inputs.

Panel of Figure 7 — Figure 7: More qualitative results on three 3D vision-language tasks (continued). The three rows correspond to object localization, region-level captioning, and spatial question answering, respectively. Color coding is the same as in Figure 5 . Our model remains effective across diverse scenes and linguistic queries.

Method	@0.5
Method	C	B-4	M	R
Scan2Cap [ 13 ]	39.08	23.32	21.97	44.48
3D-VisTA [ 59 ]	66.90	34.00	27.10	54.30
ChatScene [ 21 ]	77.19	36.34	28.01	58.12
LLaVA-3D [ 58 ]	79.21	41.12	30.21	63.41
baseline [ 54 ]	31.53	29.98	24.18	57.66

Table 9: Performance comparison on the Scan2Cap validation set.

Method	Test set						Avg.
Method	What	Is	How	Can	Which	Others	Avg.
3D-VisTA [ 59 ]	34.8	63.3	45.4	69.8	47.2	48.1	48.5
Scene-LLM [ 18 ]	40.9	69.1	45.0	70.8	47.2	52.3	54.2
ChatScene [ 21 ]	45.4	67.0	52.0	69.5	49.9	55.0	54.6
LLaVA-3D [ 58 ]	–	–	–	–	–	–	55.6
Video-3D [ 54 ]	51.1	72.4	55.5	69.8	51.3	56.0	58.6

Table 10: Performance comparison on the test set of SQA3D .

		Block 0				Block 1				Block 2				Block 3
Model	Venue	$\theta_{0}^{30}$	$\theta_{30}^{60}$	$\theta_{60}^{90}$	$\theta_{90}^{120}$	$\theta_{0}^{30}$	$\theta_{30}^{60}$	$\theta_{60}^{90}$	$\theta_{90}^{120}$	$\theta_{0}^{30}$	$\theta_{30}^{60}$	$\theta_{60}^{90}$	$\theta_{90}^{120}$	$\theta_{0}^{30}$	$\theta_{30}^{60}$	$\theta_{60}^{90}$	$\theta_{90}^{120}$
Video-3D LLM [ 55 ]	CVPR’25	75.99	38.83	20.27	10.68	80.48	52.97	32.15	17.70	75.36	49.26	34.94	21.38	71.97	46.28	34.50	22.19
3DRS [ 24 ]	NIPS’25	74.05	37.67	19.81	10.60	79.59	52.13	33.07	17.87	73.88	48.77	35.76	21.89	69.64	45.57	34.86	22.69
3D-IDE (Ours) ${}_{\text{pretrain}}$	–	74.63	38.07	19.99	10.69	81.60	53.64	33.03	17.33	77.04	52.25	37.13	22.34	72.33	47.68	35.15	22.11
3D-IDE (Ours) ${}_{\text{scratch}}$	–	74.93	38.05	19.89	10.77	81.63	53.79	33.06	17.38	77.05	52.02	37.58	23.06	72.46	47.97	36.16	23.21

Table 11: Correspondence Estimation Results for NAVI. We present the NAVI correspondence estimation results for all models. The results are presented for features extracted at different layers with performance binned for different relative viewpoint changes between image pairs. The highest performing entry in each column is bolded.

		NAVI (Test)
Model	Venue	[email protected] ∘ (%)	[email protected] ∘ (%)	Acc@30 ∘ (%)	RMSE ( ∘ ) $\downarrow$	mAcc (%) $\uparrow$
Video-3D LLM [ 55 ]	CVPR’25	$28.69$	$57.65$	$70.06$	$32.26$	$52.13$
3DRS [ 24 ]	NIPS’25	$28.61$	$57.90$	$70.48$	$31.86$	$52.33$
3D-IDE (Ours) ${}_{\text{pretrain}}$	–	$29.85$	$58.73$	$71.05$	$31.46$	$53.21$
3D-IDE (Ours) ${}_{\text{scratch}}$	–	30.15	59.01	71.32	31.46	53.49

Table 12: Surface Normal Estimation Results. We present the surface normal estimation results for all models. Higher is better for accuracy, lower is better for RMSE. The highest performing entry in each column is bolded.

Method	Unique		Multiple		Overall
Method	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]
ScanRefer [ 5 ]	76.3	53.5	32.7	21.1	41.2	27.4
MVT [ 23 ]	77.7	66.4	31.9	25.3	40.8	33.3
3DVG-Transformer [ 52 ]	81.9	60.6	39.3	28.4	47.6	34.7
ViL3DRel [ 8 ]	81.6	68.6	40.3	30.7	47.9	37.7
3DJCG [ 3 ]	83.5	64.3	41.4	30.8	49.6	37.3

Table 13: Performance comparison on the validation set of ScanRefer. “Unique” and “Multiple” depends on whether there are other objects of the same class as the target object. 211

Method	ZT w/o D	ZT w/ D	ST w/o D		ST w/ D		MT		ALL
Method	F1	F1	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]
M3DRef-CLIP [ 49 ]	81.8	39.4	53.5	47.8	34.6	30.6	43.6	37.9	42.8	38.4
D3Net [ 6 ]	81.6	32.5	–	38.6	–	23.3	–	35.0	–	32.2
3DJCG [ 3 ]	94.1	66.9	–	26.0	–	16.7	–	26.2	–	26.6
Grounded 3D-LLM [ 11 ]	–	–	–	–	–	–	–	–	45.2	40.6
PQ3D [ 60 ]	85.4	57.7	–	68.5	–	43.6	–	40.9	–	50.1

Table 14: Performance comparison on Multi3DRefer validation set. ZT: zero-target, ST: single-target, MT: multi-target, D: distractor.

Method	EM	B-1	B-2	B-3	B-4	ROUGE-L	METEOR	CIDEr
ScanQA [ 2 ]	21.05	30.24	20.40	15.11	10.08	33.33	13.14	64.86
3D-VisTA [ 59 ]	22.40	–	–	–	10.40	35.70	13.90	69.60
Oryx-34B [ 32 ]	–	38.00	24.60	–	–	37.30	15.00	72.30
LLaVA-Video-7B [ 51 ]	–	39.71	26.57	9.33	3.09	44.62	17.72	88.70
3D-LLM (Flamingo) [ 19 ]	20.40	30.30	17.80	12.00	7.20	32.30	12.20	59.20
3D-LLM (BLIP2-flant5) [ 19 ]	20.50	39.30	25.20	18.40	12.00	35.70	14.50	69.40

Table 15: Performance comparison on the validation set of ScanQA. EM indicates exact match accuracy, and B-1, B-2, B-3, B-4 denote BLEU-1, -2, -3, -4, respectively.

3D VLMs

Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.

See all 213 papers

Sep 29, 2026

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.

Jaewoo Jung, Hyeonseo Yu, Honggyu An +10

KAIST AI · ETH Zürich · Google +2

3D Spatial Reasoning 3D Scene Representation 3D Scene Understanding 3D VLMs Multimodal Spatial Reasoning

Ask PDF

Mar 6, 2026

DeepSight: Bridging Depth Maps and Language with a Depth-Driven Multimodal Model

Multimodal large language models (MLLMs) have achieved impressive performance across various tasks such as image captioning and visual question answer(VQA); however, they often struggle to accurately interpret depth information inherent in visual data. In this work, we introduce DeepSight, the first dedicated depth MLLM designed to enhance three-dimensional scene understanding. Unlike conventional methods that align RGB image encodings with text, our approach takes advantage of the unique characteristics of depth images: single-channel grayscale images where the pixel values directly reflect depth cues to improve spatial reasoning. To address challenges associated with limited depth data and the inadequacy of simple channel replication, we construct a novel depth image-text pair dataset and a depth instruction dataset. Depth maps are generated from visual images using the GLPN model, and GPT-4 is employed to curate corresponding depth instructions, an approach validated by LLaVA. Additionally, we modify the ViT encoder in CLIP to incorporate local object information, thereby capturing the subtle continuous variations of depth more effectively. To evaluate the performance of our model, we develop a comprehensive depth question answer benchmark based on existing depth image datasets, which rigorously assesses understanding in typical depth map scenarios. Experimental results demonstrate that DeepSight significantly enhances depth perception and downstream task performance, marking a substantial step forward in multimodal three-dimensional understanding.

Hao Yang, Hongbo Zhang, Yanyan Zhao +1

Harbin Institute of Technology, Harbin, China

3D VLMs Vision-Language Models 3D Scene Understanding 3D VQA

Ask PDF

Mar 29, 2026

Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM

Recent advancements in multi-modal large language models (MLLMs) have shown strong potential for 3D scene understanding. However, existing methods struggle with fine-grained object grounding and contextual reasoning, limiting their ability to interpret and interact with complex 3D environments. In this paper, we present Chat-Scene++, an MLLM framework that represents 3D scenes as context-rich object sequences. By structuring scenes as sequences of objects with contextual semantics, Chat-Scene++ enables object-centric representation and interaction. It decomposes a 3D scene into object representations paired with identifier tokens, allowing LLMs to follow instructions across diverse 3D vision-language tasks. To capture inter-object relationships and global semantics, Chat-Scene++ extracts context-rich object features using large-scale pre-trained 3D scene-level and 2D image-level encoders, unlike the isolated per-object features in Chat-Scene. Its flexible object-centric design also supports grounded chain-of-thought (G-CoT) reasoning, enabling the model to distinguish objects at both category and spatial levels during multi-step inference. Without the need for additional task-specific heads or fine-tuning, Chat-Scene++ achieves state-of-the-art performance on five major 3D vision-language benchmarks: ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D. These results highlight its effectiveness in scene comprehension, object grounding, and spatial reasoning. Additionally, without reconstructing 3D worlds through computationally expensive processes, we demonstrate its applicability to real-world scenarios using only 2D inputs.

Haifeng Huang, Yilun Chen, Zehan Wang +2

School of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China. · Shanghai AI Laboratory, Shanghai 200032, China.

3D VLMs 3D Scene Understanding 3D Scene Representation 3D Visual Grounding 3D VQA

Ask PDF

Abstract

3D VLMs

Explore similar work

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

DeepSight: Bridging Depth Maps and Language with a Depth-Driven Multimodal Model

Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM