Multimodal large language models (MLLMs) have achieved impressive performance across various tasks such as image captioning and visual question answer(VQA); however, they often struggle to accurately interpret depth information inherent in visual data. In this work, we introduce DeepSight, the first dedicated depth MLLM designed to enhance three-dimensional scene understanding. Unlike conventional methods that align RGB image encodings with text, our approach takes advantage of the unique characteristics of depth images: single-channel grayscale images where the pixel values directly reflect depth cues to improve spatial reasoning. To address challenges associated with limited depth data and the inadequacy of simple channel replication, we construct a novel depth image-text pair dataset and a depth instruction dataset. Depth maps are generated from visual images using the GLPN model, and GPT-4 is employed to curate corresponding depth instructions, an approach validated by LLaVA. Additionally, we modify the ViT encoder in CLIP to incorporate local object information, thereby capturing the subtle continuous variations of depth more effectively. To evaluate the performance of our model, we develop a comprehensive depth question answer benchmark based on existing depth image datasets, which rigorously assesses understanding in typical depth map scenarios. Experimental results demonstrate that DeepSight significantly enhances depth perception and downstream task performance, marking a substantial step forward in multimodal three-dimensional understanding.
Figures & tables
Figure 1. Stereoscopic Vision Ability Test for MLLMs. We input RGB images into the MLLMs and ask questions about distance comparisons between objects. The results show that the MLLMs have issues with its stereoscopic vision ability.
Figure 2. Benchmark construction. (a) defines the types of tasks. (b) describes how benchmark is constructed from templates.
Task
Source
Count
Percentage (%)
Scene Classification
NYU-D
1,786
13.26
Recognition
SUN-D
3,793
28.15
Distance Judgment
SUN-D
5,737
42.58
Security
SUN-D
2,157
16.01
Total
13,473
100.00
Table 1. Task statistics across datasets. We construct four categories of sub-tasks: Scene Classification, Recognition, Distance Judgment, and Security. In total, 13,473 question-answer pairs are generated. A table showing the number of tasks across datasets: NYU-D and SUN-D. Tasks include scene classification, recognition, distance judgment, and security.
Figure 3. Sample from the Depth Instruction Dataset. The left side shows a sample of caption data used for alignment, and the right side shows an instruction sample generated by GPT-4.
Figure 4. Overall model architecture. The left figure shows the Depth Encoder architecture, where DeepSight modifies the CLIP image encoder to take an additional Bbox channel along with depth convolution. The right figure illustrates the model pipeline, showing the data flow from the input question and depth image to the output answer. In the alignment stage, only the Alignment Layer is trained while keeping the Depth Encoder, Text Encoder, and LLM frozen. In the fine-tuning stage, the other modules remain frozen and additional training is performed on the LLM.
Model
Scene Classification
Recognition
Distance Judge
Security
Avg.
(a) Zero Shot
PandaGPT-7B
28.19
17.15
13.74
43.16
25.56
ImageBindLLM-7B ( Han et al., 2023 )
58.23
23.06
28.74
22.68
33.18
DeepSight-7B (Ours)
57.01
28.72
39.23
29.15
38.53
(b) Fine-tuning
PandaGPT-7B-FT
36.92
24.38
33.12
44.06
34.62
Table 2. Performance comparison of depth-aware multimodal models on the Depth Template Benchmark, covering four QA sub-tasks: Scene Classification, Recognition, Distance Judgment, and Security. We evaluate both zero-shot and fine-tuned settings using the Depth Instruction Dataset. LanguageBind is first aligned with Vicuna1.5-7B before fine-tuning. Best results in each column are highlighted in bold.
Model
NYU-D (%)
SUN-D (%)
ImageBind
54.0
35.1
LanguageBind
65.1
36.7
DeepSight-7B (Ours)
67.0
38.4
Table 3. Zero-shot scene classification on 1,000 NYU-D and 1,000 SUN-D images. Our model achieves state-of-the-art accuracy.
Train MLP
Train LLM
Distance Judge
✓
✗
46.71
✗
✓
57.36
✓
✓
63.17
Table 4. Ablation study on SFT structure. Ablation study on SFT structure. Results of different configurations evaluated on the Distance Judge sub-task of the Depth Template Benchmark. Ablation table showing distance judgment accuracy when training only MLP, only LLM, or both components.
Train Bbox Conv
Infer Bbox Conv
Distance Judge
✓
✗
58.46
✓
✓
63.17
Table 5. Ablation on the Bbox Convolution. Results of different configurations evaluated on the Distance Judge sub-task of the Depth Template Benchmark. Ablation showing the effect of training/inference with Bbox Convolution on distance judgment accuracy.
Figure 5. Sample ratio search experiment. We search sample ratio with a step of 0.05. Test metric is zero-shot Scene Classification top-1 accuracy.
Model
Scene Classification
Recognition
Distance Judge
Security
Avg.
(a) Zero-shot
LLaVA-7B
30.62
26.15
25.41
20.58
25.69
BLIP-Vicuna-7B
24.17
29.41
30.14
31.07
28.20
QWen2.5-VL-3B ( Bai et al., 2025 )
36.83
26.75
31.02
27.58
30.05
QWen2.5-VL-7B
47.59
34.17
46.90
34.25
40.73
(b) Fine-tuning
Table 6. Performance comparison of general vision-language models on the Depth Template Benchmark, covering four QA sub-tasks: Scene Classification, Recognition, Distance Judgment, and Security. We evaluate both zero-shot and fine-tuned settings using the Depth Instruction Dataset.
Figure 6. Case study. We posed questions about the depth image to multiple models, and the results show that DeepSight is better able to meet the task requirements.