DSRS: Teaching MLLMs to Perceive, Reason, and Segment in Complex Driving Scenes
Authors: Sicen Guo, Zhengfei Song, Kuanhsun Chen, Xu Huang, Ping Zhong, Wei Ye, Shaoyi Du, Rui Fan
Organizations: College of Electronic & Information Engineering, Tongji University, Shanghai 201804, China · Department of Computer Science and Technology, Central South University. · College of Electronics and Information Engineering, Tongji University, Shanghai 201804, China. He is also with the Shanghai Innovation Institute, Shanghai 200231, China · National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, the National Engineering Research Center for Visual Information and Applications, and the Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology, Shanghai Research Institute for Intelligent Autonomous Systems, Shanghai Key Laboratory of Intelligent Autonomous Systems, State Key Laboratory of Autonomous Intelligent Unmanned Systems, and Frontiers Science Center for Intelligent Autonomous Systems (Ministry of Education), Tongji University, Shanghai 201804, China, as well as with the National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’An, Shaanxi 710049, China
Existing reasoning segmentation frameworks typically employ multimodal large language models (MLLMs) with broad world knowledge to extract semantics in a human-like manner and generate corresponding segmentation masks. Nevertheless, these frameworks lack the necessary scene-specific awareness. This paper introduces driving scene reasoning segmentation (DSRS), a scene-specific task that requires models to interpret implicit instructions, infer spatial relations, detect potential risks, handle challenging conditions, and comprehend underlying traffic rules, ultimately generating accurate segmentation masks. Meanwhile, we establish a benchmark dataset to facilitate future research on the DSRS task. To pioneer this task, we propose DSRS-Net, a novel architecture with advanced reasoning segmentation capability for automated driving. Specifically, DSRS-Net seamlessly infuses geometric features for deeper reasoning segmentation, without compromising the original multimodal fusion mechanism. This strategy effectively mitigates the modality representation mismatch in MLLMs caused by RGB-only pretraining, while enhancing the integration of geometric, contextual, and linguistic features. Extensive experiments demonstrate the superior performance of DSRS-Net in both driving scene reasoning segmentation and other language-driven segmentation tasks across various public datasets. The source code and dataset will be released upon publication.
Figures & tables
Fig. 1: Comparison of the segmentation masks by LISA [ 1 ] and our DSRS-Net , facing the challenges in the driving scene reasoning segmentation task.
Fig. 2: Overview of the proposed DSRS benchmark and DSRS-Net . The left part illustrates representative driving-scene reasoning segmentation scenarios. The right part reports quantitative comparisons on the DSRS dataset.
Fig. 3: An illustration of our proposed DSRS-Net . Given the RGB image, depth map, and text query, DSRS-Net generates text output and segmentation masks. We use the lightweight LoRA strategy [ 33 ] for efficient fine-tuning.
Fig. 4: An illustration of our proposed depth-aware deformable attention module.
Fig. 5: A comprehensive analysis of our proposed DSRS dataset: (a) category composition; (b) distribution of instruction lengths, with the dashed line indicating the mean length; (c) frequency-weighted word cloud; and (d) category-wise sample counts (top) and relative mask-scale distributions, with category indices corresponding to those in (a).
Method
MLLM
Zero-shot (zs)
Fine-tuned (ft)
Val
Test
Val
Test
gIoU
cIoU
gIoU
cIoU
gIoU
cIoU
gIoU
cIoU
ReLA [ 41 ]
-
13.1
9.1
17.1
12.3
43.6
50.8
44.4
53.3
X-Decoder [ 42 ]
-
24.1
14.3
26.3
15.1
56.5
48.8
51.4
46.2
SEEM [ 43 ]
-
23.9
16.2
29.7
14.9
42.1
34.1
42.4
33.2
Grounded-SAM [ 44 ]
-
6.6
26.2
10.0
28.6
20.4
32.8
17.0
31.8
TABLE I: Comparative results on driving scene segmentation. ‘zs’ denotes zero-shot evaluation, and ‘ft’ indicates fine-tuning on the DSRS dataset. In addition, the DSRS dataset has no overlap with the initial training dataset.
Table 7
Fig. 6: Qualitative comparisons on the DSRS dataset, where ground-truth areas are highlighted with white dashed boxes and wrong areas are highlighted with blue dashed circles.
RGB Branch
Depth CLIP Branch
GPE
DHFI
gIoU
cIoU
✓
53.00
50.42
✓
✓
55.25
52.12
✓
✓
37.39
36.67
✓
✓
53.28
55.10
✓
✓
✓
55.40
56.97
TABLE IV: Ablation studies on the core designs of DSRS-Net .
Contextual Features
Geometric Features
Fusion Strategies
gIoU
cIoU
✓
Self-Attention
44.56
44.08
✓
✓
Element-wise Summation
50.47
51.12
✓
✓
Channel-wise Concatenation
51.09
51.86
✓
✓
Cross-Attention
52.78
54.05
✓
✓
DA2
55.40
56.97
TABLE V: Ablation studies on the DHFI strategy.
Fig. 7: Comparisons between driving scene reasoning segmentation and other segmentation tasks.
Dataset
Input
Output
Lang. Source
Scene
Spatial Reasoning
Traffic Rule
SQA3D [ 61 ]
Lang, Image, Point cloud, Video
Lang
Human
IS
✓
✗
MSQA [ 62 ]
Lang, Image, Point cloud
Lang
Human, LLM
IS
✓
✗
VLA‑3D [ 63 ]
Lang, Point cloud, Scene graph
Lang
Template
IS
✓
✗
Grounded 3D‑LLM [ 64 ]
Lang, Point cloud
Mask
Template, LLM
IS
✓
✗
ScanEnts3D [ 65 ]
Lang, Point cloud
BBox
Human
IS
✓
✗
EmbodiedScan [ 66 ]
Lang, RGB‑D image
BBox
Template, LLM
IS
✓
✗
TABLE VI: Comparison with Other Vision Datasets. ‘Lang’ denotes language queries, ‘Lang Source’ denotes whether language queries are human-annotated or generated via template or LLM, ‘IS’ denotes indoor scenes, ‘DS’ denotes driving scenes, and ‘MS’ denotes multi-scene.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Task type
Number of samples
ReasonSeg [ 1 ]
Open-domain reasoning segmentation
1,218
ReasonVOS [ 83 ]
Video reasoning segmentation
458
FCoT [ 47 ]
Foreground reasoning and segmentation-prompt generation
1,500
ConverSeg [ 48 ]
Conversational image segmentation
1,687
ReasonSeg-X [ 84 ]
Extended reasoning segmentation
1,169
OK-VOS [ 85 ]
Outside-knowledge video object segmentation
1,000
Appendix
TABLE VII: Scale comparison with representative reasoning and instruction segmentation datasets.
Fig. 8: Word distribution of instructions in the DSRS dataset. (a) and (c) illustrate word clouds of nouns and verbs appearing in the instructions of the DSRS dataset, respectively. (b) and (d) show the frequency distributions of representative nouns and verbs, respectively.
Fig. 9: Construction and Annotation Pipeline of Our DSRS Dataset.
Fig. 10: Mask scale distribution across 13 fine-grained categories of our proposed DSRS dataset. White circles denote the median mask scale for each category. The lower extreme and the upper extreme represent the 5th and 95th percentiles, respectively. The horizontal axis is truncated between 30% and 60% for clarity.
Model
Multimodal Encoder
LLM
Segmentation Head
Vision Encoder
Text Encoder
Projector
LISA [ 1 ]
CLIP ViT-L/14, SAM ViT-H
LLaMA Tokenizer
MLP
Vicuna 7B
SAM decoder
GSVA [ 6 ]
CLIP ViT-L/14, SAM ViT-H
LLaMA Tokenizer
MLP
Vicuna 7B
SAM decoder
PixelLM [ 45 ]
CLIP ViT-L/14
LLaMA Tokenizer
MLP
Vicuna 7B
PixelLM decoder
M 2 SA [ 22 ]
CLIP ViT-L/14, SAM ViT-H
LLaMA Tokenizer
MLP
Vicuna 7B
SAM decoder
SegAgent [ 46 ]
Qwen-VL, SClick ViT-L
Qwen-VL Tokenizer
Qwen-VL adapter, SClick prompt embedding
Qwen-VL 7B
SClick decoder
Appendix
TABLE VIII: Architectural comparison of representative language-guided segmentation models.
Fig. 11: Unsatisfactory results on the DSRS dataset: (a) areas with insufficient illumination; (b) blurred scene under rainy conditions; (c) areas with indistinct boundaries; (d) additional areas of erroneous segmentation; (e) contextual confusions. Challenging areas are highlighted with white dashed boxes.
Depth input
Perturbation
gIoU
None (RGB-only)
–
53.0
Depth Anything V2-Small
None
54.2
Depth Anything V2-Base
None
54.9
Depth Anything V2-Large
None
55.4
Depth Anything V2-Large
Gaussian ( σ=0.2 )
53.5
Depth Anything V2-Large
Gaussian ( σ=0.5 )
48.8
Appendix
TABLE IX: Influence of depth sources and Gaussian perturbations on the DSRS dataset. Gaussian noise is applied to relative-depth maps normalized to [0,1] .
Fig. 12: Visualizations of noisy depth maps.
Fig. 13: Qualitative results of depth quality analysis under low illumination.
Fig. 14: Qualitative results of depth quality analysis under strong illumination.
Method
Zero-shot (zs)
Fine-tuned (ft)
Val
Test
Val
Test
gIoU
cIoU
gIoU
cIoU
gIoU
cIoU
gIoU
cIoU
LISA-LLaVAv1 [ 1 ]
22.7
20.3
30.5
28.3
55.7
52.5
56.8
55.1
GSVA-LLaVAv1 [ 6 ]
26.1
19.4
31.1
26.3
59.7
60.8
59.9
61.0
DSRS-Net -LLaVAv1
27.3
22.7
34.2
29.1
61.6
64.5
61.0
65.0
Appendix
TABLE X: Same-backbone comparison on the DSRS dataset. DSRS-Net -LLaVAv1 adopts the LISA backbone augmented with our GPE and DHFI components.
Fig. 15: Qualitative results of depth quality analysis under visual blur.
Metric
DSRS-Net - LLaVA1.5
DSRS-Net - InternVL2.5
Depth Generation
Input resolution
336×336
896×896
518×518
Total parameters (B)
7.12
8.14
0.34
Trainable parameters (M)
318.46
82.43
N/A
FLOPs (T/image)
17.31
36.06
1.30
Latency (s/image)
2.07
3.81
0.12
Throughput (images/s)
0.48
0.26
8.65
Appendix
TABLE XI: Computational efficiency of DSRS-Net and the offline depth generation pipeline. Both DSRS-Net variants are evaluated on an A800 GPU with batch size 1. Relative depth maps are generated in advance with Depth Anything V2-Large on an RTX 3090 GPU.
Fig. 16: Qualitative results of depth quality analysis under background clutter.
Fig. 17: Qualitative results of depth quality analysis for targets with small spatial extent.
Method
MLLM
Val
Test
Overall
Short Query
Long Query
Overall
gIoU
cIoU
gIoU
cIoU
gIoU
cIoU
gIoU
cIoU
OVSeg [ 54 ]
-
28.5
18.6
18.0
15.5
28.7
22.5
26.1
20.8
ReLA [ 41 ]
-
22.4
19.9
17.6
15.0
22.6
23.8
21.3
22.0
X-Decoder [ 42 ]
-
22.6
17.9
20.4
11.6
22.2
17.5
21.7
16.3
SEEM [ 43 ]
-
25.5
21.2
20.1
11.5
25.6
20.8
24.3
18.7
Appendix
TABLE XII: Supplementary comparison on the ReasonSeg benchmark under different query lengths.
Methods
Spatial Relations
Potential Risks
Challenging Conditions
Traffic Rules
Overall
Obstacle
Road Defect
Traffic Accident
Complex Terrain
Light Condition
Weather Condition
Bicycle Lane
Bus Lane
Crosswalk
Traffic Lane
Traffic Light
Traffic Sign
PixelLM [ 45 ]
53.0
27.9
10.4
65.4
55.5
53.0
35.2
55.3
50.3
47.4
64.0
19.2
25.4
40.1
M 2 SA [ 22 ]
82.5
46.7
52.1
81.0
62.1
70.7
64.1
59.6
42.9
53.6
66.5
43.9
42.8
57.5
SegAgent [ 46 ]
80.5
17.4
0.4
80.9
47.1
58.7
33.7
22.8
6.4
20.4
32.9
12.6
16.7
30.3
ConverSeg-Net [ 48 ]
83.1
45.2
2.0
86.5
74.0
66.1
62.7
25.0
21.2
32.1
49.9
48.4
46.3
47.9
DSRS-Net (Ours)
85.0
58.8
41.5
86.5
77.8
77.9
66.5
80.9
70.4
72.6
74.5
53.6
57.3
71.2
Appendix
TABLE XIII: Per-category results on the DSRS dataset.
RGB Branch
Depth CLIP Branch
GPE
DHFI
gIoU
cIoU
Run 1
Run 2
Run 3
Mean ±Std
Run 1
Run 2
Run 3
Mean ±Std
✓
52.82
53.18
53.00
53.00_{\color[rgb]{0.5,0,0.5}\pm 0.18}
50.66
50.18
50.42
50.42_{\color[rgb]{0.5,0,0.5}\pm 0.24}
✓
✓
55.04
55.46
55.25
55.25_{\color[rgb]{0.5,0,0.5}\pm 0.21}
52.12
51.95
52.29
52.12_{\color[rgb]{0.5,0,0.5}\pm 0.17}
✓
✓
37.66
37.12
37.39
37.39_{\color[rgb]{0.5,0,0.5}\pm 0.27}
36.48
36.67
36.86
36.67_{\color[rgb]{0.5,0,0.5}\pm 0.19}
✓
✓
53.42
53.28
53.14
53.28_{\color[rgb]{0.5,0,0.5}\pm 0.14}
54.88
55.32
55.10
55.10_{\color[rgb]{0.5,0,0.5}\pm 0.22}
✓
✓
✓
55.24
55.40
55.56
\textbf{55.40}_{\color[rgb]{0.5,0,0.5}\pm 0.16}
57.10
56.84
56.97
\textbf{56.97}_{\color[rgb]{0.5,0,0.5}\pm 0.13}
Appendix
TABLE XIV: Component-wise ablation results over three independent runs on the DSRS dataset. ± Std denotes the sample standard deviation across the three runs.
Contextual Features
Geometric Features
Fusion Strategies
gIoU
cIoU
✓
Self-Attention
44.56
44.08
✓
✓
Element-wise Summation
50.47
51.12
✓
✓
Channel-wise Concatenation
51.09
51.86
✓
✓
Cross-Attention
52.78
54.05
✓
✓
DA2
55.40
56.97
Appendix
TABLE XV: Ablation studies on the DHFI strategy.
Fig. 18: Qualitative comparisons between DSRS-Net and existing language-driven segmentation networks on the DSRS dataset, where ground-truth areas are highlighted with white dashed boxes and wrong areas are highlighted with red dashed circles.
Fig. 19: Qualitative comparisons between DSRS-Net and existing language-driven segmentation networks on the DSRS dataset.
Comprehensive situational awareness is essential for autonomous vehicles operating in safety-critical environments, as it enables the identification and mitigation of potential risks. Although recent Multimodal Large Language Models (MLLMs) have shown promise on general vision-language tasks, our findings indicate that zero-shot MLLMs still underperform compared to domain-specific methods in fine-grained, spatially grounded risk assessment. To address this gap, we propose DriveSafe, a framework for risk-aware scene understanding that leverages structured natural language descriptions. Specifically, our method first generates spatially grounded captions enriched with multimodal context, including motion, spatial, and depth cues. These captions are then used for downstream risk assessment, explicitly identifying hazardous objects, their locations, and the unsafe behaviors they imply, followed by actionable safety suggestions. To further improve performance, we employ caption-risk pairings to fine-tune a lightweight adapter module, efficiently injecting domain-specific knowledge into the base LLM. By conditioning risk assessment on explicit language-based scene representations, DriveSafe achieves significant gains over both zero-shot MLLMs and prior domain-specific baselines. Exhaustive experiments on the DRAMA benchmark demonstrate state-of-the-art performance, while ablation studies validate the effectiveness of our key design choices. Project page: https://cvit.iiit.ac.in/ research/projects/cvit-projects/drivesafe
Reasoning segmentation aims to predict pixel-wise masks for targets given complex language queries. Existing approaches leverage Multimodal Large Language Models (MLLMs) for vision-language reasoning and generate intermediate target cues (e.g., points or boxes) to guide a segmentation model. However, compressing rich reasoning into sparse cues often introduces ambiguity and noise, preventing these cues from accurately preserving the reasoning intent. While multiple complementary cues can enrich target information, existing methods typically feed them jointly into a single segmentation process, allowing ambiguous or erroneous cues to affect the entire prediction. Therefore, we propose DGSeg, a reasoning segmentation framework that learns to fuse predictions guided by semantic and spatial cues. Specifically, the MLLM jointly reasons about both target identity and spatial location, producing complementary semantic and spatial cues that are fed into separate segmentation branches. Their predictions are adaptively integrated by a lightweight dynamic gating module trained with relative branch-quality supervision to suppress noisy or conflicting regions. Extensive experiments demonstrate that DGSeg consistently outperforms strong baselines on multiple benchmarks and achieves 69.6% and 67.3% gIoU on the challenging ReasonSeg validation and test splits. Code is available at https://github.com/RZZeng/DGSeg.
Ruizhe Zeng, Siyu Cao, Lu Zhang +1
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Nanjing Artificial Intelligence Research of IA, Nanjing, 211100, China
Multimodal Large Language Models (MLLMs) are rapidly becoming the intelligence brain of end-to-end autonomous driving systems. A key challenge is to assess whether MLLMs can truly understand and follow complex real-world traffic rules. However, existing benchmarks mainly focus on single-rule scenarios like traffic sign recognition, neglecting the complexity of multi-rule concurrency and conflicts in real driving. Consequently, models perform well on simple tasks but often fail or violate rules in real world complex situations. To bridge this gap, we propose DriveCombo, a text and vision-based benchmark for compositional traffic rule reasoning. Inspired by human drivers' cognitive development, we propose a systematic Five-Level Cognitive Ladder that evaluates reasoning from single-rule understanding to multi-rule integration and conflict resolution, enabling quantitative assessment across cognitive stages. We further propose a Rule2Scene Agent that maps language-based traffic rules to dynamic driving scenes through rule crafting and scene generation, enabling scene-level traffic rule visual reasoning. Evaluations of 14 mainstream MLLMs reveal performance drops as task complexity grows, particularly during rule conflicts. After splitting the dataset and fine-tuning on the training set, we further observe substantial improvements in both traffic rule reasoning and downstream planning capabilities. These results highlight the effectiveness of DriveCombo in advancing compliant and intelligent autonomous driving systems.
Enhui Ma, Jiahuan Zhang, Guantian Zheng +10
Autolab, Westlake University · Li Auto Inc · Tsinghua University +1