cs.CVNov 4, 2026

DSRS: Teaching MLLMs to Perceive, Reason, and Segment in Complex Driving Scenes

Authors: Sicen Guo, Zhengfei Song, Kuanhsun Chen, Xu Huang, Ping Zhong, Wei Ye, Shaoyi Du, Rui Fan

Organizations: College of Electronic & Information Engineering, Tongji University, Shanghai 201804, China · Department of Computer Science and Technology, Central South University. · College of Electronics and Information Engineering, Tongji University, Shanghai 201804, China. He is also with the Shanghai Innovation Institute, Shanghai 200231, China · National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, the National Engineering Research Center for Visual Information and Applications, and the Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · 

Abstract

Existing reasoning segmentation frameworks typically employ multimodal large language models (MLLMs) with broad world knowledge to extract semantics in a human-like manner and generate corresponding segmentation masks. Nevertheless, these frameworks lack the necessary scene-specific awareness. This paper introduces driving scene reasoning segmentation (DSRS), a scene-specific task that requires models to interpret implicit instructions, infer spatial relations, detect potential risks, handle challenging conditions, and comprehend underlying traffic rules, ultimately generating accurate segmentation masks. Meanwhile, we establish a benchmark dataset to facilitate future research on the DSRS task. To pioneer this task, we propose DSRS-Net, a novel architecture with advanced reasoning segmentation capability for automated driving. Specifically, DSRS-Net seamlessly infuses geometric features for deeper reasoning segmentation, without compromising the original multimodal fusion mechanism. This strategy effectively mitigates the modality representation mismatch in MLLMs caused by RGB-only pretraining, while enhancing the integration of geometric, contextual, and linguistic features. Extensive experiments demonstrate the superior performance of DSRS-Net in both driving scene reasoning segmentation and other language-driven segmentation tasks across various public datasets. The source code and dataset will be released upon publication.

Figures & tables

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. DGSeg: Dynamic Gating of Semantic-Spatial Guided Predictions for Reasoning Segmentation

    Jul 6, 2026Ruizhe Zeng, Siyu Cao, Lu Zhang +1Text-Guided Image SegmentationVLM Reasoning

  2. DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving

    Mar 2, 2026Enhui Ma, Jiahuan Zhang, Guantian Zheng +10Autonomous Driving BenchmarksVLMs for Autonomous Driving