Safety4DM: A Vision-Language Framework for Risk-Aware Dynamic Maps in Vehicle-Road-Cloud Collaborative Driving
Organizations: College of Artificial Intelligence, Tianjin University of Science and Technology, Tianjin, China · School of Resource and Environmental Sciences, Wuhan University, Wuhan, China
Abstract
Dynamic map (DM) is a core information infrastructure for vehicle-road-cloud (VRC) cooperative autonomous driving. It fuses perception data from multiple sources to overcome the inherent line-of-sight and field-of-view limits of single-vehicle perception. However, DM still falls short on driving safety in two ways. First, DM data specifications already define road conditions semantically, but no pipeline yet turns real-world road observations into fine-grained risk entities in the DM. Second, existing DM systems lack a natural language interface for driving risks. To close these gaps, we propose Safety4DM, a safety-oriented language system for VRC dynamic maps. We first build RoadRisk, a real-world road image dataset that covers four types of road conditions and four risk levels. On this dataset, we fine-tune a vision-language model (VLM) for road safety risk perception. The model outputs structured risk representations that the DM can ingest directly. We then introduce a Safety-aware Step-wise Prompting (SafeStep) mechanism as the natural language query interface of the DM. SafeStep answers queries accurately and also generates actionable driving advice. To build and evaluate this interface, we develop SafeVRC-Sim, a VRC cooperative simulation framework, and SafeDM-QA, a safety question-answering benchmark. Experiments show that even the compact Qwen3-VL-4B reaches 89.83% accuracy on risk perception after fine-tuning. SafeStep also plugs easily into multiple LLM families. With medium and large models (20B and above) from five families, it achieves over 93% natural language query accuracy in every case, which shows strong generalization.
Figures & tables
| Model | Publication | Setting | Task coverage | Scene elements | Output | ||||
| agent setting | data source | perce- ption | predic- tion | plan- ning | NL query | ||||
| DriveGPT4 [ 14 ] | IEEE RAL 2024 | Single | Real | ✓ | ✓ | ✓ | OI | Text + Control | |
| DriveVLM [ 15 ] | CoRL 2024 | Single | Real | ✓ | ✓ | ✓ | OI + EI | Text + Trajectory | |
| DriveLM [ 16 ] | ECCV 2024 | Single | Sim + Real | ✓ | ✓ | ✓ | ✓ | OI | Text + Trajectory |
| Talk2BEV [ 4 ] | ICRA 2024 | Single | Real | ✓ | ✓ | OI | Text | ||
| PromptTrack [ 17 ] | AAAI 2025 | Single | Real | ✓ | OI | 3D boxes | |||
| Condition | Level | Decision criterion |
| Normal | 0 | no floodwater, no snow cover, no obstacle |
| Floodwater | 1 | Mild: deepest point 15 cm |
| 2 | Moderate: deepest point 15–30 cm; | |
| 3 | Severe: deepest point 30 cm; | |
| Snow | 1 | Drivable lane 50% snow-covered; |
| 2 | Lane 50–100% covered; mean depth 10 cm. |
| Dimension | Category | Number |
| Scene | Urban streets | 2423 |
| Highways | 1373 | |
| Alleys | 732 | |
| Villages | 505 | |
| Tunnels | 502 | |
| Towns | 465 |
| OI | |
| Field | Description |
| id | identifier |
| t | time stamp |
| x | x coordinate |
| y | y coordinate |
| v | velocity |
| Task | Description | Task | Description |
| (1) ConditionType | road risk category | (10) Size | vehicle size |
| (2) RiskLevel | road risk level | (11) Status | signal status |
| (3) EI Count | number of road risks | (12) Distance | distance to ego |
| (4) EI Exist | presence of road risk | (13) OI Count | number of vehicles |
| (5) Velocity | vehicle velocity | (14) OI Exist | presence of vehicles |
| (6) Acceleration | vehicle acceleration | (15) TTC | time to collision |
| Query type | Task | Domain | Ego- anchored | Road- anchored | Total |
| attribute query | velocity | OI | 98 | 14 | 112 |
| acceleration | OI | 143 | 16 | 159 | |
| heading | OI | 132 | 18 | 150 | |
| color | OI | 94 | 12 | 106 | |
| classification | OI | 72 | 19 | 91 | |
| size | OI | 79 | 9 | 88 |
| Query type | Task | Domain | Ego- anchored | Road- anchored | Total |
| attribute query | velocity | OI | 280 | 24 | 304 |
| acceleration | OI | 300 | 19 | 319 | |
| heading | OI | 292 | 19 | 311 | |
| color | OI | 278 | 18 | 296 | |
| classification | OI | 275 | 22 | 297 | |
| size | OI | 276 | 24 | 300 |
| Model | Qwen3-VL-2B | Qwen3-VL-4B | Qwen3-VL-8B | ||||
| Road condition | |||||||
| Normal | 38.33 | 38.33 | 91.67 | 91.67 | 93.00 | 93.00 | +3 |
| Floodwater | 82.33 | 29.67 | 96.33 | 58.67 | 97.67 | 69.67 | 0 |
| Snow | 86.33 | 35.00 | 92.67 | 64.67 | 90.33 | 56.00 | -1 |
| Obstacle | 48.67 | 27.00 | 61.67 | 49.00 | 70.67 | 62.67 | -2 |
| Average | 63.92 | 32.50 | 85.58 | 66.00 | 87.92 | 70.33 | – |
| Model | Qwen3-VL-2B | Qwen3-VL-4B | Qwen3-VL-8B | ||||
| Road condition | |||||||
| Normal | 97.67 | 97.67 | 96.00 | 96.00 | 97.33 | 97.33 | +3 |
| Floodwater | 99.33 | 77.33 | 99.00 | 78.67 | 99.33 | 78.00 | -3 |
| Snow | 99.33 | 88.33 | 99.33 | 93.00 | 97.67 | 91.33 | 0 |
| Obstacle | 100 | 87.67 | 100 | 91.67 | 100 | 91.67 | 0 |
| Average | 99.08 | 87.75 | 98.58 | 89.83 | 98.58 | 89.58 | – |
| Safety4DM using: | DS-r1:32B | Qwen3:30B | GPT-oss:20B | Magistral:24B | Gemma3:27B | Llama3.1:8B | |||||||
| Query task | |||||||||||||
| (1) ConditionType | 100 | 100 | 100 | 100 | 100 | 100 | 99.37 | 97.33 | 100 | 98.90 | 44.74 | 44.74 | +3 |
| (2) RiskLevel | 100 | 100 | 100 | 99.25 | 100 | 100 | 98.25 | 98.25 | 100 | 99.00 | 44.25 | 44.25 | +2 |
| (3) EI Count | 99.29 | 97.98 | 99.19 | 97.78 | 100 | 98.28 | 100 | 98.28 | 99.09 | 97.07 | 85.37 | 72.25 | +1 |
| (4) EI Exist | 100 | 100 | 96.98 | 97.89 | 100 | 100 | 94.86 | 94.86 | 99.09 | 99.09 | 34.44 | 26.28 | +2 |
| (5) Velocity | 100 | 96.71 | 100 | 97.37 | 98.68 | 97.04 | 100 | 90.13 | 100 | 95.72 | 100 | 91.12 | +1 |
| Safety4DM using | Qwen3:0.6B | Qwen3:1.7B | Qwen3:4B | Qwen3:8B | Qwen3:14B | Qwen3:30B | |||||||
| Query task | |||||||||||||
| (1) ConditionType | 69.86 | 61.70 | 70.80 | 68.45 | 100 | 99.84 | 100 | 99.69 | 100 | 99.84 | 100 | 100 | +3 |
| (2) RiskLevel | 74.75 | 66.25 | 76.50 | 70.75 | 100 | 95.50 | 96.50 | 92.00 | 100 | 100 | 100 | 99.25 | +1 |
| (3) EI Count | 85.07 | 67.51 | 88.90 | 87.08 | 100 | 98.28 | 100 | 98.28 | 100 | 98.28 | 99.19 | 97.78 | 0 |
| (4) EI Exist | 53.17 | 40.03 | 26.89 | 26.59 | 96.22 | 96.07 | 96.98 | 95.17 | 100 | 100 | 96.98 | 97.89 | 0 |
| (5) Velocity | 99.01 | 43.09 | 98.68 | 79.28 | 100 | 97.04 | 100 | 92.76 | 100 | 96.05 | 100 | 97.37 | 0 |
| Safety4DM using Query task | Gemma3 family | DeepSeek-r1 family | |||||||||||||||||
| Gemma3:1B | Gemma3:4B | Gemma3:12B | Gemma3:27B | DS-r1:1.5B | DS-r1:7B | DS-r1:14B | DS-r1:32B | ||||||||||||
| (1) ConditionType | 16.64 | 4.08 | 100 | 100 | 100 | 100 | 100 | 98.90 | +1 | 30.30 | 18.52 | 20.41 | 16.01 | 100 | 100 | 100 | 100 | +1 | |
| (2) RiskLevel | 60.75 | 16.50 | 98.25 | 98.25 | 100 | 100 | 100 | 99.00 | +1 | 29.25 | 24.25 | 39.25 | 35.75 | 100 | 100 | 100 | 100 | +2 | |
| (3) EI Count | 0.00 | 9.18 | 53.68 | 50.55 | 65.09 | 64.18 | 99.09 | 97.07 | -2 | 6.46 | 4.34 | 70.13 | 59.23 | 100 | 97.07 | 99.29 | 97.98 | -1 | |
| (4) EI Exist | 0.91 | 13.44 | 91.69 | 91.54 | 98.19 | 98.19 | 99.09 | 99.09 | +1 | 12.54 | 6.50 | 34.89 | 20.39 | 99.70 | 99.70 | 100 | 100 | +1 | |
| Query task | OSP | w/o ToolBox | Safety4DM |
| (1) ConditionType | 33.1 | 34.9 | 100.0 |
| (2) RiskLevel | 24.3 | 34.0 | 100.0 |
| (3) EI Count | 90.5 | 90.8 | 98.3 |
| (4) EI Exist | 58.0 | 56.3 | 100.0 |
| (5) Velocity | 4.6 | 3.3 | 97.0 |
| (6) Acceleration | 0.3 | 1.6 | 88.1 |
| ID | Dimension | Questionnaire item | Mean±SD | Agreement(%) |
| Q1 | Information need | I need more information about road risks than my current ADAS provides. | 4.31 ± 0.80 | 80.0 |
| Q2 | Perceived usefulness | This function would help me understand risks on my current road. | 4.06 ± 0.84 | 80.0 |
| Q3 | Answer clarity | The answers shown in the examples are easy to understand. | 4.09 ± 0.95 | 77.1 |
| Q4 | Perceived reliability | I consider the answers shown in the examples to be reliable. | 4.03 ± 0.71 | 82.9 |
| Q5 | Intention to use | I would use this function if it were available in my vehicle. | 4.29 ± 0.79 | 80.0 |
| Zhengshu Zhou received the B.E. degree in software engineering from Tianjin University, Tianjin China, in 2013. He received the Master and Ph.D. degree in informatics from Nagoya University, Nagoya Japan, in 2020 and 2022, respectively. He is currently an Associate Professor with the College of Artificial Intelligence, Tianjin University of Science & Technology (TUST). Previously, he was a system engineer with the Japanese automotive industry. And he was a researcher at Hitachi Yokohama Research Laboratory, Yokohama, Japan. His research interests mainly include software engineering and autonomous driving. |
| Shaoyu Wang received the B.E. degree in computer science and technology from North China Institute of Aerospace Engineering, Langfang, China, in 2024. He is currently pursuing the M.S. degree with Tianjin University of Science and Technology, Tianjin, China. His current research interests include deep learning and autonomous driving. |
| Lu Tao is currently a tenure-track associate professor at Wuhan University. He received his B.S. and M.E. degrees in Geographic Information Systems and Navigation, Guidance, and Control from Wuhan University in 2014 and 2017, respectively. He obtained his Ph.D. degree in Informatics from Nagoya University in 2023, and was then a researcher at the Institute of Innovation for Future Society from 2023 to 2024. His research interests include vehicle motion prediction, sensor fusion, high-definition maps, dynamic maps, edge and cloud computing, and connected autonomous vehicles. |
| Xinhui Hu received the B.E. degree in computer science and technology from Hebei GEO University, Shijiazhuang, China, in 2025. She is currently pursuing the M.S. degree with Tianjin University of Science and Technology, Tianjin, China. Her research interests include deep learning and autonomous driving. |
| Jiaqi Gu is currently pursuing the M.E. degree in software engineering at the College of Artificial Intelligence, Tianjin University of Science and Technology, Tianjin, China, where he received his B.E. degree in software engineering. His research interests include software engineering, deep learning, and autonomous driving. |
| Tingting Zhao obtained her Ph.D. degree from Tokyo Institute of Technology, Japan. She is the Professor of Tianjin University of Science & Technology, China. She is the visiting researcher at RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan. She has published more than 50 papers in leading international journals or conferences of machine learning domain, including Neural Computation, Neural Networks, NeurIPS, ECML/PKDD, IJCAI, ACML and PAKDD. She is in charge of several major projects as the project leader, such as of National Natural Science Foundation of China and the project of Scientific Research Foundation for the returned overseas Chinese scholars. Her research interest mainly include machine learning, robot control, and data mining. |
Autonomous Driving
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.
See all 570 papers