OD-RASE: Ontology-Driven Risk Assessment and Safety Enhancement for Autonomous Driving

Kota Shimomura; Masaki Nambata; Atsuya Ishikawa; Ryota Mimura; Takayuki Kawabuchi; Takayoshi Yamashita; Koki Inoue

cs.CVMar 6, 2026

OD-RASE: Ontology-Driven Risk Assessment and Safety Enhancement for Autonomous Driving

Authors: Kota Shimomura, Masaki Nambata, Atsuya Ishikawa, Ryota Mimura, Takayuki Kawabuchi, Takayoshi Yamashita, Koki Inoue

Organizations: Chubu University · Elith Inc. · Honda R&D Co., Ltd.

Abstract

Although autonomous driving systems demonstrate high perception performance, they still face limitations when handling rare situations or complex road structures. Such road infrastructures are designed for human drivers, safety improvements are typically introduced only after accidents occur. This reactive approach poses a significant challenge for autonomous systems, which require proactive risk mitigation. To address this issue, we propose OD-RASE, a framework for enhancing the safety of autonomous driving systems by detecting road structures that cause traffic accidents and connecting these findings to infrastructure development. First, we formalize an ontology based on specialized domain knowledge of road traffic systems. In parallel, we generate infrastructure improvement proposals using a large-scale visual language model (LVLM) and use ontology-driven data filtering to enhance their reliability. This process automatically annotates improvement proposals on pre-accident road images, leading to the construction of a new dataset. Furthermore, we introduce the Baseline approach (OD-RASE model), which leverages LVLM and a diffusion model to produce both infrastructure improvement proposals and generated images of the improved road environment. Our experiments demonstrate that ontology-driven data filtering enables highly accurate prediction of accident-causing road structures and the corresponding improvement plans. We believe that this work contributes to the overall safety of traffic environments and marks an important step toward the broader adoption of autonomous driving systems.

Figures & tables

Figure 1: Comparison of various methods for infrastructure improvement design. (a) is based on expert knowledge, while (b) represents our proposed approach. Our method not only outputs infrastructure improvement plans for road structures that cause traffic accidents but also generates visual representations of roads after improvement.

Figure 2: Breakdown of infrastructure improvement process in field of road transportation systems, and overview of how OD-RASE Dataset is constructed on basis of it. Final set of 11 types of road structures causing traffic accidents (top) and 10 types of countermeasures (bottom).

Figure 3: Schematic diagram of the conventional infrastructure improvement process by expert.

Figure 4: Proposed ontology-driven dataset construction method. Our method allows for fully automatic generation using VLMs and enhances dataset quality and reliability through filtering with reference graphs. Additionally, such filtering further refines dataset’s overall trustworthiness.

Figure 5: OD-RASE model architecture. Each modality scene images and textual descriptions of traffic risks is encoded, and grounding block captures semantic relationships between image and text.

Vision Encoder	Text Encoder	Mapillary				BDD100K
Vision Encoder	Text Encoder	Recall	Precision	F1	Acc	Recall	Precision	F1	Acc
ResNet-50 [ 16 ]	RoBERTa-Base [ 26 ]	58.21	73.54	64.98	37.12	74.78	79.75	77.18	45.94
	Flan-T5-xl [ 9 ]	64.96	14.42	23.60	0.00	72.85	17.06	27.64	0.00
	Long-CLIP [ 54 ]	57.98	75.54	65.60	37.98	72.69	80.96	76.60	45.68
ViT-B [ 12 ]	RoBERTa-Base [ 26 ]	59.73	78.35	67.79	40.71	74.94	81.62	78.14	47.98
	Flan-T5-xl [ 9 ]	57.26	9.81	16.74	0.00	47.70	8.74	14.78	0.00

Table 1: Quantitative evaluation results for predicting infrastructure improvement proposals for road structures. The best second, third best performances are shown in First , Second , Third , respectively.

Modal	Precision	Recall	F1	Acc
Image	57.42	72.37	64.03	34.50
Text	60.02	79.76	68.50	40.63
Image & Text	64.54	77.09	70.26	42.14

Table 2: Ablation study on which modality (image, text, or both) is most effective for predicting infrastructure improvement proposals.

Figure 6: Qualitative examples of infrastructure improvement proposals for road structures. Our OD-RASE model successfully predicted relevant infrastructure improvements.

Data filtering	Precision	Recall	F1	Acc
	33.59	64.85	44.26	0.00
✓	64.54	77.09	70.26	42.14

Table 3: Ablation study on effectiveness of ontology-driven data filtering. Results indicate that filtering improved overall model performance.

Ours Baseline
Method		Recall		Precision		F1-Score		Accuracy
Vision Encoder	Text Encoder	val	test	val	test	val	test	val	test
ResNet-50 [ 16 ]	RoBERTa-Base [ 26 ]	64.38	65.49	69.08	68.91	66.65	67.16	34.97	34.52
ViT-B [ 12 ]		62.49	63.65	74.34	73.18	67.90	68.08	38.02	37.12
CLIP [ 33 ]		60.64	63.09	73.76	74.26	66.56	68.22	38.02	37.28
Long-CLIP [ 54 ]		62.68	62.34	74.91	75.57	68.25	68.32	39.29	38.96

Table 4: Quantitative evaluation of our OD-RASE model versus generalist models on infrastructure improvement proposal tasks in zero-shot setting. Model was trained on BDD100K and evaluated on Mapillary.

Ours Baseline
Method		Recall		Precision		F1-Score		Accuracy
Vision Encoder	Text Encoder	val	test	val	test	val	test	val	test
ResNet-50 [ 16 ]	RoBERTa-Base [ 26 ]	60.25	59.44	79.45	79.64	68.53	68.08	34.57	33.20
ViT-B [ 12 ]		63.00	62.21	85.46	86.12	72.53	72.24	41.00	40.62
CLIP [ 33 ]		67.66	66.86	84.51	84.87	75.15	74.80	43.63	43.17
Long-CLIP [ 54 ]		70.22	69.17	83.68	84.31	76.36	76.00	45.79	44.76

Table 5: Quantitative evaluation of our OD-RASE model versus generalist models on infrastructure improvement proposal tasks in zero-shot setting. Model was trained on Mapillary and evaluated on BDD100K.

Figure 7: Examples of layout control using Instruct Pix2Pix [ 5 ] , based on OD-RASE-predicted infrastructure improvement plans.

Model	FID $\downarrow$	Prompt Faithfulness
Model	FID $\downarrow$	None	Partial	Full
Instruct Pix2Pix	8.5	23.02	22.75	54.23

Table 6: Quantitative evaluation results of the generated images.

Figure 8: Merging and exclusion of elements in accident-causing road structures.

Figure 9: Merging and exclusion of elements in infrastructure improvement proposals.

Figure 10: Exclusion of accident-causing road structures related to corner cases.

Figure 11: Exclusion of corner-case elements in infrastructure improvement proposals.

Figure 12: Changes in data distribution before and after filtering for Mapillary Vistas. Blue: before filtering, orange: after filtering.

Figure 13: Changes in data distribution before and after filtering for BDD100K. Blue: before filtering, orange: after filtering.

Vision Encoder	Text Encoder	Mapillary				BDD100K
Vision Encoder	Text Encoder	Recall	Precision	F1	Acc	Recall	Precision	F1	Acc
ResNet-50 [ 16 ]	RoBERTa-Base [ 26 ]	74.85	82.06	79.19	64.25	86.54	88.55	87.53	73.50
	Flan-T5-xl [ 9 ]	72.07	12.50	21.30	0.00	73.94	13.54	22.89	0.00
	Long-CLIP [ 54 ]	77.17	76.96	77.06	58.92	83.35	88.74	85.96	71.04
ViT-B [ 12 ]	RoBERTa-Base [ 26 ]	71.26	78.76	68.05	63.50	88.80	86.44	87.60	72.69
	Flan-T5-xl [ 9 ]	29.01	3.76	6.65	0.00	24.25	3.33	5.85	0.00

Table 7: Quantitative evaluation results for predicting accident-causing road structures. The best, second and third best performances are shown in First , Second , Third , respectively.

Modal	Precision	Recall	F1	Acc
CLIP	54.98	72.54	62.55	32.60
Long-CLIP	66.34	57.94	61.86	28.22
Ours	76.44	83.89	79.99	65.09

Table 8: Ablation study on grounding block is most effective for predicting infrastructure improvement proposals.

Data filtering	Precision	Recall	F1	Acc
	38.25	84.33	52.63	1.01
✓	76.44	83.89	79.99	64.09

Table 9: Ablation study on effectiveness of ontology-driven data filtering. Results indicate that filtering improved overall model performance.

Ours Baseline
Method		Recall		Precision		F1-Score		Accuracy
Vision Encoder	Text Encoder	val	test	val	test	val	test	val	test
ResNet-50 [ 16 ]	Long-CLIP [ 54 ]	71.68	74.76	77.28	79.76	74.38	77.18	58.48	61.38
ViT-B [ 12 ]		76.84	77.77	79.68	80.58	78.24	79.15	62.43	63.83
CLIP [ 33 ]		74.75	75.63	80.81	82.22	77.66	78.79	61.68	63.25
Long-CLIP [ 54 ]		75.49	77.10	80.22	81.54	77.78	79.26	62.50	64.11

Table 10: Zero-shot prediction of accident-causing road structures. Models are trained on BDD100K and evaluated on Mapillary.

Ours Baseline
Method		Recall		Precision		F1-Score		Accuracy
Vision Encoder	Text Encoder	val	test	val	test	val	test	val	test
ResNet-50 [ 16 ]	Long-CLIP [ 54 ]	81.10	81.50	88.37	88.16	84.58	84.70	68.35	68.24
ViT-B [ 12 ]		75.12	76.64	82.14	83.74	78.47	80.03	63.02	65.00
CLIP [ 33 ]		81.70	82.04	90.86	90.95	86.04	86.26	71.15	71.66
Long-CLIP [ 54 ]		83.14	83.72	90.07	89.96	86.46	86.73	71.45	72.02

Table 11: Zero-shot prediction of accident-causing road structures. Model was trained on Mapillary and evaluated on BDD100K.

Figure 14: Details of the G2CoT prompt used when constructing the OD-RASE Dataset.

Ontology-Based Reasoning

See all 15 papers

DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios
May 16, 2026Sainithin Artham, Shankar Gangisetty, Avijit Dasgupta +1VLMs for Autonomous Driving Autonomous Driving Safety Evaluation
IIIT-Hyderabad, India
Data-Driven Risk Fields for Safer End-to-End Autonomous Driving
Sep 9, 2026Yuanxin Tian, Zhiyuan Liu, Jinhao Li +9End-to-End Autonomous Driving Safe Motion Planning
Tsinghua University · Yinwang · KAIST
A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous driving
Jul 8, 2026Heye Huang, Jingguang Li, Zhiyuan Zhou +4Autonomous Driving Datasets Autonomous Driving Safety Evaluation
Cho Chun Shik Graduate School of Mobility, Korea Advanced Institute of Science and Technology, Daejeon 34051, South Korea · College of Computer Science and Artificial Intelligence, Fudan University, Shanghai 200433, China · Media Lab, Massachusetts Institute of Technology, Cambridge, MA 02139, USA +2

May 16, 2026J/K move · Enter open · S save

DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios

Sainithin Artham, Shankar Gangisetty, Avijit Dasgupta, C. V. Jawahar

IIIT-Hyderabad, India

VLMs for Autonomous Driving Autonomous Driving Safety Evaluation Large Vision-Language Models Autonomous Driving Autonomous Driving Perception

Open paper AskPDF arXiv

Comprehensive situational awareness is essential for autonomous vehicles operating in safety-critical environments, as it enables the identification and mitigation of potential risks. Although recent Multimodal Large Language Models (MLLMs) have shown promise on general vision-language tasks, our findings indicate that zero-shot MLLMs still underperform compared to domain-specific methods in fine-grained, spatially grounded risk assessment. To address this gap, we propose DriveSafe, a framework for risk-aware scene understanding that leverages structured natural language descriptions. Specifically, our method first generates spatially grounded captions enriched with multimodal context, including motion, spatial, and depth cues. These captions are then used for downstream risk assessment, explicitly identifying hazardous objects, their locations, and the unsafe behaviors they imply, followed by actionable safety suggestions. To further improve performance, we employ caption-risk pairings to fine-tune a lightweight adapter module, efficiently injecting domain-specific knowledge into the base LLM. By conditioning risk assessment on explicit language-based scene representations, DriveSafe achieves significant gains over both zero-shot MLLMs and prior domain-specific baselines. Exhaustive experiments on the DRAMA benchmark demonstrate state-of-the-art performance, while ablation studies validate the effectiveness of our key design choices. Project page: https://cvit.iiit.ac.in/ research/projects/cvit-projects/drivesafe

Abstract

Ontology-Based Reasoning

Explore similar work

DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios

Data-Driven Risk Fields for Safer End-to-End Autonomous Driving

A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous driving