OD-RASE: Ontology-Driven Risk Assessment and Safety Enhancement for Autonomous Driving

Kota Shimomura; Masaki Nambata; Atsuya Ishikawa; Ryota Mimura; Takayuki Kawabuchi; Takayoshi Yamashita; Koki Inoue

cs.CVMar 6, 2026

OD-RASE: Ontology-Driven Risk Assessment and Safety Enhancement for Autonomous Driving

Authors: Kota Shimomura, Masaki Nambata, Atsuya Ishikawa, Ryota Mimura, Takayuki Kawabuchi, Takayoshi Yamashita, Koki Inoue

Organizations: Chubu University · Elith Inc. · Honda R&D Co., Ltd.

Abstract

Although autonomous driving systems demonstrate high perception performance, they still face limitations when handling rare situations or complex road structures. Such road infrastructures are designed for human drivers, safety improvements are typically introduced only after accidents occur. This reactive approach poses a significant challenge for autonomous systems, which require proactive risk mitigation. To address this issue, we propose OD-RASE, a framework for enhancing the safety of autonomous driving systems by detecting road structures that cause traffic accidents and connecting these findings to infrastructure development. First, we formalize an ontology based on specialized domain knowledge of road traffic systems. In parallel, we generate infrastructure improvement proposals using a large-scale visual language model (LVLM) and use ontology-driven data filtering to enhance their reliability. This process automatically annotates improvement proposals on pre-accident road images, leading to the construction of a new dataset. Furthermore, we introduce the Baseline approach (OD-RASE model), which leverages LVLM and a diffusion model to produce both infrastructure improvement proposals and generated images of the improved road environment. Our experiments demonstrate that ontology-driven data filtering enables highly accurate prediction of accident-causing road structures and the corresponding improvement plans. We believe that this work contributes to the overall safety of traffic environments and marks an important step toward the broader adoption of autonomous driving systems.

Figures & tables

Figure 1: Comparison of various methods for infrastructure improvement design. (a) is based on expert knowledge, while (b) represents our proposed approach. Our method not only outputs infrastructure improvement plans for road structures that cause traffic accidents but also generates visual representations of roads after improvement.

Figure 2: Breakdown of infrastructure improvement process in field of road transportation systems, and overview of how OD-RASE Dataset is constructed on basis of it. Final set of 11 types of road structures causing traffic accidents (top) and 10 types of countermeasures (bottom).

Figure 3: Schematic diagram of the conventional infrastructure improvement process by expert.

Figure 4: Proposed ontology-driven dataset construction method. Our method allows for fully automatic generation using VLMs and enhances dataset quality and reliability through filtering with reference graphs. Additionally, such filtering further refines dataset’s overall trustworthiness.

Figure 5: OD-RASE model architecture. Each modality scene images and textual descriptions of traffic risks is encoded, and grounding block captures semantic relationships between image and text.

Vision Encoder	Text Encoder	Mapillary				BDD100K
Vision Encoder	Text Encoder	Recall	Precision	F1	Acc	Recall	Precision	F1	Acc
ResNet-50 [ 16 ]	RoBERTa-Base [ 26 ]	58.21	73.54	64.98	37.12	74.78	79.75	77.18	45.94
	Flan-T5-xl [ 9 ]	64.96	14.42	23.60	0.00	72.85	17.06	27.64	0.00
	Long-CLIP [ 54 ]	57.98	75.54	65.60	37.98	72.69	80.96	76.60	45.68
ViT-B [ 12 ]	RoBERTa-Base [ 26 ]	59.73	78.35	67.79	40.71	74.94	81.62	78.14	47.98
	Flan-T5-xl [ 9 ]	57.26	9.81	16.74	0.00	47.70	8.74	14.78	0.00

Table 1: Quantitative evaluation results for predicting infrastructure improvement proposals for road structures. The best second, third best performances are shown in First , Second , Third , respectively.

Modal	Precision	Recall	F1	Acc
Image	57.42	72.37	64.03	34.50
Text	60.02	79.76	68.50	40.63
Image & Text	64.54	77.09	70.26	42.14

Table 2: Ablation study on which modality (image, text, or both) is most effective for predicting infrastructure improvement proposals.

Figure 6: Qualitative examples of infrastructure improvement proposals for road structures. Our OD-RASE model successfully predicted relevant infrastructure improvements.

Data filtering	Precision	Recall	F1	Acc
	33.59	64.85	44.26	0.00
✓	64.54	77.09	70.26	42.14

Table 3: Ablation study on effectiveness of ontology-driven data filtering. Results indicate that filtering improved overall model performance.

Ours Baseline
Method		Recall		Precision		F1-Score		Accuracy
Vision Encoder	Text Encoder	val	test	val	test	val	test	val	test
ResNet-50 [ 16 ]	RoBERTa-Base [ 26 ]	64.38	65.49	69.08	68.91	66.65	67.16	34.97	34.52
ViT-B [ 12 ]		62.49	63.65	74.34	73.18	67.90	68.08	38.02	37.12
CLIP [ 33 ]		60.64	63.09	73.76	74.26	66.56	68.22	38.02	37.28
Long-CLIP [ 54 ]		62.68	62.34	74.91	75.57	68.25	68.32	39.29	38.96

Table 4: Quantitative evaluation of our OD-RASE model versus generalist models on infrastructure improvement proposal tasks in zero-shot setting. Model was trained on BDD100K and evaluated on Mapillary.

Ours Baseline
Method		Recall		Precision		F1-Score		Accuracy
Vision Encoder	Text Encoder	val	test	val	test	val	test	val	test
ResNet-50 [ 16 ]	RoBERTa-Base [ 26 ]	60.25	59.44	79.45	79.64	68.53	68.08	34.57	33.20
ViT-B [ 12 ]		63.00	62.21	85.46	86.12	72.53	72.24	41.00	40.62
CLIP [ 33 ]		67.66	66.86	84.51	84.87	75.15	74.80	43.63	43.17
Long-CLIP [ 54 ]		70.22	69.17	83.68	84.31	76.36	76.00	45.79	44.76

Table 5: Quantitative evaluation of our OD-RASE model versus generalist models on infrastructure improvement proposal tasks in zero-shot setting. Model was trained on Mapillary and evaluated on BDD100K.

Figure 7: Examples of layout control using Instruct Pix2Pix [ 5 ] , based on OD-RASE-predicted infrastructure improvement plans.

Model	FID $\downarrow$	Prompt Faithfulness
Model	FID $\downarrow$	None	Partial	Full
Instruct Pix2Pix	8.5	23.02	22.75	54.23

Table 6: Quantitative evaluation results of the generated images.

Figure 8: Merging and exclusion of elements in accident-causing road structures.

Figure 9: Merging and exclusion of elements in infrastructure improvement proposals.

Figure 10: Exclusion of accident-causing road structures related to corner cases.

Figure 11: Exclusion of corner-case elements in infrastructure improvement proposals.

Figure 12: Changes in data distribution before and after filtering for Mapillary Vistas. Blue: before filtering, orange: after filtering.

Figure 13: Changes in data distribution before and after filtering for BDD100K. Blue: before filtering, orange: after filtering.

Vision Encoder	Text Encoder	Mapillary				BDD100K
Vision Encoder	Text Encoder	Recall	Precision	F1	Acc	Recall	Precision	F1	Acc
ResNet-50 [ 16 ]	RoBERTa-Base [ 26 ]	74.85	82.06	79.19	64.25	86.54	88.55	87.53	73.50
	Flan-T5-xl [ 9 ]	72.07	12.50	21.30	0.00	73.94	13.54	22.89	0.00
	Long-CLIP [ 54 ]	77.17	76.96	77.06	58.92	83.35	88.74	85.96	71.04
ViT-B [ 12 ]	RoBERTa-Base [ 26 ]	71.26	78.76	68.05	63.50	88.80	86.44	87.60	72.69
	Flan-T5-xl [ 9 ]	29.01	3.76	6.65	0.00	24.25	3.33	5.85	0.00

Table 7: Quantitative evaluation results for predicting accident-causing road structures. The best, second and third best performances are shown in First , Second , Third , respectively.

Modal	Precision	Recall	F1	Acc
CLIP	54.98	72.54	62.55	32.60
Long-CLIP	66.34	57.94	61.86	28.22
Ours	76.44	83.89	79.99	65.09

Table 8: Ablation study on grounding block is most effective for predicting infrastructure improvement proposals.

Data filtering	Precision	Recall	F1	Acc
	38.25	84.33	52.63	1.01
✓	76.44	83.89	79.99	64.09

Table 9: Ablation study on effectiveness of ontology-driven data filtering. Results indicate that filtering improved overall model performance.

Ours Baseline
Method		Recall		Precision		F1-Score		Accuracy
Vision Encoder	Text Encoder	val	test	val	test	val	test	val	test
ResNet-50 [ 16 ]	Long-CLIP [ 54 ]	71.68	74.76	77.28	79.76	74.38	77.18	58.48	61.38
ViT-B [ 12 ]		76.84	77.77	79.68	80.58	78.24	79.15	62.43	63.83
CLIP [ 33 ]		74.75	75.63	80.81	82.22	77.66	78.79	61.68	63.25
Long-CLIP [ 54 ]		75.49	77.10	80.22	81.54	77.78	79.26	62.50	64.11

Table 10: Zero-shot prediction of accident-causing road structures. Models are trained on BDD100K and evaluated on Mapillary.

Ours Baseline
Method		Recall		Precision		F1-Score		Accuracy
Vision Encoder	Text Encoder	val	test	val	test	val	test	val	test
ResNet-50 [ 16 ]	Long-CLIP [ 54 ]	81.10	81.50	88.37	88.16	84.58	84.70	68.35	68.24
ViT-B [ 12 ]		75.12	76.64	82.14	83.74	78.47	80.03	63.02	65.00
CLIP [ 33 ]		81.70	82.04	90.86	90.95	86.04	86.26	71.15	71.66
Long-CLIP [ 54 ]		83.14	83.72	90.07	89.96	86.46	86.73	71.45	72.02

Table 11: Zero-shot prediction of accident-causing road structures. Model was trained on Mapillary and evaluated on BDD100K.

Figure 14: Details of the G2CoT prompt used when constructing the OD-RASE Dataset.

Ontology-Based Reasoning

See all 15 papers

May 16, 2026

DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios

Comprehensive situational awareness is essential for autonomous vehicles operating in safety-critical environments, as it enables the identification and mitigation of potential risks. Although recent Multimodal Large Language Models (MLLMs) have shown promise on general vision-language tasks, our findings indicate that zero-shot MLLMs still underperform compared to domain-specific methods in fine-grained, spatially grounded risk assessment. To address this gap, we propose DriveSafe, a framework for risk-aware scene understanding that leverages structured natural language descriptions. Specifically, our method first generates spatially grounded captions enriched with multimodal context, including motion, spatial, and depth cues. These captions are then used for downstream risk assessment, explicitly identifying hazardous objects, their locations, and the unsafe behaviors they imply, followed by actionable safety suggestions. To further improve performance, we employ caption-risk pairings to fine-tune a lightweight adapter module, efficiently injecting domain-specific knowledge into the base LLM. By conditioning risk assessment on explicit language-based scene representations, DriveSafe achieves significant gains over both zero-shot MLLMs and prior domain-specific baselines. Exhaustive experiments on the DRAMA benchmark demonstrate state-of-the-art performance, while ablation studies validate the effectiveness of our key design choices. Project page: https://cvit.iiit.ac.in/ research/projects/cvit-projects/drivesafe

Sainithin Artham, Shankar Gangisetty, Avijit Dasgupta +1

IIIT-Hyderabad, India

VLMs for Autonomous Driving Autonomous Driving Safety Evaluation Large Vision-Language Models Autonomous Driving Autonomous Driving Perception

Ask PDF

Sep 9, 2026

Data-Driven Risk Fields for Safer End-to-End Autonomous Driving

Safety is a fundamental requirement for autonomous driving, yet existing end-to-end driving models still lack explicit risk-aware learning capacities. Existing rule-based risk models provide interpretable safety priors, yet their absolute risk scores depend on handcrafted functions, coefficients, and thresholds. Learning-based risk representations reduce part of this manual design, but their supervision often relies on occupancy-derived labels or heuristic cost values, which may not capture ego-conditioned planning risk. In this paper, we propose DRiF, a data-driven risk-field framework for safer end-to-end autonomous driving. DRiF learns a shared BEV feature with static map segmentation, dynamic risk prediction, and vehicle planning. For dynamic risk learning, DRiF converts rule-based safety priors into pairwise risk labels, and trains the risk field to preserve relative risk ordering instead of regressing handcrafted absolute scores. Experiments on Bench2Drive show that DRiF achieves competitive overall performance, with consistent improvements in driving score, success rate, and collision-related metrics. These results establish relative risk supervision as an effective way to connect explicit safety structure with end-to-end planning. The data and code will be publicly available.

Yuanxin Tian, Zhiyuan Liu, Jinhao Li +9

Tsinghua University · Yinwang · KAIST

End-to-End Autonomous Driving Safe Motion Planning Autonomous Driving Planning

Ask PDF

Jul 8, 2026

A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous driving

Safe autonomous driving requires both rapid responses to common high-risk events and deeper reasoning over rare, extreme long-tail scenarios in traffic safety. These scenarios are severely under-represented in naturalistic driving data, and existing trajectory and language-augmented datasets seldom provide high-risk event labels, semantic annotations, and verifiable safety signals. Here we present K-Risk, a knowledge-augmented dataset that combines structured driving trajectories with large language model generated semantic annotations for safety-critical driving scenarios. K-Risk integrates 20 human-driven and autonomous-vehicle trajectory datasets from Europe, China, and the United States, covering highways, urban freeways, intersections, and roundabouts. Using a unified risk-centric extraction pipeline, K-Risk curates 31,398 high-risk events, together with a 1,036-event extreme subset of near-collision cases. Each event is released as a synchronized trajectory, metadata, and language triplet containing structured scenario descriptions, abnormal-behavior notifications, and, for a representative subset, causal risk analyses and action recommendations validated through a closed-loop simulator with iterative reflection. By combining multi-dimensional risk annotations, interpretable language supervision, and verifiable decisions, K-Risk bridges structured traffic trajectories, semantic reasoning, and decision supervision, providing a standardized foundation for developing and evaluating next-generation risk-aware autonomous driving agents.

Heye Huang, Jingguang Li, Zhiyuan Zhou +4

Cho Chun Shik Graduate School of Mobility, Korea Advanced Institute of Science and Technology, Daejeon 34051, South Korea · College of Computer Science and Artificial Intelligence, Fudan University, Shanghai 200433, China · Media Lab, Massachusetts Institute of Technology, Cambridge, MA 02139, USA +2

Autonomous Driving Datasets Autonomous Driving Safety Evaluation Autonomous Driving

Ask PDF

Abstract

Ontology-Based Reasoning

Explore similar work

DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios

Data-Driven Risk Fields for Safer End-to-End Autonomous Driving

A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous driving