The field of Artificial Intelligence has been adopted for many application domains. Vision Language Models are one of the recently advanced AI techniques that have been explored to support automotive features such as vehicle perception, and safety assurance. However, such language models are prone to hallucinations, posing a potential threat to the safety of automotive systems that may incorporate them. Within the automotive domain, VLMs could not only hallucinate traffic objects, but could also fail to identify traffic objects that are actually present, which may potentially lead to dangerous situations. Though we have observed a growing body of literature that proposes verification and validation techniques for safe and trustworthy AI, these methods are often studied in isolation, focusing either on run-time or design-time phases. Such isolated techniques could be insufficient in safety-critical, realistic contexts such as automotive perception systems. In this paper, we analyze design-time and run-time verification and validation techniques based on a taxonomy presented by Huang et al. We present an automotive study in which a design-time qualification workflow is proposed to complement run-time monitoring. This workflow combines a fixed safety-relevant ontology-based structured annotation system together with a synonym-based evaluation process to statistically evaluate three state-of-the-art VLMs against data from the nuScenes dataset. We observed that the proposed technique enables deterministic and repeatable quantification of the hallucinations VLMs generate in automotive perception-related tasks. The proposed workflow supports model comparison and deployment-oriented engineering decisions within the design-time verification and validation process and will contribute to a holistic verification strategy that strives towards trustworthy automotive perception systems
Figure 2: The Verification and Validation technique categorization based on the taxonomy presented by Huang et al. (Image: [ 16 ] )
Figure 3: Overview diagram of the experimental setup: All curated images are passed to a set of VLMs (GPT4o and GPT4.5-preview, and LLaVA) with a prompt that guides the VLMs to detect the traffic entities that are visible. The same image set is manually evaluated by a group of human annotators to record the ground truth, ie., accurate object labels. The collected ground truth and the VLM-generated responses are statistically evaluated to analyze the performance of the VLMs. (Icons: Flaticon.com)
LLaVA
GPT4o
GPT-4.5
(a) Perfect Matches
7.8%
22.90%
28.70%
(b) Hallucinated images
92.20%
77.10%
71.30%
(c) Normalized Hallucination Rate
73.30%
50.78%
41.03%
(d) Hallucinating Objects
1418
154
252
(e) Overlooking Objects
2862
2123
1591
Table 1: Performances of VLMs under three categories: (a) Perfect matches where VLM successfully identified all hallucinations, (b) hallucinated images where VLMs returned at least one hallucination, (c) normalized hallucination rate indicating the average deviations occurred in the VLM outputs compared to ground truth and the deviations in VLM responses grouped by error categories: (d) hallucinating objects and (e) Overlooking Objects
Chi Square
797.4314
P-value
p−value≪1e−10
Effect size
0.3987
Table 2: The results of the Friedman test indicate the significance of the statistical difference of the non-normalized data.
First execution
Second execution
True Positives
57
57
False Positives
5
5
True Negatives
305
311
False Negatives
633
627
Precision
91.94%
91.94%
Recall
8.26%
8.33%
Table 3: Results of SelfCheckGPT adaptation when tested with the image subset we curated.