cs.CVSep 22, 2026

Towards Systematic Qualification of Vision-Language Models for Automotive Perception Systems

Authors: Malsha Ashani Mahawatta Dona, Konstantinos Rokanas, Alexander Säfström, Krishna Ronanki, Christian Berger

Organizations: University of Gothenburg and Chalmers University of Technology, Gothenburg, Sweden

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The field of Artificial Intelligence has been adopted for many application domains. Vision Language Models are one of the recently advanced AI techniques that have been explored to support automotive features such as vehicle perception, and safety assurance. However, such language models are prone to hallucinations, posing a potential threat to the safety of automotive systems that may incorporate them. Within the automotive domain, VLMs could not only hallucinate traffic objects, but could also fail to identify traffic objects that are actually present, which may potentially lead to dangerous situations. Though we have observed a growing body of literature that proposes verification and validation techniques for safe and trustworthy AI, these methods are often studied in isolation, focusing either on run-time or design-time phases. Such isolated techniques could be insufficient in safety-critical, realistic contexts such as automotive perception systems. In this paper, we analyze design-time and run-time verification and validation techniques based on a taxonomy presented by Huang et al. We present an automotive study in which a design-time qualification workflow is proposed to complement run-time monitoring. This workflow combines a fixed safety-relevant ontology-based structured annotation system together with a synonym-based evaluation process to statistically evaluate three state-of-the-art VLMs against data from the nuScenes dataset. We observed that the proposed technique enables deterministic and repeatable quantification of the hallucinations VLMs generate in automotive perception-related tasks. The proposed workflow supports model comparison and deployment-oriented engineering decisions within the design-time verification and validation process and will contribute to a holistic verification strategy that strives towards trustworthy automotive perception systems

Figures & tables

Figure 1: Verification framework in LLM lifecycle (Image: [ 16 ] )
Figure 2: The Verification and Validation technique categorization based on the taxonomy presented by Huang et al. (Image: [ 16 ] )
Figure 3: Overview diagram of the experimental setup: All curated images are passed to a set of VLMs (GPT4o and GPT4.5-preview, and LLaVA) with a prompt that guides the VLMs to detect the traffic entities that are visible. The same image set is manually evaluated by a group of human annotators to record the ground truth, ie., accurate object labels. The collected ground truth and the VLM-generated responses are statistically evaluated to analyze the performance of the VLMs. (Icons: Flaticon.com)
 LLaVAGPT4oGPT-4.5
(a) Perfect Matches7.8%22.90%28.70%
(b) Hallucinated images92.20%77.10%71.30%
(c) Normalized Hallucination Rate73.30%50.78%41.03%
(d) Hallucinating Objects1418154252
(e) Overlooking Objects286221231591
Table 1: Performances of VLMs under three categories: (a) Perfect matches where VLM successfully identified all hallucinations, (b) hallucinated images where VLMs returned at least one hallucination, (c) normalized hallucination rate indicating the average deviations occurred in the VLM outputs compared to ground truth and the deviations in VLM responses grouped by error categories: (d) hallucinating objects and (e) Overlooking Objects
Chi Square797.4314
P-valuep−value≪1e−10p-value\ll 1e-10
Effect size0.3987
Table 2: The results of the Friedman test indicate the significance of the statistical difference of the non-normalized data.
 First executionSecond execution
True Positives5757
False Positives55
True Negatives305311
False Negatives633627
Precision91.94%91.94%
Recall8.26%8.33%
Table 3: Results of SelfCheckGPT adaptation when tested with the image subset we curated.
More in this topic

Cultural Safety Evaluation of VLMs

See all 14 papers

Explore similar work

CardsList
Jan 13, 2026

Semantic Misalignment in Vision-Language Models under Perceptual Degradation

Vision-Language Models (VLMs) are increasingly deployed in autonomous driving and embodied AI systems, where reliable perception is critical for safe semantic reasoning and decision-making. While recent VLMs demonstrate strong performance on multimodal benchmarks, their robustness to realistic perception degradation remains poorly understood. In this work, we systematically study semantic misalignment in VLMs under controlled degradation of upstream visual perception, using semantic segmentation on the Cityscapes dataset as a representative perception module. We introduce perception-realistic corruptions that induce only moderate drops in conventional segmentation metrics, yet observe severe failures in downstream VLM behavior, including hallucinated object mentions, omission of safety-critical entities, and inconsistent safety judgments. To quantify these effects, we propose a set of language-level misalignment metrics that capture hallucination, critical omission, and safety misinterpretation, and analyze their relationship with segmentation quality across multiple contrastive and generative VLMs. Our results reveal a clear disconnect between pixel-level robustness and multimodal semantic reliability, highlighting a critical limitation of current VLM-based systems and motivating the need for evaluation frameworks that explicitly account for perception uncertainty in safety-critical applications.

Guo Cheng, Huang Li

VLM EvaluationVLM RobustnessObject Hallucination in VLMsAutonomous Driving Perception
AskPDF
Mar 5, 2026

HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token

Hallucinations remain a persistent challenge for vision-language models (VLMs), which often describe nonexistent objects or fabricate facts. Existing detection methods typically operate after text generation, making intervention both costly and untimely. We investigate whether hallucination risk can instead be predicted before any token is generated by probing a model's internal representations in a single forward pass. Across a diverse set of vision-language tasks and eight modern VLMs, including Llama-3.2-Vision, Gemma-3, Phi-4-VL, and Qwen2.5-VL, we examine three families of internal representations: (i) visual-only features without multimodal fusion, (ii) vision-token representations within the text decoder, and (iii) query-token representations that integrate visual and textual information before generation. Probes trained on these representations achieve strong hallucination-detection performance without decoding, reaching up to 0.93 AUROC on Gemma-3-12B, Phi-4-VL 5.6B, and Molmo 7B. Late query-token states are the most predictive for most models, while visual or mid-layer features dominate in a few architectures (e.g., ~0.79 AUROC for Qwen2.5-VL-7B using visual-only features). These results demonstrate that (1) hallucination risk is detectable pre-generation, (2) the most informative layer and modality vary across architectures, and (3) lightweight probes have the potential to enable early abstention, selective routing, and adaptive decoding to improve both safety and efficiency.

Sai Akhil Kogilathota, Sripadha Vallabha E G, Luzhe Sun +1

Stony Brook University · Toyota Technological Institute at Chicago

Hallucination Detection in VLMs
AskPDF
May 27, 2026

ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Driving

Vision-Language-Action (VLA) models with integrated reasoning have been proposed for end-to-end autonomous driving, assuming a tight coupling between reasoning and trajectory generation. However, the robustness of such systems under realistic input perturbations remains largely unexplored. We show that these models are highly vulnerable to realistic input perturbations, achieving up to 89% attack success rate (ASR) on reasoning and up to 72% on trajectory manipulation in closed-loop simulation, leading to increased collision rates and degraded safety metrics. Using NVIDIA's recent Alpamayo models as representative industry-developed VLAs, we conduct the first systematic black-box study of reasoning-enabled VLA models under realistic textual input corruptions, evaluating their impact on reasoning and driving behavior. We introduce a reasoning-aware evaluation framework capturing both semantic and structural aspects of reasoning, along with safety-centric measures. We also introduce a benchmark for evaluating attacks and defenses on reasoning-trajectory interactions in autonomous driving. Our results highlight the need for rigorous evaluation and improved defenses to ensure the safety of reasoning-enabled VLA systems in autonomous driving.

Mohammadreza Teymoorianfard, Jean-Philippe Monteuuis, Jonathan Petit +1

University of Massachusetts Amherst · Qualcomm

VLMs for Autonomous DrivingAutonomous Driving Safety EvaluationAutonomous DrivingAdversarial Attacks on VLA ModelsVLM ReasoningVision-Language-Action Models+2
AskPDF