In this paper, we investigate why vision-language models (VLMs) produce errors by decomposing them into two sources: visual-representation failures, in which the visual evidence required for the correct answer is not encoded, and visual-utilization failures, in which this evidence is encoded but not successfully used. We propose to estimate the visual-utilization failure rate using lightweight probes and derive theoretical bounds on the true rate that account for probe imperfections. We introduce VELens (Visual Evidence Lens), a rank-based probe that traces visual evidence for both a target object and its ground-truth attribute, such as color, material, or shape, across VLM layers. We evaluate VELens on over 6,000 images using LLaVA-1.6-7B and Qwen2.5-VL-7B. Our method achieves ROC-AUC scores of 0.95 and 0.91, respectively, and outperforms alternative probes in both low-resource and unseen-attribute settings. Using it to diagnose errors, we estimate that visual-utilization failures account for 84.4% of LLaVA errors and 70.0% of Qwen errors, with conservative lower bounds of 77.5% and 61.0%, respectively. Overall, our study identifies visual utilization as the dominant bottleneck in VLMs and suggests that a central opportunity for improvement lies in developing better mechanisms to locate, route, and use the visual information they already encode.
Figures & tables
Figure 1: Diagnosing the source of errors.
Figure 2: Overview of VELens. Top: Visual-token representations from each LLM layer are projected through the output head WV to obtain vocabulary rankings. These rankings are converted into local evidence scores and aggregated into one score per layer. The layer-wise scores serve as input to a logistic-regression classifier. Bottom: For each visual token, the vocabulary ranks of target tokens ( T+ ) and competing tokens ( T− ) are compared to produce a local evidence score.
Figure 3: Low-resource and unseen-value generalization. Left: ROC-AUC as the number of labeled examples per class varies, using Separate and averaging across color, material, and shape, with five independently sampled subsets for each size and category. Middle and right: Leave-one-value-out ROC-AUC, averaged over held-out values, using both pooling configurations. For Separate, individual probes are trained on each seen value, and their predictions are averaged for the held-out value. For All, one probe is trained jointly on all seen values and applied to the held-out value. Error bars are computed via 100 bootstraps.
Figure 4: Top: The object region is emphasized by blurring its surroundings (leftmost) or degraded by Gaussian noise, yielding the transformed image X′ (§ D.4 ). Lines report the LLaVA error rate (red) and joint visual-evidence detection among the original errors P(M(X′,YX)=1∣H(X)=1) (yellow). Bottom: VELens estimates of visual-utilization (red) and visual-representation (gray) failures. The lines and bracketed values are the utilization-failure rate bounds.
Figure 5: An example of evidence-positive error, where VELens detects both object and attribute evidence. Increasing attention to the target-object region changes the answer from wood to glass . More examples in Figure 12 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Computing object and attribute attention ratios. Top: We identify the visual tokens inside the annotated object region and the tokens in query Q representing the target object (“car”, 1) and a candidate attribute (“yellow”, 2). Bottom: The highlighted rows show attention from these concept tokens to visual tokens at one layer l and head h . For each concept, attention inside the object region is summed and divided by total image attention to obtain the inside attention ratio αl,h,jin(c) (Eq. 9 ).
Figure 7: Logit-lens projection of visual tokens inside annotated object regions. Qualitative examples from VAW in LLaVA, for three target objects: a plate (green), a hat (orange), and a bowl (purple). Top: For each object, we randomly sample five visual tokens ki from its inside-region set Kin and show, at every layer l , the top-ranked vocabulary token, i.e. the v∈V with rankl(ki,v)=1 (Eq. 14 ). Color intensity indicates the top-1 probability. Bottom: The corresponding images, each with the annotated bounding box b of its target object in the matching color. Semantics emerge at different depths across the three objects: the plate tokens are uninformative until the middle layers and then decode to plate , white and round , whereas hat and bowl are already top-ranked in the earliest layers, with red and glass following later.
Figure 9: ROC curves for attribute-evidence detection. True-positive against false-positive rate for each method, shown for LLaVA (top) and Qwen (bottom) and separately for color, material, and shape, using the Separate pooling configuration. The areas under these curves are the attribute columns of Table 1 . The dashed diagonal marks chance.
Figure 10: Controlled transformations. Five images from the VAW dataset (rows) with each transformation applied (columns): the object emphasized by blurring its surroundings ( ψ+ ), the clean image, and Gaussian noise within the bounding box b at λ=0.1 , 0.3 , and 0.9 ( ψ− ).
Figure 11: Visual-evidence development across VLM layers. Layer-wise attribute and object evidence for LLaVA (left) and Qwen (right). The figure reports the AUC obtained from each individual layer. The complete trajectory captures evidence that becomes accessible at different depths across attributes and models.
Figure 12: Additional examples of the attention intervention. Extends Figure 5 . Three evidence-positive errors, where VELens detects both object and attribute evidence (scores shown per panel), and increasing attention to the annotated target-object region corrects the answer. Each panel shows attention from the last prompt token to image patches, averaged over the last four decoder layers (head-mean). Left: no intervention. Right: prompt → inside-object visual-token attention increased at strength α=2 , chosen as described in § D.6 .
Vision-language models (VLMs) perform well on visual question answering with high-quality images but struggle when questions require knowledge beyond what is clearly and directly visible. In such settings, uncertainty quantification should not only indicate whether the model is likely to fail but also diagnose why it is uncertain, across dimensions such as perception, entity recognition, and knowledge retrieval. While prior work has focused on individual failure modes in isolation or treated incorrect answers as monolithic failures, we propose a unified framework for disentangling these failure modes and investigate whether pre-generation signals can predict these failure sources. Across a range of datasets and model families, we find a consistent pattern in VLM errors: some failures arise from visual or recognition bottlenecks, while others persist after the relevant entity is identified. Our main finding is that these failure sources can be predicted before decoding: recognition-related failures are best captured by visual-token representations, while failures that remain after recognition are better captured by prompt-conditioned hidden states. This pre-generation signal enables efficient failure-source prediction before the model produces an answer, allowing uncertain cases to be routed to targeted interventions such as image repair, entity recognition support, or external retrieval.
When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding-Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit - not the strength of encoding - better predicts grounding outcomes with a correlation of ρ= 0.847. After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering - both linear and sparse autoencoder-guided - in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.
Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2
Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.
Vision Language Models (VLMs) are well known for hallucinating non-existent objects in images. Objects with missing parts present a unique challenge for VLMs, stemming from both real-world knowledge bias and the scarcity of such images in training data. We present MissingBench-Verified, a benchmark designed to evaluate a specific and practically relevant scenario: when vision-language models fail to recognize that an essential component of an object has been removed. Across ten leading models, we observe consistent and significant failure rates that persist even when external tool evidence explicitly contradicts the model's visual perception. We further ask whether granting models access to image processing tools (e.g., cropping, contrast adjustment) enables autonomous inspection to resolve these failures. We find that existing mitigation strategies, including tool-assisted verification, autonomous visual reasoning, longer reasoning durations, and fine-tuning on an easier dataset, provide negligible improvement, indicating that this failure mode cannot be addressed through current prompting or post-hoc correction techniques. Our findings highlight a fundamental limitation of current VLM for inspection and monitoring tasks and underscore the need for architectural or training-level interventions that enable models to override internal expectations when confronted with contradictory evidence.
Wenqi Marshall Guo, Qingyun Qian, Shiyu Zhou +2
Department of CMPS, University of British Columbia, Canada · Weathon Software, Canada · Department of Computer Science, University of British Columbia, Canada