In this paper, we investigate why vision-language models (VLMs) produce errors by decomposing them into two sources: visual-representation failures, in which the visual evidence required for the correct answer is not encoded, and visual-utilization failures, in which this evidence is encoded but not successfully used. We propose to estimate the visual-utilization failure rate using lightweight probes and derive theoretical bounds on the true rate that account for probe imperfections. We introduce VELens (Visual Evidence Lens), a rank-based probe that traces visual evidence for both a target object and its ground-truth attribute, such as color, material, or shape, across VLM layers. We evaluate VELens on over 6,000 images using LLaVA-1.6-7B and Qwen2.5-VL-7B. Our method achieves ROC-AUC scores of 0.95 and 0.91, respectively, and outperforms alternative probes in both low-resource and unseen-attribute settings. Using it to diagnose errors, we estimate that visual-utilization failures account for 84.4% of LLaVA errors and 70.0% of Qwen errors, with conservative lower bounds of 77.5% and 61.0%, respectively. Overall, our study identifies visual utilization as the dominant bottleneck in VLMs and suggests that a central opportunity for improvement lies in developing better mechanisms to locate, route, and use the visual information they already encode.
Figures & tables
Figure 1: Diagnosing the source of errors.
Figure 2: Overview of VELens. Top: Visual-token representations from each LLM layer are projected through the output head WV to obtain vocabulary rankings. These rankings are converted into local evidence scores and aggregated into one score per layer. The layer-wise scores serve as input to a logistic-regression classifier. Bottom: For each visual token, the vocabulary ranks of target tokens ( T+ ) and competing tokens ( T− ) are compared to produce a local evidence score.
Figure 3: Low-resource and unseen-value generalization. Left: ROC-AUC as the number of labeled examples per class varies, using Separate and averaging across color, material, and shape, with five independently sampled subsets for each size and category. Middle and right: Leave-one-value-out ROC-AUC, averaged over held-out values, using both pooling configurations. For Separate, individual probes are trained on each seen value, and their predictions are averaged for the held-out value. For All, one probe is trained jointly on all seen values and applied to the held-out value. Error bars are computed via 100 bootstraps.
Figure 4: Top: The object region is emphasized by blurring its surroundings (leftmost) or degraded by Gaussian noise, yielding the transformed image X′ (§ D.4 ). Lines report the LLaVA error rate (red) and joint visual-evidence detection among the original errors P(M(X′,YX)=1∣H(X)=1) (yellow). Bottom: VELens estimates of visual-utilization (red) and visual-representation (gray) failures. The lines and bracketed values are the utilization-failure rate bounds.
Figure 5: An example of evidence-positive error, where VELens detects both object and attribute evidence. Increasing attention to the target-object region changes the answer from wood to glass . More examples in Figure 12 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Computing object and attribute attention ratios. Top: We identify the visual tokens inside the annotated object region and the tokens in query Q representing the target object (“car”, 1) and a candidate attribute (“yellow”, 2). Bottom: The highlighted rows show attention from these concept tokens to visual tokens at one layer l and head h . For each concept, attention inside the object region is summed and divided by total image attention to obtain the inside attention ratio αl,h,jin(c) (Eq. 9 ).
Figure 7: Logit-lens projection of visual tokens inside annotated object regions. Qualitative examples from VAW in LLaVA, for three target objects: a plate (green), a hat (orange), and a bowl (purple). Top: For each object, we randomly sample five visual tokens ki from its inside-region set Kin and show, at every layer l , the top-ranked vocabulary token, i.e. the v∈V with rankl(ki,v)=1 (Eq. 14 ). Color intensity indicates the top-1 probability. Bottom: The corresponding images, each with the annotated bounding box b of its target object in the matching color. Semantics emerge at different depths across the three objects: the plate tokens are uninformative until the middle layers and then decode to plate , white and round , whereas hat and bowl are already top-ranked in the earliest layers, with red and glass following later.
Figure 9: ROC curves for attribute-evidence detection. True-positive against false-positive rate for each method, shown for LLaVA (top) and Qwen (bottom) and separately for color, material, and shape, using the Separate pooling configuration. The areas under these curves are the attribute columns of Table 1 . The dashed diagonal marks chance.
Figure 10: Controlled transformations. Five images from the VAW dataset (rows) with each transformation applied (columns): the object emphasized by blurring its surroundings ( ψ+ ), the clean image, and Gaussian noise within the bounding box b at λ=0.1 , 0.3 , and 0.9 ( ψ− ).
Figure 11: Visual-evidence development across VLM layers. Layer-wise attribute and object evidence for LLaVA (left) and Qwen (right). The figure reports the AUC obtained from each individual layer. The complete trajectory captures evidence that becomes accessible at different depths across attributes and models.
Figure 12: Additional examples of the attention intervention. Extends Figure 5 . Three evidence-positive errors, where VELens detects both object and attribute evidence (scores shown per panel), and increasing attention to the annotated target-object region corrects the answer. Each panel shows attention from the last prompt token to image patches, averaged over the last four decoder layers (head-mean). Left: no intervention. Right: prompt → inside-object visual-token attention increased at strength α=2 , chosen as described in § D.6 .
Department of CMPS, University of British Columbia, Canada · Weathon Software, Canada · Department of Computer Science, University of British Columbia, Canada