cs.CVMar 4, 2026

When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models

Authors: Qianpu Chen, Derya Soydaner, Rob Saunders

Organizations: LIACS (Leiden Institute of Advanced Computer Science), Leiden University, Leiden, The Netherlands

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

When visual evidence is ambiguous, vision models must decide how to interpret face-like patterns. Face pareidolia, the perception of faces in non-face objects, provides a controlled probe of such decisions. We introduce a diagnostic framework that analyzes detection, localization, uncertainty and bias across class, difficulty and emotion. We evaluate six models spanning four representational regimes: vision-language models (VLMs; CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B), pure vision classification (ViT), object detection (YOLOv8), and face detection (RetinaFace). Our results reveal that uncertainty and bias are decoupled: low uncertainty can signal either safe suppression, as in detectors, or extreme over-interpretation, as in VLMs. VLMs exhibit semantic overactivation, systematically interpreting ambiguous non-human regions as Human, with LLaVA over-calling on 73% of non-human pareidolic images, especially for negative emotions. ViT instead follows an uncertainty-as-abstention strategy, remaining diffuse yet largely unbiased. Detection-based models achieve low bias through conservative priors that suppress pareidolia responses even when localization is controlled. Together, these results show that behavior under ambiguity is governed more by representation than thresholds, establishing face pareidolia as a diagnostic of semantic robustness and a source of ambiguity-aware hard negatives for vision models. Code will be released upon publication.

Figures & tables

Figure 1: Face pareidolia in an electrical outlet. The visual input is unchanged, yet observers may perceive a face, illustrating how interpretation emerges under ambiguity.
Figure 2: Example FacesInThings images ( Hamilton et al. 2024 ) . Red bounding boxes indicate face-like regions annotated by human observers in inanimate objects.
Figure 3: Unified pareidolia diagnostic pipeline. VLMs (CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B) and pure vision classifier (ViT) classify annotated regions, while object detection (YOLOv8) and face detection (RetinaFace) detect faces in full images. Predictions are mapped to a common five-class space for evaluating detection, localization, uncertainty and bias across subgroups.
Figure 4: Detection coverage vs. localization. Box-level classifiers saturate both metrics, while YOLOv8 and RetinaFace under-respond.
Figure 5: Human over-call bias on pareidolic regions. Points show False Bias Score (FBS) versus non-human →\rightarrow Human rate, revealing stronger bias in VLMs (especially LLaVA) than in ViT, YOLOv8, and RetinaFace.
Figure 6: Prediction uncertainty (RAI) on pareidolic regions. ViT is uncertain and unbiased; CLIP moderately uncertain and biased; LLaVA confident and strongly biased; detectors confident and conservative.
Figure 7: Non-human →\rightarrow Human bias by emotion. VLMs show the highest over-call rates, with CLIP especially biased on negative emotions. Other models remain near zero.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Detection by difficulty. Box-level models maintain PPDR = 1.01.0 across all difficulty levels. YOLOv8 and RetinaFace drop on harder, more ambiguous cases.
Figure 9: How CLIP-L/14 differs from CLIP-B/32 in its predictions. Each cell shows the change in row-normalized probability (CLIP-L/14 minus CLIP-B/32). CLIP-L moves probability away from Human , primarily toward Animal, reducing but not removing Human over-call bias.
Figure 10: Detection coverage by emotion for models that do not always detect. YOLOv8 changes across emotions. RetinaFace stays low. Box-level models that already reach 100%100\% detection on all emotions are omitted.
Figure 11: Detector behavior on ground-truth boxes across difficulty levels. Top: response rate (i.e., how often a model produces any Human detection on a GT box). Bottom: mean Human score when the model does respond. Each crop adds 10%10\% padding around the annotated box.
More in this topic

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 454 papers

Explore similar work

CardsList

  1. Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?

    Jun 30, 2026Ta Duc Huy, Trang Nguyen, Townim Chowdhury +5VLM EvaluationVisual Question Answering

    Australian Institute for Machine Learning, Adelaide University, Australia · Flinders University, Australia

  2. Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions

    Mar 31, 2026Xuesong Wang, Harry WangVLM RobustnessReasoning for VLMs

    Wayne State University · University of Michigan

  3. From Portraits to Pathology: Probing Diagnostic Overreach in Vision-Language Models

    Sep 2, 2026Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel +8Abstention in VLMsVision-Language Models

    Oracle AI