When visual evidence is ambiguous, vision models must decide how to interpret face-like patterns. Face pareidolia, the perception of faces in non-face objects, provides a controlled probe of such decisions. We introduce a diagnostic framework that analyzes detection, localization, uncertainty and bias across class, difficulty and emotion. We evaluate six models spanning four representational regimes: vision-language models (VLMs; CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B), pure vision classification (ViT), object detection (YOLOv8), and face detection (RetinaFace). Our results reveal that uncertainty and bias are decoupled: low uncertainty can signal either safe suppression, as in detectors, or extreme over-interpretation, as in VLMs. VLMs exhibit semantic overactivation, systematically interpreting ambiguous non-human regions as Human, with LLaVA over-calling on 73% of non-human pareidolic images, especially for negative emotions. ViT instead follows an uncertainty-as-abstention strategy, remaining diffuse yet largely unbiased. Detection-based models achieve low bias through conservative priors that suppress pareidolia responses even when localization is controlled. Together, these results show that behavior under ambiguity is governed more by representation than thresholds, establishing face pareidolia as a diagnostic of semantic robustness and a source of ambiguity-aware hard negatives for vision models. Code will be released upon publication.
Figures & tables
Figure 1: Face pareidolia in an electrical outlet. The visual input is unchanged, yet observers may perceive a face, illustrating how interpretation emerges under ambiguity.
Figure 2: Example FacesInThings images ( Hamilton et al. 2024 ) . Red bounding boxes indicate face-like regions annotated by human observers in inanimate objects.
Figure 3: Unified pareidolia diagnostic pipeline. VLMs (CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B) and pure vision classifier (ViT) classify annotated regions, while object detection (YOLOv8) and face detection (RetinaFace) detect faces in full images. Predictions are mapped to a common five-class space for evaluating detection, localization, uncertainty and bias across subgroups.
Figure 4: Detection coverage vs. localization. Box-level classifiers saturate both metrics, while YOLOv8 and RetinaFace under-respond.
Figure 5: Human over-call bias on pareidolic regions. Points show False Bias Score (FBS) versus non-human → Human rate, revealing stronger bias in VLMs (especially LLaVA) than in ViT, YOLOv8, and RetinaFace.
Figure 6: Prediction uncertainty (RAI) on pareidolic regions. ViT is uncertain and unbiased; CLIP moderately uncertain and biased; LLaVA confident and strongly biased; detectors confident and conservative.
Figure 7: Non-human → Human bias by emotion. VLMs show the highest over-call rates, with CLIP especially biased on negative emotions. Other models remain near zero.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Detection by difficulty. Box-level models maintain PPDR = 1.0 across all difficulty levels. YOLOv8 and RetinaFace drop on harder, more ambiguous cases.
Figure 9: How CLIP-L/14 differs from CLIP-B/32 in its predictions. Each cell shows the change in row-normalized probability (CLIP-L/14 minus CLIP-B/32). CLIP-L moves probability away from Human , primarily toward Animal, reducing but not removing Human over-call bias.
Figure 10: Detection coverage by emotion for models that do not always detect. YOLOv8 changes across emotions. RetinaFace stays low. Box-level models that already reach 100% detection on all emotions are omitted.
Figure 11: Detector behavior on ground-truth boxes across difficulty levels. Top: response rate (i.e., how often a model produces any Human detection on a GT box). Bottom: mean Human score when the model does respond. Each crop adds 10% padding around the annotated box.
Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.
Vision-language models can produce confident answers on visually ambiguous inputs, resulting in biased predictions. Common entropy-based methods, such as Semantic Entropy (SE), rely on output diversity. Yet our analysis shows that overconfident visual embeddings suppress output diversity under stochastic decoding, causing SE to underestimate uncertainty in such cases. Recent methods instead probe output diversity through input perturbations, including textual paraphrasing or joint text-image perturbations, and show improved performance. We study these approaches and reveals that the resulting variability is often dominated by textual changes rather than visual evidence, causing uncertainty estimates to reflect prompt sensitivity rather than visual ambiguity. We therefore propose Visual Semantic Entropy (VSE), which perturbs only the image to probe nearby visual variations while keeping the text query fixed. VSE measures uncertainty by clustering generated answers into semantic prototypes and computing the mass-weighted dispersion among them. Extensive evaluation across five modern vision-language models and five diverse VQA benchmarks demonstrates that VSE effectively captures visual ambiguity, establishing a new state-of-the-art for VLM uncertainty estimation.
Ta Duc Huy, Trang Nguyen, Townim Chowdhury +5
Australian Institute for Machine Learning, Adelaide University, Australia · Flinders University, Australia
Vision-language models (VLMs) exhibit a systematic bias when confronted with classic optical illusions: they overwhelmingly predict the illusion as "real" regardless of whether the image has been counterfactually modified. We present a tool-guided inference framework for the DataCV 2026 Challenge (Tasks I and II) that addresses this failure mode without any model training. An off-the-shelf vision-language model is given access to a small set of generic image manipulation tools: line drawing, region cropping, side-by-side comparison, and channel isolation, together with an illusion-type-routing system prompt that prescribes which tools to invoke for each perceptual question category. Critically, every tool call produces a new, immutable image resource appended to a persistent registry, so the model can reference and compose any prior annotated view throughout its reasoning chain. Rather than hard-coding illusion-specific modules, this generic-tool-plus-routing design yields strong cross-structural generalization: performance remained consistent from the validation set to a test set containing structurally unfamiliar illusion variants (e.g., Mach Bands rotated from vertical to horizontal stacking). We further report three empirical observations that we believe warrant additional investigation: (i) a strong positive-detection bias likely rooted in imbalanced illusion training data, (ii) a striking dissociation between pixel-accurate spatial reasoning and logical inference over self-generated annotations, and (iii) pronounced sensitivity to image compression artifacts that compounds false positives.
Many medical conditions require clinical assessment of symptoms, history, and functioning rather than judgments from appearance. We study whether vision-language models (VLMs) abstain from facial attribution of autism spectrum disorder (ASD), attention-deficit/hyperactivity disorder (ADHD), major depressive disorder (MDD), and generalized anxiety disorder (GAD). To support this study, we construct PARITY (Paired Assessment with Reused Identity), a demographically balanced dataset for testing whether changing facial expressions alters diagnostic judgments, while holding identity and other visual factors as constant as possible. Our comprehensive evaluation of 27 open- and closed-weight models reveals inconsistent abstention across expressions, conditions, and task formats. Models turn facial expressions into unsupported medical judgments: this behavior is widespread among open-weight VLMs and extends to safety-trained frontier models that abstain on individual portraits yet make diagnostic judgments when comparing portraits of the same identity. Safeguard checks reveal further gaps: clinical instructions do not consistently prevent these judgments, and a guardrail model rarely flags paired requests as unsafe. Semantic analysis further shows that most baseline open-model refusals fail to explain both the insufficiency of facial evidence and the need for clinical assessment. Unsupported judgments and seemingly authoritative explanations can cause distress, reinforce stigma, and offer false reassurance, underscoring the need for stronger safety measures in model training and deployment.
Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel +8