cs.CVMar 6, 2026

Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models

Authors: Nicoleta-Nina Basoc, Adrian Cosma, Emilian Radoi

Organizations: National University of Science and Technology POLITEHNICA Bucharest, Bucharest, Romania · IDSIA, The Dalle Molle Institute for Artificial Intelligence, Lugano, Switzerland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-language models (VLMs) achieve strong benchmark results, yet can exhibit systematic perceptual weaknesses: structured, large changes to pixel values can cause confident yet nonsensical predictions, even when the underlying scene remains easily recognizable to humans. We study this gap using Spatial Colour Mixing, a programmatic family of colour distortions that overlays structured patterns (in both RGB and Ostwald colour systems) onto natural images. We introduce a framework of eight spatial colour mixing variants and evaluate nine VLMs across three model families on four datasets. Across models and datasets, accuracy degrades sharply with increasing distortion, and scaling the language model does not reliably mitigate the failure. In a human study with 61 participants on an animal recognition dataset, humans substantially outperform VLMs under the same distortions. Finally, we show that a simple human-inspired preprocessing step recovers a meaningful portion of performance for several distortion types, motivating perception-aware preprocessing and tool-use as practical strategies for improving VLM robustness.

Figures & tables

Figure 1: What animals are shown in these examples? Readers are invited to distance themselves from the screen / zoom in and observe how their perception changes 1 1 1 Answer: From left to right: cat, dog, elephant, bear . Best viewed on a colour display.
Figure 2: Examples of each of our 8 spatial colour mixing illusions. Best viewed on a colour display.
Figure 3: Examples of the effect of the distortion degree on four of our proposed colour mixing illusions. Best viewed on a coloured screen.
Figure 4: Main results. Accuracy as a function of distortion degree for 9 VLMs across four datasets ( Animals , Artworks , Landmarks , MME ) under eight Spatial Colour Mixing variants. Columns group models by family ( Qwen3-VL , LLaVA , Gemma3 ); within each panel, line style indicates model scale and colour indicates the illusion type. Across datasets, accuracy degrades sharply even at low distortion degrees, and differences are driven more by model family than by language-model scale.
Fn/ImageModel OutputFn/ImageModel Output
SCMix-1LLaVA-7B : "The artist who painted the artwork shown in the image is a dog."SCMix-2gemma3-27B : "hippopotamus the image clearly depicts a hippopotamus."
SCMix-3Agemma3-12B : "cat. it’s cleverly hidden within the striped background!"Ostwald RandomQwen3-VL-30B : " 千里川土手, a location in japan."
Ostwald RandomLLaVA-NEXT-34B : "the image appears to be a low-resolution or pixelated image of a person, not an animal. it’s difficult to discern specific details due to the quality of the image."SCMix-3Bgemma3-27B : "bear the image is an optical illusion where the shape of a bear is hidden within a colorful, striped background. it takes a moment to focus and perceive the animal’s form."
Ostwald CheckerQwen3-VL-30B : "this image is a pixelated, low-resolution depiction of the statue of liberty."Ostwald Checkergemma3-12B : "based on the image, i can distinguish a skull."
Table 1: Representative distorted inputs and corresponding model outputs illustrating non-robust behaviour beyond performance degradation: e.g., off-task responses (animal names as artists), confident misidentifications, "illusion-aware" descriptions that still miss the correct concept, and occasional off-distribution generations (including foreign-language text). Best viewed on a colour display.
Figure 5: Mean per-image accuracy on the Animals dataset for three distortion degrees (2, 5, 12) and three representative illusion types ( SCMix-1 , Ostwald Checker , Ostwald Random ). Bars compare human responses to the aggregated performance of the nine evaluated VLMs.
Figure 6: Results for gemma-3-12b on the Animals dataset for preprocessing images using human-inspired low-pass filtering procedures: downscaling followed by upscaling (D/U) and applying box blurring to the distorted images using Ostwald Checker , SCMix-1 , SCMix-3A . Aggressive low-pass filtering improves performance on Ostwald Checker and SCMix-1 , but does not affect illusions with multiple lanes (e.g., SCMix-3A ).
Figure 7: Results for tool use on a subsample of Animals using Ostwald Checker perturbation using gpt-5-mini-2025-08-07 and code-interpreter . Here, "auto" refers to optional tool use. Degree "1" refers to original images.
Figure 8: Average cosine similarities between original images and their corresponding illusions based on mean-pooled embeddings generated by multiple vision encoders (CLIP-VIT, SigLIP and DINOv3). We also show SSIM and histogram correlation across distortion degrees.
More in this topic

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 454 papers

Explore similar work

CardsList

  1. VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models

    Mar 6, 2026Rohit Saxena, Alessandro Suglia, Pasquale MinerviniVLM RobustnessVLM Evaluation

    University of Edinburgh, UK

  2. DistortBench: Benchmarking Vision Language Models on Image Distortion Identification

    Apr 21, 2026Divyanshu Goyal, Akhil Eppa, Vanya Bannihatti KumarVLM EvaluationBenchmark Construction

    Adobe Inc.

  3. Same Answer, Different Representations: Hidden instability in VLMs

    Feb 6, 2026Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6VLM EvaluationVLM Robustness

    Sapienza University of Rome · University of Edinburgh · University of Oxford +5