Authors: Mohammad Asadi, Jack W. O'Sullivan, Fang Cao, Tahoura Nedaee, Kamyar Rajabalifardi, Fei-Fei Li, Ehsan Adeli, Euan Ashley
Organizations: Department of Electrical Engineering, Stanford University, CA, USA · Division of Cardiology, Department of Medicine, Stanford University, CA, USA · Department of Biomedical Data Science, Stanford University, CA, USA · Department of Biology, Stanford University, CA, USA · Department of Computer Science, Stanford University, CA, USA · Department of Psychiatry and Behavioral Sciences, Stanford University, CA, USA
Multimodal AI systems have achieved remarkable performance across a broad range of real-world tasks, yet the mechanisms underlying visual-language reasoning remain surprisingly poorly understood. We report three findings that challenge prevailing assumptions about how these systems process and integrate visual information. First, Frontier models readily generate detailed image descriptions and elaborate reasoning traces, including pathology-biased clinical findings, for images never provided; we term this phenomenon mirage reasoning. Second, without any image input, models also attain strikingly high scores across general and medical multimodal benchmarks, bringing into question their utility and design. In the most extreme case, our model achieved the top rank on a standard chest X-ray question-answering benchmark without access to any images. Third, when models were explicitly instructed to guess answers without image access, rather than being implicitly prompted to assume images were present, performance declined markedly. Explicit guessing appears to engage a more conservative response regime, in contrast to the mirage regime in which models behave as though images have been provided. These findings expose fundamental vulnerabilities in how visual-language models reason and are evaluated, pointing to an urgent need for private benchmarks that eliminate textual cues enabling non-visual inference, particularly in medical contexts where miscalibrated AI carries the greatest consequence. We introduce B-Clean as a principled solution for fair, vision-grounded evaluation of multimodal AI systems.
Figures & tables
Figure 1: Definition and quantification of the mirage effect. a, We define a mirage as an AI model describing visual features in the absence of any image input. The output of a model affected by the mirage effect, seen in isolation, is indistinguishable from that of normal visual reasoning. They both describe seen images and visual features without showing any signs of uncertainty. b, We quantify the mirage effect in a multimodal AI model by measuring the frequency of the model showing the effect across all the questions in a given category. The questions are designed to ask specifically about an image input, common in that category, without providing any images. c, We show consistently high rates of showing the mirage effect across all the tested AI models. We observe that the average rate can be higher for the newest versions of one model compared to the legacy ones, suggesting that continued training and acquiring new skills can lead to unwanted behaviors in multimodal settings.
Figure 2: The distribution of Gemini-3-Pro’s answers in response to asking for the description and diagnosis based on non-existent images of a brain MRI, chest X-ray, electrocardiogram, pathology slides, and user-taken skin pictures. We repeat each question with 200 different seeds while keeping all the other parameters unchanged. Any cases of acknowledging the lack of an image, refusing to diagnose for safety reasons, or empty responses are mentioned together as “No diagnosis found”. The diagnoses necessitating immediate follow-up actions are marked in red. We observe that although “Normal” and “No diagnosis found” cases are among the top mirage-based diagnoses, cumulatively, the pathologies are significantly more prevalent among the predictions.
Figure 3: The AI models’ answers in the mirage-mode can exhibit illusory high accuracies in the benchmark-based evaluations. a, The 4 frontier AI models of Gemini-3-Pro, Gemini-2.5-Pro, Claude-Opus-4.5, and GPT-5.1 have been tested on 3 medical benchmarks of VQA-Rad, MicroVQA, and MedXpertQA-MM, and 3 general-purpose benchmarks of MMMU-Pro, Video-MMMU, and Video-MME in original mode, as well as in mirage-mode, i.e., asking the same visual question without providing the image and without acknowledging the lack of it. All models were tested in extended thinking mode, which yielded higher accuracy across all original and mirage-mode cases. We observe that the accuracies in mirage-mode not only do not collapse, but also are competitive with the original accuracies, especially on medical benchmarks. b, Our super-guesser, trained on the public set of ReXVQA, outperforms the top AI models, as well as the average of the radiology doctors on the held-out test-set. The base model used for the super-guesser is a 3-billion-parameter text-only Queen 2.5 model, released before the ReXVQA dataset, minimizing the possibility of data contamination. c, We define the Mirage score for a benchmark-model pair as the ratio of the model’s accuracy on the benchmark in mirage-mode over the same model’s original accuracy on the benchmark. We then calculate the Mirage score for a model as the average score over all the models tested, and for a benchmark as the average over all the models. The average Mirage score of a model demonstrates the model’s dependence on the Mirage effect to answer multimodal benchmark questions. The average Mirage score of a benchmark shows the susceptibility of the benchmark to the mirage effect, i.e., the extent to which the benchmark’s questions can be answered by the AI models without the visual input.
Figure 4: Performance of GPT-5.1 on the 3 benchmarks of MicroVQA, MedXpertQA-MM, and MMMU-Pro compared in mirage-mode and guessing-mode. In guessing mode, the prompt acknowledges the lack of images and instructs the model to take the best guess based on the question. a, The accuracy decreases in all 3 of the benchmarks with awareness of the lacking image and instruction to guess. b, & c, The model in mirage-mode consistently outperforms or is comparable to the guess-mode in the medical benchmarks across all question categories. d, Mirage-mode shows a greater performance across 23 categories of MMMU-Pro, out-performed in just 5 by guess-mode, and equivalent in 2.
Figure 5: The B-Clean method helps evaluate multimodal AI models by identifying and removing all compromised questions using the mirage-mode evaluation, thereby leaving a clean version of the benchmark that exclusively evaluates visual understanding. a, We created the B-Clean versions of MMMU-Pro, MedXpertQA-MM, and MicroVQA datasets. During the cleaning, one half to three fourth of the benchmark questions were removed. b, We observe that not only do the accuracies decrease when moving from the full benchmarks to the cleaned versions, but also the rankings of the models change for 2 out of 3 models in 2 out of 3 benchmarks, and reduce the distance gaps in others.
Figure S1 : Merely adding common multimodal AI evaluation instructions, shown to improve the model performance in original (with-image) settings, substantially raises the mirage rates across all models.
Figure S2 : The effect of explicitly mentioning multimodal attachments on mirage rates.
Figure S3 : The effect of mentioning the benchmark name on model accuracy.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.