cs.CVNov 4, 2026

Trick-or-Treat: Adversarial Evaluation of Multimodal Medical Reasoning with Synthetic Images

Authors: Oishi Banerjee, Eujin Lee, Sung Eun Kim, Alexandra N. Willauer, Christine Y. Zhou, Zhizhou Yang, Pranav Rajpurkar

Organizations: Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA · Department of Computer Science, Cornell University, Ithaca, NY, USA · Division of Gastroenterology, Department of Medicine, Massachusetts General Hospital, Boston, MA, USA · Division of Pulmonary, Critical Care, and Sleep Medicine, University of Cincinnati, Cincinnati, OH, USA · 

Abstract

We used medical image synthesis to create a challenging multimodal evaluation framework that exposes hallucination behaviors in frontier models. Evaluating multimodal reasoning has historically been difficult because existing medical datasets rarely pair different image modalities, such as radiographs and external clinical photographs. To address this gap, we introduce Trick-or-Treat, an adversarial multimodal medical benchmark that pairs real chest radiographs with synthetic clinical photographs to challenge models' cross-modal reasoning. For each of a wide range of conditions, we created two pairs of images, where the pairs share a single chest radiograph but contain different external photographs. In matched pairs, the chest radiograph and photograph both contain findings suggesting the same disease (e.g., varicella pneumonia suggested by lung opacities and crusted skin lesions seen in chickenpox). In mismatched pairs, the photograph depicts an alternative clinical finding that would not lead to the same unifying diagnosis (e.g. replacing crusted skin lesions with ordinary acne). Synthetic photograph generation allows clinical findings to be varied while the person's features and photographic setting stay consistent, reducing confounding. In an investigation of three frontier models, we found that models hallucinated clinically significant photograph findings in 45-57.5% of mismatched pairs, up from 25-37.5% for the same photographs shown alone. When faced with mismatched pairs, models tended to ignore mismatches and instead hallucinated findings that better matched the radiograph (e.g. misreading acne as crusted skin lesions to justify diagnosing chickenpox). Trick-or-Treat both demonstrates concerning error pattern in frontier models' medical reasoning and provides a proof of concept for the use of synthetic images in adversarial multimodal evaluation.

Figures & tables

Explore similar work

CardsList
  1. Text Over Image: Auditing Multimodal Robustness in Synthetic Medical Image Detection

    Jun 24, 2026Ching-Hao Chiu, Hao-Wei Chung, Gelei Xu +7Multimodal RobustnessMedical VLMs

  2. Medical thinking with multiple images

    Apr 14, 2026Zonghai Yao, Benlu Wang, Yifan Zhang +8Medical Visual Question AnsweringMultimodal Reasoning