ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding

Oishi Banerjee; Sung Eun Kim; Alexandra N. Willauer; Julius M. Kernbach; Abeer Rihan Alomaish; Reema Abdulwahab S. Alghamdi; Hassan Rayhan Alomaish; Mohammed Baharoon

cs.CVMar 19, 2026

ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding

Authors: Oishi Banerjee, Sung Eun Kim, Alexandra N. Willauer, Julius M. Kernbach, Abeer Rihan Alomaish, Reema Abdulwahab S. Alghamdi, Hassan Rayhan Alomaish, Mohammed Baharoon,

Organizations: Department of Biomedical Informatics, Harvard Medical School, Boston, MA · National Strategic Technology Research Institute, Seoul National University Hospital, Seoul, Republic of Korea · Department of Medicine, Division of Gastroenterology, Massachusetts General Hospital, Boston, MA · Department of Neuroradiology, Heidelberg University Hospital, Heidelberg, Germany ·

Abstract

Everyday photographs taken with ordinary cameras are already widely used in telemedicine and other online health conversations, yet no comprehensive benchmark evaluates whether vision-language models can interpret their medical content. Analyzing these images requires both fine-grained natural image understanding and domain-specific medical reasoning, a combination that challenges both general-purpose and specialized models. We introduce ReXInTheWild, a benchmark of 955 clinician-verified multiple-choice questions spanning seven clinical topics across 484 photographs sourced from the biomedical literature. When evaluated on ReXInTheWild, leading multimodal large language models show substantial performance variation: Gemini-3 achieves 78% accuracy, followed by Claude Opus 4.5 (72%) and GPT-5 (68%), while the medical specialist model MedGemma reaches only 37%. A systematic error analysis also reveals four categories of common errors, ranging from low-level geometric errors to high-level reasoning failures and requiring different mitigation strategies. ReXInTheWild provides a challenging, clinically grounded benchmark at the intersection of natural image understanding and medical reasoning. The dataset is available on HuggingFace.

Figures & tables

Panel of Figure 1 — Figure 1: Medical photographs pose several unique challenges at the intersection of medical reasoning and natural image interpretation. Left : For example, medical models face a substantial domain shift from specialized imaging modalities [ 16 ] to photographs. In addition, photographs taken outside the clinic are likelier to have technical flaws that obscure fine-grained medical details (e.g. the abnormal lower-lid eyelashes at the bottom of “Technical Variation"). Right: ReXInTheWild tests whether models can assess the medical content in diverse natural images, including both healthy subjects and patients with medical conditions.

Figure 2: The ReXInTheWild benchmark construction pipeline. Stage A: “Image Selection" filters PubMed Central images by caption and visual content to identify suitable medical photographs. Stage B: “Question Generation" uses GPT-5 to produce candidate questions and applies automated editing to remove leading cues and refine distractors. Stage C: “Automatic and Expert Verification" automatically scores clarity and difficulty and routes high-scoring questions for a two-stage expert review process.

Figure 3: Large general-purpose models, especially Gemini-3, outperformed MedGemma, a smaller medical MLLM. Performance varied across medical categories, though model rankings generally remained consistent within each category.

Figure 4: Failure modes range from low-level errors in basic image interpretation to high-level failures in causal reasoning. Notably, models made errors when describing normal physiology (top), not only when assessing medical abnormalities.

Model	Geometric	Localization	Characterization	Causality	1+ Types	2+ Types
Claude	0.28	0.24	0.38	0.16	0.98	0.08
Gemini	0.34	0.36	0.36	0.18	1.00	0.22
GPT	0.28	0.40	0.40	0.10	1.00	0.18
Mean	0.30	0.33	0.38	0.15	0.99	0.16

Table 1: Error type fractions by model and averaged across models. Our categories provide broad coverage, with 99% of errors belonging to at least one category. They also capture relatively distinct failure modes, with only 17% of errors falling into multiple categories. Errors of all levels occurred frequently, indicating that current models still make basic geometric mistakes as well as sophisticated medical errors.

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 475 papers

Medical thinking with multiple images
Apr 14, 2026Zonghai Yao, Benlu Wang, Yifan Zhang +8Medical VQA Multimodal Reasoning
Manning College of Information and Computer Sciences, UMass Amherst · Center for Healthcare Organization and Implementation Research, VA Bedford Health Care · Department of Computer Science, Yale University +2
OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models
Jun 10, 2026Negin Baghbanzadeh, Pritam Sarkar, Michael Colacci +6Medical VQA Medical VLMs
York University · Vector Institute · University of British Columbia +5
Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models
Jul 24, 2026Bannapol Limanond, Masanori Suganuma, Takayuki OkataniMedical VLMs Multimodal Model Evaluation

Apr 14, 2026J/K move · Enter open · S save

Medical thinking with multiple images

Zonghai Yao, Benlu Wang, Yifan Zhang, Junda Wang+7

Manning College of Information and Computer Sciences, UMass Amherst · Center for Healthcare Organization and Implementation Research, VA Bedford Health Care · Department of Computer Science, Yale University+2

Medical VQA Multimodal Reasoning Multi-Image Reasoning Medical Image Benchmarks Medical Image Grounding

Open paper AskPDF arXiv

Large language models perform well on many medical QA benchmarks, but real clinical reasoning often requires integrating evidence across multiple images rather than interpreting a single view. We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images, where models must interpret each image, combine cross-view evidence, and answer diagnostic questions with intermediate supervision and step-level evaluation. The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case, substantially denser than prior work, whose expert-level benchmarks use at most 1.43 images per case. On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy, while GPT-5-mini and GPT-5-nano reach 39.7% and 30.8%. Strong open-source models lag behind, led by Qwen3.5-397B-A17B at 52.2% and Qwen3.5-27B at 50.6%. Further analysis identifies grounded multi-image reasoning as the main bottleneck: models often fail to extract, align, and compose evidence across views before higher-level inference can help. Providing expert single-image cues and cross-image summaries improves performance, whereas replacing them with self-generated intermediates reduces accuracy. Step-level analysis shows that over 70% of errors arise from image reading and cross-view integration. Scaling results further show that additional inference-time computation helps only when visual grounding is already reliable; when early evidence extraction is weak, longer reasoning yields limited or unstable gains and can amplify misread cues. These results suggest that the key challenge is not reasoning length alone, but reliable mechanisms for grounding, aligning, and composing distributed evidence across real-world multimodal clinical inputs.

Abstract

Medical VLMs

Explore similar work

Medical thinking with multiple images

OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models

Medical-Checklist: Assessing the Comprehension of Medical Images by Multimodal Models