Medical Image Benchmarks

Follow this topic Ask about this topic

Momentum

20 papers in the last four weeks, up 25% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 237

Apr 16, 2026eess.IV

CTSCAN: Evaluation Leakage in Chest CT Segmentation and a Reproducible Patient-Disjoint Benchmark

Reported chest CT segmentation performance can be strongly inflated when train and test partitions mix slices from the same study. We present CTSCAN, a reproducible multi-source chest CT benchmark and research stack designed to measure what survives under patient-disjoint evaluation. The current four-class artifact aggregates 89 cases from PleThora, MedSeg SIRM, and LongCIU, and we show that the original slice-PNG workflow induces near-complete case reuse across train, validation, and test. Using the playground environment, we run a multi-seed protocol sweep with the same FPN plus EfficientNet-B0 control configuration under slice-mixed and case-disjoint evaluation. Across 3 seeds and 12 epochs per seed, the slice-mixed protocol reaches 0.6665 foreground Dice and 0.5031 foreground IoU, whereas the case-disjoint protocol reaches 0.2066 Dice and 0.1181 IoU. Removing patient reuse therefore reduces foreground Dice by 0.4599 absolute (69.00% relative) and foreground IoU by 0.3850 absolute (76.52% relative). CTSCAN packages the corrected benchmark with deterministic split manifests, explicit weak-supervision controls, a scripted multi-seed protocol sweep, and reproducible figure generation, providing a reusable basis for patient-disjoint chest CT evaluation.

Anton Ivchenko

Data Leakage Image Segmentation Medical Image Segmentation Computed Tomography Medical Image Benchmarks

Apr 16, 2026cs.CV

CXR-LT 2026 Challenge: Multi-Center Long-Tailed and Zero Shot Chest X-ray Classification

Chest X-ray (CXR) interpretation is hindered by the long-tailed distribution of pathologies and the open-world nature of clinical environments. Existing benchmarks often rely on closed-set classes from a single institution, failing to capture the prevalence of rare diseases or the appearance of novel findings. To address this, we present the CXR-LT challenge. The first event, CXR-LT 2023, established a large-scale benchmark for long-tailed multi-label CXR classification and identified key challenges in rare disease recognition. CXR-LT 2024 further expanded the label space and introduced a zero-shot task to study generalization to unseen findings. Building on the success of CXR-LT 2023 and 2024, this third iteration of the benchmark introduces a multi-center dataset comprising over 145,000 images from PadChest and NIH Chest X-ray datasets. Additionally, all development and test sets in CXR-LT 2026 are annotated by radiologists, providing a more reliable and clinically grounded evaluation than report-derived labels. The challenge defines two core tasks this year: (1) Robust Multi-Label Classification on 30 known classes and (2) Open-World Generalization to 6 unseen (out-of-distribution) rare disease classes. This paper summarizes the overview of the CXR-LT 2026 challenge. We describe the data collection and annotation procedures, analyze solution strategies adopted by participating teams, and evaluate head-versus-tail performance, calibration, and cross-center generalization gaps. Our results show that vision-language foundation models improve both in-distribution and zero-shot performance, but detecting rare findings under multi-center shift remains challenging. Our study provides a foundation for developing and evaluating AI systems in realistic long-tailed and open-world clinical conditions.

Hexin Dong, Yi Lin, Pengyu Zhou +25

Zero-Shot Learning Long-Tailed Classification Chest X-Ray Classification Domain Generalization Open-Set Recognition Medical Image Benchmarks

Apr 16, 2026cs.CV

MetaDent: Labeling Clinical Images for Vision-Language Models in Dentistry

Vision-Language Models (VLMs) have demonstrated significant potential in medical image analysis, yet their application in intraoral photography remains largely underexplored due to the lack of fine-grained, annotated datasets and comprehensive benchmarks. To address this, we present MetaDent, a comprehensive resource that includes (1) a novel and large-scale dentistry image dataset collected from clinical, public, and web sources; (2) a semi-structured annotation framework designed to capture the hierarchical and clinically nuanced nature of dental photography; and (3) comprehensive benchmark suites for evaluating state-of-the-art VLMs on clinical image understanding. Our labeling approach combines a high-level image summary with point-by-point, free-text descriptions of abnormalities. This method enables rich, scalable, and task-agnostic representations. We curated 60,669 dental images from diverse sources and annotated a representative subset of 2,588 images using this meta-labeling scheme. Leveraging Large Language Models (LLMs), we derive standardized benchmarks: approximately 15K Visual Question Answering (VQA) pairs and an 18-class multi-label classification dataset, which we validated with human review and error analysis to justify that the LLM-driven transition reliably preserves fidelity and semantic accuracy. We then evaluate state-of-the-art VLMs across VQA, classification, and image captioning tasks. Quantitative results reveal that even the most advanced models struggle with a fine-grained understanding of intraoral scenes, achieving moderate accuracy and producing inconsistent or incomplete descriptions in image captioning. We publicly release our dataset, annotations, and tools to foster reproducible research and accelerate the development of vision-language systems for dental applications.

Meng-Xun Li, Wen-Hui Deng, Zhi-Xing Wu +6

Department of Prosthodontics, State Key Laboratory of Oral & Maxillofacial Reconstruction and Regeneration, Key Laboratory of Oral Biomedicine, Ministry of Education, Hubei Key Laboratory of Stomatology, School and Hospital of Stomatology, Wuhan University, Wuhan, Hubei, China · School of Artificial Intelligence, Wuhan University, Wuhan, Hubei, China · School of Computer Science, Wuhan University, Wuhan, Hubei, China +2

VLM Evaluation Medical Image Analysis LLM-Assisted Annotation Medical VQA Medical Image Benchmarks

Apr 16, 2026cs.AI

Rethinking Patient Education as Multi-turn Multi-modal Interaction

Most medical multimodal benchmarks focus on static tasks such as image question answering, report generation, and plain-language rewriting. Patient education is more demanding: systems must identify relevant evidence across images, show patients where to look, explain findings in accessible language, and handle confusion or distress. Yet most patient education work remains text-only, even though combined image-and-text explanations may better support understanding. We introduce MedImageEdu, a benchmark for multi-turn, evidence-grounded radiology patient education. Each case provides a radiology report with report text and case images. A DoctorAgent interacts with a PatientAgent, conditioned on a hidden profile that captures factors such as education level, health literacy, and personality. When a patient question would benefit from visual support, the DoctorAgent can issue drawing instructions grounded in the report, case images, and the current question to a benchmark-provided drawing tool. The tool returns image(s), after which the DoctorAgent produces a final multimodal response consisting of the image(s) and a grounded plain-language explanation. MedImageEdu contains 150 cases from three sources and evaluates both the consultation process and the final multimodal response along five dimensions: Consultation, Safety and Scope, Language Quality, Drawing Quality, and Image-Text Response Quality. Across representative open- and closed-source vision-language model agents, we find three consistent gaps: fluent language often outpaces faithful visual grounding, safety is the weakest dimension across disease categories, and emotionally tense interactions are harder than low education or low health literacy. MedImageEdu provides a controlled testbed for assessing whether multimodal agents can teach from evidence rather than merely answer from text.

Zonghai Yao, Zhipeng Tang, Chengtao Lin +5

1VA Bedford Health Care · 2UMass Amherst · 4Yale University +3

Multimodal Grounding Healthcare Multimodal Agents Multimodal Model Evaluation Medical Image Benchmarks

Apr 13, 2026cs.MA

VERITAS: A Multi-Agent Co-Scientist for Verifiable Image-Derived Hypothesis Testing

Scientific research based on multimodal clinical data (including medical imaging) requires coordinating clinical, radiological, programming, and biostatistical expertise, a fragmented process that bottlenecks discovery. We present VERITAS (Verifiable Epistemic Reasoning for Image-Derived Hypothesis Testing via Agentic Systems), a clinical co-scientist: a multi-agent system that autonomously tests natural-language hypotheses and produces a fully auditable evidence trail, tracing every conclusion through executable outputs from analysis plan to segmentation masks to statistical code to final verdict. Unlike prior AI-scientist systems, which mainly operate on tabular or text data, VERITAS grounds autonomous discovery directly in medical images. It decomposes the workflow into four phases handled by role-specialized agents, and introduces an epistemic evidence label framework that mechanically classifies outcomes as Supported, Refuted, Underpowered, or Invalid by jointly evaluating significance, effect direction, and study power. This distinction is critical in medical imaging, where non-significant results often reflect insufficient sample size rather than absent effects. We construct a tiered benchmark of 64 hypotheses spanning six complexity levels across cardiac and brain glioma MRI datasets. VERITAS reaches 81.4% verdict accuracy with frontier models and 71.2% with locally-hosted open-weight models (8-30B), outperforming all single-model baselines in both classes. It also produces the highest rate of independently verifiable statistical outputs (86.6%), so even its failures remain diagnosable through artifact inspection. Structured multi-agent decomposition thus substitutes for model scale while preserving the verifiability that scientific discovery demands. We release code, hypothesis bank, and evaluation pipeline at https://github.com/LucZot/veritas.

Lucas Stoffl, Benedikt Wiestler, Johannes C. Paetzold

Department of Radiology, Weill Cornell Medicine, New York, USA · Cornell Tech, New York, USA · TUM University Hospital, Munich, Germany +2

AI Agents for Scientific Discovery Healthcare Multi-Agent Systems Medical Image Benchmarks Scientific Claim Verification

Apr 7, 2026cs.CL

MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models

Medical Vision-Language Models (Med-VLMs) have achieved expert-level proficiency in interpreting diagnostic imaging. However, current models are predominantly trained on professional literature, limiting their ability to communicate findings in the lay register required for patient-centered care. While text-centric research has actively developed resources for simplifying medical jargon, there is a critical absence of large-scale multimodal benchmarks designed to facilitate lay-accessible medical image understanding. To bridge this resource gap, we introduce MedLayBench-V, the first large-scale multimodal benchmark dedicated to expert-lay semantic alignment. Unlike naive simplification approaches that risk hallucination, our dataset is constructed via a Structured Concept-Grounded Refinement (SCGR) pipeline. This method enforces strict semantic equivalence by integrating Unified Medical Language System (UMLS) Concept Unique Identifiers (CUIs) with micro-level entity constraints. MedLayBench-V provides a verified foundation for training and evaluating next-generation Med-VLMs capable of bridging the communication divide between clinical experts and patients.

Han Jang, Junhyeok Lee, Heeseong Eum +1

Seoul National University · Department of Radiology, Seoul National University Hospital · The Advanced Imaging and Computational Neuroimaging (AICON) Laboratory +2

Healthcare Medical VLMs Vision-Language Alignment Clinical NLP Medical Image Benchmarks

Mar 26, 2026eess.IV

Colon-Bench: An Agentic Workflow for Scalable Dense Lesion Annotation in Full-Procedure Colonoscopy Videos

Early screening via colonoscopy is critical for colon cancer prevention, yet developing robust AI systems for this domain is hindered by the lack of densely annotated, long-sequence video datasets. Existing datasets predominantly focus on single-class polyp detection and lack the rich spatial, temporal, and linguistic annotations required to evaluate modern Multimodal Large Language Models (MLLMs). To address this critical gap, we introduce Colon-Bench, generated via a novel multi-stage agentic workflow. Our pipeline seamlessly integrates temporal proposals, bounding-box tracking, AI-driven visual confirmation, and human-in-the-loop review to scalably annotate full-procedure videos. The resulting verified benchmark is unprecedented in scope, encompassing 528 videos, 14 distinct lesion categories (including polyps, ulcers, and bleeding), over 300,000 bounding boxes, 213,000 segmentation masks, and 133,000 words of clinical descriptions. We utilize Colon-Bench to rigorously evaluate state-of-the-art MLLMs across lesion classification, Open-Vocabulary Video Object Segmentation (OV-VOS), and video Visual Question Answering (VQA). The MLLM results demonstrate surprisingly high localization performance in medical domains compared to SAM-3. Finally, we analyze common VQA errors from MLLMs to introduce a novel "colon-skill" prompting strategy, improving zero-shot MLLM performance by up to 9.7% across most MLLMs. The dataset and the code are available at https://abdullahamdi.com/colon-bench .

Abdullah Hamdi, Changchun Yang, Xin Gao

King Abdullah University of Science and Technology (KAUST)

Video Object Segmentation Medical Image Analysis Multimodal Model Evaluation Medical Image Benchmarks

Mar 15, 2026cs.CV

Medical Image Spatial Grounding with Semantic Sampling

Vision language models (VLMs) have shown significant promise in visual grounding for images as well as videos. In medical imaging research, VLMs represent a bridge between object detection and segmentation, and report understanding and generation. However, spatial grounding of anatomical structures in the three-dimensional space of medical images poses many unique challenges. In this study, we examine image modalities, slice directions, and coordinate systems as differentiating factors for vision components of VLMs, and the use of anatomical, directional, and relational terminology as factors for the language components. We then demonstrate that visual and textual prompting systems such as labels, bounding boxes, and mask overlays have varying effects on the spatial grounding ability of VLMs. To enable measurement and reproducibility, we introduce MIS-Ground, a benchmark that comprehensively tests a VLM for vulnerabilities against specific modes of Medical Image Spatial Grounding. We release MIS-Ground to the public at github.com/asy51/mis-ground. In addition, we present MIS-SemSam, a low-cost, inference-time, and model-agnostic optimization of VLMs that improves their spatial grounding ability with the use of Semantic Sampling. We find that MIS-SemSam improves the accuracy of Qwen3-VL-32B on MIS-Ground by 13.06%.

Andrew Seohwan Yu, Mohsen Hariri, Kunio Nakamura +3

Case Western Reserve University, Cleveland OH 44106, USA · Cleveland Clinic, Cleveland OH 44106, USA

Medical VLMs Medical Image Grounding Medical Image Benchmarks

Mar 14, 2026cs.CV

Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space

Visual spatial intelligence is critical for medical image interpretation, yet remains largely unexplored in Multimodal Large Language Models (MLLMs) for 3D imaging. This gap persists due to a systemic lack of datasets featuring structured 3D spatial annotations beyond basic labels. In this study, we introduce an agentic pipeline that autonomously synthesizes spatial visual question-answering (VQA) data by orchestrating computational tools such as volume estimation and bounding boxes extraction with multi-agent collaboration and expert radiologist validation. We present SpatialMed, the first comprehensive benchmark for evaluating 3D spatial intelligence in medical MLLMs, comprising 31,253 question-answer pairs across multiple organs and tumor types. Our evaluations on 24 state-of-the-art MLLMs and extensive analyses reveal that current models lack robust spatial reasoning capabilities for medical imaging.

Quoc-Huy Trinh, Xi Ding, Yang Liu +7

3D Spatial Reasoning Multimodal Large Language Models Medical VQA Medical Image Benchmarks

Mar 13, 2026cs.CV

GP-VM $\times$ SMA: Benchmarking General-Purpose Vision Models and Specialized Architectures for 2D Medical Image Segmentation

Medical image segmentation (MIS) is a fundamental component of computer-assisted diagnosis and clinical decision support. Over the past decade, numerous architectures specifically tailored to medical imaging have emerged to address domain-specific challenges such as low contrast, small anatomical structures, and limited annotated data. In parallel, rapid progress in computer vision has produced highly capable general-purpose vision models (GP-VMs) originally designed for natural images. Despite their strong performance on standard vision benchmarks, their effectiveness for MIS remains insufficiently understood. In this controlled empirical study, we examine whether specialized medical segmentation architectures (SMAs) provide systematic advantages over modern GP-VMs for 2D MIS. We compare eleven SMAs and GP-VMs using a unified training and evaluation protocol across three heterogeneous datasets, each covering different 2D imaging modalities, class structures, and data characteristics. Beyond segmentation performance, we employ linear mixed-effects models (LMMs) for statistical assessment and qualitative Grad-CAM visualizations to investigate explainability (XAI)-related model behavior. Our results imply that, for the analyzed settings, GP-VMs achieve performance comparable to that of specialized MIS models. Moreover, XAI analyses indicate that GP-VMs are capable of identifying clinically relevant structures despite the absence of explicit domain-specific architectural priors. These findings suggest that GP-VMs can represent a viable alternative to domain-specific methods for 2D MIS, highlighting the importance of informed model selection. All code and resources are available at GitHub.

Vanessa Borst, Anna Riedmann, Samuel Kounev

Julius-Maximilians-University Würzburg, 97070 Würzburg, Germany

Image Segmentation Vision Foundation Models Medical Image Segmentation Medical Image Benchmarks

Mar 3, 2026cs.CV

Designing UNICORN: a Unified Benchmark for Imaging in Computational Pathology, Radiology, and Natural Language

Foundation models are changing the way we develop medical artificial intelligence. By learning broadly generalizable features across diverse data modalities, a single model can be rapidly adapted to address multiple modalities and tasks with minimal supervision. This potential comes with the urgent need to reliably benchmark, understand and compare the performance and clinical impact of foundation models across data modalities and clinical tasks. We introduce UNICORN, a fundamentally new benchmarking concept for medical foundation models. UNICORN brings four main contributions to medical artificial intelligence. First, a framework that enables a one-to-many benchmarking approach, where a single foundation model is tested across multiple tasks and data modalities. Here, we populate it with 20 tasks across radiology, pathology, and clinical text, covering classification, detection, segmentation, regression, and vision-language generation. Second, a publicly available evaluation platform that implements, for the first time, a two-step approach to run foundation models for data encoding followed by custom task-specific adaptation via few-shot learning and linear probing mechanisms. Third, we create a meta-model that combines state-of-the-art foundation models in pathology, radiology and language with novel task-specific adapters that address all UNICORN tasks, which we refer to as Unicorn Model-0 (UM-0). Finally, we design a novel UNICORN score to benchmark and compare model performance across all tasks. We present the results of UM-0 using sequestered test data from over 2,400 patients, 3,700 vision cases, and 2,400 clinical reports from 17 institutions across eight countries, spanning eight anatomical regions and four imaging modalities. Data, baselines, and evaluation platform are publicly accessible at unicorn.grand-challenge.org.

Michelle Stegeman, Lena Philipp, Fennie van der Graaf +20

and on behalf of the UNICORN consortium · Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands · Department of Medical Imaging, Radboud University Medical Center, Nijmegen, The Netherlands +2

Computational Pathology Medical Imaging Foundation Models Healthcare Multimodal Foundation Models Multimodal Model Evaluation Medical Image Benchmarks

Mar 2, 2026cs.CV

CTForensics: A Comprehensive Dataset and Method for AI-Generated CT Image Detection

Recent advances in generative AI have made synthetic Computed Tomography (CT) images increasingly realistic, enabling promising applications in medical data augmentation while raising serious concerns about clinical safety and data trustworthiness. Detecting AI-generated CT images remains challenging for two key reasons: existing benchmarks cover only limited generation sources, and many detectors are adapted from natural-image forensics without explicitly modeling CT-specific imaging properties. In this paper, we introduce CTForensics, a dataset for detecting AI-generated CT images. CTForensics contains 75,990 2D CT images, including a dedicated test benchmark of 29,990 balanced authentic and generated samples from ten representative CT generative models spanning GAN-based and diffusion-based paradigms. We further propose the Enhanced Spatial-Frequency CT Forgery Detector (ESF-CTFD), a CT-oriented CNN framework built around a Wavelet-Enhanced Central Stem, Multi-Scale Spatial Aggregation, and a Frequency-Aware Prediction Block. The Wavelet-Enhanced Central Stem enhances local intensity correlations and high-frequency residuals, Multi-Scale Spatial Aggregation aligns anatomical features across resolutions with lightweight residual units, and the Frequency-Aware Prediction Block models global spectral artifacts. Extensive experiments on CTForensics show that ESF-CTFD achieves 96.01% mAcc and 99.96% mAP, outperforming existing methods and maintaining strong robustness under realistic perturbations with only a 0.99% average drop. Codes will be available at https://github.com/liyih/CTForensics.

Yiheng Li, Zichang Tan, Guoqing Xu +3

School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Sangfor Technologies Inc., Shenzhen, China +1

Image Forgery Detection AI-Generated Image Detection Computed Tomography Medical Image Benchmarks

Feb 1, 2026cs.CV

MedAD-R1: Consistency-Reinforced Policy Optimization for Interpretable Medical Anomaly Detection

Medical Anomaly Detection (MedAD) offers a promising direction for medical image analysis with Large Multimodal Models (LMMs). However, progress is limited by fragmented datasets and the tendency of Supervised Fine-Tuning (SFT) to learn superficial image-text correlations rather than verifiable diagnostic reasoning. Consequently, current models often generate fluent explanations that are either insufficiently grounded in the image or inconsistent with their final answers, limiting their reliability in high-stakes medical applications. To address these issues, we introduce MedAD-38K, a large-scale, multimodal, and multicenter benchmark containing structured Visual Question Answering pairs and quality-controlled diagnostic Chain-of-Thought annotations across five core MedAD tasks. Based on this benchmark, we propose a two-stage framework. Cognitive Injection first uses SFT to inject domain-specific medical knowledge and establish a structured think-then-answer format. Consistency Group Relative Policy Optimization (Con-GRPO) then employs an Evidence-Aware Consistency Reward to reinforce reasoning that remains grounded in the image and logically supports the final answer. The resulting MedAD-R1 achieves state-of-the-art performance on MedAD-38K and consistently outperforms all evaluated baselines across five source-disjoint external datasets spanning diverse imaging modalities. Beyond accuracy, it achieves higher reasoning-answer consistency and visual-grounding scores across multiple backbones. With only 0.8B parameters, MedAD-R1 offers practical potential for resource-constrained deployment. These results demonstrate the generality of evidence-aware consistency optimization for interpretable MedAD. Project resources are available at https://github.com/zhtstar/MedAD-R1.

Haitao Zhang, Yingying Wang, Jiaxiang Wang +5

School of Informatics, Xiamen University · Institute of Artificial Intelligence, Xiamen University · Zhejiang Expressway Co., Ltd. +3

Healthcare Group Relative Policy Optimization Medical Image Anomaly Detection Multimodal Large Language Models Medical VQA Multimodal Anomaly Detection+2

Jan 13, 2026eess.IV

M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding

Chain-of-Thought (CoT) reasoning has proven effective in enhancing large language models by encouraging step-by-step intermediate reasoning, and recent advances have extended this paradigm to Multimodal Large Language Models (MLLMs). In the medical domain, where diagnostic decisions depend on nuanced visual cues and sequential reasoning, CoT aligns naturally with clinical thinking processes. However, current benchmarks for medical image understanding generally focus on the final answer while ignoring the reasoning path. Such opaque reasoning processes lack reliable bases for judgment, making it difficult to assist doctors in diagnosis. To address this gap, we introduce a new M3CoTBench benchmark specifically designed to evaluate the correctness, efficiency, impact, and consistency of CoT reasoning in medical image understanding. M3CoTBench features 1) a diverse, multi-level difficulty dataset covering 24 examination types, 2) 13 varying-difficulty tasks, 3) a suite of CoT-specific evaluation metrics (correctness, efficiency, impact, and consistency) tailored to clinical reasoning, and 4) a performance analysis of multiple MLLMs. M3CoTBench systematically evaluates CoT reasoning across diverse medical imaging tasks, revealing current limitations of MLLMs in generating reliable and clinically interpretable reasoning, and aims to foster the development of transparent, trustworthy, and diagnostically accurate AI systems for healthcare. Project page at https://juntaojianggavin.github.io/projects/M3CoTBench/.

Juntao Jiang, Jiangning Zhang, Yali Bi +7

1Zhejiang University · University of Science and Technology of China · 3East China Normal University +2

Medical VLMs Multimodal CoT Reasoning Medical Reasoning Multimodal Model Evaluation Medical Image Benchmarks

Dec 26, 2025eess.IV

Region-Specific Calibration Achieves Excellent Inter-Device Reliability for Smartphone Dermatology: A Multi-Device Benchmark on Korean Facial Skin

Background: Smartphone-based dermatology requires inter-device colorimetric reliability that holds across calibration regimes, yet quantitative multi-device benchmarks remain scarce. Materials and Methods: We analyzed matched facial images from 965 Korean subjects captured by a digital single-lens reflex (DSLR) camera, a consumer tablet, and a consumer smartphone, and evaluated two calibration methods against the DSLR reference. The methods are standard global linear Color Correction Matrix (CCM) normalization and region-specific CCM trained per anatomical region, both applied in Commission Internationale de l'Eclairage Lab* (CIELAB) space. Results: Linear CCM reduced inter-device color differences by 61-74% and placed both Melanin Index (intraclass correlation coefficient [ICC] = 0.80) and Individual Typology Angle (ITA, ICC = 0.78) in the good reliability band. Region-specific CCM raised both indices into the excellent reliability band (MI ICC = 0.95, ITA ICC = 0.93), with anatomical region exceeding the source device as the largest pre-calibration variance contributor (analysis-of-variance

η^2 = 0.18

versus 0.12). Conclusion: Consumer-device skin colorimetry therefore achieves clinically useful inter-device reliability using standard calibration, with region-aware calibration the largest remaining source of improvement.

Sungwoo Kang, Jong-Kook Kim

Department of Electrical and Computer Engineering, Korea University, Seoul, Republic of Korea · School of Electrical Engineering, Korea University, Seoul, Republic of Korea

Medical Diagnosis Medical Image Benchmarks

Dec 3, 2025cs.CV

6 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Models

Vision-language models (VLMs) are increasingly integrated into clinical workflows. However, existing benchmarks primarily assess performance on common anatomical presentations and fail to capture the challenges posed by rare variants. To address this gap, we introduce AdversarialAnatomyBench, the first benchmark comprising naturally occurring rare anatomical variants across diverse imaging modalities and anatomical regions. We call such variants that violate learned priors about "typical" human anatomy natural adversarial anatomy. Benchmarking 25 state-of-the-art VLMs with AdversarialAnatomyBench yielded three key insights. First, when queried with basic medical perception tasks, mean accuracy dropped from 71% on typical to 28% on atypical anatomy. Even the best-performing models, GPT-5, Gemini 2.5 Pro, and Llama 4 Maverick, showed performance drops of 41-51%. Second, model errors closely mirrored expected anatomical biases. Third, neither model scaling nor interventions, including bias-aware prompting and test-time reasoning, resolved these issues. These findings highlight a critical limitation in current VLMs: their poor generalization to rare anatomical presentations. AdversarialAnatomyBench provides a foundation for systematically measuring and mitigating anatomical bias in multimodal medical artificial intelligence (AI) systems.

Leon Mayer, Piotr Kalinowski, Caroline Ebersbach +6

German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany. · Medical Faculty, Heidelberg University, Germany. · Faculty of Mathematics and Computer Science, Heidelberg University, Germany. +9

VLM Evaluation VLM Robustness Healthcare OOD Generalization Medical VLMs Medical Image Benchmarks

Nov 15, 2025cs.CV

MediRound: Multi-Round Entity-Level Reasoning Segmentation in Medical Images

Despite notable progress in text-guided medical image segmentation nowadays, these methods are limited to single-round dialogues and fail to support multi-round reasoning, which is important for medical education scenarios. In this work, we introduce Multi-Round Entity-Level Medical Reasoning Segmentation (MEMR-Seg), a new task that requires generating segmentation masks through multi-round queries with entity-level reasoning, helping learners progressively develop their understanding of medical knowledge. To support this task, we construct MR-MedSeg, a large-scale dataset of 177K multi-round medical segmentation dialogues, featuring entity-based reasoning across rounds. Furthermore, we propose MediRound, an effective baseline model designed for multi-round medical reasoning segmentation. To mitigate the inherent error propagation within the chain-like pipeline of multi-round segmentation, we introduce a lightweight yet effective Judgment & Correction Mechanism during model inference. Experimental results demonstrate that our method effectively addresses the MEMR-Seg task and outperforms conventional medical referring segmentation methods. The project is available at https://github.com/Edisonhimself/MediRound.

Qinyue Tong, Ziqian Lu, Jun Liu +3

Zhejiang University, Hangzhou, Zhejiang, China · Zhejiang Sci-Tech University, Hangzhou, Zhejiang, China · National University of Singapore, Singapore

Interactive Medical Image Segmentation Text-Guided Image Segmentation Multimodal Reasoning Medical Image Segmentation Reasoning Segmentation Medical Image Benchmarks

Oct 15, 2025cs.CV

Scaling Vision Transformers for Functional MRI with Flat Maps

We study the problem of training self-supervised foundation models for functional MRI. Our main contributions are: (1) we introduce a new model family (CortexMAE) trained using the masked autoencoder framework on 2.1K hours of open fMRI data, and (2) we release the first open evaluation suite (Brainmarks) for fMRI foundation models. Our core innovation is simple: we adapt the Vision Transformer to fMRI by first converting each 3D fMRI volume to a 2D map using a cortical flat map projection. We directly compare flat maps to both parcellation and volume-based representations. While each has its advantages, flat maps generally perform best. We perform the first systematic scaling analysis for fMRI and observe strict power law scaling, albeit with limits. Finally, we use Brainmarks to do controlled benchmark comparisons. On subject-level trait prediction, we report a challenging null result: no single model achieves clear state-of-the-art performance. Moreover, all models struggle to outperform a simple functional connectivity baseline. On cognitive state decoding, we observe more robust performance, and in this setting our CortexMAE family outperforms prior models by a large margin. Code, models, and datasets are available at https://github.com/MedARC-AI/CortexMAE and https://github.com/MedARC-AI/Brainmarks.

Connor Lane, Mihir Tripathy, Leema Krishna Murali +15

MedARC · Sophont · Baylor College of Medicine +3

Medical Imaging Foundation Models Masked Autoencoders fMRI Foundation Models Neuroimaging Medical Image Benchmarks

Oct 13, 2025cs.CV

Benchmarking Deep Learning Models for Laryngeal Cancer Staging Using the LaryngealCT Dataset

Laryngeal cancer imaging research lacks standardised public datasets to enable reproducible deep learning (DL) model development. We present LaryngealCT, a curated benchmark of 1,029 computed tomography (CT) scans aggregated from six collections from The Cancer Imaging Archive (TCIA). Uniform 1 mm isotropic volumes of interest encompassing the larynx were extracted using a weakly supervised parameter search framework validated by clinical experts. Six 3D DL architectures (custom 3D CNN, ResNet18,50,101, DenseNet121 and MedicalNet-pretrained ResNet50) were benchmarked on (i) early (Tis,T1,T2) vs. advanced (T3,T4) and (ii) T4 vs. non-T4 classification tasks. On the independent test set, the 3D CNN achieved the strongest overall performance across global and per-class metrics (Accuracy 0.854, F1-macro 0.841) in early vs. advanced classification. In the T4 task, AU-ROC values exceeded 0.82 for most models, but sensitivity for T4 disease remained limited (less than or equal to 0.412), with ResNet101 showing the most promising calibrated T4 recall (0.706. Model explainability assessed using GradCAMpp with thyroid cartilage overlays for T4 classification task revealed anatomically plausible peri-cartilage activations, although spatial overlap was modest. Through open-source data, pretrained models, and integrated explainability tools, LaryngealCT offers a reproducible foundation for AI-driven research to support future clinical decision-making in laryngeal oncology.

Nivea Roy, Son Tran, Atul Sajjanhar +4

School of Information Technology, Deakin University, Burwood, Australia · Department of Head and Neck Surgery, Kasturba Medical College, Manipal Academy of Higher Education, Manipal, India · Department of Radiodiagnosis and Imaging, Kasturba Medical College, Manipal Academy of Higher Education, Manipal, India +1

Medical Image Classification Oncology Medical Image Benchmarks

Sep 24, 2025cs.LG

Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context

Public datasets of Chest X-Rays (CXRs) have long been a popular benchmark for developing machine learning (ML) computer vision models in healthcare. However, the reported strong average-case performance of these models do not necessarily reflect their actual utility when used in heterogeneous clinical settings, potentially masking weaker performance in medically significant scenarios. In this work we use clinical context to provide a more holistic evaluation of models for CXR diagnosis. In particular, we use discharge summaries, recorded prior to each CXR, to derive a ``pre-CXR'' probability of each CXR label, as a proxy for existing contextual knowledge available to clinicians when interpreting CXRs. We use this measure to probe model performance along two dimensions: First, using a stratified analysis, we show that models tend to have lower performance (as measured by AUROC and other metrics) among individuals with higher pre-CXR probability. Second, by controlling for pre-CXR probability via matching and re-weighting, we demonstrate that performance degrades when the correlation is broken between prior context and the current CXR label, suggesting that model performance is highly sensitive to the underlying distribution of clinical context. Specifically, cases with high pre-test probabilities present a fundamentally more difficult visual classification task, highlighting a gap in clinical utility when models are applied to high-risk cohorts.

Andrew Wang, Jiashuo Zhang, Michael Oberst

Brown University · Johns Hopkins University

Chest X-Ray Classification Distribution Shift Healthcare Medical Image Benchmarks

Aug 10, 2025cs.AI

EndoCogniAgent: Closed-Loop Agentic Reasoning with Self-Consistency Validation for Endoscopic Diagnosis

Endoscopic diagnosis is an iterative process in which clinicians acquire, compare, and verify local visual evidence before reaching a conclusion. Current AI systems do not adequately support this process because fine-grained evidence acquisition and multi-step reasoning remain weakly coupled, complicating reconciliation of image-derived findings with their textual interpretations. This gives rise to two failure modes, hallucinated evidence and uncorrected error accumulation, that undermine diagnostic reliability. We propose EndoCogniAgent, a closed-loop agentic framework that formulates endoscopic diagnosis as a controlled state update process for integrating complementary visual and textual evidence. At each reasoning round, a central planner selects an evidence acquisition action, specialized expert tools extract spatial and semantic observations as structured textual evidence, and a self-consistency validation mechanism examines this evidence along two dimensions, knowledge consistency against the input image and temporal consistency with prior validated findings, before updating the diagnostic state. Validated observations are admitted into the evolving state to condition subsequent planning, while insufficiently supported or conflicting findings are retained with corrective feedback that redirects the planner toward additional verification. We further introduce EndoAgentBench, a workflow-oriented benchmark comprising 6,132 question-answer pairs from 11 endoscopic datasets, to evaluate diagnostic agents across a comprehensive diagnostic chain, from fine-grained visual perception to high-level diagnostic reasoning. EndoCogniAgent achieves 85.23% overall accuracy on perception tasks and 71.13% clinical acceptance rate on reasoning tasks. Blinded clinician evaluation further shows consistent improvements in diagnostic response quality over the evaluated baselines.

Yi Tang, Kai-Ni Wang, Liang-Peng Pu +3

School of Biological Science and Medical Engineering, Southeast University, Nanjing 211189, China · Jiangsu Key Laboratory of Biomaterials and Devices, Southeast University, Nanjing 211189, China · State Key Laboratory of Digital Medical Engineering, Southeast University, Nanjing 211189, China +2

Endoscopy Medical Imaging Medical Diagnosis Agentic Reasoning Clinical Reasoning Medical Image Benchmarks+1

Jul 15, 2025cs.CV

Trexplorer Super: Topologically Correct Centerline Tree Tracking of Tubular Objects in CT Volumes

Tubular tree structures, such as blood vessels and airways, are essential in human anatomy and accurately tracking them while preserving their topology is crucial for various downstream tasks. Trexplorer is a recurrent model designed for centerline tracking in 3D medical images but it struggles with predicting duplicate branches and terminating tracking prematurely. To address these issues, we present Trexplorer Super, an enhanced version that notably improves performance through novel advancements. However, evaluating centerline tracking models is challenging due to the lack of public datasets. To enable thorough evaluation, we develop three centerline datasets, one synthetic and two real, each with increasing difficulty. Using these datasets, we conduct a comprehensive evaluation of existing state-of-the-art (SOTA) models and compare them with our approach. Trexplorer Super outperforms previous SOTA models on every dataset. Our results also highlight that strong performance on synthetic data does not necessarily translate to real datasets. The code and datasets are available at https://github.com/RomStriker/Trexplorer-Super.

Roman Naeem, David Hagerman, Jennifer Alvén +2

Chalmers University of Technology, 412 96 Gothenburg, Sweden

Medical Image Analysis 3D Medical Imaging Medical Image Benchmarks

Dec 20, 2024cs.CV

SegCol Challenge: Semantic Segmentation for Tools and Fold Edges in Colonoscopy data

Improving the reliability and completeness of colonoscopic inspection is critical for reducing missed lesions and improving colorectal cancer prevention. Reliable scene understanding is essential for navigation, reconstruction, and assessment of inspection completeness. Anatomical structures such as mucosal folds provide stable geometric cues for endoscope localization, while surgical instruments introduce dynamic occlusions that complicate visual interpretation. However, existing gastrointestinal endoscopy datasets largely focus on disease detection or artifact segmentation, leaving a gap in precise annotations of structural landmarks and instruments. We introduce SegCol, a dataset and benchmark for semantic segmentation of colon fold edges and surgical instruments derived from the EndoMapper dataset. SegCol provides manually annotated pixel-level masks for three instrument classes and thin fold-edge structures across temporally consistent image sequences. It forms the basis of the SegCol Challenge, organized as part of the EndoVis Challenge at MICCAI 2024, evaluating both supervised segmentation and annotation-efficient active learning. We further study segmentation metrics, including Dice, ODS/OIS, AP, and CLDice, under structural perturbations and different object geometries, and analyze participating methods, architectural choices, and active learning strategies. Our findings show that metric behavior strongly depends on target structure, highlighting the need for carefully selected evaluation protocols in endoscopic segmentation. Details are available at https://www.synapse.org/Synapse:syn54124209/wiki/626563, and code at https://github.com/surgical-vision/segcol_challenge.

Xinwei Ju, Rema Daher, Razvan Caramalau +7

UCL Hawkes Institute, University College London, London, UK · Department of Computer Science, University College London, London, UK · Medtronic, London, UK +3

Endoscopy Image Segmentation Active Learning Medical Image Segmentation Semantic Segmentation Medical Image Benchmarks

Dec 29, 2023cs.CV

The TopCoW Challenge -- Topology-Aware Circle of Willis Segmentation for CT and MR Angiography

The Circle of Willis (CoW) is an important network of arteries connecting major circulations of the brain. Its vascular architecture is believed to influence the risk, severity, and outcome of serious neurovascular diseases. However, characterizing the highly variable CoW anatomy remains a manual and time-consuming expert task. The CoW is commonly imaged by two non-invasive angiographic imaging modalities, magnetic resonance angiography (MRA) and computed tomography angiography (CTA), yet few datasets with annotated CoW anatomy exist, and there have been no established benchmarks for comparing CoW segmentation algorithms. We organized the TopCoW benchmark challenge alongside the release of an annotated CoW dataset with 125 paired MRA and CTA scans from the same patients. Voxel-level annotations for 13 vessel components were created using virtual reality technology and verified by clinical experts. Participants submitted algorithms for CoW segmentation and variant classification, which we evaluated on internal and external test sets comprising 226 scans from over five centers. The benchmark includes voxel-level segmentation, CoW component detection, CoW variant classification, and two clinical application tasks. We received submissions from over 250 participants across six continents. Top-performing teams achieved over 90% Dice scores for CoW segmentation, over 80% F1 scores for detecting key vessel components, and over 70% balanced accuracy in CoW variant classification across nearly all test sets. The best algorithms also supported clinically relevant downstream tasks by accurately classifying fetal-type posterior cerebral arteries and localizing aneurysms in relation to CoW anatomy. This benchmark demonstrated the utility of CoW segmentation algorithms for some downstream clinical applications with explainability.

Kaiyuan Yang, Fabio Musio, Yihui Ma +112

Image Segmentation Medical Image Segmentation Medical Image Benchmarks

Date pendingeess.IV

Subcortical Masks Generation in CT Images via Ensemble-Based Cross-Domain Label Transfer

Subcortical segmentation in neuroimages plays an important role in understanding brain anatomy and facilitating computer-aided diagnosis of traumatic brain injuries and neurodegenerative disorders. However, training accurate automatic models requires large amounts of labelled data. Despite the availability of publicly available subcortical segmentation datasets for Magnetic Resonance Imaging (MRI), a significant gap exists for Computed Tomography (CT). This paper proposes an automatic ensemble framework to generate high-quality subcortical segmentation labels for CT scans by leveraging existing MRI-based models. We introduce a robust ensembling pipeline to integrate them and apply it to unannotated paired MRI-CT data, resulting in a comprehensive CT subcortical segmentation dataset. Extensive experiments on multiple public datasets demonstrate the superior performance of our proposed framework. Furthermore, using our generated CT dataset, we train segmentation models that achieve improved performance on related segmentation tasks. To facilitate future research, we make our source code, generated dataset, and trained models publicly available at https://github.com/alxw0671/CT_Subcortical_Segmentation, marking the first open-source release for CT subcortical segmentation to the best of our knowledge.

Augustine X. W. Lee, Pak-Hei Yeung, Jagath C. Rajapakse

College of Computing and Data Science, Nanyang Technological University, Singapore

Cross-Modality Medical Image Segmentation Medical Image Segmentation Computed Tomography Medical Image Benchmarks

Date pendingcs.CV

MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation

Medical vision-language models (VLMs) are increasingly expected to support clinical workflows through diagnostic text and relevant medical images. However, current medical visual benchmarks have three recurring limitations: query-image misalignment from queries weakly grounded in specific image instances, closed-ended formats that narrow answer space and encourage shortcut-based prediction, and text-centric output paradigms that limit evaluation of image-generation and image-editing capabilities. We introduce MedGEN-Bench, a benchmark for open-ended multimodal medical generation. The evaluation snapshot reported in this manuscript comprises 6,422 image-text pairs reviewed by clinical experts and models, spanning 6 canonical imaging modalities, 15 clinical tasks, and 27 named subtasks. It includes 1,100 Visual Question Answering (VQA) pairs, 3,872 Image Editing pairs, and 1,450 Contextual Multimodal Generation pairs. MedGEN-Bench centers on contextual entanglement: dependence of an instruction's intended output on the particular image instance rather than on task wording alone. The benchmark operationalizes this concept through image-grounded instructions and extends evaluation to open-ended multimodal outputs. Its tiered evaluation protocol combines reproducible reference-based fidelity and similarity measures with a structured, checklist-guided assessment by a medical VLM judge. We evaluate 10 compositional frameworks, 2 dedicated image-editing models, 3 unified models, and 5 VLMs. The results show image-output tasks remain unsaturated. Contextual augmentation increases mean image-instruction similarity from 0.273 to 0.372, while a 1,000-case medical-expert audit shows moderate agreement between judge scores and clinician ratings. Source code and dataset are available at https://yangjj007.github.io/medgen.

Junjie Yang, Yuhao Yan, Gang Wu +12

South China University of Technology · Sun Yat-sen University · Hangzhou Dianzi University +10

Healthcare Multimodal Generation Image Generation Evaluation Image Editing Multimodal Model Evaluation Medical Image Benchmarks

Date pendingcs.CV

Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data?

As vision-language models (VLMs) are increasingly applied to medical AI, existing benchmarks mainly focus on evaluating their diagnostic ability over given medical images and texts, implicitly assuming that standardized medical images, texts, or question-answer pairs are already prepared. However, this assumption does not hold when we apply VLMs in real clinical practice, where medical data is often raw, heterogeneous, and fragmented across different sources. In this paper, we study this missing step, i.e., raw medical data standardization. Specifically, models are given raw dataset folders and evaluated on their ability to identify source formats, convert raw medical images into VLM-compatible visual inputs, extract relevant textual information, and organize the results into structured image-text pairs. To construct this Medical Data Standardization Benchmark (MDS-Bench), we manually annotate 1,939 raw medical data standardization tasks covering diverse clinical practice, radiology modalities, annotation formats, and directory layouts. Extensive experiments show that even the best performing VLM, i.e., Gemini 3 Flash, achieves only a 48.6% end-to-end success rate. Our research highlights raw medical data standardization as a critical bottleneck for medical AI diagnosis in real practice.

Xin Chen, Dongliang Xu, Cunhao Zhu +5

Shandong University · *Corresponding authors. · Stanford University

Medical VLMs Medical Image Benchmarks