Medical Image Benchmarks

Ask about this topic

Overview

Medical image benchmarks provide datasets, annotations, and shared protocols for comparing systems on tasks such as segmentation, diagnosis, image synthesis, and report generation across modalities including CT, MRI, X-rays, ultrasound, and pathology. They evaluate methods ranging from U-Net and nnU-Net models to vision transformers, foundation models, and multimodal language models, using measures such as Dice, AUROC, calibration, and expert review. A central challenge is whether strong results transfer across patients, hospitals, scanners, populations, and difficult cases; external testing, patient-level splits, and audits for leakage and shortcuts help expose gaps hidden by average scores. The field is moving toward workflow-based and evidence-grounded evaluation, but there is still tension between standardized, easily comparable scores and measures that reflect clinical safety, uncertainty, and real-world usefulness.

Latest papers

CardsList

339 papers31 papers in the last four weeks, up 7% on the four weeks before. 0.2% of all new papers.

Oct 13, 2025

Benchmarking Deep Learning Models for Laryngeal Cancer Staging Using the LaryngealCT Dataset

Laryngeal cancer imaging research lacks standardised public datasets to enable reproducible deep learning (DL) model development. We present LaryngealCT, a curated benchmark of 1,029 computed tomography (CT) scans aggregated from six collections from The Cancer Imaging Archive (TCIA). Uniform 1 mm isotropic volumes of interest encompassing the larynx were extracted using a weakly supervised parameter search framework validated by clinical experts. Six 3D DL architectures (custom 3D CNN, ResNet18,50,101, DenseNet121 and MedicalNet-pretrained ResNet50) were benchmarked on (i) early (Tis,T1,T2) vs. advanced (T3,T4) and (ii) T4 vs. non-T4 classification tasks. On the independent test set, the 3D CNN achieved the strongest overall performance across global and per-class metrics (Accuracy 0.854, F1-macro 0.841) in early vs. advanced classification. In the T4 task, AU-ROC values exceeded 0.82 for most models, but sensitivity for T4 disease remained limited (less than or equal to 0.412), with ResNet101 showing the most promising calibrated T4 recall (0.706. Model explainability assessed using GradCAMpp with thyroid cartilage overlays for T4 classification task revealed anatomically plausible peri-cartilage activations, although spatial overlap was modest. Through open-source data, pretrained models, and integrated explainability tools, LaryngealCT offers a reproducible foundation for AI-driven research to support future clinical decision-making in laryngeal oncology.

Nivea Roy, Son Tran, Atul Sajjanhar +4

School of Information Technology, Deakin University, Burwood, Australia · Department of Head and Neck Surgery, Kasturba Medical College, Manipal Academy of Higher Education, Manipal, India · Department of Radiodiagnosis and Imaging, Kasturba Medical College, Manipal Academy of Higher Education, Manipal, India +1

Medical Image ClassificationOncologyMedical Image Benchmarks
AskPDF
Sep 24, 2025

Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context

Public datasets of Chest X-Rays (CXRs) have long been a popular benchmark for developing machine learning (ML) computer vision models in healthcare. However, the reported strong average-case performance of these models do not necessarily reflect their actual utility when used in heterogeneous clinical settings, potentially masking weaker performance in medically significant scenarios. In this work we use clinical context to provide a more holistic evaluation of models for CXR diagnosis. In particular, we use discharge summaries, recorded prior to each CXR, to derive a ``pre-CXR'' probability of each CXR label, as a proxy for existing contextual knowledge available to clinicians when interpreting CXRs. We use this measure to probe model performance along two dimensions: First, using a stratified analysis, we show that models tend to have lower performance (as measured by AUROC and other metrics) among individuals with higher pre-CXR probability. Second, by controlling for pre-CXR probability via matching and re-weighting, we demonstrate that performance degrades when the correlation is broken between prior context and the current CXR label, suggesting that model performance is highly sensitive to the underlying distribution of clinical context. Specifically, cases with high pre-test probabilities present a fundamentally more difficult visual classification task, highlighting a gap in clinical utility when models are applied to high-risk cohorts.

Andrew Wang, Jiashuo Zhang, Michael Oberst

Brown University · Johns Hopkins University

Chest X-Ray ClassificationDistribution ShiftHealthcareMedical Image Benchmarks
AskPDF
Aug 15, 2025

Subcortical Masks Generation in CT Images via Ensemble-Based Cross-Domain Label Transfer

Subcortical segmentation in neuroimages plays an important role in understanding brain anatomy and facilitating computer-aided diagnosis of traumatic brain injuries and neurodegenerative disorders. However, training accurate automatic models requires large amounts of labelled data. Despite the availability of publicly available subcortical segmentation datasets for Magnetic Resonance Imaging (MRI), a significant gap exists for Computed Tomography (CT). This paper proposes an automatic ensemble framework to generate high-quality subcortical segmentation labels for CT scans by leveraging existing MRI-based models. We introduce a robust ensembling pipeline to integrate them and apply it to unannotated paired MRI-CT data, resulting in a comprehensive CT subcortical segmentation dataset. Extensive experiments on multiple public datasets demonstrate the superior performance of our proposed framework. Furthermore, using our generated CT dataset, we train segmentation models that achieve improved performance on related segmentation tasks. To facilitate future research, we make our source code, generated dataset, and trained models publicly available at https://github.com/alxw0671/CT_Subcortical_Segmentation, marking the first open-source release for CT subcortical segmentation to the best of our knowledge.

Augustine X. W. Lee, Pak-Hei Yeung, Jagath C. Rajapakse

College of Computing and Data Science, Nanyang Technological University, Singapore

Cross-Modality Medical Image SegmentationMedical Image SegmentationComputed TomographyMedical Image Benchmarks
AskPDF
Aug 10, 2025

EndoCogniAgent: Closed-Loop Agentic Reasoning with Self-Consistency Validation for Endoscopic Diagnosis

Endoscopic diagnosis is an iterative process in which clinicians acquire, compare, and verify local visual evidence before reaching a conclusion. Current AI systems do not adequately support this process because fine-grained evidence acquisition and multi-step reasoning remain weakly coupled, complicating reconciliation of image-derived findings with their textual interpretations. This gives rise to two failure modes, hallucinated evidence and uncorrected error accumulation, that undermine diagnostic reliability. We propose EndoCogniAgent, a closed-loop agentic framework that formulates endoscopic diagnosis as a controlled state update process for integrating complementary visual and textual evidence. At each reasoning round, a central planner selects an evidence acquisition action, specialized expert tools extract spatial and semantic observations as structured textual evidence, and a self-consistency validation mechanism examines this evidence along two dimensions, knowledge consistency against the input image and temporal consistency with prior validated findings, before updating the diagnostic state. Validated observations are admitted into the evolving state to condition subsequent planning, while insufficiently supported or conflicting findings are retained with corrective feedback that redirects the planner toward additional verification. We further introduce EndoAgentBench, a workflow-oriented benchmark comprising 6,132 question-answer pairs from 11 endoscopic datasets, to evaluate diagnostic agents across a comprehensive diagnostic chain, from fine-grained visual perception to high-level diagnostic reasoning. EndoCogniAgent achieves 85.23% overall accuracy on perception tasks and 71.13% clinical acceptance rate on reasoning tasks. Blinded clinician evaluation further shows consistent improvements in diagnostic response quality over the evaluated baselines.

Yi Tang, Kai-Ni Wang, Liang-Peng Pu +3

School of Biological Science and Medical Engineering, Southeast University, Nanjing 211189, China · Jiangsu Key Laboratory of Biomaterials and Devices, Southeast University, Nanjing 211189, China · State Key Laboratory of Digital Medical Engineering, Southeast University, Nanjing 211189, China +2

EndoscopyMedical ImagingMedical DiagnosisAgentic ReasoningClinical ReasoningMedical Image Benchmarks+1
AskPDF
Jul 15, 2025

Trexplorer Super: Topologically Correct Centerline Tree Tracking of Tubular Objects in CT Volumes

Tubular tree structures, such as blood vessels and airways, are essential in human anatomy and accurately tracking them while preserving their topology is crucial for various downstream tasks. Trexplorer is a recurrent model designed for centerline tracking in 3D medical images but it struggles with predicting duplicate branches and terminating tracking prematurely. To address these issues, we present Trexplorer Super, an enhanced version that notably improves performance through novel advancements. However, evaluating centerline tracking models is challenging due to the lack of public datasets. To enable thorough evaluation, we develop three centerline datasets, one synthetic and two real, each with increasing difficulty. Using these datasets, we conduct a comprehensive evaluation of existing state-of-the-art (SOTA) models and compare them with our approach. Trexplorer Super outperforms previous SOTA models on every dataset. Our results also highlight that strong performance on synthetic data does not necessarily translate to real datasets. The code and datasets are available at https://github.com/RomStriker/Trexplorer-Super.

Roman Naeem, David Hagerman, Jennifer Alvén +2

Chalmers University of Technology, 412 96 Gothenburg, Sweden

Medical Image Analysis3D Medical ImagingMedical Image Benchmarks
AskPDF
May 21, 2025

Benchmarking Classification Tasks of Chest X-ray AI Across Multinational Public and Private Datasets: a retrospective diagnostic accuracy study

Chest X-ray (CXR) AI models are increasingly proposed for triage, screening, and rule-out, where the operating point is fixed to miss few abnormalities. Most evaluations instead report threshold-free discrimination on a few public benchmarks, leaving clinically usable performance and generalisability to unseen institutional cohorts poorly characterised. We retrospectively evaluated 10 published CXR classification models on 12 datasets from six countries: seven public benchmarks and five institutional cohorts from China, Japan, and Germany. Models spanned convolutional networks, vision-language models, a question-answering foundation model, and a repurposed encoder. Only image-level classification was assessed; segmentation, detection, and report generation were out of scope. Source labels were harmonised to a common vocabulary. The primary outcome was specificity at 95% sensitivity; PPV at 95% sensitivity and AUROC were secondary outcomes. Estimates weighted each diagnostic task equally, with 95% CIs. MAVL and MedKLIP had the highest overall specificity at 95% sensitivity (0.37 [95% CI 0.31-0.44] and 0.37 [0.30-0.44]) and kept that ranking on tasks evaluable for every model. Their PPV was 0.18 (0.13-0.24) and 0.19 (0.13-0.25): most flagged examinations would be false positives. Specificity was lower on institutional cohorts than on public benchmarks for most models, by 0.13 for MAVL, 0.15 for DenseNet, and 0.09 for MedKLIP. No model was best for every finding, country, lesion extent, or prevalence stratum; specificity fell from 0.34 for diffuse to 0.15 for focal findings. At 95% sensitivity, the best current models clear only about a third of truly negative examinations, and public-benchmark performance overstates what they achieve on unseen institutional data. Operating-point-aware evaluation and external validation on institutional cohorts should precede deployment.

Qinmei Xu, Yiheng Li, Wenya Zhao +21

Stanford Center for Biomedical Informatics Research (BMIR), Department of Medicine, Stanford University, Stanford, CA, USA · Department of Radiology, the Second Hospital of Nanjing, Nanjing University of Chinese Medicine, Nanjing, China · Stanford Breast Imaging, Department of Radiology, Stanford University, Stanford, CA, USA +3

Chest X-Ray ClassificationMedical Image BenchmarksMedical DiagnosisDomain Generalization
AskPDF
Mar 29, 2025

OncoReg: Medical Image Registration for Oncological Challenges

In modern cancer research, the vast volume of medical data generated is often underutilised due to challenges related to patient privacy. The OncoReg Challenge addresses this issue by enabling researchers to develop and validate image registration methods through a two-phase framework that ensures patient privacy while fostering the development of more generalisable AI models. Phase one involves working with a publicly available dataset, while phase two focuses on training models on a private dataset within secure hospital networks. OncoReg builds upon the foundation established by the Learn2Reg Challenge by incorporating the registration of interventional cone-beam computed tomography with standard planning fan-beam CT images in radiotherapy. Accurate image registration is crucial in oncology, particularly for dynamic treatment adjustments in image-guided radiotherapy, where precise alignment is necessary to minimise radiation exposure to healthy tissues while effectively targeting tumours. This work details the methodology and data behind the OncoReg Challenge and provides a comprehensive analysis of the competition entries and results. Findings reveal that feature extraction plays a pivotal role in this registration task. A new method emerging from this challenge demonstrated its versatility, while established approaches continue to perform comparably to newer techniques. Both deep learning and classical approaches still play significant roles in image registration, with the combination of methods, particularly in feature extraction, proving most effective.

Wiebke Heyer, Yannic Elser, Lennart Berkel +15

Institute of Medical Informatics, University of Lübeck, Lübeck, Germany · Institute of Radiology and Nuclear Medicine, University Hospital Schleswig-Holstein, Lübeck, Germany · Department of Biomedical Engineering and Center for Biotechnology and Interdisciplinary Studies, Rensselaer Polytechnic Institute, Troy, NY, USA +8

Medical Image RegistrationCross-Modal Image RegistrationMedical Image Benchmarks
AskPDF
Dec 20, 2024

SegCol Challenge: Semantic Segmentation for Tools and Fold Edges in Colonoscopy data

Improving the reliability and completeness of colonoscopic inspection is critical for reducing missed lesions and improving colorectal cancer prevention. Reliable scene understanding is essential for navigation, reconstruction, and assessment of inspection completeness. Anatomical structures such as mucosal folds provide stable geometric cues for endoscope localization, while surgical instruments introduce dynamic occlusions that complicate visual interpretation. However, existing gastrointestinal endoscopy datasets largely focus on disease detection or artifact segmentation, leaving a gap in precise annotations of structural landmarks and instruments. We introduce SegCol, a dataset and benchmark for semantic segmentation of colon fold edges and surgical instruments derived from the EndoMapper dataset. SegCol provides manually annotated pixel-level masks for three instrument classes and thin fold-edge structures across temporally consistent image sequences. It forms the basis of the SegCol Challenge, organized as part of the EndoVis Challenge at MICCAI 2024, evaluating both supervised segmentation and annotation-efficient active learning. We further study segmentation metrics, including Dice, ODS/OIS, AP, and CLDice, under structural perturbations and different object geometries, and analyze participating methods, architectural choices, and active learning strategies. Our findings show that metric behavior strongly depends on target structure, highlighting the need for carefully selected evaluation protocols in endoscopic segmentation. Details are available at https://www.synapse.org/Synapse:syn54124209/wiki/626563, and code at https://github.com/surgical-vision/segcol_challenge.

Xinwei Ju, Rema Daher, Razvan Caramalau +7

UCL Hawkes Institute, University College London, London, UK · Department of Computer Science, University College London, London, UK · Medtronic, London, UK +3

EndoscopyImage SegmentationActive LearningMedical Image SegmentationSemantic SegmentationMedical Image Benchmarks
AskPDF
Dec 29, 2023

The TopCoW Challenge -- Topology-Aware Circle of Willis Segmentation for CT and MR Angiography

The Circle of Willis (CoW) is an important network of arteries connecting major circulations of the brain. Its vascular architecture is believed to influence the risk, severity, and outcome of serious neurovascular diseases. However, characterizing the highly variable CoW anatomy remains a manual and time-consuming expert task. The CoW is commonly imaged by two non-invasive angiographic imaging modalities, magnetic resonance angiography (MRA) and computed tomography angiography (CTA), yet few datasets with annotated CoW anatomy exist, and there have been no established benchmarks for comparing CoW segmentation algorithms. We organized the TopCoW benchmark challenge alongside the release of an annotated CoW dataset with 125 paired MRA and CTA scans from the same patients. Voxel-level annotations for 13 vessel components were created using virtual reality technology and verified by clinical experts. Participants submitted algorithms for CoW segmentation and variant classification, which we evaluated on internal and external test sets comprising 226 scans from over five centers. The benchmark includes voxel-level segmentation, CoW component detection, CoW variant classification, and two clinical application tasks. We received submissions from over 250 participants across six continents. Top-performing teams achieved over 90% Dice scores for CoW segmentation, over 80% F1 scores for detecting key vessel components, and over 70% balanced accuracy in CoW variant classification across nearly all test sets. The best algorithms also supported clinically relevant downstream tasks by accurately classifying fetal-type posterior cerebral arteries and localizing aneurysms in relation to CoW anatomy. This benchmark demonstrated the utility of CoW segmentation algorithms for some downstream clinical applications with explainability.

Kaiyuan Yang, Fabio Musio, Yihui Ma +112

Image SegmentationMedical Image SegmentationMedical Image Benchmarks
AskPDF