CT-Bench: A Benchmark for Multimodal Lesion Understanding in Computed Tomography

Qingqing Zhu; Qiao Jin; Tejas S. Mathai; Yin Fang; Zhizheng Wang; Yifan Yang; Maame Sarfo-Gyamfi; Benjamin Hou

cs.CVFeb 16, 2026

CT-Bench: A Benchmark for Multimodal Lesion Understanding in Computed Tomography

Authors: Qingqing Zhu, Qiao Jin, Tejas S. Mathai, Yin Fang, Zhizheng Wang, Yifan Yang, Maame Sarfo-Gyamfi, Benjamin Hou,

Abstract

Artificial intelligence (AI) can automatically delineate lesions on computed tomography (CT) and generate radiology report content, yet progress is limited by the scarcity of publicly available CT datasets with lesion-level annotations. To bridge this gap, we introduce CT-Bench, a first-of-its-kind benchmark dataset comprising two components: a Lesion Image and Metadata Set containing 20,335 lesions from 7,795 CT studies with bounding boxes, descriptions, and size information, and a multitask visual question answering benchmark with 2,850 QA pairs covering lesion localization, description, size estimation, and attribute categorization. Hard negative examples are included to reflect real-world diagnostic challenges. We evaluate multiple state-of-the-art multimodal models, including vision-language and medical CLIP variants, by comparing their performance to radiologist assessments, demonstrating the value of CT-Bench as a comprehensive benchmark for lesion analysis. Moreover, fine-tuning models on the Lesion Image and Metadata Set yields significant performance gains across both components, underscoring the clinical utility of CT-Bench.

Figures & tables

Dataset	Image Data Type	Textual Annotations	Size	Notes
ROCOv2	CT, MRI, PET, Ultrasound, X-ray	General captions	$\sim$ 80k	Image-text pairs
PMC-OA	Biomedical images	Captions from PubMed	1M+	Image-text pairs
CT-RATE	CT (Thoracic, non-contrast)	Basic radiology reports	25k	3D CT-report pairs
DeepLesion	CT (BBox available)	No textual annotations	$\sim$ 32k	Images only
MEDICAT	Radiology, histology, scope procedures	Captions, references	7.5k	subfigure-subcaption pairs
Ours (Lesion& Metadata)	CT (Bbox available)	Lesion descriptions and size info from PACS	$\sim$ 20k	Image-text pairs

Table 1: Comparison of medical imaging datasets.

Dataset	Size (Images)	Modality	Answer Type	QA Curation	H.N. *
VQA-RAD	0.3k	MRI, CT, X-ray, etc.	Open/MC	Manually curated	No
PathVQA	5k	Microscopy Images	Open	Textbook-based, manually verified	No
SLAKE	0.7k	MRI, CT, X-ray, etc.	Open/MC	Manually curated	No
VQA-Med	5k	MRI, CT, X-ray, etc.	Open	Manually curated	No
PMC-VQA	149k	Mixed	Open/MC	ChatGPT-generated	Yes
Ours (QA)	2.8k	CT (BBox available)	MC	Template-based, manually verified	Yes

Table 2: Comparison of medical VQA datasets in terms of size, imaging modality, answer type, QA curation approach, and the inclusion of hard negatives. * H.N. = Hard Negatives; Open = open-ended questions; MC = multiple choice.

Dataset	Patients	Male	Female	Studies	Lesions
Training	2,835	1,516	1,319	5,875	15,380
Validation	342	181	161	667	1,758
Test	616	367	249	1,253	3,197
Total	3,793	2,064	1,729	7,795	20,335

Table 3: Summary of the CT-Bench: Lesion Image & Metadata Set.

Modality	Img2txt		CT2txt		Txt2img		Txt2bbox		Img2attrib		CT2attrib		Img2size		Total
	w/o	w/	w/o	w/	w/o	w/	w/o	w/	w/o	w/	w/o	w/	w/o	w/	w/o	w/
Validation	100	100	100	100	100	100	-	50	50	50	50	50	50	50	450	500
Test	200	200	200	200	200	200	-	100	100	100	100	100	100	100	900	1,000

Table 4: Distribution of the CT-Bench: QA Benchmark Component across seven VQA task types, split by validation and test sets, and by presence of BBox annotations. Abbreviations used: w/o = Without BBox, w/ = With BBox.

Model	BLEU-1		METEOR		ROUGE-1		BERT(F1)		CosineSim
	w/o	w/	w/o	w/	w/o	w/	w/o	w/	w/o	w/
Unsupervised
RadFM	0.0713	0.0779	0.0984	0.1034	0.0573	0.0604	0.8518	0.8525	0.3262	0.3325
Dragonfly	0.0842	0.1685	0.1045	0.2242	0.0307	0.1250	0.8430	0.8681	0.2638	0.4082
Llava_med	0.0756	0.0802	0.1555	0.1666	0.0748	0.0804	0.8462	0.8484	0.3914	0.4014
GPT-4V	0.0656	0.0916	0.1405	0.1915	0.0586	0.0819	0.8424	0.8549	0.3895	0.4456

Table 5: Summary of performance for models with and without BBoxes. The highest values for unsupervised models are highlighted in bold, and the best fine-tuned models are underlined.

Model

Img2txt

CT2txt

Txt2img

Txt2bbox

Img2attrib

CT2attrib

Img2size

Average

w/o

Total

Unsupervised

Random

25.0\pm 6.0

25.0\pm 6.0

25.0\pm 6.0

25.0\pm 6.0

25.0\pm 6.0

25.0\pm 6.0

–

25.0\pm 8.49

25.0\pm 8.49

25.0\pm 8.49

25.0\pm 8.49

25.0\pm 8.49

25.0\pm 8.49

25.0\pm 8.49

25.0

25.0

25.0

PMC-CLIP

30.0\pm 6.35

33.0\pm 6.52

30.5\pm 6.38

30.5\pm 6.38

24.5\pm 5.96

24.5\pm 5.96

–

19.0\pm 7.69

27.0\pm 8.7

28.0\pm 8.8

26.0\pm 8.6

29.0\pm 8.89

20.0\pm 7.84

16.0\pm 7.19

27.0

26.8

26.9

BiomedCLIP

\mathbf{38.0\pm 6.73}

\mathbf{42.5\pm 6.85}

\mathbf{37.0\pm 6.69}

\mathbf{42.0\pm 6.84}

26.5\pm 6.12

\mathbf{34.5\pm 6.59}

–

35.0\pm 9.35

44.0\pm 9.73

\mathbf{57.0\pm 9.7}

37.0\pm 9.46

\mathbf{42.0\pm 9.67}

30.0\pm 8.98

38.0\pm 9.51

\mathbf{34.9}

\mathbf{41.0}

\mathbf{38.1}

RadFM

25.0\pm 6.0

25.0\pm 6.0

25.0\pm 6.0

24.5\pm 5.96

3.5\pm 2.55

5.0\pm 3.02

–

3.0\pm 3.34

18.0\pm 7.53

18.0\pm 7.53

17.0\pm 7.36

19.0\pm 7.69

0.0\pm 0.0

0.0\pm 0.0

15.8

14.9

15.3

Table 6: Accuracy performance (%) of different models across QA Benchmark tasks, reported with 95% confidence intervals based on the binomial proportion for each task. Per-task results include the mean

\pm

confidence interval. For aggregate metrics ( Average ), we report point estimates of the mean accuracy across tasks. Since the average spans heterogeneous tasks with different sample sizes, binomial confidence intervals are not directly applicable; instead, averages are presented as summary statistics for overall comparison. A Random baseline (25%) is included as a reference for chance-level performance.

Role	Img2txt		CT2txt		Txt2img		Txt2bbox		Img2attrib		CT2attrib		Img2size		Average
	w/o	w/	w/o	w/	w/o	w/	w/o	w/	w/o	w/	w/o	w/	w/o	w/	w/o	w/	Total
Senior Radiologist 1	100.0	90.0	80.0	60.0	80.0	70.0	–	100.0	60.0	100.0	40.0	100.0	40.0	100.0	66.0	90.0	79.20
Senior Radiologist 2	70.0	100.0	80.0	70.0	80.0	70.0	–	100.0	60.0	100.0	60.0	100.0	80.0	100.0	70.5	90.5	81.50
Junior Doctor (Radiology Exp.)	50.0	80.0	70.0	50.0	30.0	40.0	–	80.0	60.0	80.0	40.0	100.0	0.0	40.0	45.5	68.5	58.15

Table 7: Human evaluation accuracy (%) across CT-Bench tasks by clinical role.

Model	Output	Alignment	Key Observations
Ground Truth	A nodule in the superior segment of the left lower lobe.	-	-
RadFM	There are multiple nodular opacities scattered throughout the bilateral lungs. The largest is in the right upper lobe with a draining vessel.	Misaligned	Overinterprets findings (multiple nodules, feeding vessels) and incorrectly identifies a larger nodule in the right upper lobe.
TunedRadFM(w)	Left lower lobe nodule.	Most aligned	Correctly identifies a nodule in the left lower lobe but lacks the specific location (superior segment).
LLAVA-med	The image is a computed tomography (CT) scan of the chest. It shows a large mass in the right upper lobe of the lung.	Misaligned	Mislocalizes the finding to the right upper lobe and describes it as a ”large mass” rather than a nodule.
GPT	The image shows a transverse section of a chest CT scan with a well-circumscribed, round, hyperdense lesion in the right lung parenchyma.	Misaligned	Incorrectly identifies the lesion in the right lung; lacks diagnostic detail or specificity.
Gemini	There is a small, well-defined nodule in the right lower lobe. The nodule is approximately 5 mm in diameter. No other significant findings.	Misaligned	Mislocalizes the nodule to the right lower lobe; adds size information not in the ground truth.

Table 8: Comparison of model outputs for lesion localization and description.

Figure 1: Overview of CT-Bench. (a) Example of data annotation, showing the transformation of original reports from PACS into detailed descriptions enriched with size information for CT-Bench: Lesion Image & Metadata Set. (b) Analysis of description: word count and value distribution. (c) Distribution of lesion locations. (d) CT-Bench: components of the QA benchmark cases.

Figure 2: Heatmap of performance change with BBox: model performance differences across tasks when BBox annotations are used. Positive values indicate performance gains due to BBox, while negative values indicate performance drops.

Figure 3: Comparison of model performance on single-slice (Img2txt / Img2attrib) versus multi-slice context tasks (Context2txt / Context2attrib).

Figure 4: Illustration of the selection process for “hard negative” cases using the BiomedCLIP model and MD validation.

Medical Image Benchmarks

Medical image benchmarks compare AI methods on curated clinical images, tasks, and evaluation protocols to measure accuracy, robustness, and clinical usefulness.

See all 337 papers

BenchX: Benchmarking AI Models for Cancer Detection and Localization with Demographic and Protocol Biases
Jun 23, 2026Qi Chen, Wenxuan Li, Pedro R. A. S. Bassi +14Distribution Shift Medical Image Analysis
Johns Hopkins University · German Cancer Research Center · University Hospital Basel +10
AMPLIFAI: A Multiphase CT Dataset for Benchmarking Clinical Reasoning in LI-RADS Assessment of Liver Lesions
Aug 14, 2026Pranav Kulkarni, Nikhil Shah, Amritansh Suryavanshi +9Medical Image Benchmarks
University of Maryland Institute for Health Computing, North Bethesda, MD 20852 · Department of Computer Science, University of Maryland, College Park, MD 20742 · Department of Diagnostic Radiology and Nuclear Medicine, University of Maryland School of Medicine, Baltimore, MD 21201 +2
DeepTumorVQA: A Hierarchical 3D CT Benchmark for Stage-Wise Evaluation of Medical VLMs and Tool-Augmented Agents
May 10, 2026Yixiong Chen, Wenjie Xiao, Pedro R. A. S. Bassi +7VLM Evaluation Tool Use in VLMs
Johns Hopkins University · University of Bologna · Center for Biomolecular Nanotechnologies, Istituto Italiano di Tecnologia +3

Jun 23, 2026J/K move · Enter open · S save

BenchX: Benchmarking AI Models for Cancer Detection and Localization with Demographic and Protocol Biases

Qi Chen, Wenxuan Li, Pedro R. A. S. Bassi, Xinze Zhou+13

Johns Hopkins University · German Cancer Research Center · University Hospital Basel+10

Distribution Shift Medical Image Analysis Algorithmic Fairness Fairness in Medical Image Classification Oncology Computed Tomography+2

Open paper AskPDF arXiv

Artificial intelligence (AI) has achieved remarkable success in medical imaging, but it is widely recognized that these models often perform inconsistently across real-world clinical settings. Such inconsistencies occur when patient demographics and imaging protocols vary, for example, in detecting small tumors, analyzing scans from different contrast phases, or evaluating patients of different ages or sexes. To quantify these inconsistencies, we develop a large-scale, open benchmark of 85,355 CT scans that systematically evaluates 12 tumor-detection AI models across tumor size, location, patient subgroup, and imaging protocol. We leverage large language models (LLMs) to extract and organize subgroup information from clinical data, which makes the analysis both scalable and reproducible. Our benchmark reveals that current state-of-the-art AI models, optimized for average accuracy, perform poorly in rare or underrepresented subgroups, such as young, female African Americans. However, collecting sufficient annotated data for these rare cases is often impractical. The benchmark provides a foundation for building more reliable and robust AI models for tumor detection and highlighting the need for rigorous, subgroup-level evaluation in medical imaging and computer vision. Datasets, code

Abstract

Medical Image Benchmarks

Explore similar work

BenchX: Benchmarking AI Models for Cancer Detection and Localization with Demographic and Protocol Biases

AMPLIFAI: A Multiphase CT Dataset for Benchmarking Clinical Reasoning in LI-RADS Assessment of Liver Lesions

DeepTumorVQA: A Hierarchical 3D CT Benchmark for Stage-Wise Evaluation of Medical VLMs and Tool-Augmented Agents