Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT
Authors: Simon Ging, Philipp Arnold, Sebastian Walter, Hani Alnahas, Hannah Bast, Elmar Kotter, Jiancheng Yang, Behzad Bozorgtabar, +1 more
Organizations: Computer Vision Group, University of Freiburg, Germany · Adaptive & Agentic AI · Adaptive & Agentic AI (A3) Lab, Aarhus University, Denmark · Department of Radiology, Medical Center -- University of Freiburg, Germany · Department of Radiology, Medical Center – University of Freiburg, Germany · Chair of Algorithms and Data Structures, University of Freiburg, Germany · ELLIS Institute Finland · School of Electrical Engineering, Aalto University, Finland
Recent 3D CT vision-language models align volumes with reports via contrastive pretraining, but typically rely on limited public data and provide only coarse global supervision. We train a 3D CT vision-language model on 98k report-volume pairs (50k patients) collected at a single hospital, combined with public datasets, using SigLIP-style contrastive pretraining together with prompt-based disease supervision in the shared vision-text embedding space. On CT-RATE, our model achieves state-of-the-art text-to-image retrieval (R@10 31.5 vs. 22.2) and competitive disease classification (AUC 83.8 vs. 83.8), with consistent results on Rad-ChestCT (AUC 77.0 vs. 77.3). We further observe that radiologists routinely reference specific images within their reports (e.g., ``series X, image Y''), linking textual descriptions to precise axial locations. We automatically mine 262k such snippet-slice pairs and introduce the task of intra-scan snippet localization -- predicting the axial depth referred to by a text snippet -- reducing mean absolute error to 36.3 mm at 12 mm feature resolution, compared with 67.0 mm for the best baseline. Adding this localization objective leaves retrieval and classification broadly unchanged within confidence bounds, yielding a single unified model for retrieval, classification, and intra-scan grounding.
Figures & tables
Figure 1: Overview of the RadFinder architecture and training pipeline. Left (Vision): A 3D CT scan is processed by a frozen parallel window encoder, followed by a trainable encoder to extract both a global volume embedding and local slice-level embeddings. Right (Language): A trainable text encoder processes full radiology reports, localized text snippets (e.g., the sentence that references the fourth slice out of the 32 slices in this example scan), and LLM-extracted positive/negative disease prompts. Center (Training): The model is optimized via three contrastive objectives in a shared embedding space: a global Contrastive Loss aligning the full scan and report, an intra-scan Localization Loss aligning text snippets with their specific slices, and a Disease Prompt Loss aligning the global scan with corresponding disease prompt descriptions
Data
CT-RATE
RC
Merlin R@1
A
B
C
D
R@10
MAP@5
AUC
AUC
Findings
Impr.
Random Chance
0.3
0.1
50.0
50.0
0.8
0.8
CT-CLIP [ 6 ]
✓
5.0
68.3
73.1
62.9
—
—
MedVista3D-ViT [ 11 ]
✓
10.7
—
77.8
—
—
—
MedVista3D-UniMISS
✓
8.7
—
78.2
—
—
—
Merlin [ 2 ]
✓
2.7
62.6
72.8
64.4
59.4
19.4
Table 1: Text-to-image retrieval on CT-RATE, Rad-ChestCT (RC) and Merlin validation sets. Training datasets: A: RefCT (ours, 78k), B: CT-RATE [ 6 ] (47k), C: Merlin [ 2 ] (15k), D: INSPECT [ 7 ] (19k). ∗ fVLM additionally trains on MedVL-CT69 [ 16 ] (272k). We highlight values within the confidence interval of the highest score separately for the comparison of our main model with baselines (top) and our model with our ablations (bottom).
MAE (mm)
< 6mm
< 18mm
< 30mm
Random slice
126.9
5.7
4.7
1.1
11.8
1.6
17.8
2.0
Middle slice
95.8
3.6
4.6
1.2
13.0
1.8
20.8
2.2
CLIP ViT-B/32 [ 13 ]
124.6
5.5
3.2
0.8
9.4
1.4
14.8
1.7
BiomedCLIP [ 21 ]
86.6
5.2
8.3
1.3
20.2
1.9
30.5
2.2
MedSigLIP-448 [ 15 ]
75.6
5.0
9.7
1.4
26.2
2.1
40.1
2.3
SigLIP2 [ 17 ]
82.4
4.7
7.8
1.3
19.3
1.8
31.5
2.3
Table 2: Results for snippet localization on the RefCT test set. ∗ SigLIP2 finetuned on RefCT snippet–slice pairs via contrastive loss.
Vision-language pre-training (VLP) holds great promise for general-purpose medical AI by leveraging radiology reports as rich textual supervision, yet existing methods struggle with 3D CT imaging due to inefficient visual backbones and coarse semantic alignment. To address these issues, we propose a tailored VLP framework featuring three key components: (1) a CNN-ViT hybrid encoder that replaces ViT's patch embedding with a 3D CNN backbone to efficiently capture local anatomical details while preserving global attention and compatibility with pre-trained cross-modal priors; (2) a disease-level contrastive learning mechanism using learnable query tokens to dynamically extract disease-specific semantics from full reports and align them with corresponding visual features, thereby disentangling distinct diseases within the same anatomical region; and (3) a diagnosis-aware prompt strategy that employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap and enhance zero-shot diagnostic reliability. Our model achieves state-of-the-art performance on CT-RATE (84.4% AUC, +5.1%) and Rad-ChestCT (75.4% AUC, +5.4%), with even larger gains (+9.8% AUC) on a challenging 60-disease benchmark, and demonstrates strong transferability to radiology report generation, underscoring the generality and clinical utility of our approach.
Bowen Shi, Weiwei Cao, Ruifeng Yuan +5
DAMO Academy, Alibaba Group · Hupan Lab, 310023, Hangzhou, China · Shanghai Jiao Tong University, China +2
We present NV-Reason-CT, a generative vision--language model for chest and abdominal CT combining native 3D visual encoding with radiologist-guided reasoning. The model couples a native 3D vision transformer with a language model, passing all visual tokens and their explicit 3D coordinates into language decoding without further spatial token merging. This retains volumetric spatial information within the vision encoder and through the language model's positional encoding during joint processing with text. We train on a curated corpus of approximately 550,000 multimodal instruction examples from 70,111 unique CT image inputs, combining standardized reports, abnormality-focused and anatomy-specific questions, multi-turn interactions, and radiologist-authored reasoning from recorded and transcribed expert CT interpretations. Expert annotations provide direct supervision and guide additional report-grounded synthetic reasoning. End-to-end supervised fine-tuning (SFT) is followed by Group Relative Policy Optimization (GRPO), with verifiable rewards over chest and abdominal abnormality sets. The model supports abnormality classification, report generation, and interactive reasoning with reviewable observations, differential diagnoses, and uncertainty. Evaluation spans public CT benchmarks and a held-out NIH cohort. On CT-RATE, NV-Reason-CT achieves a macro-F1 of 0.614 and macro-AUROC of 0.871 without a task-specific classification head; generated reports achieve a report-derived macro-F1 of 0.592. In a preliminary study with expert radiologists, AI-assisted review received favorable confidence ratings and was associated with a 50% reduction in average reported interpretation and reporting time. We release the model and training code to support reproducible research on explainable AI for volumetric medical imaging.
Andriy Myronenko, Dong Yang, Yucheng Tang +15
NVIDIA, Santa Clara, CA · NIH, Bethesda, MD · University of Oxford, Oxford, UK +4
Vision-language contrastive pretraining has become the dominant recipe for 3D medical foundation models, leveraging the large volumes of paired scans and reports produced in clinical practice. However, medical images usually span dozens of organs, and radiological reports are much longer than typical natural image captions and are composed of multiple structured sections. CLIP-style pretraining compresses this structure by encoding each modality into a single global token, at the risk of losing important details. We introduce ConQuer (Concept Queries), an image-text pretraining method that augments CLIP's global alignment with a set of localized alignments, one per concept. ConQuer splits the report into concept-specific sections and learns cross-attention queries that pool the matching image features without using any segmentation mask or spatial supervision. Contrastive learning is then applied independently for each concept. Concepts can be any unit of semantic localization; here, they are anatomical regions, one query per organ or gross body region. As a byproduct, each query learns attention maps focused on its concept, providing built-in spatial interpretability. We use ConQuer to train Jolia, a 3D CT foundation model on chest and abdominal CT. Jolia consistently outperforms a CLIP baseline on findings classification, report generation, and cross-center transfer, and sets a new state of the art across multiple public benchmarks. Jolia's weights are available at https://huggingface.co/raidium/Jolia
Julien Khlaut, Charles Corbière, Baptiste Callard +9
Raidium, Paris, 75014, France · Department of Vascular and Oncological Interventional Radiology,2026 Hôpital Européen Georges Pompidou, AP-HP, Paris, France · Faculté de Santé, Université Paris-Cité, Paris, France +3