Organizations: Biomaps, UMR1281, INSERM, CNRS, CEA, Université Paris-Saclay, Villejuif, France · Guerbet Research, Villepinte, France · Department of Diagnostic Oncologic Imaging, Gustave Roussy Institute, Villejuif, France · Department of Radiology, Pellegrin University Hospital, Bordeaux, France · Bordeaux Institute of Oncology, BRIC U1312, INSERM, Team SARCOTARGET, Bordeaux, France
Purpose : To develop and evaluate CAPTOR, a deep-learning vision-language foundation model for oncologic chest-abdomen-pelvis (CAP) CT interpretation. Materials & Methods : CAPTOR (Chest-Abdomen-Pelvis foundaTion model for Oncologic Radiology) is a contrastive vision-language model, trained using a pretraining strategy for oncologic CAP CT (in single- or multiple-series acquisitions), on a multicenter retrospective dataset from North America, Europe and Asia, predominantly from tumor lesion examinations. CAPTOR was compared with state-of-the-art CT foundation models on three downstream tasks: disease detection, survival prediction using TumorImagingBench and image retrieval. Classification and retrieval were both evaluated on held-out internal and public test sets, stratified by anatomical coverage and indication. Bootstrap resampling was used to estimate confidence intervals, and pairwise comparisons were performed using permutation tests. Results : CAPTOR was trained and tested on 25,512 and 10,794 CT-report pairs respectively. TumorImagingBench datasets comprised between 134 and 421 studies each. CAPTOR outperformed all baseline models, on the classification task, on the CAP subset (AUC 0.767, 95%CI: 0.734-0.793, vs. 0.695, 95%CI: 0.645-0.736 for the next-best, p<0.001) and oncologic subset (AUC 0.784, 95%CI: 0.759-0.805, vs. 0.734, 95%CI: 0.703-0.767 p<0.001), ranked first on three of the four oncologic prognostic benchmarks, and achieved superior retrieval performance on CAP and oncologic subsets. Conclusion : A foundation model trained on oncology-enriched CAP CT data outperformed relevant state-of-the-art models in its target clinical context, underscoring the importance of clinically aligned pretraining. We release our preprocessing, training and inference code : https://github.com/Guerbet-AI/CAPTOR and model weights : https://huggingface.co/Guerbet-AI/CAPTOR
Figures & tables
Figure 1: Overview of the CAPTOR pipeline. CT volumes are cropped into 12 anatomical sub-regions using TotalSegmentator28 masks (A); regional crops are paired with the Findings section of the corresponding report and CAPTOR is trained with a rotating-region contrastive objective (B). At inference, TotalSegmentator masks are used to generate regional crops, and each CT yields one embedding per region (C). The resulting frozen image and text encoders are evaluated via XGBoost classification, k-nearest neighbor (kNN) survival prediction, and cross-modal retrieval (D, E, F). MAP@K = mean average precision at K, with K the number of retrieved images, std = Study
Figure 2: Flow chart showing the datasets used for training CAPTOR and for training/testing downstream tasks. CAP= Chest-Abdomen-Pelvis, AP= Abdomen-Pelvis
Table 1: Composition of the internal and public datasets used for training and testing. Percentages are reported at the study level. Train/test sets for TumorImagingBench1 are not included, as their original training/test splits were not used: survival prediction was instead assessed via k-nearest-neighbor classifiers across 10 random splits (Materials and Methods). The median age is shown, with inter-quartile range in brackets. CTPA = CT pulmonary angiogram; F=Female, M=Male, NSCLC = non-small cell lung cancer; GTV = gross tumor volume; CRLM = colorectal liver metastases.
Figure 3: Bar plot showing the classification scores of CAPTOR and five state-of-the art foundation models on our test set, stratified by visible anatomical region and clinical indication. The number of studies in each subset is in parentheses. One XGBoost was trained for each medical condition and model using the image embedding extracted with the model’s image encoder. For each subset and model, the macro-AUC is the unweighted mean of the AUC scores for each medical condition relevant to the subset among the 26 conditions studied. The confidence intervals and the pairwise comparison were obtained with a patient-level bootstrap and a patient-level permutation test respectively, both based on 2,000 iterations. AP = Abdomen-Pelvis, CAP=Chest-Abdomen-Pelvis. *** = p < 0.001, ** = p < 0.01, * = p < 0.05
Figure 4: Per-condition classification performance of CAPTOR and five baselines on the oncological subset, for each of the 26 medical conditions studied. One XGBoost was trained for each medical condition and model, using the appropriate image embedding (i.e. the corresponding regional crop for CAPTOR and the corresponding series for the baselines) extracted with the model’s image encoder.
Figure 5: Two-year survival-prediction performance (AUC) of CAPTOR and five baseline models on four oncologic TumorImagingBench1 cohorts (NSCLC-Radiomics21, NSCLC-Radiogenomics22, C4KC-KiTS23, Colorectal-Liver-Metastases24), using kNN classifiers on frozen image embeddings, averaged over 10 random splits. Statistical significance was assessed using a permutation test (2,000 permutations). The number of studies for each dataset is indicated between parentheses. ** = p < 0.01, * = p < 0.05
Datasets Task
Subset
CAPTOR
CAPTOR-100K
Whole
0.849 (0.828–0.866)
0.846 (0.824–0.862)
Chest
0.838 (0.789–0.865)
0.828 (0.778–0.860)
Global test set Medical condition classification
AP
0.807 (0.797–0.816)
0.808 (0.798–0.818)
CAP
0.767 (0.734–0.793)
0.744 (0.703–0.779)
Oncological
0.784 (0.759–0.805)
0.771 (0.743–0.796)
Non-oncological
0.859 (0.833–0.879)
0.856 (0.831–0.874)
Table 3: Effect of training-data volume on CAPTOR performance. Comparison of CAPTOR and CAPTOR-100K performances across all downstream tasks with 95%confidence intervals in parentheses. Retrieval results are shown for K=10, with K the number of retrieved images considered.
Figure 6: Classification performance (macro-AUC) as a function of labeled training-set fraction (10-100%) for CAPTOR, CAPTOR-100K, and five baseline models, on (left) the whole test set, (middle) the CAP subset, and (right) the oncologic subset.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Radar plot comparing the per-condition classification F1-score of Mistral Large 3, GPT-5.4, and GPT-4o mini on the CT-RATE dataset, evaluated against the dataset’s original labels (extracted with a fine-tuned RadBERT-RoBERTa-4m on 1,000 reports).
Figure 8: Bar plot showing the classification scores of CAPTOR, CAPTOR-100K and five state-of-the art foundation models, on our test set stratified by anatomical region visible and indication. One XGBoost was trained for each medical condition and model using the image embedding extracted with the model’s image encoder. For each subset and model, the macro-AUC is the unweighted mean of the AUC scores for each medical condition relevant to the subset among the 26 conditions we studied. The confidence intervals and the pairwise comparison were obtained with a patient-level bootstrap and a patient-level permutation test respectively, both of size 2,000. ***= p < 0.001, ** = p < 0.01, * = p < 0.05, the stars are the color of the comparison model (orange for CAPTOR-100K and blue for CAPTOR)
Figure 9: Per-condition classification performance of CAPTOR, CAPTOR-100K, and baselines for the 26 medical conditions, shown separately for the CAP subset (left) and the oncological subset (right). One XGBoost was trained for each medical condition and model using the appropriate image embedding (i.e. the right regional crop for CAPTOR and the right series for the baselines) extracted with the model’s image encoder
Figure 10: Two-year survival-prediction performance (AUC) of CAPTOR, CAPTOR-100K and five baseline models on four oncologic TumorImagingBench cohorts (NSCLC-Radiomics, NSCLC-Radiogenomics, C4KC-KiTS, Colorectal-Liver-Metastases), using kNN classifiers on frozen image embeddings averaged over 10 random splits. Statistical significance was assessed using a permutation test (2,000 permutations).
Figure 11: Classification macro-AUC of CAPTOR, CAPTOR-100K, and five baseline models for each abnormality label on the external, non-oncologic CT-RATE [ 11 ] and RAD-Chest CT [ 33 ] datasets, using each dataset’s original labels. One XGBoost was trained for each medical condition and model using the image embedding extracted with the model’s image encoder. For each subset and model, the macro-AUC is the unweighted mean of the AUC scores for each medical condition relevant to the subset among the 26 conditions we studied. The confidence intervals and the pairwise comparison were obtained with a patient-level bootstrap and a patient-level permutation test respectively, both of size 2,000. *** = p < 0.001, ** = p < 0.01, * = p < 0.05
Accurate prognosis prediction is important for treatment planning in lung cancer, but deep learning-driven survival modelling is often limited by the scarcity of curated imaging cohorts with reliable outcome data. This study evaluates whether representations from a domain-specific foundation model can be used for multimodal survival prediction in data-constrained clinical settings. We assess the foundation model CT-CLIP as a feature extractor for pretreatment computed tomography images and clinical variables from 242 diagnosed lung cancer patients. The evaluation includes adaptation strategies based on frozen encoders, full fine-tuning, and low-rank adaptation, together with modality ablations and comparisons with clinical and multimodal baselines. The results show that a frozen CT-CLIP model combined with a trainable lightweight survival head outperforms the clinical baseline and achieves comparable or improved performance relative to other multimodal approaches, and separates patients into clinically meaningful high- and low-risk groups.
Sofie Allgöwer, Mikael Johansson, Andreas Hallqvist +4
Chalmers University of Technology, Gothenburg, Sweden · Umeå University, Umeå, Sweden · Sahlgrenska University Hospital, Gothenburg, Sweden +1
There is substantial interest in developing artificial intelligence systems to support radiologists across tasks ranging from segmentation to report generation. Existing computed tomography (CT) foundation models have largely focused on building generalist vision-language systems capable of tasks such as question answering and report generation. However, training reliable vision-language systems requires paired image-text data at a scale that remains unavailable in CT. Moreover, adapting the underlying visual representations to downstream tasks typically requires partial or full backbone fine-tuning, a computationally demanding process inaccessible to many research groups. Instead, foundation models should prioritise learning robust visual representations that enable efficient transfer to new tasks with minimal labelled data and without backbone fine-tuning. We present VoxelFM, a 3D CT foundation model trained with self-distillation using the DINO framework, which learns semantically rich features without language supervision. We evaluated VoxelFM across seven categories of clinically relevant downstream tasks using frozen backbone representations with lightweight probes: classification, regression, survival analysis, instance retrieval, localisation, segmentation, and report generation. VoxelFM matched or outperformed four existing CT foundation models across all task categories. Despite receiving no language supervision during pre-training, VoxelFM surpassed models explicitly trained with language-alignment objectives, including on report generation. Our results indicate that current CT foundation models perform significantly better as feature extractors for lightweight probes rather than as vision encoders for vision-language models. Model weights and training code are publicly available.
Rubén Moreno-Aguado, Alba Magallón, Victor Moreno +2
Bioengineering Department and Imperial-X, Imperial College London, London, UK · Department of Computer Science, University of Manchester, Manchester, UK · Oncology Data Analytics Program, Catalan Institute of Oncology, Barcelona, Spain +6
CT vision-language foundation models have demonstrated promising performance across downstream tasks, but are typically trained with whole-volume representations that dilute fine-grained anatomical signals. Fine-grained vision-language pre-training addresses this by aligning anatomy-level visual features with anatomy-specific text, but in doing so discards the global context that whole-volume models provide. Furthermore, existing fine-grained approaches train from scratch, making them computationally expensive. We introduce Anatomy Contextualized Adaptation (ACA), a lightweight framework that adapts frozen CT foundation model representations for anatomy-level vision-language alignment while enhancing global contextualization. ACA uses TotalSegmentator to decompose CT volumes into anatomy-level embeddings, which are refined via a transformer that captures cross-anatomy relationships, and aligned to both per-anatomy and scan-level text extracted from radiology reports. Evaluated on Merlin and CT-RATE, ACA consistently outperforms both the frozen foundation model baselines and existing fine-grained methods in zero-shot finding classification, while requiring less than one hour of training once embeddings are cached. The attention weights learned by ACA's inter-anatomy transformer additionally indicate plausible cross-anatomy context routing. Altogether, these results support ACA as a lightweight approach for adapting CT foundation models to anatomically grounded vision-language alignment while preserving and enhancing global anatomical context.
Roshan Kenia, Stephanie L McNamara, William Lotter
Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Data Science, Dana-Farber Cancer Institute, Boston, MA · Department of Radiology, Massachusetts General Hospital, Boston, MA +2