Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Tasks
Authors: Rubén Moreno-Aguado, Alba Magallón, Victor Moreno, Yingying Fang, Guang Yang
Organizations: Bioengineering Department and Imperial-X, Imperial College London, London, UK · Department of Computer Science, University of Manchester, Manchester, UK · Oncology Data Analytics Program, Catalan Institute of Oncology, Barcelona, Spain · Colorectal Cancer Group, ONCOBELL Program, Institut d’Investigació Biomèdica de Bellvitge, Barcelona, Spain · Consortium for Biomedical Research in Epidemiology and Public Health, Madrid, Spain · Department of Clinical Sciences, Faculty of Medicine and Health Sciences, Universitat de Barcelona Institute of Complex Systems, University of Barcelona, Barcelona, Spain · National Heart and Lung Institute, Imperial College London, London, UK · Cardiovascular Research Centre, Royal Brompton Hospital, London, UK · School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK
There is substantial interest in developing artificial intelligence systems to support radiologists across tasks ranging from segmentation to report generation. Existing computed tomography (CT) foundation models have largely focused on building generalist vision-language systems capable of tasks such as question answering and report generation. However, training reliable vision-language systems requires paired image-text data at a scale that remains unavailable in CT. Moreover, adapting the underlying visual representations to downstream tasks typically requires partial or full backbone fine-tuning, a computationally demanding process inaccessible to many research groups. Instead, foundation models should prioritise learning robust visual representations that enable efficient transfer to new tasks with minimal labelled data and without backbone fine-tuning. We present VoxelFM, a 3D CT foundation model trained with self-distillation using the DINO framework, which learns semantically rich features without language supervision. We evaluated VoxelFM across seven categories of clinically relevant downstream tasks using frozen backbone representations with lightweight probes: classification, regression, survival analysis, instance retrieval, localisation, segmentation, and report generation. VoxelFM matched or outperformed four existing CT foundation models across all task categories. Despite receiving no language supervision during pre-training, VoxelFM surpassed models explicitly trained with language-alignment objectives, including on report generation. Our results indicate that current CT foundation models perform significantly better as feature extractors for lightweight probes rather than as vision encoders for vision-language models. Model weights and training code are publicly available.
Figures & tables
Task
Dataset
Metric
VoxelFM
Merlin
RadFM
M3D
CT-CLIP
Classification
CT-RATE
AUROC
0.870 (0.006)
∗
0.798 (0.006)
0.759 (0.002)
0.740 (0.008)
0.574 (0.002)
Merlin
AUROC
0.797 (0.005)
0.810 (0.005)
0.668 (0.006)
0.649 (0.006)
0.538 (0.006)
RSNA-STR
AUROC
0.760 (0.017)
∗
0.597 (0.019)
0.570 (0.019)
0.529 (0.019)
0.542 (0.019)
Mycobacterial
AUROC
0.799 (0.031)
0.750 (0.035)
0.768 (0.033)
0.722 (0.036)
0.639 (0.040)
iCTCF-Covid
AUROC
0.845 (0.027)
∗
0.760 (0.033)
0.655 (0.038)
0.518 (0.042)
0.517 (0.042)
iCTCF-Severity
AUROC
0.792 (0.054)
0.752 (0.057)
0.742 (0.057)
0.547 (0.061)
0.515 (0.060)
Table 1: Performance comparison across seven downstream tasks. Results are reported for classification (AUROC), regression (MAE), survival analysis (AUROC and C-index), retrieval (Recall@10), localisation (MAE), segmentation (DICE), and report generation (F1). For regression and localisation, lower MAE is better ( ↓ ). All other metrics are higher-is-better ( ↑ ). Classification results on CT-RATE and Merlin are macro-averaged over all abnormality classes. TotalSegmentator segmentation results are reported as both micro- and macro-averaged DICE. Values in parenthesis are standard errors. Bold values denote the best-performing model in each row. ∗ denotes that the difference in performance between VoxelFM and next best model is statistically significant (p < 0.05).
Figure 1: Overview of the evaluation protocol. All evaluations use pre-computed embeddings from a pre-trained encoder backbone. The first common step is (A): embeddings are extracted by the backbone and cached for downstream use. (B) Classification and regression: two methods are available depending on the token type used. For the class token, a two-layer MLP is applied, and for patch tokens, a Q-Former with cross-attention is used instead. (C) Instance retrieval: class tokens are ranked against a query token by cosine similarity. (D) Localisation: patch tokens are decoded to normalised 3D coordinates via multi-head self-attention, with a softmax operation used to weight the positional coordinates. (E) Segmentation: patch tokens are reshaped into a 3D feature map and decoded through 3D convolutional and upsampling layers. (F) Report generation: patch tokens are projected via cross-attention and combined with a system prompt before being passed to a large language model (LLM).
Figure 2: Bar chart summary of model performance across six tasks. Results are shown for classification (AUROC), regression (MAE ↓ ), survival analysis (AUROC), localisation (MAE ↓ ), segmentation (DICE), and retrieval (Recall@10), with 95% confidence intervals displayed for each model. TotalSegmentator segmentation is reported as micro-averaged DICE. CT-RATE and Merlin classification results are macro-averaged over all abnormality classes. See Table 1 for full numerical results.
Figure 3: Per-abnormality breakdown of classification and report generation performance across 18 findings. (Left) Binary classification F1 scores (threshold =0.5 ) for each of the 18 abnormalities in the CT-RATE dataset, where an individual Q-Former probe is trained per label as described in Figure 1 . (Right) Corresponding report generation F1 scores for the same 18 CT-RATE abnormality labels, aligned row-by-row with the left panel. A red cross marks the best-performing classifier result in each row.
Figure 4: Effect of downstream dataset size. Q-Former probes are trained on various fractions of labelled training data (20%–100%) and evaluated on a fixed held-out test set. (Left) iCTCF-Covid. (Right) RSNA-STR. Error bars represent 95% confidence intervals.
Figure 5: Comparison of inference strategies and feature aggregation methods. (Left) MLP applied to the class token versus a single-layer Q-Former applied to patch tokens for classification tasks. CT-RATE and Merlin results are macro-averaged over their respective abnormality labels. Error bars represent 95% confidence intervals. (Right) Chunked 2.5D versus full 3D inference across classification tasks.
Figure 6: Overview of the VoxelFM pre-training framework. Student and teacher networks share identical ViT-based architectures with class and patch token heads. The teacher network is updated via an exponential moving average of the student parameters. For each CT volume, two global and eight local crops are generated. Only the student processes local crops, and random masking is applied to its global crops. Training minimises a DINO loss ( LCLS ) on class tokens and an iBOT loss ( LPatch ) on masked patch tokens, both using cross-entropy against teacher outputs. Both output heads consist of an MLP with a bottleneck layer followed by a prototype layer. The CT render is from NSCLC-Radiomics [ 2 ] .
Dataset
Location
Clinical Focus
Studies
CT-RATE [ 23 ]
Lung
Lung Abnormalities
25,692
INSPECT [ 29 ]
Lung
Pulmonary Embolism
23,203
NLST [ 45 ]
Lung
Lung Cancer
13,587
RIDER-Lung-PET-CT [ 43 ]
Lung
Lung Cancer
413
Lung-PET-CT-Dx [ 35 ]
Lung
Lung Cancer
268
Anti-PD-Lung [ 41 ]
Lung
Lung Cancer
211
Table 2: Datasets used for training the foundation model.
Dataset
Targets
Studies
CT-RATE [ 23 ]
Radiology Reports, 18 Abnormalities
27,256
Merlin [ 9 ]
Radiology Reports, 31 Abnormalities
25,494
RSNA-STR [ 16 ]
Has Pulmonary Embolism
7,280
iCTCF [ 46 ]
SARS-CoV-2 nucleic acids, Morbidity
1,321
Mycobacterial [ 24 ]
Tuberculosis
1,301
TotalSegmentator [ 65 ]
Semantic segmentation 117 classes
1,204
Table 3: Datasets used for downstream evaluations. The number of studies does not include augmentations and test samples. Due to its large size, the NLST only represents a sample of the full dataset.
Model
Pre-training
Training Data
Resolution
Patch Size
Architecture
RadFM
Text generation
16M 2D + 500K 3D (multi-modal)
64x256x256
4x32x32
3D ViT
Merlin
CLIP + Supervised
15K CT
160x224x224
—
ResNet152
M3D
CLIP
120K CT
32x256x256
4x16x16
3D ViT
CT-CLIP
CLIP
26K CT
240x480x480
10x20x20
2D/3D ViT
VoxelFM
Self-supervised
137K CT
112x112x112
14x14x14
3D ViT
Table 4: Comparison of baseline foundation models and VoxelFM. Overview of pre-training strategies, training data, input resolution, patch size, and backbone architecture for each model. RadFM is trained on 2D and 3D modalities (X-ray, CT, MRI, PET). VoxelFM (bottom) is our proposed model.
While foundation models in radiology are expected to be applied to various clinical tasks, computational cost constraints remain a major challenge when training on 3D-CT volumetric data. In this study, we propose TotalFM, a radiological foundation model that efficiently learns the correspondence between 3D-CT images and linguistic expressions based on the concept of organ separation, utilizing a large-scale dataset of 140,000 series. By automating the creation of organ volume and finding-sentence pairs through segmentation techniques and Large Language Model (LLM)-based radiology report processing, and by combining self-supervised pre-training via VideoMAE with contrastive learning using volume-text pairs, we aimed to balance computational efficiency and representation capability. In zero-shot organ-wise lesion classification tasks, the proposed model achieved higher F1 scores in 83% (5/6) of organs compared to CT-CLIP and 64% (9/14) of organs compared to Merlin. These results suggest that the proposed model exhibits high generalization performance in a clinical evaluation setting using actual radiology report sentences. Furthermore, in zero-shot finding-wise lesion classification tasks, our model achieved a higher AUROC in 83% (25/30) of finding categories compared to Merlin. We also confirmed performance comparable to existing Vision-Language Models (VLMs) in radiology report generation tasks. Our results demonstrate that the organ-separated learning framework can serve as a realistic and effective design guideline for the practical implementation of 3D-CT foundation models. The source code and pretrained models are publicly available at https://github.com/jichi-labo/TotalFM.
Kohei Yamamoto, Tomohiro Kikuchi
Department of Radiology, Jichi Medical University · Data Science Center, Jichi Medical University
CT vision-language foundation models have demonstrated promising performance across downstream tasks, but are typically trained with whole-volume representations that dilute fine-grained anatomical signals. Fine-grained vision-language pre-training addresses this by aligning anatomy-level visual features with anatomy-specific text, but in doing so discards the global context that whole-volume models provide. Furthermore, existing fine-grained approaches train from scratch, making them computationally expensive. We introduce Anatomy Contextualized Adaptation (ACA), a lightweight framework that adapts frozen CT foundation model representations for anatomy-level vision-language alignment while enhancing global contextualization. ACA uses TotalSegmentator to decompose CT volumes into anatomy-level embeddings, which are refined via a transformer that captures cross-anatomy relationships, and aligned to both per-anatomy and scan-level text extracted from radiology reports. Evaluated on Merlin and CT-RATE, ACA consistently outperforms both the frozen foundation model baselines and existing fine-grained methods in zero-shot finding classification, while requiring less than one hour of training once embeddings are cached. The attention weights learned by ACA's inter-anatomy transformer additionally indicate plausible cross-anatomy context routing. Altogether, these results support ACA as a lightweight approach for adapting CT foundation models to anatomically grounded vision-language alignment while preserving and enhancing global anatomical context.
Roshan Kenia, Stephanie L McNamara, William Lotter
Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Data Science, Dana-Farber Cancer Institute, Boston, MA · Department of Radiology, Massachusetts General Hospital, Boston, MA +2
Recent breakthroughs in self-supervised learning (SSL), such as the Latent-Euclidean Joint-Embedding Predictive Architecture (LeJEPA), alongside successes in integrating visual encoders with language models, have driven the demand for adaptable, high-capacity vision encoders in Computed Tomography (CT). In this work, we explore 2D slice-based architectures as a flexible alternative to native 3D models for processing volumetric CT data. Using the CT-RATE dataset, we trained DALE-CT (Depth-Aware Latent-Euclidean Computed Tomography), a 2D model family built entirely from scratch using LeJEPA, and compared its performance against a continually pre-trained DINOv2 baseline. To enhance representation quality, we developed a novel 3D depth-aware pre-training strategy anchored by dense auxiliary supervision from both automated anatomical masks and human-annotated abnormalities. Under linear probe evaluation with Multiple Instance Learning (MIL) for multi-abnormality detection, the frozen backbone of this dual-supervised model (DALE-CT-2S) achieves a Macro AUROC of 0.833. This performance demonstrates near-parity with state-of-the-art 3D vision-language models, achieved entirely from scratch with significantly less data and no textual supervision. To ensure reproducibility, all training code, evaluation scripts, and model weights have been made publicly available.
Evan W. Damron, Mahmut S. Gokmen, Mitchell A. Klusty +3
College of Medicine Office of Research, University of Kentucky, Lexington, KY 40506 USA