Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Tasks

Rubén Moreno-Aguado; Alba Magallón; Victor Moreno; Yingying Fang; Guang Yang

cs.CVApr 5, 2026

Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Tasks

Authors: Rubén Moreno-Aguado, Alba Magallón, Victor Moreno, Yingying Fang, Guang Yang

Organizations: Bioengineering Department and Imperial-X, Imperial College London, London, UK · Department of Computer Science, University of Manchester, Manchester, UK · Oncology Data Analytics Program, Catalan Institute of Oncology, Barcelona, Spain · Colorectal Cancer Group, ONCOBELL Program, Institut d’Investigació Biomèdica de Bellvitge, Barcelona, Spain · Consortium for Biomedical Research in Epidemiology and Public Health, Madrid, Spain · Department of Clinical Sciences, Faculty of Medicine and Health Sciences, Universitat de Barcelona Institute of Complex Systems, University of Barcelona, Barcelona, Spain · National Heart and Lung Institute, Imperial College London, London, UK · Cardiovascular Research Centre, Royal Brompton Hospital, London, UK · School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK

Abstract

There is substantial interest in developing artificial intelligence systems to support radiologists across tasks ranging from segmentation to report generation. Existing computed tomography (CT) foundation models have largely focused on building generalist vision-language systems capable of tasks such as question answering and report generation. However, training reliable vision-language systems requires paired image-text data at a scale that remains unavailable in CT. Moreover, adapting the underlying visual representations to downstream tasks typically requires partial or full backbone fine-tuning, a computationally demanding process inaccessible to many research groups. Instead, foundation models should prioritise learning robust visual representations that enable efficient transfer to new tasks with minimal labelled data and without backbone fine-tuning. We present VoxelFM, a 3D CT foundation model trained with self-distillation using the DINO framework, which learns semantically rich features without language supervision. We evaluated VoxelFM across seven categories of clinically relevant downstream tasks using frozen backbone representations with lightweight probes: classification, regression, survival analysis, instance retrieval, localisation, segmentation, and report generation. VoxelFM matched or outperformed four existing CT foundation models across all task categories. Despite receiving no language supervision during pre-training, VoxelFM surpassed models explicitly trained with language-alignment objectives, including on report generation. Our results indicate that current CT foundation models perform significantly better as feature extractors for lightweight probes rather than as vision encoders for vision-language models. Model weights and training code are publicly available.

Figures & tables

Task	Dataset	Metric	VoxelFM		Merlin	RadFM	M3D	CT-CLIP
Classification	CT-RATE	AUROC	0.870 (0.006)	$*$	0.798 (0.006)	0.759 (0.002)	0.740 (0.008)	0.574 (0.002)
	Merlin	AUROC	0.797 (0.005)		0.810 (0.005)	0.668 (0.006)	0.649 (0.006)	0.538 (0.006)
	RSNA-STR	AUROC	0.760 (0.017)	$*$	0.597 (0.019)	0.570 (0.019)	0.529 (0.019)	0.542 (0.019)
	Mycobacterial	AUROC	0.799 (0.031)		0.750 (0.035)	0.768 (0.033)	0.722 (0.036)	0.639 (0.040)
	iCTCF-Covid	AUROC	0.845 (0.027)	$*$	0.760 (0.033)	0.655 (0.038)	0.518 (0.042)	0.517 (0.042)
	iCTCF-Severity	AUROC	0.792 (0.054)		0.752 (0.057)	0.742 (0.057)	0.547 (0.061)	0.515 (0.060)

Table 1: Performance comparison across seven downstream tasks. Results are reported for classification (AUROC), regression (MAE), survival analysis (AUROC and C-index), retrieval (Recall@10), localisation (MAE), segmentation (DICE), and report generation (F1). For regression and localisation, lower MAE is better (

\downarrow

). All other metrics are higher-is-better (

\uparrow

). Classification results on CT-RATE and Merlin are macro-averaged over all abnormality classes. TotalSegmentator segmentation results are reported as both micro- and macro-averaged DICE. Values in parenthesis are standard errors. Bold values denote the best-performing model in each row.

*

denotes that the difference in performance between VoxelFM and next best model is statistically significant (p

<

0.05).

Figure 1: Overview of the evaluation protocol. All evaluations use pre-computed embeddings from a pre-trained encoder backbone. The first common step is (A): embeddings are extracted by the backbone and cached for downstream use. (B) Classification and regression: two methods are available depending on the token type used. For the class token, a two-layer MLP is applied, and for patch tokens, a Q-Former with cross-attention is used instead. (C) Instance retrieval: class tokens are ranked against a query token by cosine similarity. (D) Localisation: patch tokens are decoded to normalised 3D coordinates via multi-head self-attention, with a softmax operation used to weight the positional coordinates. (E) Segmentation: patch tokens are reshaped into a 3D feature map and decoded through 3D convolutional and upsampling layers. (F) Report generation: patch tokens are projected via cross-attention and combined with a system prompt before being passed to a large language model (LLM).

Figure 2: Bar chart summary of model performance across six tasks. Results are shown for classification (AUROC), regression (MAE $\downarrow$ ), survival analysis (AUROC), localisation (MAE $\downarrow$ ), segmentation (DICE), and retrieval (Recall@10), with 95% confidence intervals displayed for each model. TotalSegmentator segmentation is reported as micro-averaged DICE. CT-RATE and Merlin classification results are macro-averaged over all abnormality classes. See Table 1 for full numerical results.

Figure 3: Per-abnormality breakdown of classification and report generation performance across 18 findings. (Left) Binary classification F1 scores (threshold $=0.5$ ) for each of the 18 abnormalities in the CT-RATE dataset, where an individual Q-Former probe is trained per label as described in Figure 1 . (Right) Corresponding report generation F1 scores for the same 18 CT-RATE abnormality labels, aligned row-by-row with the left panel. A red cross marks the best-performing classifier result in each row.

Figure 4: Effect of downstream dataset size. Q-Former probes are trained on various fractions of labelled training data (20%–100%) and evaluated on a fixed held-out test set. (Left) iCTCF-Covid. (Right) RSNA-STR. Error bars represent 95% confidence intervals.

Figure 5: Comparison of inference strategies and feature aggregation methods. (Left) MLP applied to the class token versus a single-layer Q-Former applied to patch tokens for classification tasks. CT-RATE and Merlin results are macro-averaged over their respective abnormality labels. Error bars represent 95% confidence intervals. (Right) Chunked 2.5D versus full 3D inference across classification tasks.

Figure 6: Overview of the VoxelFM pre-training framework. Student and teacher networks share identical ViT-based architectures with class and patch token heads. The teacher network is updated via an exponential moving average of the student parameters. For each CT volume, two global and eight local crops are generated. Only the student processes local crops, and random masking is applied to its global crops. Training minimises a DINO loss ( $\mathcal{L}_{\text{CLS}}$ ) on class tokens and an iBOT loss ( $\mathcal{L}_{\text{Patch}}$ ) on masked patch tokens, both using cross-entropy against teacher outputs. Both output heads consist of an MLP with a bottleneck layer followed by a prototype layer. The CT render is from NSCLC-Radiomics [ 2 ] .

Dataset	Location	Clinical Focus	Studies
CT-RATE [ 23 ]	Lung	Lung Abnormalities	25,692
INSPECT [ 29 ]	Lung	Pulmonary Embolism	23,203
NLST [ 45 ]	Lung	Lung Cancer	13,587
RIDER-Lung-PET-CT [ 43 ]	Lung	Lung Cancer	413
Lung-PET-CT-Dx [ 35 ]	Lung	Lung Cancer	268
Anti-PD-Lung [ 41 ]	Lung	Lung Cancer	211

Table 2: Datasets used for training the foundation model.

Dataset	Targets	Studies
CT-RATE [ 23 ]	Radiology Reports, 18 Abnormalities	27,256
Merlin [ 9 ]	Radiology Reports, 31 Abnormalities	25,494
RSNA-STR [ 16 ]	Has Pulmonary Embolism	7,280
iCTCF [ 46 ]	SARS-CoV-2 nucleic acids, Morbidity	1,321
Mycobacterial [ 24 ]	Tuberculosis	1,301
TotalSegmentator [ 65 ]	Semantic segmentation 117 classes	1,204

Table 3: Datasets used for downstream evaluations. The number of studies does not include augmentations and test samples. Due to its large size, the NLST only represents a sample of the full dataset.

Model	Pre-training	Training Data	Resolution	Patch Size	Architecture
RadFM	Text generation	16M 2D + 500K 3D (multi-modal)	64x256x256	4x32x32	3D ViT
Merlin	CLIP + Supervised	15K CT	160x224x224	—	ResNet152
M3D	CLIP	120K CT	32x256x256	4x16x16	3D ViT
CT-CLIP	CLIP	26K CT	240x480x480	10x20x20	2D/3D ViT
VoxelFM	Self-supervised	137K CT	112x112x112	14x14x14	3D ViT

Table 4: Comparison of baseline foundation models and VoxelFM. Overview of pre-training strategies, training data, input resolution, patch size, and backbone architecture for each model. RadFM is trained on 2D and 3D modalities (X-ray, CT, MRI, PET). VoxelFM (bottom) is our proposed model.

Explore similar work

TotalFM: An Organ-Separated 3D-CT Foundation Model Leveraging Large-Scale Routine Clinical Radiology Data
Jan 1, 2026Kohei Yamamoto, Tomohiro Kikuchi3D Foundation Models Medical Imaging Foundation Models
Department of Radiology, Jichi Medical University · Data Science Center, Jichi Medical University
Anatomy Contextualized Adaption of CT Foundation Models
Jul 29, 2026Roshan Kenia, Stephanie L McNamara, William LotterMedical Imaging Foundation Models VLM Adaptation
Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Data Science, Dana-Farber Cancer Institute, Boston, MA · Department of Radiology, Massachusetts General Hospital, Boston, MA +2
DALE-CT: Depth-Aware Foundation Models for Computed Tomography
Jun 5, 2026Evan W. Damron, Mahmut S. Gokmen, Mitchell A. Klusty +3Medical Imaging Foundation Models Self-Supervised Pre-Training
College of Medicine Office of Research, University of Kentucky, Lexington, KY 40506 USA

Jan 1, 2026J/K move · Enter open · S save

TotalFM: An Organ-Separated 3D-CT Foundation Model Leveraging Large-Scale Routine Clinical Radiology Data

Kohei Yamamoto, Tomohiro Kikuchi

Department of Radiology, Jichi Medical University · Data Science Center, Jichi Medical University

3D Foundation Models Medical Imaging Foundation Models Radiology VLMs Medical Image Classification 3D VLMs

Open paper AskPDF arXiv

While foundation models in radiology are expected to be applied to various clinical tasks, computational cost constraints remain a major challenge when training on 3D-CT volumetric data. In this study, we propose TotalFM, a radiological foundation model that efficiently learns the correspondence between 3D-CT images and linguistic expressions based on the concept of organ separation, utilizing a large-scale dataset of 140,000 series. By automating the creation of organ volume and finding-sentence pairs through segmentation techniques and Large Language Model (LLM)-based radiology report processing, and by combining self-supervised pre-training via VideoMAE with contrastive learning using volume-text pairs, we aimed to balance computational efficiency and representation capability. In zero-shot organ-wise lesion classification tasks, the proposed model achieved higher F1 scores in 83% (5/6) of organs compared to CT-CLIP and 64% (9/14) of organs compared to Merlin. These results suggest that the proposed model exhibits high generalization performance in a clinical evaluation setting using actual radiology report sentences. Furthermore, in zero-shot finding-wise lesion classification tasks, our model achieved a higher AUROC in 83% (25/30) of finding categories compared to Merlin. We also confirmed performance comparable to existing Vision-Language Models (VLMs) in radiology report generation tasks. Our results demonstrate that the organ-separated learning framework can serve as a realistic and effective design guideline for the practical implementation of 3D-CT foundation models. The source code and pretrained models are publicly available at https://github.com/jichi-labo/TotalFM.

Abstract

Explore similar work

TotalFM: An Organ-Separated 3D-CT Foundation Model Leveraging Large-Scale Routine Clinical Radiology Data

Anatomy Contextualized Adaption of CT Foundation Models

DALE-CT: Depth-Aware Foundation Models for Computed Tomography