Impact of Pretraining Length, Model Size, and Annotation Budget on Downstream Clinical Tasks of Foundation Models in Radiotherapy
Organizations: Department of Computer Science, University of Copenhagen, Denmark · Department of Oncology, Copenhagen University Hospital, Denmark
Abstract
Cone-beam CT (CBCT) scans acquired during radiotherapy capture changes such as tumor shrinkage and radiation-induced tissue changes, offering great potential for foundation models that help monitor individual treatment responses. However, to our knowledge, no large-scale pretrained models for radiotherapy CBCT exist, and evaluating modeling choices is difficult since downstream tasks are not known in advance. We examine how pretraining length, model size, and finetuning strategy relate to downstream clinical performance, and how metrics tracked during pretraining correlate with it. We pretrain vision transformer encoders with masked image modeling on about 150,000 raw 3D radiotherapy CBCT scans, tracking representation quality via validation loss, effective rank, and linear probing. Downstream tasks are classification of feeding tube presence ( CBCTs, patients) and clinical characteristics (T-, N-stage, sex, age, tumor site; patients) in head and neck cancer. We compare low-rank adaptation (LoRA) and full finetuning against no pretraining, varying annotated dataset size. Pretext validation loss was the most consistent predictor of downstream performance; effective rank and linear probing were also associated, but not consistently across model sizes and finetuning methods. LoRA performed comparably to full finetuning while training only 0.5-2.5% of the parameters. Fully finetuned models consistently outperformed models without pretraining across all sizes, with AUROC 0.91-0.97 vs. 0.58-0.85 for feeding tube classification and 0.99-1.00 vs. 0.71-0.97 for sex classification. Model size had only a small, inconsistent effect, suggesting smaller models suffice at lower computational cost. Code and pretrained models (non-commercial license): https://github.com/alexCortinaU/RT-CBCT_SSL
Figures & tables
| A. Pretraining set | |||||
|---|---|---|---|---|---|
| Total | Lymphoma | H&N | Esophagus | Lung | |
| Train (95%) | |||||
| No. patients | 7,004 | 1,130 | 2,166 | 1,049 | 2,659 |
| No. RT plans | 7,592 | 1,251 | 2,317 | 1,089 | 2,935 |
| No. CBCTs | 145,027 | 16,655 | 66,260 | 18,595 | 43,517 |
| Validation (5%) | |||||
| Full finetuning | LoRA | ||
|---|---|---|---|
| Model | Metric | [95% CI] | [95% CI] |
| ViT-t | Normalized RankMe | 0.636 [-0.245, 0.987] | 0.321 [-0.667, 0.887] |
| Linear probing AUC | 0.539 [-0.223, 0.912] | 0.588 [-0.170, 0.962] | |
| Validation loss | -0.818 [-0.988, -0.304] | -0.976 [-1.000, -0.810] | |
| ViT-s | Normalized RankMe | 0.927 [0.660, 1.000] | 0.891 [0.487, 1.000] |
| Linear probing AUC | 0.818 [0.208, 1.000] | 0.758 [0.202, 0.962] |
Medical Imaging Foundation Models
Medical imaging foundation models learn reusable representations for tasks across clinical images, modalities, and institutions.
See all 253 papers