Few-Shot Continual Learning for 3D Brain MRI with Frozen Foundation Models

Chi-Sheng Chen; Xinyu Zhang; Guan-Ying Chen; Qiuzhe Xie; Fan Zhang; En-Jui Kuo

eess.IVFeb 26, 2026

Few-Shot Continual Learning for 3D Brain MRI with Frozen Foundation Models

Authors: Chi-Sheng Chen, Xinyu Zhang, Guan-Ying Chen, Qiuzhe Xie, Fan Zhang, En-Jui Kuo

Organizations: Independent Researcher, USA · Indiana University at Bloomington, United States · Independent Researcher, Taiwan · National Taiwan University, Taiwan · Boise State University, United States · National Yang Ming Chiao Tung University, Taiwan

Abstract

Foundation models pretrained on large-scale 3D medical imaging data face challenges when adapted to multiple downstream tasks under continual learning with limited labeled data. We address few-shot continual learning for 3D brain MRI by combining a frozen pretrained backbone with task-specific Low-Rank Adaptation (LoRA) modules. Tasks arrive sequentially -- tumor segmentation (BraTS) and brain age estimation (IXI) -- with no replay of previous task data. Each task receives a dedicated LoRA adapter; only the adapter and task-specific head are trained while the backbone remains frozen, thereby eliminating catastrophic forgetting by design (BWT=0). In continual learning, sequential full fine-tuning suffers severe forgetting (T1 Dice drops from 0.80 to 0.16 after T2), while sequential linear probing achieves strong T1 (Dice 0.79) but fails on T2 (MAE 1.45). Our LoRA approach achieves the best balanced performance across both tasks: T1 Dice 0.62 $\pm$ 0.07, T2 MAE 0.16 $\pm$ 0.05, with zero forgetting and $<$ 0.1% trainable parameters per task, though with noted systematic age underestimation in T2 (Wilcoxon $p<0.001$ ). Frozen foundation models with task-specific LoRA adapters thus offer a practical solution when both tasks must be maintained under few-shot continual learning.

Figures & tables

Panel of Figure 1 — Figure 1: Framework: frozen pretrained backbone $f_{\theta}$ with task-specific LoRA adapters $\phi_{k}$ and heads $h_{k}$ . Gray = frozen; blue = trainable per task. At inference for T1 (resp. T2): backbone + $\phi_{1}$ + $h_{1}$ (resp. $\phi_{2}$ + $h_{2}$ ).

Method	T1 Dice $\uparrow$	T2 MAE $\downarrow$	T1 after T2	BWT
LoRA (enc+dec)	$0.60\pm 0.08$	$0.012\pm 0.003$	(=T1)	$\mathbf{0.00}$
Sequential Linear	$0.79\pm 0.01$	$1.45\pm 0.03$	$0.78\pm 0.01$	$-0.01\pm 0.01$
Sequential FT	$0.80\pm 0.02$	$0.005^{\dagger}\pm 0.003$	$0.16\pm 0.19$	$-0.65\pm 0.17$
EWC	$0.79\pm 0.02$	$0.001^{\dagger}\pm 0.001$	$0.15\pm 0.24$	$-0.65\pm 0.23$
LwF	$0.80\pm 0.02$	$0.020\pm 0.009$	$0.25\pm 0.22$	$-0.56\pm 0.24$
Replay	$0.79\pm 0.01$	$0.021\pm 0.013$	$0.01\pm 0.01$	$-0.78\pm 0.02$

Table 1: Main continual learning results (mean

\pm

std over seeds 42–44, n_shot=32). BWT negative = forgetting. † Seq FT and EWC T2 MAE likely overfit on few-shot validation (MAE

<

0.01 implausible; per-task linear achieves 0.063 on full eval).

Figure 2: Task 1 tumor segmentation (BraTS): representative sample. Input (T2-FLAIR), Ground Truth, LoRA prediction, overlay (red=GT, green=pred). Full six-sample figure in Appendix (Fig. 4 ).

Figure 3: Task 2 brain age regression (IXI): representative sample. Orthogonal MRI views (Axial, Coronal, Sagittal) and predicted vs. actual age. Full six-sample figure in Appendix (Fig. 5 ).

Method	T1 Dice $\uparrow$	T2 MAE $\downarrow$	Setting
LoRA (enc+dec)	$0.60\pm 0.08$	$\mathbf{0.012}\pm 0.003$	Continual
EWC	$0.79\pm 0.02$	$0.001^{\dagger}\pm 0.001$	Continual
LwF	$0.80\pm 0.02$	$0.020\pm 0.009$	Continual
Replay	$0.79\pm 0.01$	$0.021\pm 0.013$	Continual
Sequential Linear	$0.79\pm 0.01$	$1.45\pm 0.03$	Continual
Sequential FT	$0.80\pm 0.02$	$0.005^{\dagger}\pm 0.003$	Continual (forgets)

Table 2: Extended comparison. Per-task: separate model per task (patch=32, epochs=50; not continual). EWC and Seq FT T2 MAE likely overfit (see Table 1 ).

LoRA Config	T1 Dice $\uparrow$	T2 MAE $\downarrow$
Encoder-only	0.19	0.20
Encoder+decoder	0.50	0.20

Table 3: LoRA placement ablation. Enc+dec = encoder+decoder.

n_shot	T1 Dice $\uparrow$	T2 MAE $\downarrow$
16	$0.45\pm 0.04$	$0.33\pm 0.01$
32	$0.62\pm 0.07$	$0.16\pm 0.05$

Table 4: Shot count ablation (LoRA enc+dec, seeds 42–43 for n_shot=16).

Method	T1 Dice $\uparrow$	T2 MAE $\downarrow$	T1 after T2	BWT
EWC	$0.84\pm 0.02$	$0.001^{\dagger}\pm 0.0003$	$0.07\pm 0.10$	$-0.77\pm 0.11$
LwF	$0.82\pm 0.01$	$0.038\pm 0.014$	$0.25\pm 0.41$	$-0.58\pm 0.42$

Table 5: n_shot=64 results (EWC, LwF; seeds 42–44). † EWC T2 MAE likely overfits.

Method	T1 Dice $\uparrow$	T2 MAE $\downarrow$	T2 after T1	BWT
LoRA (enc+dec)	$0.50\pm 0.08$	$0.005\pm 0.002$	—	$\mathbf{0.00}$
Sequential Linear	$0.78\pm 0.01$	$0.26\pm 0.003$	$0.36\pm 0.05$	$0.10\pm 0.05$
Sequential FT	$0.81\pm 0.01$	$0.001^{\dagger}\pm 0.0003$	$7.17\pm 1.41$	$7.16\pm 1.41$

Table 6: Phase 3 (t32_t3t2, T3

\to

T2 order, n_shot=32). BWT = T2 MAE after T1 minus T2 MAE after T2 (positive = T3 forgetting). † Seq FT T2 MAE likely overfits.

Seed	T1 Dice $\uparrow$	T2 MAE $\downarrow$
42	0.523	0.014
43	0.598	0.008
44	0.680	0.012

Table 7: Per-seed LoRA results (enc+dec, n_shot=32).

Task	Params (trainable)	GPU peak (MB)	Time (sec)
T1 (segmentation)	46,872	1,670	$\sim$ 4k–26k
T2 (regression)	28,177	688	$\sim$ 95–780

Table 8: Resource usage (LoRA, n_shot=32).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Task 1 tumor segmentation (BraTS): full six-sample stack. Input (T2-FLAIR), Ground Truth, LoRA prediction, overlay (red=GT, green=pred). Samples selected by highest slice-level Dice.

Figure 5: Task 2 brain age regression (IXI): full six-sample stack. Orthogonal MRI views and regression results. Samples selected by lowest prediction error.

Figure 6: Task 1 failure cases: under-segmentation (BraTS). Six samples with lowest Dice. Red = ground truth; green = LoRA prediction.

Figure 7: Task 2 failure cases: age underestimation (IXI). Six samples with highest $|$ predicted $-$ true $|$ error.

Explore similar work

A Generalizable Feature Extractor for Alzheimer's-Related Brain MRI Tasks
Sep 4, 2026Reza Rajabli, D. Louis CollinsMedical Imaging Foundation Models Transfer Learning
McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University · Department of Biomedical Engineering, McGill University
A continually expandable foundation model for brain MRI
Aug 8, 2026Michail Mamalakis, Carmen Jimenez-Mesa, Yonghao Li +8Medical Imaging Foundation Models Continual Learning
1*Cancer Research UK Cambridge Institute, University of Cambridge Li Ka Shing Centre, Robinson Way, Cambridge, CB2 0RE, Cambridgeshire, United Kingdom. · Department of Computer Science and Technology, University of Cambridge, 15 JJ Thomson Ave, Cambridge, CB3 0FD, Cambridgeshire, United Kingdom. · Department of Psychiatry, University of Cambridge, Hills Road, Cambridge, CB2 2QQ, Cambridgeshire, United Kingdom. +7
BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis
Jul 20, 2026Moona Mazher, Abdul Qayyum, Steven A. Niederer +1Medical Imaging Foundation Models Self-Supervised Pre-Training
UCL Hawkes Institute, Department of Computer Science, University College London, London, United Kingdom · National Heart & Lung Institute, Imperial College London, London, United Kingdom · Division of Cardiovascular Medicine, Stanford Cardiovascular Institute, Stanford University, California, USA

Sep 4, 2026J/K move · Enter open · S save

A Generalizable Feature Extractor for Alzheimer's-Related Brain MRI Tasks

Reza Rajabli, D. Louis Collins

McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University · Department of Biomedical Engineering, McGill University

Medical Imaging Foundation Models Transfer Learning Cross-Domain Transfer Learning Alzheimer's Disease Classification Alzheimer's Disease Progression Prediction

Open paper AskPDF arXiv

When there is not enough labeled data to properly train deep learning models, transfer learning can help. We still do not fully understand how effective it is in neuroimaging, especially for Alzheimer's disease research. It is also not clear if these transferred models can work on new datasets without being retrained for each specific task. We evaluate whether a compact, supervised pretrained model can serve as a reusable foundation model for downstream neuroimaging tasks. We freeze the 7.18 million weights of a 3D CNN previously trained for brain-age prediction, and adapt it to each task using Low-Rank Adaptation (LoRA), requiring only ~1% additional trainable parameters. We evaluate generalizability in six experiments. Adapting the model to classify cognitively normal versus Dementia on ADNI gave an AUC of 0.964 on held-out folds (Experiment #1). Applying that adapted model unchanged to OASIS-3, with no retraining, gave an AUC of 0.871 (Experiment #2). Reusing its output logit together with age and a cognitive score distinguished stable from progressing MCI with an AUC of 0.828 (Experiment #3). Adapting the same backbone to predict amyloid positivity from structural MRI gave an AUC of 0.804 (Experiment #4). Finally, the same approach estimated ICV-normalized hippocampal and white matter hypointensity volumes directly from the T1w image, with R^2 of 0.80 and 0.91 respectively, tasks normally addressed with much larger U-Net networks (Experiments #5 and #6). A compact model supervised on brain age can therefore serve as a reusable backbone, adapting to each task with ~1% additional parameters and transferring to an unseen cohort without any training. Our findings suggest that a carefully trained brain age model can serve as an effective foundation model for Alzheimer's related tasks, even under strict data constraints.

Abstract

Explore similar work

A Generalizable Feature Extractor for Alzheimer's-Related Brain MRI Tasks

A continually expandable foundation model for brain MRI

BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis