Convergent Performance, Divergent Mechanisms in LoRA-Adapted Vision Foundation Models for Brain MRI
Authors: Cecilia Diana-Albelda, Fiona Kekwick, Raghav Mehta, Aleksandra Kulbaka, Paul M. Matthews, Jesus Bescos, Álvaro García-Martín, Wenjia Bai, Bernhard Kainz
Organizations: Department of Electronic Technology and Communications, Universidad Autónoma de Madrid, Madrid, Spain · Department of Computing, Imperial College London, London, United Kingdom · Department of Brain Sciences, Imperial College London, London, United Kingdom · UK Dementia Research Institute Centre, Imperial College London, London, United Kingdom · Rosalind Franklin Institute, Didcot, United Kingdom · Department of Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany
Adapting Vision Foundation Models (VFMs) to medical imaging via LoRA has become standard practice, yet model selection is guided almost entirely by downstream task metrics, treating backbones with similar performance as interchangeable. We investigate whether equal performance implies equal internal representational shift. We analyse four ViT-B/16 backbones, DINOv3, SigLIP2, MerMED-FM, and BiomedCLIP, pre-trained with different objectives (self-distillation vs. contrastive vision-language) and on different data domains (natural vs. medical images). We adapt each to brain tumour segmentation and evaluate on six datasets covering both segmentation and Alzheimer's disease classification. Using attention divergence, weight-change analysis, and activation patching, we find that the four models converge to similar segmentation performance (mean Dice 78-81%) through markedly different internal dynamics, yet diverge in cross-task transfer: segmentation adaptation improves Alzheimer's disease classification for DINOv3 but degrades it for the other three backbones. We reveal a dissociation between modification magnitude and causal importance: the most heavily modified layers are not the causally relevant ones, with early-to-mid layers identified as sufficient for segmentation recovery. Restricting LoRA to these layers matches or approaches full-layer segmentation while reducing classification degradation in the three affected backbones, providing a mechanistically grounded strategy that is more parameter-efficient and more robust than standard full-layer adaptation.
Figures & tables
Figure 1: Overview of the adaptation and analysis pipeline. (Panel A) Four brain MRI modalities (T1n, T1c, T2f, T2w) are fused via a trainable 1×1 convolution into a 3-channel input, processed by a frozen ViT-B/16 encoder with trainable LoRA modules applied to the query (Q) and value (V) projections of every transformer block in the default all-layer configuration. Components marked with (fusion convolution, LoRA modules, and segmentation head) are jointly trained on BraTS-GLI. (Panel B) Segmentation is evaluated on three BraTS datasets. For classification, the full encoder (including LoRA weights) remains frozen and only a logistic regression classifier ( ) is fitted on each classification dataset using mean-pooled features from ℓ12 ; the fusion convolution is bypassed (T1n input only). Four mechanistic interpretability tools are applied per layer to analyse the adaptation process: JS divergence, weight-change ratio, off-brain attention fraction, and activation patching.
Dice (%) ↑
HD95 (mm) ↓
Backbone
Adaptor
WT
TC
ENH
Mean
WT
TC
ENH
Self-distillation Learning
DINOv3 [ 12 ]
Frozen
80.23 ± 0.02
75.60 ± 0.04
57.30 ± 0.33
71.04 ± 0.11
11.10 ± 0.73
6.72 ± 0.44
5.45 ± 0.16
LoRA ( r=4 )
86.91 ± 0.04
84.20 ± 0.11
70.51 ± 0.37
80.54 ± 0.13
6.02 ± 0.33
4.74 ± 0.42
4.28 ± 0.10
LoRA ( r=8 )
87.10 ± 0.06
84.16 ± 0.06
71.01 ± 0.43
80.76 ± 0.15
5.67 ± 0.10
4.53 ± 0.09
4.19 ± 0.16
LoRA ( r=12 )
87.29 ± 0.04
84.16 ± 0.10
71.20 ± 0.71
80.88 ± 0.24
5.81 ± 0.04
5.04 ± 0.27
4.21 ± 0.09
Table 1: In-distribution segmentation performance on BraTS-GLI test set. Four ViT-B/16 backbones are evaluated in two configurations: Frozen (encoder fully frozen, only the segmentation head is trained) and LoRA (encoder adapted via LoRA at ranks r∈{4,8,12,16} , trained jointly with the segmentation head). Dice coefficient (%) and Hausdorff distance at the 95th percentile (HD95, mm) are reported for three tumour sub-regions: whole tumour (WT = enhancing + necrotic + edema), tumour core (TC = enhancing + necrotic), and enhancing tumour (ENH). All values are mean ± std across 3 random seeds. Best result per model/metric combination is shown in bold .
BraTS-Africa
BraTS-PED
Backbone
Adaptor
WT
TC
ENH
WT
TC
ENH
Self-distillation Learning
DINOv3 [ 12 ]
Frozen
69.70 ± 2.21
58.50 ± 0.07
41.55 ± 0.79
55.44 ± 0.42
35.23 ± 1.43
29.49 ± 1.36
LoRA ( r=8 )
85.55 ± 0.30
73.72 ± 0.33
66.94 ± 0.54
79.81 ± 0.79
47.33 ± 1.88
41.88 ± 0.63
MerMED-FM [ 14 ]
Frozen
76.19 ± 0.48
56.39 ± 0.20
41.85 ± 0.79
57.13 ± 1.44
33.70 ± 0.90
28.15 ± 0.38
LoRA ( r=8 )
83.73 ± 0.29
72.85 ± 0.30
64.77 ± 0.08
77.88 ± 0.69
45.72 ± 0.46
40.36 ± 1.41
Table 2: Out-of-distribution segmentation performance (Dice %). All models are trained on BraTS-GLI and evaluated on BraTS-Africa (Sub-Saharan African adult glioma) and BraTS-PED (pediatric glioma) without retraining. Dice coefficient (%) is reported for three tumour sub-regions: whole tumour (WT = enhancing + necrotic + edema), tumour core (TC = enhancing + necrotic), and enhancing tumour (ENH). Only r=8 is reported. All values are mean ± std across 3 random seeds. Best result per column in bold .
Figure 2: Qualitative segmentation results for a representative axial slice from each dataset. Tumour sub-regions (enhancing, necrotic core, peritumoural edema) are overlaid on the T2f input. Each row corresponds to one dataset (BraTS-GLI, BraTS-Africa, BraTS-PED). For each backbone, the frozen ( F ) and LoRA-adapted ( A ) predictions are shown side by side. LoRA adaptation visibly refines region boundaries across all four backbones.
Backbone
Adaptor
OASIS-1 (binary)
ADNI (binary)
ADNI (3-class)
AUC
BAcc (%)
AUC
BAcc (%)
AUC
BAcc (%)
Self-distillation Learning
DINOv3 [ 12 ]
Frozen
0.839 ± 0.026
71.40 ± 2.43
0.747 ± 0.026
69.03 ± 1.18
0.606 ± 0.023
43.47 ± 2.93
LoRA ( r=8 )
0.868 ± 0.029
75.44 ± 3.13
0.799 ± 0.017
72.40 ± 1.03
0.637 ± 0.012
45.94 ± 0.98
MerMED-FM [ 14 ]
Frozen
0.878 ± 0.020
78.68 ± 1.22
0.835 ± 0.038
74.02 ± 4.10
0.654 ± 0.008
47.88 ± 2.71
LoRA ( r=8 )
0.830 ± 0.019
72.77 ± 1.12
0.772 ± 0.027
71.17 ± 1.43
0.603 ± 0.013
42.79 ± 2.55
Table 3: Alzheimer’s disease classification via linear probing. Each backbone is evaluated in two configurations: Frozen (original pre-trained encoder) and LoRA ( r=8 , adapted for segmentation on BraTS-GLI). Features are mean-pooled from the final transformer block and aggregated per subject; a logistic regression classifier is fitted on the training set. AUC and balanced accuracy (BAcc) are reported on OASIS-1 (binary AD classification) and ADNI (binary and 3-class). All values are mean ± std across 3 folds for the frozen models, and 3 folds and 3 seeds for the LoRA trained models. Best result per column in bold .
Figure 3: Per-layer mechanistic analysis of LoRA adaptation. (a) Attention Jensen-Shannon divergence between frozen and adapted attention distributions. (b) LoRA weight-change ratio ( ∥ΔW∥F/∥W∥F ), shown separately for query (Q, solid) and value (V, dashed) projections on a log scale. (c) Frozen off-brain attention fraction, measuring the fraction of attention mass directed to off-brain patches before adaptation. All metrics are computed on the BraTS-GLI test set and averaged across patients and MRI slices. Shaded bands show ±1 standard deviation across 3 seeds (absent in (c), which uses only pre-trained weights).
Figure 4: Last-layer attention maps for a BraTS-GLI patient. Each column shows one backbone in its frozen ( F ) and LoRA-adapted ( A ) configuration. Maps show block-12 attention from the CLS token to the patch tokens, averaged over heads; since SigLIP2 has no CLS token, its map shows the attention each patch receives, averaged over all patch queries. Each panel uses its own colour scale, so only spatial patterns are comparable. The frozen medical-domain models (MerMED-FM, BiomedCLIP) attend to anatomical structures, whereas the frozen natural-domain models (DINOv3, SigLIP2) distribute attention more diffusely. After adaptation, all four models redirect attention toward the tumour.
Figure 5: Activation patching results across segmentation and classification datasets. For each block ℓ , the adapted activation is injected into an otherwise frozen forward pass and the resulting task metric is recorded. Note that R(ℓ) is cumulative: the patched activation at block ℓ already incorporates LoRA from all preceding blocks. Top row (a–c): normalised segmentation recovery R(ℓ) on three BraTS datasets; values above zero indicate improvement over the frozen baseline, and values below zero indicate degradation. Bottom row (d–f): absolute classification AUC on three Alzheimer’s disease benchmarks (not normalised, since the frozen-to-adapted gap varies in sign across backbones); the logistic regression probe is refit on the patched features for each block. All metrics are averaged across patients and MRI slices. Shaded bands show ±1 standard deviation across 3 seeds.
Figure 6: Layer-selective LoRA performance across segmentation and classification datasets. Each backbone is evaluated with three LoRA configurations: early (blocks 1–6, diagonal hatching), all (blocks 1–12, solid fill), and late (blocks 7–12, dotted hatching). Black stars indicate frozen baseline performance; when absent, it falls below the plotted range. Upper row: mean Dice (%) on three segmentation datasets. Bottom row: AUC on three Alzheimer’s disease classification benchmarks. The rightmost column shows each model’s average across the three datasets in each task category.
Vision foundation model adaptation tailors pretrained image and video models to new tasks and domains while balancing accuracy, efficiency, and generalization.
Low-rank adaptation (LoRA) is usually tied to the pretrained backbone on which it is learned. We ask whether LoRAs trained independently for the same task on different pretrained backbones can be summed and applied unchanged to all contributing backbones. We call this construction LoRA superposition. Across seven ViT-B/16 CLIP backbones and eight image-classification tasks, the exact sum improves all 56 backbone-task pairs and achieves a mean accuracy gain of 30.21 percentage points over the unadapted backbone. The exact sum achieves a mean accuracy only 0.28 percentage points below that of individually trained solo LoRAs. In contrast, a leave-one-model-out (LOMO) sum produces a mean gain of only 0.06 percentage points over the unadapted backbones. Backbone-specific solo LoRAs are nearly orthogonal in parameter space, while pairwise cross-backbone responses are strongly concentrated on the diagonal. These results support a component-wise non-interference hypothesis, under which the frozen backbone acts as an implicit selector of its own component without explicit routing or cross-model alignment.
Keigo Watanabe, Hiroshi Kera, Kazuhiko Kawamoto
Chiba University, Chiba, Japan · National Institute of Informatics, Tokyo, Japan
Foundation models pretrained on large-scale 3D medical imaging data face challenges when adapted to multiple downstream tasks under continual learning with limited labeled data. We address few-shot continual learning for 3D brain MRI by combining a frozen pretrained backbone with task-specific Low-Rank Adaptation (LoRA) modules. Tasks arrive sequentially -- tumor segmentation (BraTS) and brain age estimation (IXI) -- with no replay of previous task data. Each task receives a dedicated LoRA adapter; only the adapter and task-specific head are trained while the backbone remains frozen, thereby eliminating catastrophic forgetting by design (BWT=0). In continual learning, sequential full fine-tuning suffers severe forgetting (T1 Dice drops from 0.80 to 0.16 after T2), while sequential linear probing achieves strong T1 (Dice 0.79) but fails on T2 (MAE 1.45). Our LoRA approach achieves the best balanced performance across both tasks: T1 Dice 0.62±0.07, T2 MAE 0.16±0.05, with zero forgetting and <0.1% trainable parameters per task, though with noted systematic age underestimation in T2 (Wilcoxon p<0.001). Frozen foundation models with task-specific LoRA adapters thus offer a practical solution when both tasks must be maintained under few-shot continual learning.
Chi-Sheng Chen, Xinyu Zhang, Guan-Ying Chen +3
Independent Researcher, USA · Indiana University at Bloomington, United States · Independent Researcher, Taiwan +3
The advent of vision foundation models, notably the Segment Anything Model (SAM), has catalyzed significant advancements in natural image segmentation. However, their direct transfer to medical imaging remains severely bottlenecked by profound domain gaps, such as cross-modality and cross-center shifts. Existing Parameter-Efficient Fine-Tuning (PEFT) methods facilitate the adaptation of SAM to medical domains; nevertheless, they frequently suffer from performance degradation under severe distribution shifts. This vulnerability primarily stems from the implicit entanglement of heterogeneous frequency components within a shared low-rank subspace, which directly exacerbates sub-optimal structural alignment and localized boundary blurring. To overcome this representational bottleneck, we propose the Fourier-Adaptive Nonlinear Low-Rank Adaptor (FAN-LoRA), a novel frequency-decoupled fine-tuning architecture. FAN-LoRA explicitly separates the optimization space by employing a B-spline-driven low-pass branch for global structural alignment, synergistically coupled with a discrete Fourier high-pass branch for local textural compensation. Extensive experiments across three challenging cross-modality and cross-center benchmarks demonstrate that FAN-LoRA consistently outperforms state-of-the-art PEFT baselines. Compared to the strongest competitors, our method achieves consistent improvements in average Dice scores and notable reductions in boundary errors, while maintaining a compact module size without compromising computational efficiency.
Ziquan Liu, Zhewei Zhu, Xuyang Shi
School of Information and Control Engineering, Southwest University of Science and Technology Mianyang, China