FAN-LoRA: A Fourier-Adaptive Nonlinear Low-Rank Adaptor for Medical Foundation Model Domain Adaptation

Ziquan Liu; Zhewei Zhu; Xuyang Shi

cs.CVAug 27, 2026

FAN-LoRA: A Fourier-Adaptive Nonlinear Low-Rank Adaptor for Medical Foundation Model Domain Adaptation

Authors: Ziquan Liu, Zhewei Zhu, Xuyang Shi

Organizations: School of Information and Control Engineering, Southwest University of Science and Technology Mianyang, China

Abstract

The advent of vision foundation models, notably the Segment Anything Model (SAM), has catalyzed significant advancements in natural image segmentation. However, their direct transfer to medical imaging remains severely bottlenecked by profound domain gaps, such as cross-modality and cross-center shifts. Existing Parameter-Efficient Fine-Tuning (PEFT) methods facilitate the adaptation of SAM to medical domains; nevertheless, they frequently suffer from performance degradation under severe distribution shifts. This vulnerability primarily stems from the implicit entanglement of heterogeneous frequency components within a shared low-rank subspace, which directly exacerbates sub-optimal structural alignment and localized boundary blurring. To overcome this representational bottleneck, we propose the Fourier-Adaptive Nonlinear Low-Rank Adaptor (FAN-LoRA), a novel frequency-decoupled fine-tuning architecture. FAN-LoRA explicitly separates the optimization space by employing a B-spline-driven low-pass branch for global structural alignment, synergistically coupled with a discrete Fourier high-pass branch for local textural compensation. Extensive experiments across three challenging cross-modality and cross-center benchmarks demonstrate that FAN-LoRA consistently outperforms state-of-the-art PEFT baselines. Compared to the strongest competitors, our method achieves consistent improvements in average Dice scores and notable reductions in boundary errors, while maintaining a compact module size without compromising computational efficiency.

Figures & tables

Fig. 1: Overall architecture of the proposed Fourier-Adaptive Nonlinear Low-Rank Adaptor (FAN-LoRA). The module fundamentally decouples spatial gradients by projecting input features into two parallel branches: a B-spline-driven Adaptive Nonlinear Layer (ANL) for low-frequency global structural alignment, and an explicit discrete Fourier branch for high-frequency local detail compensation.

Fig. 2: Visualization of the discrete Fourier projection basis matrix $\mathbf{U}_{\text{high}}$ . The columns are generated via sinusoidal sampling along discrete integer dimensions, capturing distinct high-frequency components to effectively extract texture residuals without excessive parameter overhead.

Source	Target	Modality	Anatomy
MM-WHS 2017 MR	MM-WHS 2017 CT	MR $\rightarrow$ CT	Heart
Promise 12	NCI-ISBI	MRI (1.5T/3T)	Prostate
FLARE 22	CHAOS	CT $\rightarrow$ MRI	Abdomen

TABLE I: Summary of datasets and domain adaptation settings.

Method	CHAOS
	Liver	R.kidney	L.kidney	Spleen	Avg.	HD95 $\downarrow$
Zero-shot	90.9	90.5	89.8	93.6	91.2	5.03
Fine-tuning	72.6	87.5	84.4	88	83.1	11.21
LoRA	94.3	91.6	91.5	94.6	93	2.84
FourierFT	94.7	92.5	91.4	94.5	93.3	2.71
DeLoRA	94.4	92.9	91.6	94.2	93.3	2.66

TABLE II: Quantitative comparison of different PEFT methods on the CHAOS target domain (MRI)

Method	NCI-ISBI 1.5T				NCI-ISBI 3T
	PZ	CG	Avg.	HD95 $\downarrow$	PZ	CG	Avg.	HD95 $\downarrow$
Zero-shot	67.2	87.7	77.5	37.68	66.4	89.9	78.2	31.52
Fine-tuning	65.8	90.2	78	34.3	58.9	91.7	75.3	33.69
LoRA	66.3	93.5	79.9	30.04	60.2	93.1	76.7	31.99
FourierFT	68.2	92.7	80.5	27.87	62.8	93.1	78	27.51
DeLoRA	65.7	93.2	79.5	30.4	65.5	92.8	79.2	27.12

TABLE III: Quantitative comparison of different PEFT methods on the NCI-ISBI target domain across 1.5T and 3T scanners

Method	MM-WHS 17 CT
	LAC	LVC	MYO	AA	Avg.	HD95 $\downarrow$
Zero-shot	68.5	91.3	87.2	90.7	84.4	30.07
Fine-tuning	70.2	91.9	87.3	91.3	85.2	28.79
LoRA	74.1	91.8	88.5	91	86.4	27.84
FourierFT	86.3	91.9	89.3	91.4	89.7	12.97
DeLoRA	73.8	91.6	88.3	91.2	86.5	26.95

TABLE IV: Quantitative comparison of different PEFT methods on the MM-WHS 2017 CT target domain

Method	FLARE 22
	Liver	R.kidney	Spleen	Pancreas	Aorta	IVC	RAG	LAG	Gallbladder	Esophagus	Stomach	L.kidney	Avg.	HD95 $\downarrow$
Zero-shot	96	95.8	96.2	83.2	94.3	92	56.2	67.2	88.8	78.3	90.7	68.9	83.97	11.02
Fine-tuning	96.2	96.4	97.4	83.5	95.9	93.4	77.2	77.3	90.3	88.7	92.9	80.3	89.13	4.88
LoRA	96.4	96.6	96.7	84.3	94.7	94.2	68.8	67.6	91	89.1	93.4	74	87.23	7.1
FourierFT	95.9	95.8	95.5	87.4	93.6	93.9	71.7	74.4	90.6	88.7	92.3	78.9	88.23	5.32
DeLoRA	96.3	96.4	96.4	86.4	94.7	94.4	75.2	76	90.9	88.5	92.9	81	89.09	4.34

TABLE V: Performance retention evaluation of different PEFT methods on the FLARE 22 source domain (CT)

Method	Promise 12		MM-WHS 17 MR
	prostate	HD95 $\downarrow$	LAC	LVC	MYO	AA	Avg.	HD95 $\downarrow$
Zero-shot	92.3	4.58	52.5	80.8	82.9	73.3	72.38	42.15
Fine-tuning	94.2	3.12	84.5	83.2	88.7	85.2	85.4	20.03
LoRA	94	3.45	80.2	83.2	87.9	82.9	83.6	20.55
FourierFT	94.1	2.85	81.1	82.8	87.2	82.1	83.37	23.46
DeLoRA	94.1	3.11	85.2	82.8	88.4	79.3	83.9	21.28

TABLE VI: Performance retention evaluation of different PEFT methods on the Promise 12 and MM-WHS 2017 MR source domains

Method	Trainable Parameters
LoRA	454,656
FourierFT	24,000
DeLoRA	442,392
SHiRA	442,368
FreqFiT	592128
AuroRA	111,072

TABLE VII: Comparison of trainable parameters for different PEFT adapters when integrated into the MedSAM image encoder.

Fig. 3: Training and validation loss curves on the FLARE 22 and Promise 12 datasets, initialized with SAM pre-trained weights. Four methods are compared: Fine-tuning, AuroRA, FourierFT, and FAN‑LoRA. On both datasets, Fine-tune exhibits signs of overfitting. In contrast, compared with AuroRA and FourierFT, FAN‑LoRA demonstrates a smoother convergence trajectory and exhibits less overfitting.

Fig. 4: Qualitative comparison of different domain adaptation methods. From left to right: input image, ground truth, FAN-LoRA (ours), AuroRA, DeLoRA, FourierFT, LoRA, and SHiRA. FAN-LoRA delivers sharper boundaries and better-preserved fine structures under domain shift, while competing methods tend to over-smooth or produce noisy artifacts. Red boxes highlight representative errors, such as missed structures and false positives.

Base LoRA	Low-pass	High-pass	NCI-ISBI 1.5T		NCI-ISBI 3T
Base LoRA	Low-pass	High-pass	Avg.	HD95 $\downarrow$	Avg.	HD95 $\downarrow$
✓			78.6	30.51	78.1	28.41
✓	✓		79.1	29.08	78.6	28.19
✓		✓	79.5	30.19	78.7	27.13
✓	✓	✓	80.9	26.96	79.6	26.48

TABLE VIII: Ablation study of the core low-pass and high-pass modules in FAN-LoRA on the NCI-ISBI target domain

$U_{\text{high}},V_{\text{high}}$	$S_{\text{freq}}$	NCI-ISBI 1.5T		NCI-ISBI 3T
$U_{\text{high}},V_{\text{high}}$	$S_{\text{freq}}$	Avg.	HD95 $\downarrow$	Avg.	HD95 $\downarrow$
Random & Frozen		75.9	36.09	77.3	30.63
Learnable		78.9	35.25	78.05	28.75
Fixed Sinusoidal		79.6	28.63	78.9	27.11
Random & Frozen	✓	77.3	37.24	78.3	29.36
Learnable	✓	79.2	30.58	78.6	28.97

TABLE IX: Ablation study on high-pass branch components on the NCI-ISBI target domain.

Rank	Frequency	NCI-ISBI 1.5T				NCI-ISBI 3T
		PZ	CG	Avg.	HD95 $\downarrow$	PZ	CG	Avg.	HD95 $\downarrow$
4	16	66.2	92.2	79.2	30.38	65.1	92.9	79	27.71
8		66.1	93.8	80	30.18	59.6	93.2	76.4	32.77
16		65.7	92.9	79.3	30.68	61.2	93.4	77.3	31.42
32		65.8	93.2	79.5	33.44	59	93.3	76.15	33.47
2	32	66.7	93.2	80	29.57	59.3	92.6	76	32.15

TABLE X: Hyperparameter sensitivity analysis of rank dimension and frequency sampling on the NCI-ISBI target domain

Explore similar work

When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning
Aug 21, 2026Marko Haralović, Sounic Akkaraju, Carlo Baretta +2Vision Foundation Model Adaptation Medical Imaging Foundation Models
University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, Croatia · University of Twente, Enschede, The Netherlands
Frequency Adapter with SAM for Generalized Medical Image Segmentation
May 11, 2026Phuoc-Nguyen Bui, Van-Nguyen Pham, Duc-Tai Le +2Frequency-Domain Feature Learning Domain Generalization
Sungkyunkwan University, Korea
Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentation
Jun 7, 2026Xingyue Zhao, Wenke Huang, Linghao Zhuang +7Vision Foundation Model Adaptation Non-IID Federated Learning
Department of Neurosurgery, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · College of Computing and Data Science, Nanyang Technological University, Singapore +4

Aug 21, 2026J/K move · Enter open · S save

When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning

Marko Haralović, Sounic Akkaraju, Carlo Baretta, Vasil Zapryanov+1

University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, Croatia · University of Twente, Enschede, The Netherlands

Vision Foundation Model Adaptation Medical Imaging Foundation Models Medical Image Segmentation OOD Generalization Parameter-Efficient Fine-Tuning Prompt Robustness

Open paper AskPDF arXiv

Foundation models for medical image segmentation, like prompt-based MedSAM, generalize well across domains and modalities, often in zero or few-shot setups. However, their performance depends on the quality of prompts and the adaptation of the models to custom datasets. This work systematically examines how MedSAM generalizes across diverse medical imaging benchmarks, with six adaptation strategies: full-model and encoder-only LoRA, shallow and deep visual prompt tuning (VPT), and decoder-only and full fine-tuning. Models are trained on the International Skin Imaging Collaboration Challenge (ISIC 2018) dataset and evaluated under clean and increasingly noisy prompts on IN and Out-of-Distribution (OOD) datasets: close-OOD PH2 (dermoscopy), far-OOD BUSI (Breast Ultrasound Images Dataset) and CBIS-DDSM (Curated Breast Imaging Subset of the Digital Database for Screening Mammography). We show that adaptation improves performance on IN and close-OOD data but often reduces performance on far-OOD data. Full fine-tuning provides the best tradeoff, while encoder-only LoRA is the strongest parameter-efficient alternative, outperforming standard LoRA and VPT under far-OOD shifts. Using Centered Kernel Alignment (CKA), we show that far-OOD degradation is strongly associated with drift in decoder representations, whereas encoder similarity alone does not explain robustness. This suggests encoder-only LoRA provides stronger robustness than standard LoRA by adapting the encoder to distribution shift in visual features, while preserving the decoder pathway. We further show that random 0-100 pixel jitter on prompts produces more robust and better performing models. We thus conclude that robust MedSAM adaptation requires the combined consideration of prompt noise exposure, domain shift, and representation preservation. We release our code: https://github.com/ImSounic/medsam-vpt

Abstract

Explore similar work

When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning

Frequency Adapter with SAM for Generalized Medical Image Segmentation

Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentation