TAPE: A two-stage parameter-efficient adaptation framework for foundation models in OCT-OCTA analysis
Authors: Xiaofei Su, Zengshuo Wang, Minghe Sun, Xin Zhao, Mingzhu Sun
Organizations: Nankai University Optometry & Vision Science Institute, National Key Laboratory of Intelligent Tracking and Forecasting for Infectious Diseases, Tianjin Key Laboratory of Intelligent Robotic, Institute of Robotics and Automatic Information System, Nankai University, Tianjin, 300350, China · Institute of Intelligence Technology and Robotic Systems, Shenzhen Research Institute of Nankai University, Shenzhen, 518083, China
Automated analysis of optical coherence tomography (OCT) and OCT angiography (OCTA) images is critical for robust ophthalmic diagnosis. Existing mainstream methods trained from scratch rely heavily on massive data and model scale, thereby hindering their practical deployment in resource-constrained clinical settings. Although transfer learning based on foundation models (FMs) is promising, it still faces significant challenges: domain shift and task misalignment. To address these, we propose TAPE: A Two-stage Adaptation Framework via Parameter-Efficient Fine-tuning, which strategically decouples adaptation into domain alignment and task fitting for downstream segmentation. The domain adaptation stage notably applies parameter-efficient fine-tuning (PEFT) in the context of masked image modeling for medical image domain adaptation, a novel approach to the best of our knowledge. Applying TAPE to retinal layer segmentation on both universal (masked auto-encoder, MAE) and specialized (RETFound) FMs, it demonstrates superior parameter efficiency and achieves state-of-the-art generalization performance across diverse pathologies.
Figures & tables
Figure 1: Proposed TAPE architecture which contains domain adaptation (Stage I) and task adaptation (Stage II). Domain adapter is trained on MIM-based SSL task to add knowledge of target data domain for FM. Task adapter is trained on downstream task, to fit FM for layer segmentation task.
Foundation Model
Methods
Trainable Params
Train Loss
Test Loss
MAE
FFT
329.81 M (100%)
0.1886
0.3801
LoRA
3.15 M (0.94%)
0.1834
0.3085
ViT-Adapter
0.84 M (0.25%)
0.184
0.3109
VPT
10.24 K (0.003%)
0.1836
0.3179
RETFound
FFT
329.54 M (100%)
0.1227
0.2708
LoRA
3.14 M (0.94%)
0.1311
0.25
Table 1: Comparison of different PEFT methods and FFT on foundation models using MSE loss.
Figure 2: TAPE segments retinal layers while achieving superior performance. Rows correspond to the NORMAL class and three disease classes (DR, RVO, and AMD), and columns show labels and segmentation maps generated by TAPE, STL-OCT, STL, FFT-DA, FFT-TA, DLoRA, and TLoRA. Segmentation targets include the internal limiting membrane (ILM), inner plexiform layer (IPL), outer plexiform layer (OPL), inner segment/outer segment (ISOS), retinal pigment epithelium (RPE), and Bruch’s membrane (BM).
Foundation model
Method
AMD
DR
RVO
NORMAL
ALL
mDice
mIoU
mDice
mIoU
mDice
mIoU
mDice
mIoU
mDice
mIoU
MAE
STL- OCT
88.72
81.54
91.77
85.86
86.99
78.62
95.21
91.15
92.62
87.25
STL
89.06
82.05
91.85
85.96
86.92
78.55
95.33
91.37
92.77
87.49
FFT DA
89.41
82.39
91.9
86.04
86.64
78.22
95.21
91.15
92.78
87.45
FFT TA
87.81
80.47
90.92
84.61
85.62
76.85
94.85
90.52
91.98
86.34
DLoRA
89.8
82.97
92.05
86.25
87.13
78.88
95.34
91.38
92.99
87.78
Table 2: Comparison of different PEFT methods on foundation models for diverse pathologies. The values of mDice and mIoU are presented as percentages.
ReLayNet
EMVNet
MGUNet
mDice
mIoU
mDice
mIoU
mDice
mIoU
91.62
85.92
92.98
88.14
93.04
87.99
MAE-STL
MAE-DLoRA
MAE-TAPE
mDice
mIoU
mDice
mIoU
mDice
mIoU
92.77
87.49
92.99
87.78
93.47
88.58
RETFound-STL
RETFound- DLoRA
RETFound- TAPE
Table 3: Comparison of ReLayNet, EMVNet, and MGUNet with FM-based methods.
Retinal vessel segmentation in multi-domain deployment requires a source model to adapt to domains that differ in imaging conditions and annotation conventions. Conventional parameter-efficient fine-tuning reduces target-specific storage, but its highly restricted adaptation subspace can be insufficient for reconstructing thin, connected vascular structures. We therefore ask how target-specific capacity should be allocated so that topology-aware supervision remains effective under a strict per-domain parameter budget. Based on this principle, we propose TAPDecoderFT, a topology-responsive, role-structured adaptation framework. Specifically, TAPDecoderFT shares a fixed source parameter state across deployment domains, uses low-rank residuals for target-specific private/fusion feature mixing, and retains a trainable dense-reconstruction path comprising the decoder, output head, and refinement module. To promote structurally faithful predictions, the compact target state is jointly optimized with a region-overlap and topology-aware objective that encourages centerline continuity and thin-branch recovery. It improves both DSC and clDice over GenericLoRA-r4 and narrow TAP-r4 in all six directions and is comparable to full fine-tuning.
Optical Coherence Tomography (OCT) has become one of the most used imaging modality in ophthalmology. It provides high-resolution, non-invasive visualization of retinal microarchitecture. The automated analysis of OCT images through representation learning has emerged as a central research frontier. This has mainly been driven by the clinical need to process large acquisition volumes. The objective is to reduce the reliance on expert annotation, and improve diagnostic consistency across devices and populations. This survey provides a comprehensive and structured review of representation learning methods for retinal OCT image analysis. It covers the period from early deep learning approaches to the most recent developments in foundation models and vision-language systems. We organize the literature along a principled taxonomy of learning paradigms, encompassing supervised learning with CNN-based and transformer-based architectures, self-supervised and semi-supervised methods, generative approaches, as well as 3D volumetric modeling, multimodal representation learning, and large-scale pretrained foundation models. For each paradigm, we analyze the core methodological contributions, identify persistent limitations, and trace the connections between successive approaches. We further provide a structured overview of publicly available OCT datasets, discuss evaluation protocol considerations, and present a unified problem formulation that situates each learning paradigm within a common mathematical framework. Building on this analysis, we identify and discuss the most pressing open research directions emerging in the literature. This includes volumetric foundation model pretraining, uncertainty-aware representation learning, federated and privacy-preserving training, fairness and bias mitigation, concept-based interpretability,...
Hedi Tabia, Désiré Sidibé, Nawres Khlifa +6
IBISC Univ. Evry Université Paris-Saclay · University of Tunis El Manar · ESIEE Paris - Université Gustave Eiffel +2
The advent of vision foundation models, notably the Segment Anything Model (SAM), has catalyzed significant advancements in natural image segmentation. However, their direct transfer to medical imaging remains severely bottlenecked by profound domain gaps, such as cross-modality and cross-center shifts. Existing Parameter-Efficient Fine-Tuning (PEFT) methods facilitate the adaptation of SAM to medical domains; nevertheless, they frequently suffer from performance degradation under severe distribution shifts. This vulnerability primarily stems from the implicit entanglement of heterogeneous frequency components within a shared low-rank subspace, which directly exacerbates sub-optimal structural alignment and localized boundary blurring. To overcome this representational bottleneck, we propose the Fourier-Adaptive Nonlinear Low-Rank Adaptor (FAN-LoRA), a novel frequency-decoupled fine-tuning architecture. FAN-LoRA explicitly separates the optimization space by employing a B-spline-driven low-pass branch for global structural alignment, synergistically coupled with a discrete Fourier high-pass branch for local textural compensation. Extensive experiments across three challenging cross-modality and cross-center benchmarks demonstrate that FAN-LoRA consistently outperforms state-of-the-art PEFT baselines. Compared to the strongest competitors, our method achieves consistent improvements in average Dice scores and notable reductions in boundary errors, while maintaining a compact module size without compromising computational efficiency.
Ziquan Liu, Zhewei Zhu, Xuyang Shi
School of Information and Control Engineering, Southwest University of Science and Technology Mianyang, China