TAPE: A two-stage parameter-efficient adaptation framework for foundation models in OCT-OCTA analysis
Authors: Xiaofei Su, Zengshuo Wang, Minghe Sun, Xin Zhao, Mingzhu Sun
Organizations: Nankai University Optometry & Vision Science Institute, National Key Laboratory of Intelligent Tracking and Forecasting for Infectious Diseases, Tianjin Key Laboratory of Intelligent Robotic, Institute of Robotics and Automatic Information System, Nankai University, Tianjin, 300350, China · Institute of Intelligence Technology and Robotic Systems, Shenzhen Research Institute of Nankai University, Shenzhen, 518083, China
Automated analysis of optical coherence tomography (OCT) and OCT angiography (OCTA) images is critical for robust ophthalmic diagnosis. Existing mainstream methods trained from scratch rely heavily on massive data and model scale, thereby hindering their practical deployment in resource-constrained clinical settings. Although transfer learning based on foundation models (FMs) is promising, it still faces significant challenges: domain shift and task misalignment. To address these, we propose TAPE: A Two-stage Adaptation Framework via Parameter-Efficient Fine-tuning, which strategically decouples adaptation into domain alignment and task fitting for downstream segmentation. The domain adaptation stage notably applies parameter-efficient fine-tuning (PEFT) in the context of masked image modeling for medical image domain adaptation, a novel approach to the best of our knowledge. Applying TAPE to retinal layer segmentation on both universal (masked auto-encoder, MAE) and specialized (RETFound) FMs, it demonstrates superior parameter efficiency and achieves state-of-the-art generalization performance across diverse pathologies.
Figures & tables
Figure 1: Proposed TAPE architecture which contains domain adaptation (Stage I) and task adaptation (Stage II). Domain adapter is trained on MIM-based SSL task to add knowledge of target data domain for FM. Task adapter is trained on downstream task, to fit FM for layer segmentation task.
Foundation Model
Methods
Trainable Params
Train Loss
Test Loss
MAE
FFT
329.81 M (100%)
0.1886
0.3801
LoRA
3.15 M (0.94%)
0.1834
0.3085
ViT-Adapter
0.84 M (0.25%)
0.184
0.3109
VPT
10.24 K (0.003%)
0.1836
0.3179
RETFound
FFT
329.54 M (100%)
0.1227
0.2708
LoRA
3.14 M (0.94%)
0.1311
0.25
Table 1: Comparison of different PEFT methods and FFT on foundation models using MSE loss.
Figure 2: TAPE segments retinal layers while achieving superior performance. Rows correspond to the NORMAL class and three disease classes (DR, RVO, and AMD), and columns show labels and segmentation maps generated by TAPE, STL-OCT, STL, FFT-DA, FFT-TA, DLoRA, and TLoRA. Segmentation targets include the internal limiting membrane (ILM), inner plexiform layer (IPL), outer plexiform layer (OPL), inner segment/outer segment (ISOS), retinal pigment epithelium (RPE), and Bruch’s membrane (BM).
Foundation model
Method
AMD
DR
RVO
NORMAL
ALL
mDice
mIoU
mDice
mIoU
mDice
mIoU
mDice
mIoU
mDice
mIoU
MAE
STL- OCT
88.72
81.54
91.77
85.86
86.99
78.62
95.21
91.15
92.62
87.25
STL
89.06
82.05
91.85
85.96
86.92
78.55
95.33
91.37
92.77
87.49
FFT DA
89.41
82.39
91.9
86.04
86.64
78.22
95.21
91.15
92.78
87.45
FFT TA
87.81
80.47
90.92
84.61
85.62
76.85
94.85
90.52
91.98
86.34
DLoRA
89.8
82.97
92.05
86.25
87.13
78.88
95.34
91.38
92.99
87.78
Table 2: Comparison of different PEFT methods on foundation models for diverse pathologies. The values of mDice and mIoU are presented as percentages.
ReLayNet
EMVNet
MGUNet
mDice
mIoU
mDice
mIoU
mDice
mIoU
91.62
85.92
92.98
88.14
93.04
87.99
MAE-STL
MAE-DLoRA
MAE-TAPE
mDice
mIoU
mDice
mIoU
mDice
mIoU
92.77
87.49
92.99
87.78
93.47
88.58
RETFound-STL
RETFound- DLoRA
RETFound- TAPE
Table 3: Comparison of ReLayNet, EMVNet, and MGUNet with FM-based methods.