UniPrompt-CL: Efficient Single-Inference Continual Learning for Medical Image Classification

Gyutae Oh; Jitae Shin

cs.LGAug 14, 2025

UniPrompt-CL: Efficient Single-Inference Continual Learning for Medical Image Classification

Authors: Gyutae Oh, Jitae Shin

Organizations: Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419, Republic of Korea

Abstract

Medical AI systems must remain reliable as data distributions shift across hospitals, imaging devices, and patient populations, yet privacy constraints often prevent storing or revisiting past patient data. We present UniPrompt-CL, a rehearsal-free domain-incremental learning framework for medical image classification using a single backbone and a single inference. Rather than relying on multiple backbones, repeated ViT inferences, or independent layer-wise prompt pools, UniPrompt-CL shares a unified prompt pool across transformer layers and expands it with only a few prompts per new domain. This reduces cross-layer redundancy while enabling efficient adaptation. On diabetic retinopathy, UniPrompt-CL improves average accuracy by 2.0 percentage points and substantially reduces forgetting relative to the strongest backbone-matched baseline. On skin cancer, it achieves competitive accuracy at the lowest training cost among DINOv2-B configurations. A controlled comparison varying only pool organization shows that the benefit arises from retention rather than learning each new domain: it is largest on the medical benchmarks, smaller on OfficeHome, and reverses on a heterogeneous six-stage DomainNet stream. These results suggest that cross-layer prompt sharing with lightweight expansion is practical for medical models requiring repeated retraining under limited on-site compute, while identifying the domain shifts under which it helps.

Figures & tables

Figure 1: Representative examples of natural images used in prior PCL studies and medical images used in this study, highlighting the standardized acquisition conditions and fine-grained visual differences in medical datasets.

Figure 2: Overall architecture. At each layer the [CLS] token serves as a query; the prompt-pool integration module retrieves a prompt from the unified pool and fuses it with the layer representation. At each new stage a small number of prompts is added, while all previously learned prompts remain frozen.

Method	GFLOPs $\downarrow$	Diabetic Retinopathy		Skin Cancer
Method	GFLOPs $\downarrow$	AvgACC $\uparrow$	AvgF $\downarrow$	AvgACC $\uparrow$	AvgF $\downarrow$
OS-Prompt [ 7 ]	34.26	0.666 $\pm$ 0.062	0.132 $\pm$ 0.003	0.682 $\pm$ 0.041	0.135 $\pm$ 0.075
MoE-Adapters [ 16 ]	105.64	0.716 $\pm$ 0.141	0.080 $\pm$ 0.092	0.597 $\pm$ 0.165	0.040 $\pm$ 0.016
Coda-Prompt [ 14 ]	134.33	0.688 $\pm$ 0.079	0.140 $\pm$ 0.044	0.713 $\pm$ 0.040	0.041 $\pm$ 0.013
DualPrompt [ 15 ]	105.05	0.467 $\pm$ 0.159	0.291 $\pm$ 0.065	0.637 $\pm$ 0.065	0.012 $\pm$ 0.039
DER++ [ 1 ]	168.02	0.607 $\pm$ 0.121	0.288 $\pm$ 0.160	0.722 $\pm$ 0.057	0.099 $\pm$ 0.023

Table 1: Continual learning performance and training cost.

*

marks a DINOv2-B backbone, added for the applicable prompt-based baselines so that backbone-level and method-level gains can be separated; the others use their original configuration. GFLOPs is the per-step training cost (forward and backward) and depends on the method and backbone, not the benchmark. BWT is omitted: it coincides with

-\mathrm{AvgF}

throughout these sequences (Sup. F).

\dagger

denotes the proposed model; bold is the best value per benchmark.

Figure 3: Inter-stage similarity of incrementally added prompts. Newly added prompts exhibit limited overlap with prompts from earlier stages, consistent with each expansion contributing complementary stage-specific directions rather than duplicating existing ones. We treat this as qualitative motivation, validated empirically in Sec. 5 .

Sequence	Stages	AvgACC $\uparrow$		AvgF $\downarrow$
Sequence	Stages	U / L	$\Delta$	U / L	$\Delta$
DR	3	0.844 / 0.810	$+3.44$	0.079 / 0.172	$-9.28^{\ast}$
Skin	3	0.732 / 0.720	$+1.20$	0.049 / 0.137	$-8.75$
OfficeHome	4	0.809 / 0.802	$+0.68$	0.114 / 0.147	$-3.29$
DomainNet	6	0.645 / 0.649	$-0.44$	0.506 / 0.476	$+3.02^{\ast}$

Table 2: Unified (U) vs. layer-wise (L) prompt pools; only the pool organization differs. The layer-wise control spreads the same

L_{p}=500

prompts over the five prompted layers, so prompt parameters and unique prompt counts match. Rows (DR: diabetic retinopathy; Skin: skin cancer) list the two medical sequences followed by the two natural-image sequences. Means over three seeds;

\Delta=\mathrm{U}-\mathrm{L}

in points, with ∗ marking a

95\%

paired CI excluding zero. These are ablations within our framework, so absolute values are not comparable to published DomainNet or OfficeHome results.

Figure 4: Ablation analysis of the surrogate selection loss

\mathcal{L}_{s}

and its weight

\lambda

on APTOS

\rightarrow

DDR

\rightarrow

DRD. All values are computed on the validation split, not on the test split. FAA and FAF denote the final average accuracy and final average F1-score across all stages, respectively. Markers show the mean over three seeds and error bars one standard deviation. The selected setting,

\lambda=0.001

, attains the best value on both measures; performance declines again at larger

\lambda

Figure 5: Prompt expansion ratio sensitivity on the APTOS

\rightarrow

DDR

\rightarrow

DRD sequence. All values are computed on the validation split of each dataset, not on the test split. Markers show the mean over three seeds and error bars one standard deviation; the horizontal axis is categorical, with the six evaluated ratios at equal spacing. The highlighted

20\%

ratio attains the best value on all four measures. Beyond it, average performance is flat while forgetting continues to grow.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Cross-layer same-index redundancy analysis of OS-Prompt. (A) Prompts and keys with the same index across different prompted layers exhibit consistently high absolute cosine similarity across continual learning stages. (B) At the final stage, same-index cross-layer similarity is close to one, while within-layer prompt/key similarity remains close to zero. (C) The index-wise distribution confirms that high same-index redundancy is observed for most prompt indices, rather than a small subset.

Dataset	Normal (0)	NPDR (1)	PDR (2)
APTOS [ 5 ]	1805	1562	295
DDR [ 9 ]	6266	5343	913
DRD [ 4 ]	5000	8608	708

Appendix

Table 3: Distribution of diabetic retinopathy datasets used for training.

Dataset	Benign	Malignant
ISIC [ 12 ]	584	584
HAM [ 3 ]	1113	1113
DERM7 [ 6 ]	252	252

Appendix

Table 4: Distribution of skin cancer datasets used for external validation.

Model	Parameters
facebook/dinov2-small	22.1 M
facebook/dinov2-base	86.6 M
facebook/dinov2-large	304 M
facebook/dinov2-giant	1.14 B

Appendix

Table 5: Parameter counts for facebook/dinov2 models.

\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{CE}}+\lambda\mathcal{L}_{s}

Appendix

Algorithm 1 UniPrompt-CL Training Procedure with Fixed Per-Stage Prompt Expansion

Figure 7: Representative examples from the three skin cancer datasets used for the controlled external validation: ISIC [ 12 ] , HAM [ 3 ] , and DERM7 [ 6 ] . The samples illustrate the diversity in lesion appearance and acquisition conditions across datasets.

Item	Value
GPU	2 $\times$ V100
RAM	256GB
Learning Rate	$0.001$
Scheduler	Cosine Scheduler
Number of Prompts ( $L_{p}$ )	500
Prompt Dimension ( $D$ )	768

Appendix

Table 6: Experiment settings.

Method	Hyperparameter	Value
CODA-Prompt	Prompt pool size ( coda_pool_size )	100
	Prompt length ( coda_prompt_length )	4
	Ortho. regularization coeff. ( coda_ortho_mu )	0.1
DualPrompt	Prompt length ( dual_prompt_length )	4
	Prompt pool size ( dual_pool_size )	100
	Top- $k$ prompts ( dual_top_k )	5

Appendix

Table 7: Key hyperparameters for prompt-based continual learning baselines.

CL-Type (Ref)	Model	APTOS		DDR		DRD
		Acc	F1	Acc	F1	Acc	F1
PCL (CVPR2026)	CPS-Prompt [ 10 ]	0.676 $\pm$ 0.046	0.651 $\pm$ 0.041	0.694 $\pm$ 0.011	0.660 $\pm$ 0.018	0.647 $\pm$ 0.010	0.579 $\pm$ 0.006
PCL (ECCV2024)	OS-Prompt [ 7 ]	0.687 $\pm$ 0.011	0.637 $\pm$ 0.042	0.693 $\pm$ 0.013	0.648 $\pm$ 0.043	0.619 $\pm$ 0.002	0.568 $\pm$ 0.063
PCL (ECCV2024)	OS-Prompt++ [ 7 ]	0.743 $\pm$ 0.104	0.686 $\pm$ 0.072	0.697 $\pm$ 0.032	0.655 $\pm$ 0.032	0.623 $\pm$ 0.005	0.565 $\pm$ 0.023
Arch-CL (CVPR2024)	MoE-Adapters [ 16 ]	0.835 $\pm$ 0.001	0.742 $\pm$ 0.001	0.747 $\pm$ 0.004	0.694 $\pm$ 0.013	0.564 $\pm$ 0.002	0.478 $\pm$ 0.006
PCL (CVPR2023)	Coda-Prompt [ 14 ]	0.682 $\pm$ 0.022	0.646 $\pm$ 0.022	0.721 $\pm$ 0.001	0.697 $\pm$ 0.023	0.663 $\pm$ 0.015	0.557 $\pm$ 0.061

Appendix

Table 8: Final Accuracy (Acc) and F1-Score (F1) Results After the Final Stage and Performance Comparison with Other PCL Models. The symbol

\dagger

denotes our proposed model. PCL denotes prompt-based continual learning; Arch-CL refers to architecture-based continual learning; Reh-CL means rehearsal-based continual learning; and Reg-CL indicates regularization-based continual learning. ( bold indicates the highest performance;)

		Evaluation
		OS-Prompt++ (Reference-ViT training)						UniPrompt-CL † (Single-pass training)
Training	Dataset	APTOS		DDR		DRD		APTOS		DDR		DRD
		Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
Stage 1	APTOS	0.868	0.753	0.565	0.474	0.409	0.354	0.901	0.767	0.601	0.447	0.453	0.381
Stage 2	DDR	0.707	0.638	0.797	0.748	0.508	0.413	0.866	0.663	0.878	0.844	0.636	0.534
Stage 3	DRD	0.743	0.686	0.697	0.655	0.623	0.565	0.849	0.761	0.772	0.723	0.701	0.656

Appendix

Table 9: Stage-wise comparison of catastrophic forgetting during continual learning. Red indicates performance on the current training dataset, blue indicates performance on previously learned datasets, and black indicates performance on unseen future datasets. The symbol

\dagger

denotes the proposed model.

Figure 8: Sensitivity of

\mathrm{CARA}_{p}

over

p\in\{0,0.25,0.5,0.75,1.0\}

on the DR and skin cancer benchmarks. UniPrompt-CL is highlighted with star markers and the dashed vertical line indicates

p=0.5

. V and D denote ViT-B and DINOv2-B, respectively.

Method	GFLOPs $\downarrow$	$\mathbf{CARA}_{0.5}\uparrow$
		DR	Skin Cancer
OS-Prompt [ 7 ]	34.26	0.099	0.101
MoE-Adapters [ 16 ]	105.64	0.064	0.056
Coda-Prompt [ 14 ]	134.33	0.051	0.059
DualPrompt [ 15 ]	105.05	0.032	0.061
DER++ [ 1 ]	168.02	0.033	0.050

Appendix

Table 10: Auxiliary

\mathrm{CARA}_{0.5}

values for every configuration in Table 1, retained here for completeness.

*

marks a DINOv2-B backbone and

\dagger

the proposed model. AvgACC, AvgF, and GFLOPs, from which these values are computed, are reported separately in Table 1; the main paper draws its conclusions from those quantities rather than from this metric.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 304 papers

From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation
Apr 19, 2026Evren Çetinkaya, Sangmin Lee, Jung Uk Kim +2Domain Generalization Medical Image Segmentation
Technical University of Munich · Korea University · Kyung Hee University +1
Bi-CRCL: Bidirectional Conservative-Radical Complementary Learning with Pre-trained Foundation Models for Class-incremental Medical Image Analysis
Mar 24, 2026Xinyao Wu, Zhe Xu, Cheng Chen +3Class-Incremental Learning Medical Image Analysis
Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Department of Radiation Oncology, Columbia University Irving Medical Center and Data Science Institute, Columbia University, New York, NY, USA. · Department of Electrical and Electronic Engineering and School of Biomedical Engineering, The University of Hong Kong, Hong Kong, China. +2
Residual SODAP: Residual Self-Organizing Domain-Adaptive Prompting with Structural Knowledge Preservation for Continual Learning
Mar 13, 2026Gyutae Oh, Jungwoo Bae, Jitae ShinContinual Learning Domain-Incremental Learning
Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419, Republic of Korea

Apr 19, 2026J/K move · Enter open · S save

From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation

Evren Çetinkaya, Sangmin Lee, Jung Uk Kim, Hong Joo Lee+1

Technical University of Munich · Korea University · Kyung Hee University+1

Domain Generalization Medical Image Segmentation Visual Prompt Learning

Open paper AskPDF arXiv

Visual prompting has emerged as a powerful method for adapting pre-trained models to new domains without updating model parameters. However, existing prompting methods typically optimize a single prompt per domain and apply it uniformly to all inputs, limiting their ability to generalize under intra and inter-domain variability, which is especially critical in the medical field. To address this, we propose APEX, an Adaptive Prompt EXtraction framework that retrieves input-specific prompts from a learnable prompt memory. The memory stores diverse, domain-discriminative prompt representations and is queried via domain features extracted from the Fourier spectrum. To learn robust and discriminative domain features, we introduce a novel Low-Frequency Feature Contrastive (LFC) learning framework that clusters representations from the same domain while separating those from different domains. Extensive experiments on two medical segmentation tasks demonstrate that APEX significantly improves generalization across both seen and unseen domains. Furthermore, it complements any existing backbones and consistently enhances performance, confirming its effectiveness as a plug-and-play prompting solution in medical fields. The code is available at https://github.com/cetinkayaevren/apex/

Abstract

Continual Learning

Explore similar work

From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation

Bi-CRCL: Bidirectional Conservative-Radical Complementary Learning with Pre-trained Foundation Models for Class-incremental Medical Image Analysis

Residual SODAP: Residual Self-Organizing Domain-Adaptive Prompting with Structural Knowledge Preservation for Continual Learning