Bi-CRCL: Bidirectional Conservative-Radical Complementary Learning with Pre-trained Foundation Models for Class-incremental Medical Image Analysis

Xinyao Wu; Zhe Xu; Cheng Chen; Jiawei Ma; Yefeng Zheng; Raymond Kai-yu Tong

cs.CVMar 24, 2026

Bi-CRCL: Bidirectional Conservative-Radical Complementary Learning with Pre-trained Foundation Models for Class-incremental Medical Image Analysis

Authors: Xinyao Wu, Zhe Xu, Cheng Chen, Jiawei Ma, Yefeng Zheng, Raymond Kai-yu Tong

Organizations: Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Department of Radiation Oncology, Columbia University Irving Medical Center and Data Science Institute, Columbia University, New York, NY, USA. · Department of Electrical and Electronic Engineering and School of Biomedical Engineering, The University of Hong Kong, Hong Kong, China. · Department of Computer Science, City University of Hong Kong, Hong Kong, China. · Department of Artificial Intelligence, Westlake University, Hangzhou, China.

Abstract

Class-incremental learning (CIL) in medical image-guided diagnosis requires retaining prior diagnostic knowledge while adapting to newly emerging disease categories, which is critical for scalable clinical deployment. This problem is particularly challenging due to heterogeneous data and privacy constraints that prevent memory replay. Although pretrained foundation models (PFMs) have advanced general-domain CIL, their potential in medical imaging remains underexplored, where domain-specific adaptation is essential yet difficult due to anatomical complexity and inter-institutional heterogeneity. To address this gap, we conduct a systematic benchmark of recent PFM-based CIL methods and propose Bidirectional Conservative-Radical Complementary Learning (Bi-CRCL), a dual-learner framework inspired by complementary learning systems. Bi-CRCL integrates a conservative learner that preserves prior knowledge through stability-oriented updates and a radical learner that rapidly adapts to new categories via plasticity-oriented learning. A bidirectional interaction mechanism enables forward transfer and backward consolidation, allowing continual integration of new knowledge while mitigating catastrophic forgetting. During inference, outputs from both learners are adaptively fused for robust predictions. Experiments on five medical imaging datasets demonstrate consistent improvements over state-of-the-art methods under diverse settings, including cross-dataset shifts and varying task configurations.

Figures & tables

Fig. 1: Illustration of conventional vs. PFM-based medical CIL. Conventional CIL trains models from scratch, similar to teaching an infant to learn from zero experience. PFM-based CIL builds on PFMs, analogous to guiding an experienced adult to adapt efficiently to new tasks. Both paradigms share the fundamental challenge of catastrophic forgetting when learning new classes sequentially.

Fig. 2: Overview of our Bi-CRCL framework. (a) Initialized Domain Alignment: At task $t=1$ , the general-domain PFM adapts to the medical domain via adapter tuning to enable efficient PFM transfer while preserving generalization; (b) Continual Bidirectional Complementary Learning: For $t>1$ , a radical learner incrementally learns new classes and exchanges knowledge with the conservative learner through bidirectional consolidation and re-initialization, emulating hippocampus–neocortex coordination; (c) Collaborative Inference: During inference, outputs from both learners are projected via analytical classifiers and adaptively fused to produce robust task-agnostic predictions.

Dataset	Classes	Training set	Test set	Task Num.	Size
Colon [ 15 ]	9	70,000	30,000	4	$224\times 224$
Blood [ 2 ]	8	11,965	5,127	4	$360\times 363$
Skin8 [ 29 ]	8	3,555	705	4	$[600,1024]$
MedMNIST-Sub [ 41 ]	36	302,002	75,659	4	$28\times 28$
COVID (CT&X-ray) [ 30 ]	11	3,939	1,072	6	$224\times 224$

TABLE I: Overview of medical image classification datasets.

Fig. 3: An illustration of the diverse medical imaging tasks addressed by our model, spanning lesion classification (dermoscopic images), organ classification (CT scans), blood cell classification (microscopy), tissue classification (histopathology), and disease detection (CT & X-ray scans), with representative example classes from each dataset.

Method	Colon		Blood		Skin8		MedMNIST-Sub		COVID (CT&X-ray)
Method	$Acc_{\text{Avg}}$	$Acc_{\text{Last}}$	$Acc_{\text{Avg}}$	$Acc_{\text{Last}}$	$Acc_{\text{Avg}}$	$Acc_{\text{Last}}$	$Acc_{\text{Avg}}$	$Acc_{\text{Last}}$	$Acc_{\text{Avg}}$	$Acc_{\text{Last}}$
Joint Training	-	99.98	-	99.61	-	67.73	-	73.61	-	92.43
Finetune	38.65	10.43	37.47	14.54	39.62	17.87	29.18	5.66	26.55	10.54
FOSTER † [ 31 ]	90.81	86.69	87.70	90.24	55.00	39.01	58.24	31.46	71.65	55.50
iCaRL † [ 24 ]	80.50	78.13	81.41	81.57	58.45	39.57	68.23	38.44	70.10	65.21
DER † [ 40 ]	85.92	85.63	86.48	87.10	48.57	23.55	69.97	42.11	31.40	27.05

TABLE II: Performance on five medical datasets. “

\dagger

” indicates the necessity of replaying prior data. The best and second-best results are bolded and underlined , respectively.

Fig. 4: The performance curves across learning sessions on five medical datasets. Bi-CRCL consistently achieves the best overall performance and exhibits minimal degradation as class number increases.

PFM	Method	Colon		Blood		Skin8		MedMNIST-Sub		COVID (CT&X-rays)
PFM	Method	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)
RAD-DINO [ 21 ]	Joint Training	-	86.34	-	65.83	-	42.20	-	56.27	-	70.80
	Finetuning	38.06	11.57	44.84	36.57	31.76	15.32	48.80	33.53	24.39	6.25
	SimpleCIL [ 48 ]	53.44	46.67	43.14	25.29	34.82	21.56	48.62	38.01	57.09	46.60
	ADAM-Adapter [ 48 ]	54.92	48.58	42.50	24.55	35.40	21.99	49.91	38.06	55.51	39.46
	MOS [ 27 ]	24.52	8.77	31.68	9.08	24.09	5.53	22.95	8.23	21.28	11.01

TABLE III: Performance comparison using different PFMs (medical-specific vs. general-domain) across five benchmark datasets. The best results of each PFM are bolded .

Dataset	Setting	Exclusion	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)
Skin8	Bi-CRCL	None	74.59	61.99
	Abla-1	Initialized Adaptation	70.96	59.69
	Abla-2	Bidirectional Knowledge Interaction	74.83	61.42
	Abla-3	Conservative Learner Inference	73.82	60.43
	Abla-4	Radical Learner Inference	74.49	61.84
COVID	Bi-CRCL	None	96.12	88.15

TABLE IV: Ablation analysis on three representative and challenging datasets (Skin8, COVID and MedMNIST-Sub). The best and second-best results are bolded and underlined , respectively.

Method	Order	Colon		Blood		Skin8		MedMNIST-Sub		COVID (CT&X-rays)
Method	Order	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)
Finetuning	-	38.65	10.43	37.47	14.54	39.62	17.87	29.18	5.66	26.55	10.54
Finetuning	reversed	47.80	26.52	49.67	22.97	25.32	14.75	31.13	18.45	34.47	11.19
SimpleCIL [ 48 ]	-	90.10	85.41	83.85	79.79	56.61	38.30	68.07	50.63	75.84	57.37
SimpleCIL [ 48 ]	reversed	90.17	85.91	89.54	79.85	53.79	38.30	49.58	50.44	77.88	66.79
SSIAT [ 28 ]	-	75.36	66.34	86.00	84.63	60.46	41.99	59.43	25.79	72.00	60.17

TABLE V: Performance comparison across five benchmark datasets using a reversed task order. The best and second-best results are bolded and underlined , respectively.

Method	Task Number	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)
SimpleCIL [ 48 ]	4	68.07	50.63
SimpleCIL [ 48 ]	8	51.41	50.44
MOS [ 27 ]	4	74.59	51.80
MOS [ 27 ]	8	80.96	61.83
Bi-CRCL (ours)	4	84.22	69.71
Bi-CRCL (ours)	8	86.48	70.28

TABLE VI: Effect of task-split granularity on Bi-CRCL performance (MedMNIST-Sub dataset).

Dataset	Replay	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)
Colon	$\checkmark$	99.06	98.43
	$\times$	99.12	98.51
Blood	$\checkmark$	98.06	97.53
	$\times$	98.08	97.56
Skin8	$\checkmark$	74.56	61.99
	$\times$	74.59	61.99

TABLE VII: Empirical analysis of memory replay in Bi-CRCL.

Method	$Acc_{\text{Avg}}$ (%)	$Acc_{\text{Last}}$ (%)
Joint Training	-	71.65
SimpleCIL [ 48 ]	57.50	55.41
ADAM-Adapter [ 48 ]	63.18	55.52
SLCA [ 44 ]	38.34	33.72
SSIAT [ 28 ]	53.97	28.41
MOS [ 27 ]	70.66	62.05

TABLE VIII: Performance comparison on inter-dataset continual learning. The best results are bolded .

Explore similar work

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning
Nov 14, 2025Lan Li, Tao Hu, Da-Wei Zhou +3Class-Incremental Learning VLM Adaptation
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China
From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning
Feb 24, 2026Ruiqi Liu, Boyu Diao, Hangda Liu +3Class-Incremental Learning Incremental Learning
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
MedCRP-CL: Continual Medical Image Segmentation via Bayesian Nonparametric Semantic Modality Discovery
May 19, 2026Ziyuan GaoInteractive Medical Image Segmentation Image Segmentation
University College London, London, United Kingdom

Nov 14, 2025J/K move · Enter open · S save

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning

Lan Li, Tao Hu, Da-Wei Zhou, Jia-Qi Yang+2

School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China

Class-Incremental Learning VLM Adaptation Incremental Learning

Open paper AskPDF arXiv

Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge. Vision-language models such as CLIP offer strong transferable representations via multi-modal supervision, making them promising for CIL. However, applying CLIP to CIL poses two major challenges: (1) adapting to downstream tasks often requires additional learnable modules, increasing model complexity and susceptibility to forgetting; and (2) while multi-modal representations offer complementary strengths, existing methods have yet to fully realize their potential in effectively integrating visual and textual modalities. To address these issues, we propose BOFA (Bridge-layer Orthogonal Fusion for Adaptation), a novel framework for CIL. BOFA confines all model adaptation exclusively to CLIP's existing cross-modal bridge-layer, thereby adding no extra parameters or inference cost. To prevent forgetting within this layer, it leverages Orthogonal Low-Rank Fusion, a mechanism that constrains parameter updates to a low-rank ``safe subspace" mathematically constructed to be orthogonal to past task features. This ensures stable knowledge accumulation without data replay. Furthermore, BOFA employs a cross-modal hybrid prototype that synergizes stable textual prototypes with visual counterparts derived from our stably adapted bridge-layer, enhancing classification performance. Extensive experiments on standard benchmarks show that BOFA achieves superior accuracy and efficiency compared to existing methods.

Abstract

Explore similar work

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning

From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning

MedCRP-CL: Continual Medical Image Segmentation via Bayesian Nonparametric Semantic Modality Discovery