One Adapter for All: Towards Unified Representation in Step-Imbalanced Class-Incremental Learning

Xiaoyan Zhang; Jiangpeng He

cs.CVMar 10, 2026

One Adapter for All: Towards Unified Representation in Step-Imbalanced Class-Incremental Learning

Authors: Xiaoyan Zhang, Jiangpeng He

Organizations: University of Michigan, Ann Arbor, Michigan, U.S.A · Indiana University, Bloomington, Indiana, U.S.A

Abstract

Class-incremental learning (CIL) aims to acquire new classes over time while retaining prior knowledge, yet most setups and methods assume balanced task streams. In practice, the number of classes per task often varies significantly. We refer to this as step imbalance, where large tasks that contain more classes dominate learning and small tasks inject unstable updates. Existing CIL methods assume balanced tasks and therefore treat all tasks uniformly, producing imbalanced updates that degrade overall learning performance. To address this challenge, we propose One-A, a unified and imbalance-aware framework that incrementally merges task updates into a single adapter, maintaining constant inference cost. One-A performs asymmetric subspace alignment to preserve dominant subspaces learned from large tasks while constraining low-information updates within them. An information-adaptive weighting balances the contribution between base and new adapters, and a directional gating mechanism selectively fuses updates along each singular direction, maintaining stability in head directions and plasticity in tail ones. Across multiple benchmarks and step-imbalanced streams, One-A achieves competitive accuracy with significantly low inference overhead, showing that a single, asymmetrically fused adapter can remain both adaptive to dynamic task sizes and efficient at deployment.

Figures & tables

Figure 1: Necessity of step-imbalanced continual learning and overview of our method. (a) Task splitting increases incremental steps, resulting in higher computation, more parameters, and stronger interference, which harms accuracy. (b) Direct merging averages task-specific parameters without alignment. (c) Our method aligns task subspaces via SVD and performs direction-aware fusion to preserve dominant knowledge while integrating information into a unified adapter with balanced stability and plasticity.

Figure 2: Performance comparison under balanced and step-imbalanced settings. All methods show clear degradation under step-imbalanced tasks.

Figure 3: Overview of One-A. For each layer, we decompose the large-task adapter via SVD to extract its dominant subspace $U_{b}\Sigma_{b}$ and project the small-task adapter for subspace alignment ( $V_{a\rightarrow b}$ ). A global fusion is first performed between aligned components to obtain $V_{\mathrm{fused}}$ , followed by direction-wise gating $g_{i}$ on the update $(V_{\mathrm{fused}}-V_{b})$ , which adaptively modulates each singular direction to balance stability and plasticity and yields the final merged adapter $\Delta W_{m}$ .

Method	CIFAR100 (T=10)		CUB (T=20)		ImageNet-A (T=10)		ImageNet-R (T=40)
Method	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$
L2P [ 54 ]	64.63 ± 1.33	76.69 ± 1.02	53.04 ± 1.35	69.00 ± 1.08	34.52 ± 2.40	46.51 ± 3.08	45.79 ± 1.48	58.26 ± 1.48
DualPrompt [ 53 ]	71.43 ± 1.35	81.12 ± 0.46	53.76 ± 0.72	70.28 ± 1.41	32.08 ± 2.29	42.59 ± 2.64	45.13 ± 1.42	55.25 ± 1.19
CODA-Prompt [ 41 ]	81.40 ± 0.61	87.55 ± 0.34	66.02 ± 1.70	77.08 ± 0.76	44.37 ± 0.90	53.46 ± 1.41	53.18 ± 1.75	62.13 ± 0.79
Aper-adapter [ 57 ]	86.47 ± 1.87	91.04 ± 2.41	87.31 ± 0.22	91.38 ± 0.82	50.74 ± 1.84	61.31 ± 3.04	54.61 ± 0.14	63.36 ± 1.58
EASE [ 59 ]	86.90 ± 0.73	91.71 ± 0.88	83.12 ± 0.23	87.05 ± 2.75	56.11 ± 0.54	64.52 ± 1.03	67.02 ± 0.31	72.98 ± 0.83

Table 1: Last task accuracy (

A_{T}

) and average accuracy (

\bar{A}

) on multiple datasets with the step imbalance factor

\gamma=0.01

and different task numbers. All results are averaged over 5 runs with mean ± standard deviation. Red denotes the best result and Blue denotes the second best in each column.

Method	$\gamma=0.001$				$\gamma=0.05$
	$T=10$		$T=20$		$T=10$		$T=20$
	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$
Aper-adapter [ 57 ]	84.91	90.31	82.41	87.68	86.32	90.34	83.05	88.32
EASE [ 59 ]	87.52	91.98	84.79	90.42	87.02	91.74	84.42	89.23
SEMA [ 49 ]	84.70	88.58	74.85	81.86	85.63	91.18	81.48	87.85
ACMap [ 12 ]	83.43	90.15	83.80	88.84	85.81	91.21	84.25	89.08

Table 2: Last task accuracy (

A_{T}

) and average accuracy (

\bar{A}

) on multiple datasets with different step imbalance factors

\gamma

and task numbers

T

Figure 4: Accuracy at each step on ImageNet-A under different step-imbalance ratios $\gamma\in\{0.001,0.005,0.01,0.05\}$ .

Figure 5: Forgetting curves under different configurations, along with the corresponding $\bar{A}$ .

ASA	GW	DG	CIFAR-100		ImageNet-A
ASA	GW	DG	$T=10$	$T=20$	$T=10$	$T=20$
			72.43	78.34	45.82	47.60
$\checkmark$			75.82	81.27	48.02	48.39
	$\checkmark$		82.46	80.74	51.32	49.24
$\checkmark$	$\checkmark$		84.31	82.76	53.65	52.01
$\checkmark$	$\checkmark$	$\checkmark$	88.39	85.32	59.25	56.62

Table 3: Ablation study of each component of One-A on CIFAR100 and ImageNet-A.

Method	$A_{T}$ (%)	$w\bar{A}$ (%)	FLOPs	#Adapters
Per-task adapter	43.15	55.67	$39.19\times$	$40(T)$
MOS [ 44 ]	67.73	74.04	$40.60\times$	$40(T)$
ACMap [ 12 ]	61.32	68.56	$0.99\times$	1
One-A (Ours)	70.55	76.95	$1.\times$	1

Table 4: Comparison of merging strategies on SI-CIL.

Panel of Figure 6 — Figure 6: Illustration of the subspace behavior during adapter merging.

Panel of Figure 7 — Figure 7: Sensitivity analysis on the gating parameters. Last-step accuracy ( $A_{T}$ ) under different quantile threshold $\tau_{q}$ and gating sharpness $\kappa$ .

Figure 8: Generation of the step-imbalanced task sequence. Left: head-to-tail allocation based on exponential class ratios. Right: random permutation producing the final task order.

Figure 9: Illustration of long-tailed CIL vs. SI-CIL. We show an example with

T{=}10

tasks and

C{=}100

classes. Top (Long-tailed CIL): each task contains the same number of classes, while the number of samples per class is long-tailed. Bottom (SI-CIL): the number of samples per class is fixed, while the number of classes varies significantly across incremental steps.

Method	CIFAR100	ImageNet-A
Norm-based	86.23	55.17
SV-based	85.83	53.65
Class count-based	88.58	59.64

Table 5: Comparison of information proxies for global fusion in One-A under the SI-CIL setting. Results are last-step accuracy (%) on CIFAR100 and ImageNet-A.

Method	CIFAR100	ImageNet-A
w/o $\mathcal{L}_{\mathrm{ctr}}$	88.34	59.38
w/o $E(t)$	88.26	58.99
One-A	88.58	59.64

Table 6: Ablation of the contrastive loss

\mathcal{L}_{\mathrm{ctr}}

and the size-dependent epoch schedule

E(t)

in One-A under the SI-CIL setting. Results are last-step accuracy (%) on CIFAR100 and ImageNet-A.

Dataset	LR	Weight Decay	Batch Size	$E_{0}$	$r$
CIFAR100	0.01	$5\times 10^{-4}$	128	25	128
CUB	0.07	0.0005	16	10	128
ImageNet-A	0.08	0.003	64	40	128
ImageNet-R	0.03	0.005	24	30	64

Table 7: Dataset-specific hyperparameters used in both main and supplementary experiments.

Figure 10: Class-per-task distributions for the descending SI-CIL configurations.

Method	CIFAR100 (T=10)	CUB (T=20)	ImageNet-A (T=10)	ImageNet-R (T=40)
L2P [ 54 ]	76.16 ± 1.76	53.47 ± 2.39	36.71 ± 1.29	46.43 ± 1.01
DualPrompt [ 53 ]	71.38 ± 0.54	55.19 ± 0.98	33.16 ± 2.12	46.28 ± 0.60
CODA-Prompt [ 41 ]	80.01 ± 0.48	64.83 ± 0.95	45.93 ± 0.37	54.73 ± 0.22
Aper-adapter [ 57 ]	88.24 ± 0.72	87.67 ± 0.19	53.63 ± 0.60	62.79 ± 0.34
EASE [ 59 ]	86.56 ± 1.16	84.31 ± 0.45	56.16 ± 0.28	65.00 ± 0.77
SEMA [ 49 ]	88.31 ± 0.74	67.88 ± 1.97	43.95 ± 1.98	64.92 ± 1.02

Table 8: Last-task accuracy (

A_{T}

) under the descending scenario with step-imbalance factor

\gamma=0.01

. All results are averaged over 5 runs with mean

\pm

standard deviation. Red denotes the best result and Blue denotes the second best in each column. Numbers in parenthesis indicate the improvement over the second-best method.

Method	T=5		T=10		T=20		T=40
Method	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$
L2P [ 54 ]	62.05 ± 1.81	74.68 ± 1.62	59.22 ± 1.14	73.24 ± 0.38	57.28 ± 0.87	70.04 ± 0.27	55.19 ± 0.95	66.77 ± 1.03
DualPrompt [ 53 ]	63.07 ± 0.49	75.41 ± 0.75	57.51 ± 0.96	72.52 ± 1.70	65.09 ± 1.12	77.32 ± 0.45	56.25 ± 1.89	71.15 ± 1.80
CODA-Prompt [ 41 ]	78.50 ± 0.42	86.14 ± 0.35	74.26 ± 0.15	83.30 ± 0.64	70.96 ± 1.43	81.85 ± 0.29	65.60 ± 1.18	77.53 ± 0.14
Aper-adapter [ 57 ]	87.67 ± 0.19	91.29 ± 0.74	87.28 ± 0.14	91.79 ± 0.63	86.89 ± 0.32	91.76 ± 0.57	87.19 ± 0.01	91.93 ± 0.63
EASE [ 59 ]	84.34 ± 0.33	89.51 ± 0.73	86.01 ± 0.23	90.83 ± 0.61	87.11 ± 0.13	91.79 ± 0.52	86.79 ± 0.39	91.88 ± 0.43

Table 9: Last-task accuracy (

A_{T}

) and average accuracy

\bar{A}

under the balanced scenario with CUB under different task numbers. All results are averaged over 5 runs with mean

\pm

standard deviation. Red denotes the best result and Blue denotes the second best in each column.

Method	CIFAR100 (T=10)		CUB (T=20)
Method	$A_{T}$	$\bar{A}$	$A_{T}$	$\bar{A}$
Aper-adapter [ 57 ]	78.91 ± 0.63	85.24 ± 2.02	79.30 ± 0.02	85.78 ± 0.59
EASE [ 59 ]	81.20 ± 1.53	88.50 ± 0.95	79.40 ± 0.27	86.59 ± 0.73
InfLoRA [ 28 ]	71.69 ± 1.55	79.81 ± 2.42	49.32 ± 1.93	62.52 ± 2.99
SEMA [ 49 ]	64.41 ± 0.85	74.98 ± 2.15	49.06 ± 0.16	62.00 ± 1.53
ACMap [ 12 ]	80.50 ± 0.13	87.11 ± 0.83	79.47 ± 0.04	85.90 ± 0.63

Table 10: Last task accuracy (

A_{T}

) and average accuracy (

\bar{A}

) under the long-tailed step-imbalance setting.

Figure 11: Adapter rank sensitivity on ImageNet-A with $T=10$ tasks.

Figure 12: Inference accuracy and efficiency on CIFAR100 under SI-CIL

Adapter-Based Continual Learning

See all 17 papers

Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learning
May 28, 2026Xiang Tan, Run He, Yawen Cui +6Class-Incremental Learning Incremental Learning
South China University of Technology · Hong Kong Polytechnic University · Agency for Science, Technology and Research +3
One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning
Aug 31, 2026Yunxiang Fu, Meng Lou, Yizhou YuClass-Incremental Learning Incremental Learning
School of Computing and Data Science The University of Hong Kong
From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning
Feb 24, 2026Ruiqi Liu, Boyu Diao, Hangda Liu +3Adapter-Based Continual Learning Class-Incremental Learning
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

May 28, 2026J/K move · Enter open · S save

Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learning

Xiang Tan, Run He, Yawen Cui, Mengchen Zhao+5

South China University of Technology · Hong Kong Polytechnic University · Agency for Science, Technology and Research+3

Class-Incremental Learning Incremental Learning Catastrophic Forgetting

Open paper AskPDF arXiv

Class-Incremental Learning (CIL) with pre-trained models (PTMs) aims to sequentially adapt PTMs to new categories without forgetting old knowledge. Built upon PTMs, existing adapter-based methods mainly train models via distinct task-specific adapters, and present a uniform knowledge allocation for each adapter during inference. However, this allocation mechanism ignores the nature of task discrepancy and leads to suboptimal utilization of adapters. Also, under CIL constraint, an allocator is prone to forgetting when tasks evolve. To address these issues, we propose a Non-Forgetting Allocation with Bi-Level Competition (NoFA-BC). NoFA-BC constructs a non-forgetting allocator (NFA) by transforming the allocator training into a recursive least-squares problem and achieves an allocator equivalent to that trained with all data. Based on the NFA, a Bi-Level Competition (BLC) including an intra-task level Winner-Takes-All (WTA) mechanism and inter-task Last-Ones-Fall (LOF) elimination is proposed to provide better allocation of adapter knowledge. WTA extracts the most significant logit within a task to represent the adapter's contribution and LOF suppresses the irrelevant adapters. With BLC, participation ratio of each adapter can be tailored for each input. Moreover, a Stability Enhancement (SE) process is incorporated to further improve the performance of old tasks.

Abstract

Adapter-Based Continual Learning

Explore similar work

Non-Forgetting Knowledge Allocation with Bi-level Competition for Class-Incremental Learning

One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning

From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning