AIR: Analytic Imbalance Rectifier for Continual Learning

Di Fang; Yinan Zhu; Zhiping Lin; Cen Chen; Ziqian Zeng; Huiping Zhuang

cs.LGAug 19, 2024

AIR: Analytic Imbalance Rectifier for Continual Learning

Authors: Di Fang, Yinan Zhu, Zhiping Lin, Cen Chen, Ziqian Zeng, Huiping Zhuang

Organizations: School of Future Technology, South China University of Technology, Guangzhou, China · Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, Guangzhou, China

Abstract

Continual learning (CL) agents incrementally learn from sequentially arriving data and adapt to the dynamic, ever-changing nature of real-world environments. However, many existing CL methods suffer performance degradation in evolving, imbalanced data streams due to limited adaptation to changing class frequencies or ineffective use of mixed data from new and previously observed classes. To deal with these challenges, we propose an analytic imbalance rectifier (AIR) algorithm for real-world CL. AIR is an online exemplar-free approach with a frozen backbone as the feature extractor and a closed-form incremental classifier whose weight equals the joint-learning weight for the same class-weighted ridge objective. AIR addresses class imbalance with an analytic reweighting module (ARM) that calculates a reweighting factor for each class in the loss function to equalize total sample weights across classes. Under long-tailed class-incremental learning, AIR leads 28 baselines in aggregate accuracy and exemplar-free methods in aggregate macro F1, gaining 3.21% accuracy and 2.14% macro F1 over the respective strongest exemplar-free baselines. Under the Si-Blurry setting with recurring classes, AIR leads 15 exemplar-based and exemplar-free baselines, gaining 2.32% aggregate accuracy and 1.27% aggregate macro F1 over the strongest baseline. One-sided paired tests support positive mean absolute gains in these four comparisons (Holm-adjusted p<0.006).

Figures & tables

Method	Score	CIFAR-100 (LT)						ImageNet-R (LT)						Time
		Ascending		Descending		Shuffled		Ascending		Descending		Shuffled
		$\mathcal{A}_{\text{avg}}$	$\mathcal{A}_{\text{last}}$	$\mathcal{A}_{\text{avg}}$	$\mathcal{A}_{\text{last}}$	$\mathcal{A}_{\text{avg}}$	$\mathcal{A}_{\text{last}}$	$\mathcal{A}_{\text{avg}}$	$\mathcal{A}_{\text{last}}$	$\mathcal{A}_{\text{avg}}$	$\mathcal{A}_{\text{last}}$	$\mathcal{A}_{\text{avg}}$	$\mathcal{A}_{\text{last}}$
DGR [ 11 ]	61.01 $\pm$ 0.45	47.49 $\pm$ 0.77	57.85 $\pm$ 0.24	85.67 $\pm$ 0.27	65.07 $\pm$ 0.44	67.67 $\pm$ 3.45	60.09 $\pm$ 1.66	44.14 $\pm$ 1.54	54.35 $\pm$ 0.77	72.12 $\pm$ 0.35	56.55 $\pm$ 0.32	63.60 $\pm$ 1.31	57.50 $\pm$ 0.39	9.19
MVP-R [ 27 ]	61.39 $\pm$ 0.66	64.49 $\pm$ 0.57	54.57 $\pm$ 0.59	83.62 $\pm$ 0.25	70.99 $\pm$ 0.68	67.60 $\pm$ 2.88	59.39 $\pm$ 2.88	51.16 $\pm$ 0.58	46.42 $\pm$ 0.50	69.46 $\pm$ 0.22	58.96 $\pm$ 0.36	59.59 $\pm$ 1.33	50.42 $\pm$ 0.70	6.84
CLIB [ 19 ]	62.48 $\pm$ 0.55	62.86 $\pm$ 1.08	62.36 $\pm$ 0.79	84.15 $\pm$ 0.30	70.03 $\pm$ 0.26	71.63 $\pm$ 2.41	65.69 $\pm$ 1.50	54.84 $\pm$ 0.59	47.56 $\pm$ 0.40	66.17 $\pm$ 0.29	53.51 $\pm$ 0.34	60.34 $\pm$ 1.00	50.67 $\pm$ 0.64	0.51
PRS [ 17 ]	64.21 $\pm$ 0.61	49.02 $\pm$ 0.67	66.42 $\pm$ 0.77	81.29 $\pm$ 0.59	64.45 $\pm$ 0.60	70.46 $\pm$ 2.53	69.81 $\pm$ 1.97	46.39 $\pm$ 0.68	56.60 $\pm$ 1.72	73.66 $\pm$ 0.39	66.45 $\pm$ 0.50	64.60 $\pm$ 1.30	61.34 $\pm$ 0.89	7.20

Table 1: LT-CIL accuracy (%). The first block uses exemplars, with fixed-memory methods before growing-memory methods: fixed memories retain 500 samples on CIFAR-100 or 1,000 on ImageNet-R, and growing memories retain 5/class. The second block and AIR are exemplar-free. Bold / underlined mark the best/second-best exemplar-free results. Values are mean

\pm

standard error over 6 runs. Time is the mean training time in s/1,000 samples across datasets and orders (lower is better).

Method	Score	CIFAR-100 (LT)						ImageNet-R (LT)						Time
		Ascending		Descending		Shuffled		Ascending		Descending		Shuffled
		$\mathcal{F}_{\text{avg}}$	$\mathcal{F}_{\text{last}}$	$\mathcal{F}_{\text{avg}}$	$\mathcal{F}_{\text{last}}$	$\mathcal{F}_{\text{avg}}$	$\mathcal{F}_{\text{last}}$	$\mathcal{F}_{\text{avg}}$	$\mathcal{F}_{\text{last}}$	$\mathcal{F}_{\text{avg}}$	$\mathcal{F}_{\text{last}}$	$\mathcal{F}_{\text{avg}}$	$\mathcal{F}_{\text{last}}$
DGR [ 11 ]	59.41 $\pm$ 0.47	44.42 $\pm$ 0.90	55.22 $\pm$ 0.27	85.39 $\pm$ 0.27	62.13 $\pm$ 0.49	65.24 $\pm$ 3.64	57.63 $\pm$ 1.45	42.68 $\pm$ 1.37	52.31 $\pm$ 0.78	71.66 $\pm$ 0.42	56.38 $\pm$ 0.24	63.00 $\pm$ 1.44	56.81 $\pm$ 0.41	9.19
MVP-R [ 27 ]	61.08 $\pm$ 0.74	62.66 $\pm$ 0.76	51.87 $\pm$ 0.47	83.02 $\pm$ 0.25	67.98 $\pm$ 0.70	64.74 $\pm$ 3.35	57.57 $\pm$ 2.91	53.49 $\pm$ 0.52	47.10 $\pm$ 0.56	71.13 $\pm$ 0.27	59.36 $\pm$ 0.37	61.12 $\pm$ 1.24	52.94 $\pm$ 0.77	6.84
CLIB [ 19 ]	63.91 $\pm$ 0.63	61.44 $\pm$ 1.30	64.25 $\pm$ 0.71	84.02 $\pm$ 0.30	67.87 $\pm$ 0.35	69.49 $\pm$ 2.96	65.64 $\pm$ 1.51	58.17 $\pm$ 0.58	54.47 $\pm$ 0.52	68.17 $\pm$ 0.36	55.13 $\pm$ 0.29	62.63 $\pm$ 0.92	55.59 $\pm$ 0.55	0.51
PRS [ 17 ]	64.04 $\pm$ 0.65	47.39 $\pm$ 0.77	66.78 $\pm$ 0.66	79.91 $\pm$ 0.63	59.03 $\pm$ 0.79	68.05 $\pm$ 2.85	69.37 $\pm$ 1.98	49.75 $\pm$ 0.71	61.61 $\pm$ 1.39	73.86 $\pm$ 0.38	62.43 $\pm$ 0.26	66.66 $\pm$ 1.17	63.63 $\pm$ 0.67	7.20

Table 2: LT-CIL macro F1 (%). The experimental settings, layout, methods, memory grouping, seeds, and timing convention follow Table 1 . Score is the unweighted mean of

\mathcal{F}_{\text{avg}}

and

\mathcal{F}_{\text{last}}

over both datasets and all three orders. Bold / underlined mark the best/second-best exemplar-free results; values are mean

\pm

standard error over six runs.

Method	Type	Score	CIFAR-100			ImageNet-R			CORe50			CUB-200-2011			Time
			AUC	Avg	Last	AUC	Avg	Last	AUC	Avg	Last	AUC	Avg	Last
O-LoRA [ 35 ]	$\mathcal{A}$	54.95 $\pm$ 0.98	68.15 $\pm$ 2.52	68.02 $\pm$ 1.83	68.20 $\pm$ 2.69	61.45 $\pm$ 0.98	60.85 $\pm$ 1.06	53.54 $\pm$ 1.70	47.06 $\pm$ 2.17	50.73 $\pm$ 3.33	50.84 $\pm$ 3.78	40.54 $\pm$ 0.71	47.50 $\pm$ 1.66	42.60 $\pm$ 1.07	5.10
	$\mathcal{F}$	50.54 $\pm$ 1.14	64.44 $\pm$ 3.07	65.09 $\pm$ 2.25	65.83 $\pm$ 2.98	59.57 $\pm$ 1.18	59.49 $\pm$ 1.14	53.91 $\pm$ 1.50	40.23 $\pm$ 2.62	43.44 $\pm$ 4.13	44.41 $\pm$ 4.56	33.30 $\pm$ 0.95	40.97 $\pm$ 2.17	35.80 $\pm$ 1.43
UniCIL [ 37 ]	$\mathcal{A}$	57.39 $\pm$ 4.70	78.00 $\pm$ 0.97	72.96 $\pm$ 1.45	85.38 $\pm$ 0.34	37.34 $\pm$ 15.94	38.28 $\pm$ 16.41	38.83 $\pm$ 16.87	49.06 $\pm$ 16.10	49.84 $\pm$ 16.21	49.93 $\pm$ 16.92	62.77 $\pm$ 2.05	63.29 $\pm$ 2.97	63.02 $\pm$ 4.83	7.95
	$\mathcal{F}$	54.42 $\pm$ 4.78	75.27 $\pm$ 1.47	69.64 $\pm$ 1.60	85.35 $\pm$ 0.32	35.47 $\pm$ 15.85	36.26 $\pm$ 16.20	37.91 $\pm$ 16.93	46.49 $\pm$ 16.42	47.47 $\pm$ 16.76	48.78 $\pm$ 17.38	56.09 $\pm$ 2.53	56.39 $\pm$ 3.43	57.90 $\pm$ 5.94
EWC++ [ 18 ]	$\mathcal{A}$	72.55 $\pm$ 0.48	72.44 $\pm$ 1.22	70.95 $\pm$ 0.86	69.91 $\pm$ 0.79	61.96 $\pm$ 0.76	61.62 $\pm$ 1.19	54.16 $\pm$ 0.58	81.28 $\pm$ 0.68	81.81 $\pm$ 0.67	82.54 $\pm$ 1.29	77.49 $\pm$ 1.02	81.10 $\pm$ 0.98	75.28 $\pm$ 0.51	0.50

Table 3: Si-Blurry accuracy (

\mathcal{A}

) and macro F1 (

\mathcal{F}

), in % (mean

\pm

standard error, 6 runs). The first block retains samples, ordered by increasing Accuracy Score; the rest are exemplar-free. Fixed memories hold 500/1,000/250/1,000 samples in dataset-column order; growing memories hold 5/class. O-LoRA retains four hard samples only for MAS estimation. Bold / underlined : best/second-best exemplar-free results. Time is the mean training time in s/1,000 samples across datasets (lower is better).

	$\mathcal{A}_{\text{avg}}$	$\mathcal{A}_{\text{last}}$	$\mathcal{A}_{\text{avg}}$	$\mathcal{A}_{\text{last}}$	$\mathcal{A}_{\text{avg}}$	$\mathcal{A}_{\text{last}}$
Method	Ascending		Descending		Shuffled
ACIL w/o ARM	73.28 $\pm$ 0.71	62.00 $\pm$ 0.44	83.99 $\pm$ 0.20	62.00 $\pm$ 0.44	69.66 $\pm$ 1.45	62.00 $\pm$ 0.44
ACIL w/ ARM	77.21 $\pm$ 0.74 (+3.93 $\pm$ 0.09)	72.60 $\pm$ 0.57 (+10.60 $\pm$ 0.31)	87.93 $\pm$ 0.14 (+3.94 $\pm$ 0.13)	72.60 $\pm$ 0.57 (+10.60 $\pm$ 0.31)	77.13 $\pm$ 1.64 (+7.47 $\pm$ 0.23)	72.60 $\pm$ 0.57 (+10.60 $\pm$ 0.31)
DS-AL w/o ARM	73.90 $\pm$ 0.75	62.66 $\pm$ 0.46	84.70 $\pm$ 0.21	64.41 $\pm$ 0.52	70.97 $\pm$ 1.51	63.63 $\pm$ 0.56
DS-AL w/ ARM	75.61 $\pm$ 0.77 (+1.71 $\pm$ 0.04)	69.32 $\pm$ 0.57 (+6.66 $\pm$ 0.26)	86.73 $\pm$ 0.16 (+2.03 $\pm$ 0.09)	69.88 $\pm$ 0.56 (+5.47 $\pm$ 0.21)	74.58 $\pm$ 1.65 (+3.61 $\pm$ 0.19)	69.60 $\pm$ 0.59 (+5.97 $\pm$ 0.20)

Table 4: Accuracy (%) after adding ARM to ACIL and DS-AL on CIFAR-100 LT-CIL (

\rho=500

). Leading values are six-seed means

\pm

standard errors; parentheses on the ARM rows report the paired improvement as mean

\pm

standard error over the same seeds.

Method	$\bm{A}$	$\bm{C}$	$\bm{W}_{k-1}$	Total
ACIL & GACL	$\Theta(f^{2})$	-	$\Theta(fC_{k})$	$\Theta(f^{2}+fC_{k})$
RanPAC	$\Theta(f^{2})$	$\Theta(fC_{k})$	$\Theta(fC_{k})$	$\Theta(f^{2}+fC_{k})$
AIR	$\Theta(f^{2})$	$\Theta(fC_{k})$	$\Theta(fC_{k})$	$\Theta(f^{2}+fC_{k})$
G-AIR	$\Theta(f^{2}C_{k})$	$\Theta(fC_{k})$	$\Theta(fC_{k})$	$\Theta(f^{2}C_{k}+fC_{k})$

Table 5: Storage costs of retained matrices

\bm{A}

\bm{C}

, and

\bm{W}_{k-1}

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Setting	Phase-end
comparisons	$\max\epsilon_{W}$	Final accuracy (%)
			CL	Joint
LT-CIL ascending	40	$5.02\times 10^{-14}$	76.11	76.11
LT-CIL descending	40	$4.58\times 10^{-14}$	76.11	76.11
LT-CIL shuffled	40	$6.03\times 10^{-14}$	76.11	76.11
Si-Blurry	10	$5.81\times 10^{-14}$	84.31	84.31

Appendix

Table A.1: Joint–continual equivalence on cached CIFAR-100 features. Comparison counts equal phases

\times

two seeds; errors are maxima across comparisons. Final test accuracies average the two seeds for continual (CL) and joint learning.

Explore similar work

Realistic Continual Learning Approach using Pre-trained Models
Apr 11, 2024Nadia Nasri, Carlos Gutiérrez-Álvarez, Sergio Lafuente-Arroyo +2Efficient Continual Learning Continual Learning Methods
CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework
Sep 15, 2026Yunxiang Fu, Meng Lou, Zicheng Liao +1Efficient Continual Learning Domain Incremental Learning
Data-Free Reservoir Features for Efficient Long-Horizon Cold-Start Continual Learning
Jun 25, 2026Augustinas Jučas, Yangchen PanDomain Incremental Learning Efficient Continual Learning

Apr 11, 2024 · cs.LGJ/K move · Enter open · S save

Realistic Continual Learning Approach using Pre-trained Models

Nadia Nasri, Carlos Gutiérrez-Álvarez, Sergio Lafuente-Arroyo, Saturnino Maldonado-Bascón+1

University of Alcala, Department of Signal Theory and Communications, GRAM Research group, Alcala de Henares, 28805, Spain · Univesity of Alicante, Department of Computer Science and Artificial Intelligence, San Vicente del Raspeig, 03690, Spain

Continual learning (CL) evaluates adaptability in learning solutions to retain knowledge. Our research addresses the challenge of catastrophic forgetting, where models lose proficiency in previously learned tasks as they acquire new ones. While numerous solutions have been proposed, existing experimental setups often rely on idealized class-incremental learning scenarios. We introduce Realistic Continual Learning (RealCL), a novel CL paradigm where class distributions across tasks are random. We also present CLARE (Continual Learning Approach with pRE-trained models for RealCL scenarios), a pre-trained model-based solution designed to integrate new knowledge while preserving past learning. Our contributions include pioneering RealCL as a generalization of traditional CL setups, proposing CLARE as an adaptable approach for RealCL tasks, and conducting extensive experiments demonstrating its effectiveness across various RealCL scenarios. Notably, CLARE outperforms existing models on RealCL benchmarks, highlighting its versatility in unpredictable learning environments. Code to reproduce all our experiments can be found at https://github.com/gramuah/clare.

Efficient Continual Learning Continual Learning Methods Catastrophic Forgetting Model Pretraining

Open paper Ask about it PDF

Abstract

Explore similar work

Realistic Continual Learning Approach using Pre-trained Models

CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework

Data-Free Reservoir Features for Efficient Long-Horizon Cold-Start Continual Learning