Temporal Imbalance of Positive and Negative Supervision in Class-Incremental Learning

Jinge Ma; Fengqing Zhu

cs.LGMar 2, 2026

Temporal Imbalance of Positive and Negative Supervision in Class-Incremental Learning

Authors: Jinge Ma, Fengqing Zhu

Organizations: Purdue University, West Lafayette, Indiana, U.S.A.

Abstract

With the widespread adoption of deep learning in visual tasks, Class-Incremental Learning (CIL) has become an important paradigm for handling dynamically evolving data distributions. However, CIL faces the core challenge of catastrophic forgetting, often manifested as a prediction bias toward new classes. Existing methods mainly attribute this bias to intra-task class imbalance and focus on corrections at the classifier head. In this paper, we highlight an overlooked factor -- temporal imbalance -- as a key cause of this bias. Earlier classes receive stronger negative supervision toward the end of training, leading to asymmetric precision and recall. We establish a temporal supervision model, formally define temporal imbalance, and propose Temporal-Adjusted Loss (TAL), which uses a temporal decay kernel to construct a supervision strength vector and dynamically reweight the negative supervision in cross-entropy loss. Theoretical analysis shows that TAL degenerates to standard cross-entropy under balanced conditions and effectively mitigates prediction bias under imbalance. Extensive experiments demonstrate that TAL significantly reduces forgetting and improves performance on multiple CIL benchmarks, underscoring the importance of temporal modeling for stable long-term learning.

Figures & tables

Figure 1: Illustration of temporal imbalance, which differs from class imbalance. In Task 2, although old classes A and B have the same number of training samples, the earlier old class suffers from more severe forgetting than the later one.

Panel of Figure 2 — Figure 2: (a) An example of Temporal Imbalance: it illustrates, in the CIL setting, how the cumulative number of positive samples $S[n]$ for two classes evolves as the number of training batches $n$ increases. Since the positive samples of class 11 appear earlier than those of class 45, the result shows that $S_{11}[n]\geq S_{45}[n]$ . (b)&(c) Effects of Temporal Imbalance: (b) Shows, at the end of training, the correlation between the Precision/Recall of each class and its $Q$ value when ER is trained with the cross-entropy loss. For clearer visualization, the normalized value $\tilde{Q}=Q-Q_{\min}$ is used, where $Q_{\min}$ denotes the minimum $Q$ across all classes. (c) Demonstrates that, at the end of training, the Precision–Recall asymmetry induced by temporal imbalance is a common phenomenon across different CIL methods. Note that all classes shown in the figure are old classes, so there is no class imbalance in the final task.

Method	CIFAR-100				ImageNet-100				Food101
Method	10-task		20-task		10-task		20-task		10-task
	$A_{Mean}$	$A_{Last}$	$A_{Mean}$	$A_{Last}$	$A_{Mean}$	$A_{Last}$	$A_{Mean}$	$A_{Last}$	$A_{Mean}$	$A_{Last}$
iCaRL [ 29 ]	58.76 $\pm$ 0.12	45.39 $\pm$ 0.22	55.98 $\pm$ 0.14	43.20 $\pm$ 0.24	43.71 $\pm$ 0.12	24.38 $\pm$ 0.24	33.45 $\pm$ 0.14	17.62 $\pm$ 0.28	63.46 $\pm$ 0.12	50.13 $\pm$ 0.22
+ Ours	60.82 $\pm$ 0.10	47.36 $\pm$ 0.18	58.68 $\pm$ 0.12	44.79 $\pm$ 0.20	52.19 $\pm$ 0.10	32.78 $\pm$ 0.20	42.54 $\pm$ 0.12	23.90 $\pm$ 0.24	68.57 $\pm$ 0.10	56.03 $\pm$ 0.18
FOSTER [ 44 ]	62.04 $\pm$ 0.42	49.30 $\pm$ 0.64	56.99 $\pm$ 0.46	47.34 $\pm$ 0.66	48.39 $\pm$ 0.46	20.88 $\pm$ 0.70	35.67 $\pm$ 0.48	15.28 $\pm$ 0.72	63.88 $\pm$ 0.46	52.60 $\pm$ 0.66
+ Ours	64.80 $\pm$ 0.40	52.76 $\pm$ 0.60	58.21 $\pm$ 0.42	49.28 $\pm$ 0.64	51.49 $\pm$ 0.42	23.38 $\pm$ 0.66	38.07 $\pm$ 0.46	19.72 $\pm$ 0.70	64.61 $\pm$ 0.42	53.90 $\pm$ 0.60

Table 1: Average accuracy (

A_{Mean}

) and last accuracy (

A_{Last}

) under 10-task and 20-task settings across three datasets. “+ Ours” indicates the baseline method with the Temporal-Adjusted Loss applied. All methods use a replay buffer of 20 exemplars per class.

Figure 3: This figure shows the performance of different baselines on each class when Task id = 3 under the 10-task setting on ImageNet-100. It can be observed that earlier classes exhibit lower average class accuracy (i.e., recall), similar to Fig 2 (c). TAL reduces the sensitivity of old classes to negative supervision while increasing that of new classes. As a result, TAL improves the recall of most old classes at the cost of lowering the recall of new classes slightly.

Figure 4: Forgetting curves with representative CIL baselines and datasets.

Figure 5: This figure illustrates the impact of applying TAL on the feature space of iCaRL. We visualize the distribution of test images from several new and old classes in the ResNet18 feature space on ImageNet100 at Task ID = 3.

$r$	$\lambda$
	0.99		0.995		0.999		0.9995
	$A_{\text{Mean}}$	$A_{\text{Last}}$	$A_{\text{Mean}}$	$A_{\text{Last}}$	$A_{\text{Mean}}$	$A_{\text{Last}}$	$A_{\text{Mean}}$	$A_{\text{Last}}$
0.2	61.30	44.39	61.06	44.05	60.69	43.54	60.26	43.18
0.5	62.12	45.36	61.27	45.05	61.72	44.50	61.36	44.28
1.0	62.60	46.13	63.36	47.64	62.46	46.66	61.79	45.44
2.0	62.51	45.38	60.24	45.80	62.85	46.16	62.50	44.85

Table 2: Ablations on

\lambda

and

r

. We report both average accuracy (

A_{\text{Mean}}

) and last accuracy (

A_{\text{Last}}

Figure 6: Forgetting curves with different replay buffer sizes

Figure 7: The impact of different

r

values on the accuracy of each class, arranged in learning order at the end of the last task. Here,

\Delta

Accuracy is defined as the difference between the class accuracy of TAL and that of CE ( i.e. ,

\Delta\text{Accuracy}=\text{Accuracy}_{\text{TAL}}-\text{Accuracy}_{\text{CE}}

). It can be observed that TAL does not simply apply uniform protection to all old classes. In the right figure, the more recent old classes (80–89) are even suppressed.

Method	CE (min)	TAL (min)	% $\Delta$ (TAL vs CE)
ER	42.5	42.8	+0.7%
BiC	208.9	211.7	+1.3%
iCaRL	44.2	45.2	+2.3%
FOSTER	66.7	67.8	+1.6%
MEMO	79.6	78.1	-1.9%
DER	81.3	82.1	+1.0%

Table 3: Comparison of total training time on CIFAR-100 between Cross-Entropy (CE) and TAL. The third column reports the relative change (%) in total training time.

Batch	Method	5 cls	20 cls	100 cls	500 cls
32	CE	10.89	10.78	10.77	10.81
32	TAL	42.77	42.87	42.77	43.03
64	CE	10.83	10.86	10.76	10.81
64	TAL	42.90	42.85	43.13	43.04
128	CE	10.72	10.74	10.75	10.79
128	TAL	42.89	42.80	42.91	43.63

Table 4: Per-batch loss computation time comparison between Cross-Entropy (CE) and TAL under different batch sizes and numbers of classes. Units in

10^{-2}

ms.

Method	CUB-LT [ 42 ]				ImageNetR-LT [ 16 ]				Food101-LT [ 15 ]
Method	5-task		10-task		5-task		10-task		5-task		10-task
	$A_{Mean}$	$A_{Tail}$	$A_{Mean}$	$A_{Tail}$	$A_{Mean}$	$A_{Tail}$	$A_{Mean}$	$A_{Tail}$	$A_{Mean}$	$A_{Tail}$	$A_{Mean}$	$A_{Tail}$
L2P (CVPR’22) [ 46 ]	74.22	57.20	68.91	46.05	70.03	56.32	69.94	54.52	55.36	29.33	47.46	16.11
+ TAL	75.91	61.19	70.71	49.80	72.84	63.50	71.63	60.47	58.53	40.93	50.84	23.84
DualPrompt (ECCV’22) [ 45 ]	73.85	51.30	69.87	50.75	65.40	48.28	65.14	46.52	52.24	25.55	49.84	22.56
+ TAL	75.06	54.65	70.68	52.50	66.45	52.04	66.00	50.06	54.28	30.01	50.77	24.89

Table 5: Average accuracy (

A_{Mean}

) and tail class accuracy (

A_{Tail}

) under 5-task and 10-task settings across three long-tailed datasets. +TAL implies that cross entropy loss is replaced with TAL in the PTM-based methods.

Figure 8: Sample-Class Distribution of Long-Tailed Datasets

5 classes
Dataset	Method	Batch=64	Batch=128	Batch=256
CIFAR	CE	88.56	88.68	85.88
CIFAR	TAL	89.36	89.24	89.36
ImageNet	CE	96.83	94.84	96.43
ImageNet	TAL	97.62	95.24	97.22
10 classes

Table 6: Average accuracy (%) comparison between Cross-Entropy (CE) and TAL under different batch sizes and class numbers on CIFAR and ImageNet. Classes are randomly chosen with 10 different random seeds .

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 160 papers

May 5, 2026

Dynamic Distillation and Gradient Consistency for Robust Long-Tailed Incremental Learning

The task of Long-tailed Class Incremental Learning (LT-CIL) addresses the sequential learning of new classes from datasets with imbalanced class distributions. This scenario intensifies the fundamental problem of catastrophic forgetting, inherent to continual learning, with the dual challenges of under-learning minority classes and overfitting majority classes. To tackle these combined issues, this paper proposes two main techniques. First, we introduce gradient consistency regularization, which leverages the moving average of gradients to suppress abrupt fluctuations and stabilize the training process. Second, we dynamically adjust the weight of the distillation loss by measuring the degree of class imbalance with normalized entropy. This adaptive weighting establishes an optimal balance between retaining old knowledge and acquiring new information. Experiments on the CIFAR-100-LT, ImageNetSubset-LT, and Food101-LT benchmarks show that our method achieves consistent accuracy improvements of up to 5.0%. Furthermore, we demonstrate dramatic gains in the challenging 'In-ordered' setting, where tasks progress from majority to minority classes, highlighting our method's robustness in mitigating forgetting under unfavorable learning dynamics. This enhanced performance is achieved without a significant increase in computational overhead, demonstrating the practicality of our framework.

Taigo Sakai, Kazuhiro Hotta

Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya 468-8502, Japan

Class-Incremental Learning Long-Tail Learning Adaptive Loss Weighting Incremental Learning Knowledge Distillation Consistency Regularization

Ask PDF

Apr 3, 2026

Free-Flow Class-Incremental Learning: Towards Robust CIL under Variable Class Arrivals

Class-incremental learning (CIL) is commonly evaluated under predefined schedules with fixed or nearly equal class increments, leaving irregular class-arrival scenarios underexplored. However, practical CIL systems may need to update whenever new categories emerge, without forcing them into balanced task partitions. We formalize this setting as Free-Flow Class-Incremental Learning (FFCIL), where the number of newly arriving classes can vary substantially across learning stages. We show that variable class arrivals alter the class composition of incremental training, the reliability of new-class classifier statistics, and the consistency of representations learned across increments, causing clear performance degradation in both conventional and pre-trained model (PTM)-based CIL methods. To improve robustness under FFCIL, we introduce a general framework consisting of Class-Wise Mean (CWM), which replaces instance-wise loss aggregation with class-wise averaging; Dynamic Intervention Weight Alignment (DIWA), which adjusts new-class weight calibration according to the current increment size; and Head-Agnostic Alignment (HA), which performs feature-level correction for PTM-based methods using current and previous-class feature supervision. Extensive experiments across diverse methods, datasets, and class-arrival schedules demonstrate the general impact of FFCIL and the consistent effectiveness of our framework.

Zhiming Xu, Baile Xu, Jian Zhao +2

National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · School of Electronic Science and Engineering, Nanjing University, China +1

Class-Incremental Learning

Ask PDF

May 14, 2026

Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learning

Neural networks suffer from catastrophic forgetting in class-incremental learning (CIL) settings. Rehearsal

\unicode{x2013}

replaying a subset of past samples

\unicode{x2013}

is a well-established mitigation strategy. However, recent results suggest that, despite balanced rehearsal allocation, some classes are forgotten substantially more than others. Despite its relevance, this imbalanced forgetting phenomenon remains underexplored. This work shows that imbalanced forgetting arises systematically and severely in rehearsal-based CIL and investigates it extensively. Specifically, we construct, from a principled analysis, three last-layer coefficients that capture different gradient-level sources of interference affecting each past class during an incremental step. We then demonstrate that, together, they reliably predict how past classes will rank in terms of forgetting at the end of that step. While predictive performance alone does not establish causality, these results support the interpretation of the coefficients as a plausible mechanistic account linking last-layer gradient-level interactions during training to class-level forgetting outcomes. Notably, one coefficient

\unicode{x2013}

capturing self-induced interference

\unicode{x2013}

emerges as the strongest predictor, with controlled experiments providing evidence consistent with this coefficient being influenced by the new-class interference coefficient. Overall, our findings provide valuable insights and suggest promising directions for mitigating imbalanced forgetting by reducing class-wise disparities in the identified sources of interference.

Alberto Tamajo, Srinandan Dasmahapatra, Rahman Attar

School of Electronics and Computer Science, University of Southampton, SO17 1BJ Southampton, United Kingdom

Class-Incremental Learning Gradient Interference Replay-Based Continual Learning Incremental Learning Catastrophic Forgetting

Ask PDF

Abstract

Class-Incremental Learning

Explore similar work

Dynamic Distillation and Gradient Consistency for Robust Long-Tailed Incremental Learning

Free-Flow Class-Incremental Learning: Towards Robust CIL under Variable Class Arrivals

Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learning