Universal Incremental Learning: Mitigating Confusion from Inter- and Intra-task Distribution Randomness

Yi Zhou; Jinsong Shi; Sheng Luo; Tao Zhou

cs.CVMar 10, 2025

Universal Incremental Learning: Mitigating Confusion from Inter- and Intra-task Distribution Randomness

Authors: Yi Zhou, Jinsong Shi, Sheng Luo, Tao Zhou

Organizations: School of Computer Science and Engineering, Southeast University, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Nanjing University of Science and Technology, Nanjing, China

Abstract

Existing incremental learning (IL) methods rely on strong assumptions: incoming tasks either introduce exclusively new classes or new domains (i.e., Class IL and Domain IL), or increment by a fixed scale in a class- and domain-agnostic manner (i.e., Versatile IL, VIL). These assumptions severely limit their applicability in dynamic, unpredictable real-world scenarios. In this work, we investigate Universal Incremental Learning (UIL), a more realistic setting where the model has no prior knowledge of which new classes or domains will appear across sequential tasks, nor the scale of these increments within each task. This uncertainty hinders the model's ability to confidently acquire knowledge across varying task distributions and to learn uniformly from imbalanced intra-task distributions. Consequently, UIL introduces a more general IL scenario, where the model must cope with confusion stemming from both inter-task and intra-task distribution randomness. To mitigate both sources of confusion, we propose MiCo, a simple yet effective framework for UIL. At the inter-task level, we employ a multi-objective learning scheme to enforce accurate and deterministic predictions, further enhanced by a direction recalibration module that alleviates conflicting gradient updates. At the intra-task level, we introduce a magnitude recalibration module to address asymmetrical optimization caused by imbalanced class distributions. Extensive experiments on four benchmarks demonstrate that MiCo significantly outperforms existing state-of-the-art methods in both the proposed UIL and existing VIL scenarios.

Figures & tables

Figure 1: Comparison of existing IL scenarios and our proposed UIL scenario on a 6-class dataset with 4 domains. The horizontal and vertical axes denote class and domain indices, respectively. Each incremental task is outlined in black. CIL, DIL, and VIL assume that each incoming task introduces only class increments, domain increments, or both class and domain increments, respectively. In contrast, UIL represents a more general IL scenario by simultaneously introducing randomness in both the type and scale of increments, where the model has no prior knowledge of which new classes or domains will emerge across sequential tasks, nor of the number of new classes or domains within each task.

Setting	Increment Characteristics (Type & Scale)	Task Boundary	Core Challenge
Traditional IL (CIL/DIL)	Fixed type, Fixed scale	Explicit	Catastrophic Forgetting (CF)
VIL	Random type, Fixed scale	Explicit	CF + Type Randomness
GCL / Task-free CL	Random type, Continuous stream	No / Blurry	CF + Blurry Boundaries
LT-CIL	Class only, Imbalanced scale	Explicit	CF + Class Imbalance
UIL (Ours)	Random type, Random scale	Explicit	CF + Inter/Intra-task Confusion

Table 1: Comparison of UIL with representative Incremental Learning settings.

Figure 2: Analysis of prediction distribution entropy (PDE) in the proposed UIL scenario and its correlation with test loss/accuracy. Left: Comparison of PDE distributions between UIL and VIL. UIL exhibits higher mean and variance (UIL: Mean=2.89, Var=0.71; VIL: Mean=2.21, Var=0.45), indicating greater prediction confusion. Right: Samples grouped by entropy intervals during inference. Lower-entropy samples consistently yield lower test loss and higher accuracy, validating entropy minimization as an effective strategy for improving model reliability.

Figure 3: Correlation between normalized gradient magnitude and average accuracy across class groups G1–G7 (grouped by emergence order). Blue bars denote normalized gradient magnitude; the red dashed line denotes average accuracy (%). The clear positive trend indicates that classes receiving larger gradient updates achieve higher accuracy, revealing an optimization bias toward head classes under imbalanced distributions.

Figure 4: Overview of the proposed UIL scenario (left) and the MiCo framework (right). During training on the current task $\mathcal{D}_{t}$ , MiCo employs a multi-objective learning scheme ( $\mathcal{L}_{ce}+\gamma\mathcal{L}_{em}$ ) to enforce accurate and deterministic predictions. A direction recalibration module mitigates gradient conflicts between $\vec{g}_{ce}$ and $\vec{g}_{em}$ , while a magnitude recalibration module rescales gradient norms (e.g., $\lVert\boldsymbol{g}_{ce}\rVert$ , $\lVert\boldsymbol{g}_{em}\rVert$ ) to alleviate asymmetrical optimization induced by imbalanced class distributions. The recalibrated gradient $\hat{\boldsymbol{g}}_{f}$ , combining updated direction $\vec{g}_{f}$ and magnitude $\lVert\hat{\boldsymbol{g}}_{f}\rVert$ , is used for parameter updates. During inference, the model is evaluated on all accumulated tasks $\mathcal{D}_{\text{test}}=\{\mathcal{D}_{1},\dots,\mathcal{D}_{t}\}$ . Note: A cumulative known-class binary mask is applied to logits. During training, classes outside the current task are set to $-\infty$ ; during inference, the mask contains all classes learned so far and does not require oracle task identity.

Figure 5: Internal mechanisms of the MiCo modules. (a) DIR resolves gradient conflicts between $\mathcal{L}_{ce}$ and $\mathcal{L}_{em}$ via CAGrad-based projection to obtain the recalibrated direction $\vec{g}_{f}$ . (b) MAG rebalances asymmetric class-wise gradient magnitudes through min-max inverse scaling to amplify tail-class signals.

Method	Versatile-IL (VIL)			Universal-IL (UIL)			Mean Acc ( $\uparrow$ )
	AVA( $\uparrow$ )	WA( $\uparrow$ )	F( $\downarrow$ )	AVA( $\uparrow$ )	WA( $\uparrow$ )	F( $\downarrow$ )
			iDigits
Fine-tuning	36.3 $\pm$ 0.1	32.6 $\pm$ 1.0	21.5 $\pm$ 5.5	46.5 $\pm$ 0.5	50.0 $\pm$ 2.4	18.4 $\pm$ 13.0	41.4 $\pm$ 0.3
EWC [ 25 ]	37.4 $\pm$ 2.1	34.4 $\pm$ 2.3	28.0 $\pm$ 3.5	61.4 $\pm$ 2.5	56.5 $\pm$ 2.8	20.0 $\pm$ 4.2	48.5 $\pm$ 2.3
LwF [ 28 ]	38.2 $\pm$ 2.0	35.1 $\pm$ 2.2	26.0 $\pm$ 3.2	63.9 $\pm$ 2.4	58.8 $\pm$ 2.6	19.0 $\pm$ 3.8	51.4 $\pm$ 2.2
L2P [ 51 ]	57.2 $\pm$ 3.0	52.1 $\pm$ 3.7	32.1 $\pm$ 4.1	74.3 $\pm$ 1.3	74.1 $\pm$ 3.0	18.0 $\pm$ 9.1	65.8 $\pm$ 2.2

Table 3: Comparison with state-of-the-art IL methods on the UIL and VIL scenarios. Results (%) are reported in terms of AVA, WA, and Forgetting (

F

) as mean

\pm

standard deviation across three random seeds. The AVA across both scenarios is averaged as Mean Acc (last column). Best and second-best results are marked in bold and underline , respectively.

Method	Versatile-IL (VIL)			Universal-IL (UIL)			Mean Acc ( $\uparrow$ )
	AVA( $\uparrow$ )	WA( $\uparrow$ )	F( $\downarrow$ )	AVA( $\uparrow$ )	WA( $\uparrow$ )	F( $\downarrow$ )
Fine-tuning	38.2 $\pm$ 1.2	36.5 $\pm$ 1.5	22.3 $\pm$ 4.2	42.1 $\pm$ 1.8	44.2 $\pm$ 2.1	19.5 $\pm$ 8.3	40.2 $\pm$ 1.5
EWC [ 25 ]	40.5 $\pm$ 2.1	38.2 $\pm$ 2.3	26.1 $\pm$ 3.8	44.5 $\pm$ 2.5	43.1 $\pm$ 2.8	21.2 $\pm$ 5.1	42.5 $\pm$ 2.3
LwF [ 28 ]	41.2 $\pm$ 2.0	39.1 $\pm$ 2.2	24.5 $\pm$ 3.5	46.4 $\pm$ 2.4	44.5 $\pm$ 2.6	20.1 $\pm$ 4.8	43.8 $\pm$ 2.2
L2P [ 51 ]	43.8 $\pm$ 3.1	41.5 $\pm$ 3.4	27.2 $\pm$ 4.2	44.8 $\pm$ 2.3	44.1 $\pm$ 2.6	19.5 $\pm$ 6.5	44.3 $\pm$ 2.7
DGR [ 16 ]	42.1 $\pm$ 2.6	40.8 $\pm$ 2.4	28.5 $\pm$ 3.9	44.3 $\pm$ 3.0	43.5 $\pm$ 3.3	22.1 $\pm$ 4.5	43.2 $\pm$ 2.8

Table 4: Comparison with state-of-the-art IL methods on ImageNet-R dataset under UIL and VIL scenarios. Results (%) are reported as mean

\pm

standard deviation across three random seeds. Best and second-best results are marked in bold and underline , respectively.

Method			iDigits			CORe50			DomainNet
MOBJ	DIR	MAG	AVA( $\uparrow$ )	WA( $\uparrow$ )	F( $\downarrow$ )	AVA( $\uparrow$ )	WA( $\uparrow$ )	F( $\downarrow$ )	AVA( $\uparrow$ )	WA( $\uparrow$ )	F( $\downarrow$ )
			78.3 $\pm$ 3.2	79.5 $\pm$ 6.1	14.3 $\pm$ 7.9	64.9 $\pm$ 5.6	64.7 $\pm$ 6.9	20.5 $\pm$ 4.3	46.9 $\pm$ 3.8	55.2 $\pm$ 1.4	11.6 $\pm$ 2.6
✓			80.4 $\pm$ 4.6	81.0 $\pm$ 5.9	13.5 $\pm$ 9.5	65.0 $\pm$ 3.2	65.5 $\pm$ 4.4	19.7 $\pm$ 2.3	49.8 $\pm$ 3.0	55.7 $\pm$ 1.4	16.6 $\pm$ 1.4
		✓	81.3 $\pm$ 4.4	82.4 $\pm$ 6.3	13.0 $\pm$ 9.1	65.9 $\pm$ 3.8	65.9 $\pm$ 4.6	18.3 $\pm$ 4.2	50.2 $\pm$ 3.1	54.5 $\pm$ 1.4	14.9 $\pm$ 1.9
✓		✓	84.0 $\pm$ 3.0	83.9 $\pm$ 4.7	12.4 $\pm$ 6.9	67.1 $\pm$ 2.5	66.9 $\pm$ 2.1	17.1 $\pm$ 2.8	53.1 $\pm$ 2.6	55.9 $\pm$ 0.3	14.5 $\pm$ 0.7
✓	✓		83.1 $\pm$ 3.9	82.5 $\pm$ 6.0	13.4 $\pm$ 7.4	66.4 $\pm$ 1.2	67.3 $\pm$ 2.3	19.0 $\pm$ 2.3	51.2 $\pm$ 3.1	55.2 $\pm$ 0.3	15.8 $\pm$ 5.1

Table 5: Ablation experiments of each component in UIL. MOBJ: multi-objective learning scheme; DIR: direction recalibration module; MAG: magnitude recalibration module. We progressively integrate each component from baseline to full MiCo. DIR operates in conjunction with MOBJ to reduce conflicting gradients. Results (%) are reported as mean

\pm

standard deviation across three random seeds.

Figure 6: Effect of the direction recalibration (DIR) module on the final task of iDigits (5-task split). DIR consistently improves Acc@1 and reduces forgetting, confirming its role in stabilizing multi-objective optimization.

Method	Head Acc	Mid Acc	Tail Acc	Overall Acc
Baseline	46.0 $\pm$ 1.6	40.0 $\pm$ 1.8	31.0 $\pm$ 2.0	42.1 $\pm$ 1.8
MAG	47.2 $\pm$ 1.5	42.4 $\pm$ 1.6	39.3 $\pm$ 1.9	44.9 $\pm$ 1.9
MiCo	49.6 $\pm$ 1.2	45.1 $\pm$ 1.3	41.8 $\pm$ 1.6	49.8 $\pm$ 1.0

Table 6: Head/mid/tail analysis on ImageNet-R under the UIL setting. Classes are grouped by frequency into head (top 40%), mid (middle 40%), and tail (bottom 20%) groups. MAG consistently improves all groups, with the most significant gain on tail classes. Overall accuracy (%) is computed on all test samples, not as the unweighted average of group accuracies.

Figure 7: Class-frequency distribution of ImageNet-R under the UIL split. Classes are sorted in descending order by training sample count, revealing a pronounced long-tail distribution.

Method	Train Loss $\downarrow$	Test Loss $\downarrow$	AVA $\uparrow$	WA $\uparrow$	F $\downarrow$
CE	0.137 $\pm$ 0.076	1.687 $\pm$ 0.089	81.1 $\pm$ 8.6	82.5 $\pm$ 6.5	17.0 $\pm$ 11.4
Focal	0.007 $\pm$ 0.006	1.987 $\pm$ 0.041	76.0 $\pm$ 7.6	79.3 $\pm$ 4.0	17.4 $\pm$ 9.1
LDAM	2.067 $\pm$ 1.018	2.007 $\pm$ 0.048	77.5 $\pm$ 10.2	79.8 $\pm$ 6.9	20.9 $\pm$ 12.7
MiCo	0.160 $\pm$ 0.088	1.654 $\pm$ 0.054	84.6 $\pm$ 4.1	83.9 $\pm$ 6.2	11.5 $\pm$ 4.3

Table 7: Comparison with imbalance-aware losses on iDigits under the UIL setting. Results (%) are reported as mean

\pm

standard deviation over three random seeds. Lower is better for Train Loss, Test Loss, and Forgetting; higher is better for AVA and WA. MiCo achieves the lowest test loss and best overall performance, demonstrating superior generalization and knowledge retention.

Method	AVA ( $\uparrow$ )	WA ( $\uparrow$ )	Forgetting ( $\downarrow$ )	$\Delta$ vs. Fine-tuning
Vanilla Fine-tuning	32.4 $\pm$ 1.8	34.1 $\pm$ 2.1	28.6 $\pm$ 3.5	–
ICON	39.7 $\pm$ 1.5	41.2 $\pm$ 1.8	22.4 $\pm$ 2.8	+7.3
PSR	38.5 $\pm$ 1.6	39.8 $\pm$ 1.9	24.2 $\pm$ 3.1	+6.1
CL-LoRA	39.4 $\pm$ 1.4	40.7 $\pm$ 1.7	23.5 $\pm$ 2.8	+7.0
DUCT	37.8 $\pm$ 1.7	39.2 $\pm$ 2.0	25.1 $\pm$ 3.2	+5.4
MISA	40.2 $\pm$ 1.3	41.8 $\pm$ 1.6	21.9 $\pm$ 2.5	+7.8

Table 8: Performance comparison under the Strictly Disjoint UIL setting on DomainNet.

$\gamma$	iDigits			CORe50			DomainNet
	AVA( $\uparrow$ )	WA( $\uparrow$ )	F( $\downarrow$ )	AVA( $\uparrow$ )	WA( $\uparrow$ )	F( $\downarrow$ )	AVA( $\uparrow$ )	WA( $\uparrow$ )	F( $\downarrow$ )
1.0	85.1 $\pm$ 4.3	84.0 $\pm$ 5.3	12.7 $\pm$ 7.9	66.3 $\pm$ 3.4	66.6 $\pm$ 2.8	17.2 $\pm$ 2.1	47.3 $\pm$ 3.9	54.4 $\pm$ 0.5	11.1 $\pm$ 1.7
0.1	80.5 $\pm$ 7.9	80.8 $\pm$ 8.4	15.8 $\pm$ 10.2	66.5 $\pm$ 4.5	66.4 $\pm$ 5.6	17.8 $\pm$ 5.2	52.0 $\pm$ 1.2	55.5 $\pm$ 0.7	15.3 $\pm$ 4.9
0.01	84.6 $\pm$ 4.1	83.9 $\pm$ 6.2	11.5 $\pm$ 4.3	67.4 $\pm$ 2.3	66.4 $\pm$ 2.0	16.4 $\pm$ 3.2	55.2 $\pm$ 2.3	56.1 $\pm$ 0.1	15.0 $\pm$ 2.8
0.001	83.2 $\pm$ 2.1	82.1 $\pm$ 5.8	13.6 $\pm$ 7.6	66.5 $\pm$ 2.1	66.2 $\pm$ 0.3	22.0 $\pm$ 0.6	52.0 $\pm$ 0.1	54.9 $\pm$ 0.1	19.1 $\pm$ 1.3

Table 9: Ablation of the balancing coefficient

\gamma

in the multi-objective learning scheme. We evaluate

\gamma\in\{1.0,0.1,0.01,0.001\}

across three benchmarks.

\gamma=0.01

achieves the best trade-off between accuracy and forgetting, and is adopted as the default setting. Results (%) are reported as mean

\pm

standard deviation across three random seeds.

Figure 8: Sensitivity analysis of $\beta$ and $\gamma$ on iDigits under UIL. Black dashed boxes highlight flat high-performance regions, while the blue box marks the grid-search optimum.

Method	Train GFLOPs/iter $\downarrow$	Iter Time (s) $\downarrow$	Epoch Time (s) $\downarrow$	Peak Memory (MB) $\downarrow$	Total Time (s) $\downarrow$
Baseline	1265.9	0.248 $\pm$ 0.039	1308.8 $\pm$ 269.0	2239.7 $\pm$ 0.6	42867.7 $\pm$ 7237.5
ICON	1785.2	0.198 $\pm$ 0.017	1047.5 $\pm$ 183.1	3996.0 $\pm$ 0.0	36046.7 $\pm$ 2422.8
MiCo	1420.5	0.181 $\pm$ 0.030	949.6 $\pm$ 183.0	2268.0 $\pm$ 0.0	30101.7 $\pm$ 4674.2

Table 10: Efficiency comparison on iDigits under the UIL setting with five tasks. Results are reported as mean

\pm

standard deviation over three random seeds. Iter Time and Epoch Time denote the average training time per iteration and per epoch, respectively. Lower is better for all metrics. MiCo achieves the best efficiency across FLOPs, time, and memory while maintaining superior accuracy.

Scale (Classes/Task)	Method	AVA ( $\uparrow$ )	F ( $\downarrow$ )	$\Delta$ vs. ICON
8 ( $T=40$ )	ICON	52.3 $\pm$ 1.5	15.2 $\pm$ 2.6	–
	MiCo w/o DIR	53.1 $\pm$ 1.6	14.5 $\pm$ 2.4	+0.8
	MiCo w/o MAG	53.8 $\pm$ 1.4	13.9 $\pm$ 2.5	+1.5
	MiCo	55.2 $\pm$ 1.2	12.8 $\pm$ 2.1	+2.9
11 ( $T=30$ )	ICON	48.6 $\pm$ 1.8	19.4 $\pm$ 3.1	–
	MiCo w/o DIR	49.5 $\pm$ 1.7	18.2 $\pm$ 2.9	+0.9

Table 11: Scalability analysis on DomainNet under varying increment scales. Smaller

T

dictates a larger number of classes/domains per task, intensifying the distribution randomness.

Incremental Learning

Incremental learning studies how models update from sequential data while retaining earlier knowledge as classes, tasks, or domains change.

See all 146 papers

Free-Flow Class-Incremental Learning: Towards Robust CIL under Variable Class Arrivals
Apr 3, 2026Zhiming Xu, Baile Xu, Jian Zhao +2Class-Incremental Learning
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · School of Electronic Science and Engineering, Nanjing University, China +1
Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning
Jul 20, 2026Kai Jiang, Zisong Lin, Hongyuan Zhang +2Class-Incremental Learning Metric Learning
National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China · The University of Hong Kong, Hong Kong SAR, China · Institute of Artificial Intelligence (TeleAI) of China Telecom, China
Dynamic Distillation and Gradient Consistency for Robust Long-Tailed Incremental Learning
May 5, 2026Taigo Sakai, Kazuhiro HottaClass-Incremental Learning Long-Tail Learning
Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya 468-8502, Japan

Apr 3, 2026J/K move · Enter open · S save

Free-Flow Class-Incremental Learning: Towards Robust CIL under Variable Class Arrivals

Zhiming Xu, Baile Xu, Jian Zhao, Furao Shen+1

National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · School of Electronic Science and Engineering, Nanjing University, China+1

Class-Incremental Learning

Open paper AskPDF arXiv

Class-incremental learning (CIL) is commonly evaluated under predefined schedules with fixed or nearly equal class increments, leaving irregular class-arrival scenarios underexplored. However, practical CIL systems may need to update whenever new categories emerge, without forcing them into balanced task partitions. We formalize this setting as Free-Flow Class-Incremental Learning (FFCIL), where the number of newly arriving classes can vary substantially across learning stages. We show that variable class arrivals alter the class composition of incremental training, the reliability of new-class classifier statistics, and the consistency of representations learned across increments, causing clear performance degradation in both conventional and pre-trained model (PTM)-based CIL methods. To improve robustness under FFCIL, we introduce a general framework consisting of Class-Wise Mean (CWM), which replaces instance-wise loss aggregation with class-wise averaging; Dynamic Intervention Weight Alignment (DIWA), which adjusts new-class weight calibration according to the current increment size; and Head-Agnostic Alignment (HA), which performs feature-level correction for PTM-based methods using current and previous-class feature supervision. Extensive experiments across diverse methods, datasets, and class-arrival schedules demonstrate the general impact of FFCIL and the consistent effectiveness of our framework.

Abstract

Incremental Learning

Explore similar work

Free-Flow Class-Incremental Learning: Towards Robust CIL under Variable Class Arrivals

Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning

Dynamic Distillation and Gradient Consistency for Robust Long-Tailed Incremental Learning