Organizations: Southern University of Science and Technology · Technical University of Munich · Zhejiang University · City University of Hong Kong · Shanghai Jiao Tong University
Lifelong learning aims to preserve knowledge acquired from previous tasks while incorporating knowledge from a sequence of new tasks. However, most prior work explores only streams of homogeneous tasks (e.g., only classification tasks) and neglects the scenario of learning across heterogeneous tasks that possess different structures of outputs. In this work, we formalize this broader setting as lifelong heterogeneous learning (LHL). Departing from conventional lifelong learning, the task sequence of LHL spans different task types, and the learner needs to retain heterogeneous knowledge for different output space structures. To instantiate the LHL, we focus on LHL in the context of dense prediction (LHL4DP), a realistic and challenging scenario. To this end, we propose the Heterogeneity-Aware Distillation (HAD) method, an exemplar-free approach that preserves previously gained heterogeneous knowledge by self-distillation in each training phase. The proposed HAD comprises two complementary components, including a distribution-balanced heterogeneity-aware distillation loss to alleviate the global imbalance of prediction distribution and a salience-guided heterogeneity-aware distillation loss that concentrates learning on informative edge pixels extracted with the Sobel operator. Extensive experiments demonstrate that the proposed HAD method significantly outperforms existing methods in this new scenario.
Figures & tables
Figure 1: Vanilla training under LHL4DP. To assess the impact of catastrophic forgetting, we shuffle the learning sequences of three DP tasks. Each figure illustrates how the performance of a given task varies as the training phase proceeds, where the number in the horizontal axis denotes the task index in each sequence of three DP tasks. The performance metric is indicated above each column. The symbol ↑ ( ↓ ) signifies that a higher (lower) value denotes better performance.
Figure 2: The training pipeline of the proposed HAD method in the t -th training phase. The HAD method uses the distribution-balanced and salience-guided distillation loss to mitigate forgetting of previous tasks Tj(j<t) , all of which are calculated on the pseudo-labels generated by the frozen teacher model Fjt−1 . Adapting to the new task Tt is achieved by the task-specific loss function Lnew .
Figure 3: An illustration of the distribution imbalance in pseudo-labels. The number of pixels in the semantic segmentation task is counted per class. In the depth estimation task, we divide the range of pseudo-labels given by the teacher model into ten equal intervals, each of which is a group, and then the ten groups are sorted based on the number of pixels in each group.
Method
Segmentation
Depth
Surface Normal
Δvm↑
MR ↓
mIoU ↑
Pix Acc ↑
Abs Err ↓
Rel Err ↓
Angle Distance
Within t∘
Mean ↓
Median ↓
11.25 ↑
22.5 ↑
30 ↑
ResNet-18
Vanilla training
17.49
46.81
0.9609
0.3328
32.45
26.92
20.72
42.56
54.73
+0.00%
5.67
Joint training
41.84
66.14
0.5793
0.2201
31.53
25.78
22.38
44.54
56.36
+40.83%
1.00
EWC
32.17
57.21
0.9586
0.3493
37.52
33.08
13.47
33.37
45.35
+24.39%
6.33
iCaRL
21.78
53.00
1.3093
0.4561
33.07
27.73
19.45
41.20
53.43
−4.82%
6.67
Table 1: Performance on 3 tasks ( i.e. , 13-class semantic segmentation, depth estimation, and surface normal prediction) after the last training phase of the NYUv2 dataset across different encoders. The best results are shown in bold . ↑ ( ↓ ) means that the higher (lower) the value, the better the performance.
Method
Seg. ↓
Dep. ↓
Normal ↓
E.-2D ↓
Res. ↓
K.-2D ↓
E.-3D ↓
E. D. ↓
Curv. ↓
K.-3D ↓
Δvm↑
MR ↓
Vanilla
0.7282
0.2673
0.2869
0.1627
0.4658
0.5143
0.5214
0.1968
1.8655
0.3863
+0.00%
6.20
Joint
0.1615
0.1071
0.1281
0.1434
0.1487
0.2969
0.3244
0.1054
1.3501
0.3149
+44.57%
1.00
EWC
0.4701
0.2358
0.1945
0.1692
0.3109
0.5327
0.5525
0.2286
1.9835
0.4347
+6.41%
6.00
iCaRL
0.5704
0.2247
0.2214
0.1846
0.3386
0.5022
0.5126
0.1918
1.7441
0.3965
+8.47%
4.20
LwF
0.6482
0.2628
0.2484
0.1774
0.4718
0.5298
0.5137
0.1879
1.9212
0.4208
+0.68%
6.40
DER
0.5993
0.2872
0.2315
0.1518
0.4584
0.5068
0.4564
0.2674
1.8488
0.3762
+1.94%
4.90
Table 2: Performance on 10 tasks: semantic segmentation (Seg.), depth estimation (Dep.), surface normal estimation (Normal), edge-2D detection (E.-2D), reshading (Res.), keypoint-2D detection (K.-2D), edge-3D detection (E.-3D), Euclidean distance (E. D.), curvatures (Curv.), and keypoint-3D detection (K.-3D) on the Taskonomy dataset. The best results are shown in bold . ↑ ( ↓ ) means that the higher (lower) the value, the better the performance.
Figure 4: The comparison between training separate models and the proposed HAD. Each figure illustrates the performance improvement of the HAD method in the LHL4DP scenario of a given task. The symbol ↑ ( ↓ ) signifies that a higher (lower) value denotes better performance.
Figure 5: Visualization of the raw data (left), the gradient magnitude map of its prediction (middle), and the gradient magnitude map of the loss map (right).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Comparison between Lifelong Learning and Lifelong Heterogeneous Learning for Dense Prediction (LHL4DP). (a) CIL incrementally recognizes all encountered classes, while (b) LHL4DP progressively addresses all encountered heterogeneous dense prediction tasks.
Dataset
Task Name
Task Type
Output Channel
Loss Function
NYUv2
Semantic Segmentation
Classification
13
Cross-Entropy Loss
Depth Estimation
Regression
1
L1 Loss
Surface Normal Prediction
Regression
3
Cosine Distance Loss
CityScapes
Semantic Segmentation
Classification
7
Cross-Entropy Loss
Depth Estimation
Regression
1
L1 Loss
Taskonomy
Semantic Segmentation
Classification
8
Cross-Entropy Loss
Appendix
Table 5: Task-specific forms of Lnew for heterogeneous dense prediction tasks.
Figure 7: Results of different methods during each training phase on the Taskonomy dataset.
Method
NYUv2
CityScapes
Taskonomy
Resnet-18
Resnet-50
Resnet-18
Resnet-50
Resnet-18
EWC
109
1010
106
103
109
iCaRL
0.01
0.1
0.01
0.1
1
LWF
5
5
0.01
0.1
0.1
DER
0.01
0.1
1
1
0.1
SGP
0.1
10
10
1000
100
Appendix
Table 6: Hyperparameter of different methods.
Method
Segmentation
Depth
Surface Normal
ΔbT↑
MR ↓
mIoU ↑
Pix Acc ↑
Abs Err ↓
Rel Err ↓
Angle Distance
Within t∘
Mean ↓
Median ↓
11.25 ↑
22.5 ↑
30 ↑
Sequence 2
Vanilla training
33.77
60.36
1.0261
0.3592
40.76
34.74
10.16
30.98
43.17
+0.00%
6.00
Joint training
41.84
66.14
0.5793
0.2201
31.53
25.78
22.38
44.54
56.36
+3.09%
1.00
EWC
29.78
56.75
0.9217
0.3218
39.15
33.31
10.48
31.98
44.95
−0.77%
5.33
iCaRL
23.87
53.78
1.5976
0.5474
35.87
33.19
11.60
31.55
44.63
−27.11%
7.67
Appendix
Table 7: Performance on the NYUv2 dataset using Resnet-18 with a shuffled task sequence after the last training phase. The best results for each task are shown in bold . ↑ ( ↓ ) means that the higher (lower) the value, the better the performance.
Method
Segmentation → Depth
Depth → Segmentation
Segmentation
Depth
ΔbT↑
MR ↓
Segmentation
Depth
ΔbT↑
MR ↓
mIoU ↑
Pix Acc ↑
Abs Err ↓
Rel Err ↓
mIoU ↑
Pix Acc ↑
Abs Err ↓
Rel Err ↓
Vanilla training
58.40
86.84
0.0203
50.0861
+0.00%
6.50
68.44
91.45
0.0456
77.8347
+0.00%
5.50
Joint training
71.38
92.15
0.0164
43.7236
+15.06%
1.00
71.38
92.15
0.0164
43.7236
+28.23%
1.00
EWC
66.14
90.01
0.0203
56.8526
+0.85%
6.50
65.56
90.06
0.0217
51.9560
+19.98%
5.50
iCaRL
67.10
91.09
0.0204
50.2167
+4.76%
5.50
57.24
87.82
0.0221
58.8481
+13.90%
7.00
Appendix
Table 8: Performance on two tasks after the last training phase (i.e., 7-class semantic segmentation and depth estimation) of the CityScapes dataset using ResNet-18 under two different sequences.
Method
Segmentation
Depth
ΔbT↑
MR ↓
mIoU ↑
Pix Acc ↑
Abs Err ↓
Rel Err ↓
Vanilla training
64.93
88.93
0.0168
40.0604
+0.00%
5.50
Joint training
76.49
93.91
0.0155
45.7162
+4.26%
3.00
EWC
69.44
90.60
0.0154
41.4725
+3.41%
3.50
iCaRL
72.80
92.60
0.0166
47.3969
−0.22%
7.00
LwF
76.07
93.66
0.0175
43.5094
+2.42%
5.00
Appendix
Table 9: Performance on two tasks after the last training phase (i.e., 7-class semantic segmentation and depth estimation) of the CityScapes dataset using Resnet-50 . The best results for each task are shown in bold . ↑ ( ↓ ) means that the higher (lower) the value, the better the performance.
Method
Segmentation
Depth
Surface Normal
Δvm↑
MR ↓
mIoU ↑
Pix Acc ↑
Abs Err ↓
Rel Err ↓
Angle Distance
Within t∘
Mean ↓
Median ↓
11.25 ↑
22.5 ↑
30 ↑
Vanilla training
17.49
46.81
0.9609
0.3328
32.45
26.92
20.72
42.56
54.73
+0.00%
5.67
Joint training
41.84
66.14
0.5793
0.2201
31.53
25.78
22.38
44.54
56.36
+40.83%
1.00
Local POD w. fro
32.25
58.11
0.8651
0.3063
35.43
30.40
16.15
37.23
49.36
+25.51%
3.33
Local POD w. L1
32.56
58.70
0.9240
0.3289
36.70
32.22
14.72
34.67
46.62
+25.77%
5.00
Appendix
Table 10: Performance on 3 tasks ( i.e. , 13-class semantic segmentation, depth estimation, and surface normal prediction) after the last training phase of the NYUv2 dataset compared with feature distillation methods. The best results are shown in bold . ↑ ( ↓ ) means that the higher (lower) the value, the better the performance.
Continual learning requires a model to keep acquiring new skills and knowledge without compromising its existing capabilities. Parametric methods suffer from catastrophic forgetting and loss of plasticity. Non-parametric methods, such as prompts, skills, and harnesses, leverage in-context learning but are limited by context length and lossy plain-text compression. To effectively combine their complementary strengths, we propose HERD (Harness-Evolving Recursive Distillation), a hybrid continual learning framework that learns efficiently in the harness and internalizes experience into cumulative capabilities via distillation. On the harness side, HERD continually refines its skills to capture reusable patterns, while retaining high-residual examples that the skills fail to capture in a bounded memory. On the parameter side, HERD performs drift-constrained distillation with dual anchors, substantially reducing forgetting over long recursive runs. On the 25-stage task stream of TRACE+17, under fixed memory and compute budgets, HERD matches the strongest parametric baseline in new-task acquisition while exhibiting the least forgetting and the strongest retention of general capabilities.
Xun Sun, Thanh Thi Nguyen, Thuy-Trang Vu
Monash University · University of the Sunshine Coast
Hypergraph knowledge distillation aims to retain the predictive performance of a hypergraph neural network (HNN) teacher while reducing inference costs through a lightweight student model. In this work, we observe that HNNs exhibit substantially lower prediction performance on heterophilic nodes connected through semantically diverse hyperedges, indicating that the reliability of teacher knowledge varies across nodes. Motivated by this observation, we propose HADES, a heterophily-aware adaptive distillation method for hypergraph neural networks. HADES quantifies node heterophily and leverages it as an estimate of teacher reliability to modulate the transfer of teacher knowledge during distillation. Experimental results on real-world hypergraphs demonstrate that HADES consistently improves student performance across different HNN teachers and distillation objectives. In many cases, the resulting student models surpass the predictive performance of their teachers while achieving up to 12.3 times faster inference.
Joohee Cho, David Yoon Suk Kang, Yunyong Ko
Chung-Ang University · Seoul, South Korea · Chungbuk National University +1
Continual learning (CL) is commonly studied under the assumption that sequential tasks are semantically related or structurally similar. However, in highly heterogeneous settings, where tasks differ substantially in reasoning patterns and input-output formats, existing methods often suffer from catastrophic forgetting and inefficient capacity allocation. To address this challenge, we propose Task-differentiated Atomic Skill Expansion and Routing (TASER), a CL framework that jointly determines how many new atomic skills to introduce for each task and which skills to activate. The framework first uses atomic skill incremental learning to dynamically expand capacity based on task divergence and model uncertainty. It then applies orthogonality-enhanced skill detection to ensure these skills remain semantically distinct and independently reusable. Finally, a skill dynamic routing mechanism composes task-relevant skills through lightweight task-conditioned gating. We further introduce HeteroCLBench, a highly heterogeneous benchmark for CL, comprising 19 diverse tasks across 9 cognitive dimensions under a standardized sequential protocol. Experiments on HeteroCLBench show that TASER consistently outperforms strong baselines by improving plasticity and reducing catastrophic forgetting.
Jiacheng Wang, Xinjia He, Qi Ding +5
East China Normal University · Fudan University · Ocean University of China