HAD: Heterogeneity-Aware Distillation for Lifelong Heterogeneous Learning

Xuerui Zhang; Xuehao Wang; Zhan Zhuang; Linglan Zhao; Ziyue Li; Xinmin Zhang; Zhihuan Song; Yu Zhang

cs.CVMar 27, 2026

HAD: Heterogeneity-Aware Distillation for Lifelong Heterogeneous Learning

Authors: Xuerui Zhang, Xuehao Wang, Zhan Zhuang, Linglan Zhao, Ziyue Li, Xinmin Zhang, Zhihuan Song, Yu Zhang

Organizations: Southern University of Science and Technology · Technical University of Munich · Zhejiang University · City University of Hong Kong ·

Abstract

Lifelong learning aims to preserve knowledge acquired from previous tasks while incorporating knowledge from a sequence of new tasks. However, most prior work explores only streams of homogeneous tasks (e.g., only classification tasks) and neglects the scenario of learning across heterogeneous tasks that possess different structures of outputs. In this work, we formalize this broader setting as lifelong heterogeneous learning (LHL). Departing from conventional lifelong learning, the task sequence of LHL spans different task types, and the learner needs to retain heterogeneous knowledge for different output space structures. To instantiate the LHL, we focus on LHL in the context of dense prediction (LHL4DP), a realistic and challenging scenario. To this end, we propose the Heterogeneity-Aware Distillation (HAD) method, an exemplar-free approach that preserves previously gained heterogeneous knowledge by self-distillation in each training phase. The proposed HAD comprises two complementary components, including a distribution-balanced heterogeneity-aware distillation loss to alleviate the global imbalance of prediction distribution and a salience-guided heterogeneity-aware distillation loss that concentrates learning on informative edge pixels extracted with the Sobel operator. Extensive experiments demonstrate that the proposed HAD method significantly outperforms existing methods in this new scenario.

Figures & tables

Figure 1: Vanilla training under LHL4DP. To assess the impact of catastrophic forgetting, we shuffle the learning sequences of three DP tasks. Each figure illustrates how the performance of a given task varies as the training phase proceeds, where the number in the horizontal axis denotes the task index in each sequence of three DP tasks. The performance metric is indicated above each column. The symbol

\uparrow

(

\downarrow

) signifies that a higher (lower) value denotes better performance.

Figure 2: The training pipeline of the proposed HAD method in the $t$ -th training phase. The HAD method uses the distribution-balanced and salience-guided distillation loss to mitigate forgetting of previous tasks $\mathcal{T}_{j}\ (j<t)$ , all of which are calculated on the pseudo-labels generated by the frozen teacher model $\mathcal{F}^{t-1}_{j}$ . Adapting to the new task $\mathcal{T}_{t}$ is achieved by the task-specific loss function $\mathcal{L}_{\mathrm{new}}$ .

Panel of Figure 3 — Figure 3: An illustration of the distribution imbalance in pseudo-labels. The number of pixels in the semantic segmentation task is counted per class. In the depth estimation task, we divide the range of pseudo-labels given by the teacher model into ten equal intervals, each of which is a group, and then the ten groups are sorted based on the number of pixels in each group.

	Method	Segmentation		Depth		Surface Normal					$\Delta_{v}^{m}\uparrow$	MR $\downarrow$
		mIoU ${\uparrow}$	Pix Acc $\uparrow$	Abs Err $\downarrow$	Rel Err $\downarrow$	Angle Distance		Within $t^{\circ}$
		mIoU ${\uparrow}$	Pix Acc $\uparrow$	Abs Err $\downarrow$	Rel Err $\downarrow$	Mean $\downarrow$	Median $\downarrow$	11.25 $\uparrow$	22.5 $\uparrow$	30 $\uparrow$
ResNet-18	Vanilla training	17.49	46.81	0.9609	0.3328	32.45	26.92	20.72	42.56	54.73	$+0.00\%$	5.67
	Joint training	41.84	66.14	0.5793	0.2201	31.53	25.78	22.38	44.54	56.36	$+40.83\%$	1.00
	EWC	32.17	57.21	0.9586	0.3493	37.52	33.08	13.47	33.37	45.35	$+24.39\%$	6.33
	iCaRL	21.78	53.00	1.3093	0.4561	33.07	27.73	19.45	41.20	53.43	$-4.82\%$	6.67

Table 1: Performance on 3 tasks ( i.e. , 13-class semantic segmentation, depth estimation, and surface normal prediction) after the last training phase of the NYUv2 dataset across different encoders. The best results are shown in bold .

\uparrow

(

\downarrow

) means that the higher (lower) the value, the better the performance.

Method	Seg. $\downarrow$	Dep. $\downarrow$	Normal $\downarrow$	E.-2D $\downarrow$	Res. $\downarrow$	K.-2D $\downarrow$	E.-3D $\downarrow$	E. D. $\downarrow$	Curv. $\downarrow$	K.-3D $\downarrow$	$\Delta_{v}^{m}\uparrow$	MR $\downarrow$
Vanilla	0.7282	0.2673	0.2869	0.1627	0.4658	0.5143	0.5214	0.1968	1.8655	0.3863	$+0.00\%$	6.20
Joint	0.1615	0.1071	0.1281	0.1434	0.1487	0.2969	0.3244	0.1054	1.3501	0.3149	$+44.57\%$	1.00
EWC	0.4701	0.2358	0.1945	0.1692	0.3109	0.5327	0.5525	0.2286	1.9835	0.4347	$+6.41\%$	6.00
iCaRL	0.5704	0.2247	0.2214	0.1846	0.3386	0.5022	0.5126	0.1918	1.7441	0.3965	$+8.47\%$	4.20
LwF	0.6482	0.2628	0.2484	0.1774	0.4718	0.5298	0.5137	0.1879	1.9212	0.4208	$+0.68\%$	6.40
DER	0.5993	0.2872	0.2315	0.1518	0.4584	0.5068	0.4564	0.2674	1.8488	0.3762	$+1.94\%$	4.90

Table 2: Performance on 10 tasks: semantic segmentation (Seg.), depth estimation (Dep.), surface normal estimation (Normal), edge-2D detection (E.-2D), reshading (Res.), keypoint-2D detection (K.-2D), edge-3D detection (E.-3D), Euclidean distance (E. D.), curvatures (Curv.), and keypoint-3D detection (K.-3D) on the Taskonomy dataset. The best results are shown in bold .

\uparrow

(

\downarrow

) means that the higher (lower) the value, the better the performance.

Figure 4: The comparison between training separate models and the proposed HAD. Each figure illustrates the performance improvement of the HAD method in the LHL4DP scenario of a given task. The symbol $\uparrow$ ( $\downarrow$ ) signifies that a higher (lower) value denotes better performance.

Figure 5: Visualization of the raw data (left), the gradient magnitude map of its prediction (middle), and the gradient magnitude map of the loss map (right).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Comparison between Lifelong Learning and Lifelong Heterogeneous Learning for Dense Prediction (LHL4DP). (a) CIL incrementally recognizes all encountered classes, while (b) LHL4DP progressively addresses all encountered heterogeneous dense prediction tasks.

Dataset	Task Name	Task Type	Output Channel	Loss Function
NYUv2	Semantic Segmentation	Classification	13	Cross-Entropy Loss
	Depth Estimation	Regression	1	$L_{1}$ Loss
	Surface Normal Prediction	Regression	3	Cosine Distance Loss
CityScapes	Semantic Segmentation	Classification	7	Cross-Entropy Loss
CityScapes	Depth Estimation	Regression	1	$L_{1}$ Loss
Taskonomy	Semantic Segmentation	Classification	8	Cross-Entropy Loss

Appendix

Table 5: Task-specific forms of

\mathcal{L}_{\mathrm{new}}

for heterogeneous dense prediction tasks.

Figure 7: Results of different methods during each training phase on the Taskonomy dataset.

Method	NYUv2		CityScapes		Taskonomy
Method	Resnet-18	Resnet-50	Resnet-18	Resnet-50	Resnet-18
EWC	$10^{9}$	$10^{10}$	$10^{6}$	$10^{3}$	$10^{9}$
iCaRL	0.01	0.1	0.01	0.1	1
LWF	5	5	0.01	0.1	0.1
DER	0.01	0.1	1	1	0.1
SGP	0.1	10	10	1000	100

Appendix

Table 6: Hyperparameter of different methods.

	Method	Segmentation		Depth		Surface Normal					$\Delta_{b}^{T}\uparrow$	MR $\downarrow$
		mIoU ${\uparrow}$	Pix Acc $\uparrow$	Abs Err $\downarrow$	Rel Err $\downarrow$	Angle Distance		Within $t^{\circ}$
		mIoU ${\uparrow}$	Pix Acc $\uparrow$	Abs Err $\downarrow$	Rel Err $\downarrow$	Mean $\downarrow$	Median $\downarrow$	11.25 $\uparrow$	22.5 $\uparrow$	30 $\uparrow$
Sequence 2	Vanilla training	33.77	60.36	1.0261	0.3592	40.76	34.74	10.16	30.98	43.17	$+0.00\%$	6.00
	Joint training	41.84	66.14	0.5793	0.2201	31.53	25.78	22.38	44.54	56.36	$+3.09\%$	1.00
	EWC	29.78	56.75	0.9217	0.3218	39.15	33.31	10.48	31.98	44.95	$-0.77\%$	5.33
	iCaRL	23.87	53.78	1.5976	0.5474	35.87	33.19	11.60	31.55	44.63	$-27.11\%$	7.67

Appendix

Table 7: Performance on the NYUv2 dataset using Resnet-18 with a shuffled task sequence after the last training phase. The best results for each task are shown in bold .

\uparrow

(

\downarrow

) means that the higher (lower) the value, the better the performance.

Method	Segmentation $\mathrm{\rightarrow}$ Depth						Depth $\mathrm{\rightarrow}$ Segmentation
	Segmentation		Depth		$\Delta_{b}^{T}\uparrow$	MR $\downarrow$	Segmentation		Depth		$\Delta_{b}^{T}\uparrow$	MR $\downarrow$
	mIoU ${\uparrow}$	Pix Acc $\uparrow$	Abs Err $\downarrow$	Rel Err $\downarrow$	$\Delta_{b}^{T}\uparrow$	MR $\downarrow$	mIoU ${\uparrow}$	Pix Acc $\uparrow$	Abs Err $\downarrow$	Rel Err $\downarrow$	$\Delta_{b}^{T}\uparrow$	MR $\downarrow$
Vanilla training	58.40	86.84	0.0203	50.0861	$+0.00\%$	6.50	68.44	91.45	0.0456	77.8347	$+0.00\%$	5.50
Joint training	71.38	92.15	0.0164	43.7236	$+15.06\%$	1.00	71.38	92.15	0.0164	43.7236	$+28.23\%$	1.00
EWC	66.14	90.01	0.0203	56.8526	$+0.85\%$	6.50	65.56	90.06	0.0217	51.9560	$+19.98\%$	5.50
iCaRL	67.10	91.09	0.0204	50.2167	$+4.76\%$	5.50	57.24	87.82	0.0221	58.8481	$+13.90\%$	7.00

Appendix

Table 8: Performance on two tasks after the last training phase (i.e., 7-class semantic segmentation and depth estimation) of the CityScapes dataset using ResNet-18 under two different sequences.

Method	Segmentation		Depth		$\Delta_{b}^{T}\uparrow$	MR $\downarrow$
Method	mIoU ${\uparrow}$	Pix Acc $\uparrow$	Abs Err $\downarrow$	Rel Err $\downarrow$	$\Delta_{b}^{T}\uparrow$	MR $\downarrow$
Vanilla training	64.93	88.93	0.0168	40.0604	$+0.00\%$	5.50
Joint training	76.49	93.91	0.0155	45.7162	$+4.26\%$	3.00
EWC	69.44	90.60	0.0154	41.4725	$+3.41\%$	3.50
iCaRL	72.80	92.60	0.0166	47.3969	$-0.22\%$	7.00
LwF	76.07	93.66	0.0175	43.5094	$+2.42\%$	5.00

Appendix

Table 9: Performance on two tasks after the last training phase (i.e., 7-class semantic segmentation and depth estimation) of the CityScapes dataset using Resnet-50 . The best results for each task are shown in bold .

\uparrow

(

\downarrow

) means that the higher (lower) the value, the better the performance.

Method	Segmentation		Depth		Surface Normal					$\Delta_{v}^{m}\uparrow$	MR $\downarrow$
	mIoU ${\uparrow}$	Pix Acc $\uparrow$	Abs Err $\downarrow$	Rel Err $\downarrow$	Angle Distance		Within $t^{\circ}$
	mIoU ${\uparrow}$	Pix Acc $\uparrow$	Abs Err $\downarrow$	Rel Err $\downarrow$	Mean $\downarrow$	Median $\downarrow$	11.25 $\uparrow$	22.5 $\uparrow$	30 $\uparrow$
Vanilla training	17.49	46.81	0.9609	0.3328	32.45	26.92	20.72	42.56	54.73	$+0.00\%$	5.67
Joint training	41.84	66.14	0.5793	0.2201	31.53	25.78	22.38	44.54	56.36	$+40.83\%$	1.00
Local POD w. fro	32.25	58.11	0.8651	0.3063	35.43	30.40	16.15	37.23	49.36	$+25.51\%$	3.33
Local POD w. $L_{1}$	32.56	58.70	0.9240	0.3289	36.70	32.22	14.72	34.67	46.62	$+25.77\%$	5.00

Appendix

Table 10: Performance on 3 tasks ( i.e. , 13-class semantic segmentation, depth estimation, and surface normal prediction) after the last training phase of the NYUv2 dataset compared with feature distillation methods. The best results are shown in bold .

\uparrow

(

\downarrow

) means that the higher (lower) the value, the better the performance.

Panel of Table 11 — Table 11: $\Delta_{b}^{T}$ of different hyperparameter $k,\tau$ .

$\alpha$	1	2	3	4	5	6
$\Delta_{b}^{T}\uparrow$	-5.79	-2.71	0.00	-0.60	-1.10	-0.83

Appendix

Table 12:

\Delta_{b}^{T}

of different hyperparameter

\alpha

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 814 papers

HERD: Harness-Evolving Recursive Distillation for Continual Learning
Oct 22, 2026Xun Sun, Thanh Thi Nguyen, Thuy-Trang VuContinual Learning Continual Learning for LLM Agents
Monash University · University of the Sunshine Coast
Heterophily-Aware Adaptive Knowledge Distillation for Hypergraph Neural Networks
Jun 8, 2026Joohee Cho, David Yoon Suk Kang, Yunyong KoEfficient Neural Network Inference Hypergraph Neural Networks
Chung-Ang University · Seoul, South Korea · Chungbuk National University +1
Task-Differentiated Atomic Skill Expansion and Routing for Continual Learning Across Highly Heterogeneous Tasks
Jun 19, 2026Jiacheng Wang, Xinjia He, Qi Ding +5Dynamic Neural Networks Continual Learning
East China Normal University · Fudan University · Ocean University of China

Oct 22, 2026J/K move · Enter open · S save

HERD: Harness-Evolving Recursive Distillation for Continual Learning

Xun Sun, Thanh Thi Nguyen, Thuy-Trang Vu

Monash University · University of the Sunshine Coast

Continual Learning Continual Learning for LLM Agents Storage-Efficient Continual Learning Knowledge Distillation

Open paper AskPDF arXiv

Continual learning requires a model to keep acquiring new skills and knowledge without compromising its existing capabilities. Parametric methods suffer from catastrophic forgetting and loss of plasticity. Non-parametric methods, such as prompts, skills, and harnesses, leverage in-context learning but are limited by context length and lossy plain-text compression. To effectively combine their complementary strengths, we propose HERD (Harness-Evolving Recursive Distillation), a hybrid continual learning framework that learns efficiently in the harness and internalizes experience into cumulative capabilities via distillation. On the harness side, HERD continually refines its skills to capture reusable patterns, while retaining high-residual examples that the skills fail to capture in a bounded memory. On the parameter side, HERD performs drift-constrained distillation with dual anchors, substantially reducing forgetting over long recursive runs. On the 25-stage task stream of TRACE+17, under fixed memory and compute budgets, HERD matches the strongest parametric baseline in new-task acquisition while exhibiting the least forgetting and the strongest retention of general capabilities.

Abstract

Knowledge Distillation

Explore similar work

HERD: Harness-Evolving Recursive Distillation for Continual Learning

Heterophily-Aware Adaptive Knowledge Distillation for Hypergraph Neural Networks

Task-Differentiated Atomic Skill Expansion and Routing for Continual Learning Across Highly Heterogeneous Tasks