Collaborative Adaptive Curriculum for Progressive Knowledge Distillation

Jing Liu; Zhenchao Ma; Han Yu; Bobo Ju; Wenliang Yang; Chengfang Li; Bo Hu; Liang Song

cs.LGMar 19, 2026

Collaborative Adaptive Curriculum for Progressive Knowledge Distillation

Authors: Jing Liu, Zhenchao Ma, Han Yu, Bobo Ju, Wenliang Yang, Chengfang Li, Bo Hu, Liang Song

Organizations: Fudan University · The University of British Columbia · Duke Kunshan University · Shanghai Shentong Metro Co., Ltd. ·

Abstract

Recent advances in collaborative knowledge distillation have demonstrated cutting-edge performance for resource-constrained distributed multimedia learning scenarios. However, achieving such competitiveness requires addressing a fundamental mismatch: high-dimensional teacher knowledge complexity versus heterogeneous client learning capacities, which currently prohibits deployment in edge-based visual analytics systems. Drawing inspiration from curriculum learning principles, we introduce Federated Adaptive Progressive Distillation (FAPD), a consensus-driven framework that orchestrates adaptive knowledge transfer. FAPD hierarchically decomposes teacher features via PCA-based structuring, extracting principal components ordered by variance contribution to establish a natural visual knowledge hierarchy. Clients progressively receive knowledge of increasing complexity through dimension-adaptive projection matrices. Meanwhile, the server monitors network-wide learning stability by tracking global accuracy fluctuations across a temporal consensus window, advancing curriculum dimensionality only when collective consensus emerges. Consequently, FAPD provably adapts knowledge transfer pace while achieving superior convergence over fixed-complexity approaches. Extensive experiments on three datasets validate FAPD's effectiveness: it attains 3.64% accuracy improvement over FedAvg on CIFAR-10, demonstrates 2x faster convergence, and maintains robust performance under extreme data heterogeneity (α=0.1), outperforming baselines by over 4.5%.

Figures & tables

Fig. 1 : Illustration of FAPD featuring three components: (a) server-side hierarchical knowledge decomposition via PCA generating rotation matrix

R

, (b) consensus-driven adaptive curriculum controller monitoring accuracy history

\mathcal{A}

to determine dimension

k_{t}

, and (c) client-side progressive knowledge distillation projecting features through

P_{t}

to compute multi-objective loss

\mathcal{L}_{c}

Method	CIFAR-10	CIFAR-100	Tiny-ImageNet
FedAvg [ 21 ]	85.78	61.26	43.35
FedProx [ 35 ]	85.68	61.40	43.48
MOON [ 36 ]	86.10	61.48	43.78
FedNH [ 10 ]	86.25	61.43	44.12
FedRCL [ 11 ]	86.89	62.36	44.89
FedCDA [ 13 ]	87.11	61.95	44.56

TABLE I : Accuracy (%) of comparison across datasets (

\alpha

=0.5).

Fig. 2 : Result of local epoch count analysis on CIFAR-10/100.

ID	HKD	CDC	PKD	$\mathcal{L}_{CE}$	$\mathcal{L}_{KD}$	$\mathcal{L}_{CL}$	CIFAR-10	CIFAR-100
1	✗	✗	✗	✓	✗	✗	85.78	61.26
2	✗	✗	✗	✓	✗	✓	86.23	61.67
3	✓	✗	✗	✓	✓	✗	86.12	61.74
4	✓	✗	✗	✓	✓	✓	87.23	62.18
5	✓	✓	✓	✓	✓	✗	87.89	63.05
6	✓	✓	✓	✓	✓	✓	89.42	63.84

TABLE II : Ablation study on components and loss functions.

Fig. 3 : Result of client scalability analysis on CIFAR-10/100.

Dataset	Method	$\alpha=0.1$	$\alpha=0.2$	$\alpha=0.5$	$\alpha=1.0$
CIFAR-10	FedAvg [ 21 ]	81.35	84.12	85.78	87.12
	FAPD nadpt	83.12	85.78	87.23	88.45
	FAPD ncont	84.56	86.89	87.89	89.12
	FAPD (Ours)	85.87	88.34	89.42	90.18
CIFAR-100	FedAvg [ 21 ]	56.12	58.45	61.26	62.89
	FAPD nadpt	57.45	59.78	62.18	63.56

TABLE III : Accuracy (%) under varying

\alpha

level.

Fig. 4 : t-SNE visualization comparing feature distributions.

Federated Knowledge Distillation

Federated knowledge distillation transfers predictive behavior among data-isolated clients to train useful models without sharing raw data.

See all 36 papers

Adaptive Heterogeneous Compression for Resource-Efficient Federated Knowledge Distillation
Aug 16, 2026Chenwang Liu, Yijun Liu, Chang Liu +2Federated Knowledge Distillation Federated Learning
School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China · School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search
May 20, 2026Chaimaa Medjadji, Sylvain Kubler, Yves Le Traon +3Communication-Efficient Distributed Training Federated Knowledge Distillation
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Luxembourg · Blekinge Institute of Technology, Sweden · ADSLabs, Umea University, Sweden
Progressive $^2$ : A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression
Jul 31, 2026Tiancong Cheng, Ying Zhang, Zhiwen Yu +2Model Compression Knowledge Distillation
School of Computer Science, Northwestern Polytechnical University, Xi’an 710129, China

Aug 16, 2026J/K move · Enter open · S save

Adaptive Heterogeneous Compression for Resource-Efficient Federated Knowledge Distillation

Chenwang Liu, Yijun Liu, Chang Liu, Xu Zhang+1

School of Information Engineering, Guangdong University of Technology, Guangzhou 510006, China · School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

Federated Knowledge Distillation Federated Learning Communication-Efficient Distributed Training Heterogeneous Federated Learning

Open paper AskPDF arXiv

Federated learning (FL) enables privacy-preserving distributed model training but faces challenges from heterogeneous model architectures and limited communication resources at the network edge. Federated knowledge distillation (FedKD) alleviates model heterogeneity by combining prototype-wise parameter aggregation and knowledge transfer across heterogeneous models. However, transmitting gradients still introduces considerable communication overhead, while existing compression approaches typically apply a uniform strategy across clients and ignore their diverse model characteristics and resource capacities. To address this issue, we propose a heterogeneous compression framework for FedKD that enables each client to select a compression strategy from a candidate strategy set. We formulate the compression strategy selection problem as a non-stationary stochastic multi-armed bandit (MAB), where each arm corresponds to a compression strategy. An efficiency-aware reward is designed by jointly considering local optimization improvement, global knowledge alignment, and execution time. Based on this formulation, we develop an Adaptive heterogeneouS Compression algorithm for fEderated kNowledge Distillation (ASCEND), which employs an exponential moving average (EMA)-enhanced

ε

-greedy policy to balance exploration and exploitation. Experimental results on multiple datasets demonstrate that ASCEND effectively adapts to heterogeneous model and resource settings, reducing communication overhead and training time while maintaining competitive model accuracy.

Abstract

Federated Knowledge Distillation

Explore similar work

Adaptive Heterogeneous Compression for Resource-Efficient Federated Knowledge Distillation

Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search

Progressive2^22: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression

Progressive $^2$ : A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression