SPADE-KD: Peer-to-Peer Collaborative Learning under Heterogeneous Settings

Harikrishna Kuttivelil; Katia Obraczka

cs.LGOct 15, 2026

SPADE-KD: Peer-to-Peer Collaborative Learning under Heterogeneous Settings

Authors: Harikrishna Kuttivelil, Katia Obraczka

Organizations: University of California, Santa Cruz

Abstract

Peer-to-peer collaborative learning is an emerging distributed deep learning paradigm that eliminates the need for a central aggregator and disseminator, avoiding the single-point-of-failure and bottleneck issues they can bring. Many such approaches use parameter averaging, requiring clients to have homogeneous model architectures, whereas others use knowledge distillation as an alternative while leaning on shared public data or centralized control infrastructure. Many of these approaches also make a one-size-fits-all assumption, i.e., jointly training a single, globally optimal solution for all clients, when in reality, clients' underlying data distributions might differ such that a globally optimal solution may not be locally optimal for all clients. To enable collaborative learning in environments with both model architecture and data distribution heterogeneity, we present SPADE-KD -- a fully decentralized model distillation framework that optimizes collaboration towards personalized rather than global performance. SPADE-KD leverages two mechanisms -- adaptive distillation and utility-driven peer selection. Adaptive distillation modulates the strength of each incoming model transfer by the performance advantage of the incoming model compared to the native model measured by their task losses, such that a client will distill more aggressively from incoming models that outperform it on its own local data. Utility-driven peer selection uses the loss-informed utility derived from adaptive distillation to rank prospective partners, focusing on exchanges that measurably improve the local model while suppressing low-utility transfers. Using federated classification tasks, we demonstrate that SPADE-KD consistent outperforms naive decentralized distillation with lower loss and faster convergence and that its individual components are more effective together than independently.

Figures & tables

Symbol	Description
$n$	Number of clients in the network
$M_{i}$	Neighbors of client $i$ (candidate peers)
$m_{i}$	Number of peers contacted per round, $\|\mathcal{C}_{i}\|$
$\mathcal{C}_{i}$	Selected peers of client $i$ , $\mathcal{C}_{i}\subseteq M_{i}$
$D_{i}$	Private local dataset of client $i$
$\theta_{i}$	Native model of client $i$

Table 1: Summary of notation.

Scenario	$n$	$m$	$\epsilon$	$\tau_{\mathrm{pre}}$	$\tau$
MNIST	100	10	3	$\leq$ 10	40
FEMNIST	80	10	20	$\leq$ 40	60
SpeechCommands	80	5	2	$\leq$ 15	60

Table 2: Experimental configuration for each scenario.

	FEMNIST						MNIST						SpeechCommands
	Mean		$P_{10}$		$P_{90}$		Mean		$P_{10}$		$P_{90}$		Mean		$P_{10}$		$P_{90}$
Method	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$
D-KD	$0.620$	$-$	$0.346$	$-$	$0.969$	$-$	$0.432$	$-$	$0.224$	$-$	$0.691$	$-$	$1.095$	$-$	$0.393$	$-$	$1.804$	$-$
D-KD-PS	$0.566$	$+8.7$	$0.314$	$+9.4$	$0.890$	$+8.2$	$0.344$	$+20.4$	$0.185$	$+17.4$	$0.527$	$+23.8$	$1.086$	$+0.9$	$0.383$	$+2.6$	$1.821$	$-0.9$
D-AKD	$0.586$	$+5.4$	$0.300$	$+13.5$	$0.953$	$+1.7$	$0.399$	$+7.7$	$0.204$	$+8.9$	$0.655$	$+5.2$	$1.075$	$+1.9$	$0.352$	$+10.5$	$1.826$	$-1.2$
$\spadesuit$ -KD	$0.548$	$+11.6$	$0.276$	$+20.2$	$0.894$	$+7.8$	$0.333$	$+23.0$	$0.168$	$+25.0$	$0.524$	$+24.3$	$1.073$	$+2.1$	$0.349$	$+11.4$	$1.831$	$-1.5$

Table 3: \par Final-round client loss (

\mathcal{L\par}

) and percentage improvement (

\Delta\%\par

) of the utility-based peer selection ablation (D-KD-PS), the adaptive knowledge distillation ablation (D-AKD), and the complete SPADE-KD (

\spadesuit

-KD) method versus the naive decentralized KD (D-KD) baseline, at the mean,

P_{10}

, and

P_{90}

across clients.

Panel of Figure 3 — Figure 3: \par Client loss on the FEMNIST scenario (naturally heterogeneous image data): mean, $P_{10}$ , and $P_{90}$ across clients.

Panel of Figure 6 — Figure 6: \par Client loss on the MNIST scenario (artificially heterogeneous image data): mean, $P_{10}$ , and $P_{90}$ across clients.

Panel of Figure 9 — Figure 9: \par Client loss on the SpeechCommands scenario (naturally heterogeneous audio data): mean, $P_{10}$ , and $P_{90}$ across clients.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10: \par $\spadesuit$ -KD client-recorded scores over the course of collaborative training in the FEMNIST scenario.

Panel of Figure 11 — Figure 11: \par $\spadesuit$ -KD client-recorded scores over the course of collaborative training in the MNIST scenario.

Panel of Figure 12 — Figure 12: \par $\spadesuit$ -KD client-recorded scores over the course of collaborative training in the SpeechCommands scenario.

Panel of Figure 18 — Figure 18: \par Clients’ loss and accuracy on the FEMNIST scenario (naturally heterogeneous image data).

Panel of Figure 24 — Figure 24: \par Clients’ loss and accuracy on the MNIST scenario (artificially heterogeneous image data).

Panel of Figure 30 — Figure 30: \par Clients’ loss and accuracy on the SpeechCommands scenario (naturally heterogeneous audio data).

	FEMNIST						MNIST						SpeechCommands
	Mean		$P_{10}$		$P_{90}$		Mean		$P_{10}$		$P_{90}$		Mean		$P_{10}$		$P_{90}$
Method	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$
FedKD	$0.708$	$-14.3$	$0.433$	$-25.2$	$1.063$	$-9.7$	$0.442$	$-2.3$	$0.249$	$-11.6$	$0.679$	$+1.8$	$1.093$	$+0.3$	$0.445$	$-13.1$	$1.828$	$-1.3$
D-KD	$0.620$	$-$	$0.346$	$-$	$0.969$	$-$	$0.432$	$-$	$0.224$	$-$	$0.691$	$-$	$1.095$	$-$	$0.393$	$-$	$1.804$	$-$
D-KD-PS	$0.566$	$+8.7$	$0.314$	$+9.4$	$0.890$	$+8.2$	$0.344$	$+20.4$	$0.185$	$+17.4$	$0.527$	$+23.8$	$1.086$	$+0.9$	$0.383$	$+2.6$	$1.821$	$-0.9$
D-AKD	$0.586$	$+5.4$	$0.300$	$+13.5$	$0.953$	$+1.7$	$0.399$	$+7.7$	$0.204$	$+8.9$	$0.655$	$+5.2$	$1.075$	$+1.9$	$0.352$	$+10.5$	$1.826$	$-1.2$

Appendix

Table 4: \par Final-round client loss (lower is better) and percentage improvement

\Delta\%\par

versus the naive decentralized KD (D-KD) baseline, at the mean,

P_{10}

, and

P_{90}

across clients.

\Delta\%\par>0

means lower loss than D-KD.

	FEMNIST						MNIST						SpeechCommands
	Mean		$P_{10}$		$P_{90}$		Mean		$P_{10}$		$P_{90}$		Mean		$P_{10}$		$P_{90}$
Method	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$
FedKD	$0.828$	$-1.9$	$0.678$	$-1.9$	$0.956$	$-0.5$	$0.883$	$+0.2$	$0.815$	$+1.7$	$0.944$	$+0.2$	$0.654$	$-1.4$	$0.378$	$-5.7$	$0.918$	$-0.2$
D-KD	$0.844$	$-$	$0.692$	$-$	$0.961$	$-$	$0.881$	$-$	$0.802$	$-$	$0.942$	$-$	$0.663$	$-$	$0.401$	$-$	$0.920$	$-$
D-KD-PS	$0.854$	$+1.2$	$0.727$	$+5.1$	$0.964$	$+0.3$	$0.898$	$+1.9$	$0.846$	$+5.5$	$0.950$	$+0.9$	$0.663$	$+0.0$	$0.387$	$-3.3$	$0.910$	$-1.2$
D-AKD	$0.843$	$-0.1$	$0.699$	$+1.1$	$0.961$	$+0.0$	$0.887$	$+0.7$	$0.822$	$+2.5$	$0.946$	$+0.5$	$0.662$	$-0.2$	$0.399$	$-0.4$	$0.915$	$-0.6$

Appendix

Table 5: \par Final-round client accuracy (higher is better) and percentage improvement

\Delta\%\par

versus the naive decentralized KD (D-KD) baseline, at the mean,

P_{10}

, and

P_{90}

across clients.

\Delta\%\par>0

means higher accuracy than D-KD.

Federated Knowledge Distillation

Federated knowledge distillation transfers predictive behavior among data-isolated clients to train useful models without sharing raw data.

See all 36 papers

Sep 1, 2026

Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network

Decentralized, serverless learning increasingly connects devices running different architectures, where the standard tool, decentralized SGD, is undefined as models with different parameter counts cannot be averaged. Knowledge distillation (KD) exchanges soft predictions rather than weights and sidesteps this obstacle, yet convergence theory for fully decentralized, asynchronous peer-to-peer (P2P) KD is lacking. We provide one, relocating consensus from parameter space to function (output) space: a KD event is a geometric contraction operator in logit space on the peers' predictive distributions, which we analyse in the Hilbert space of predictions on a reference measure. Under standard smoothness/variance assumptions and two realizability assumptions, one bridging parameter SGD to the functional step and one controlling restricted task/KD alignment, the time-averaged functional stationarity and function-space disagreement converge at rate

O(1/(ηT))

to an

O(η)+O(B_f^2)+O(ζ_f^2)

neighbourhood. Here

B_f

is the distance from the task optimum to the peers' reachable classes and

ζ_f

measures persistent local-task heterogeneity. Across homogeneous, width-heterogeneous, and mixed-family networks of the experiments, KD contracts function disagreement by

40-61\times

, while isolated training does not. The sampled stationarity diagnostic has late transient exponents

0.99-1.90

on the shared-skeleton main runs, and the four-point step-size sweep exhibits the predicted transient: neighbourhood tradeoff.

Lucas Qingyang Fang, Tiyao Liu, Jinhao Jing +4

University of California, Santa Cruz, USA · China University of Petroleum, China · The Chinese University of Hong Kong, Shenzhen, China +2

Decentralized Learning Cross-Architecture Knowledge Distillation Knowledge Distillation

Ask PDF

Sep 24, 2026

SPADE-DFL: Communication-Efficient Decentralized Federated Learning via Derivative-Free Linearized ADMM

Reducing communication in derivative-free decentralized learning requires controlling the disagreement accumulated over multiple local updates. This paper develops SPADE-DFL, a primal--dual method that allows the number of local function-value updates between neighbor exchanges to grow with the computation budget while preserving the nonprivate convergence order. For smooth nonconvex objectives under uniform query-moment bounds, the prescribed nonprivate schedule achieves a time-averaged stationarity and consensus bound of

\mathcal{O}(T^{-1/3})

using only

Θ(T^{2/3})

communication rounds, where

T

is the number of local updates per client. For private training, the accumulated data-dependent increment is isolated from the graph correction, allowing one protected state per client and round to generate all outgoing messages. We prove client-level differential privacy for the full interactive transcript and quantify the resulting optimization error over a finite horizon. Experiments on four classification tasks show that SPADE-DFL achieves higher mean test accuracy than existing decentralized learning methods.

Mengli Wei, Mengkai Zhu, Jiawen Chen +2

School of Mathematics, Southeast University, Nanjing 210096, China

Communication-Efficient Distributed Training Black-Box Optimization Alternating Direction Method of Multipliers Decentralized Learning Differentially Private Federated Learning Derivative-Free Optimization+1

Ask PDF

Sep 30, 2024

Collaborative Knowledge Distillation via a Learning-by-Education Node Community

A novel Learning-by-Education Node Community framework (LENC) for Collaborative Knowledge Distillation (CKD) is presented, which facilitates continual collective learning through effective knowledge exchanges among diverse deployed Deep Neural Network (DNN) peer nodes. These DNNs dynamically and autonomously adopt either the role of a student, seeking knowledge, or that of a teacher, imparting knowledge, fostering a collaborative learning environment. The proposed framework triggers knowledge transfer via autonomous teacher discovery and stream-driven DNN distillation as needed, while enhancing their learning capabilities and promoting their collaboration. LENC addresses the challenges of handling diverse training data distributions and the limitations of individual DNN node learning abilities. \hl{It enables the exploitation of selected peer-teacher knowledge upon learning a new task and mitigates catastrophic forgetting in DNN nodes.} \hl{Additionally, it supports task-boundary-free continual adaptation in distributed settings via autonomous role assignment and modular forgetting mitigation, as DNN nodes receive no explicit task-boundary metadata during deployment.} Experimental evaluation on a proof-of-concept implementation demonstrates the LENC framework's functionalities and benefits across multiple DNN learning and inference scenarios. The conducted experiments showcase its ability to gradually improve the average test accuracy of the community of interacting DNN nodes in image classification problems, by appropriately leveraging the collective knowledge of all node peers. The LENC framework achieves strong performance in on-line unlabelled CKD.

Anestis Kaimakamidis, Ioannis Mademlis, Ioannis Pitas

Aristotle University of Thessaloniki, Thessaloniki, 54124, Greece

Online Knowledge Distillation Continual Learning Decentralized Learning Multi-Teacher Knowledge Distillation Knowledge Distillation Catastrophic Forgetting

Ask PDF

Abstract

Federated Knowledge Distillation

Explore similar work

Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network

SPADE-DFL: Communication-Efficient Decentralized Federated Learning via Derivative-Free Linearized ADMM

Collaborative Knowledge Distillation via a Learning-by-Education Node Community