SPADE-KD: Peer-to-Peer Collaborative Learning under Heterogeneous Settings

Harikrishna Kuttivelil; Katia Obraczka

cs.LGOct 15, 2026

SPADE-KD: Peer-to-Peer Collaborative Learning under Heterogeneous Settings

Authors: Harikrishna Kuttivelil, Katia Obraczka

Organizations: University of California, Santa Cruz

Abstract

Peer-to-peer collaborative learning is an emerging distributed deep learning paradigm that eliminates the need for a central aggregator and disseminator, avoiding the single-point-of-failure and bottleneck issues they can bring. Many such approaches use parameter averaging, requiring clients to have homogeneous model architectures, whereas others use knowledge distillation as an alternative while leaning on shared public data or centralized control infrastructure. Many of these approaches also make a one-size-fits-all assumption, i.e., jointly training a single, globally optimal solution for all clients, when in reality, clients' underlying data distributions might differ such that a globally optimal solution may not be locally optimal for all clients. To enable collaborative learning in environments with both model architecture and data distribution heterogeneity, we present SPADE-KD -- a fully decentralized model distillation framework that optimizes collaboration towards personalized rather than global performance. SPADE-KD leverages two mechanisms -- adaptive distillation and utility-driven peer selection. Adaptive distillation modulates the strength of each incoming model transfer by the performance advantage of the incoming model compared to the native model measured by their task losses, such that a client will distill more aggressively from incoming models that outperform it on its own local data. Utility-driven peer selection uses the loss-informed utility derived from adaptive distillation to rank prospective partners, focusing on exchanges that measurably improve the local model while suppressing low-utility transfers. Using federated classification tasks, we demonstrate that SPADE-KD consistent outperforms naive decentralized distillation with lower loss and faster convergence and that its individual components are more effective together than independently.

Figures & tables

Symbol	Description
$n$	Number of clients in the network
$M_{i}$	Neighbors of client $i$ (candidate peers)
$m_{i}$	Number of peers contacted per round, $\|\mathcal{C}_{i}\|$
$\mathcal{C}_{i}$	Selected peers of client $i$ , $\mathcal{C}_{i}\subseteq M_{i}$
$D_{i}$	Private local dataset of client $i$
$\theta_{i}$	Native model of client $i$

Table 1: Summary of notation.

Scenario	$n$	$m$	$\epsilon$	$\tau_{\mathrm{pre}}$	$\tau$
MNIST	100	10	3	$\leq$ 10	40
FEMNIST	80	10	20	$\leq$ 40	60
SpeechCommands	80	5	2	$\leq$ 15	60

Table 2: Experimental configuration for each scenario.

	FEMNIST						MNIST						SpeechCommands
	Mean		$P_{10}$		$P_{90}$		Mean		$P_{10}$		$P_{90}$		Mean		$P_{10}$		$P_{90}$
Method	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$	$\mathcal{L\par}$	$\Delta\%\par$
D-KD	$0.620$	$-$	$0.346$	$-$	$0.969$	$-$	$0.432$	$-$	$0.224$	$-$	$0.691$	$-$	$1.095$	$-$	$0.393$	$-$	$1.804$	$-$
D-KD-PS	$0.566$	$+8.7$	$0.314$	$+9.4$	$0.890$	$+8.2$	$0.344$	$+20.4$	$0.185$	$+17.4$	$0.527$	$+23.8$	$1.086$	$+0.9$	$0.383$	$+2.6$	$1.821$	$-0.9$
D-AKD	$0.586$	$+5.4$	$0.300$	$+13.5$	$0.953$	$+1.7$	$0.399$	$+7.7$	$0.204$	$+8.9$	$0.655$	$+5.2$	$1.075$	$+1.9$	$0.352$	$+10.5$	$1.826$	$-1.2$
$\spadesuit$ -KD	$0.548$	$+11.6$	$0.276$	$+20.2$	$0.894$	$+7.8$	$0.333$	$+23.0$	$0.168$	$+25.0$	$0.524$	$+24.3$	$1.073$	$+2.1$	$0.349$	$+11.4$	$1.831$	$-1.5$

Table 3: \par Final-round client loss (

\mathcal{L\par}

) and percentage improvement (

\Delta\%\par

) of the utility-based peer selection ablation (D-KD-PS), the adaptive knowledge distillation ablation (D-AKD), and the complete SPADE-KD (

\spadesuit

-KD) method versus the naive decentralized KD (D-KD) baseline, at the mean,

P_{10}

, and

P_{90}

across clients.

Panel of Figure 3 — Figure 3: \par Client loss on the FEMNIST scenario (naturally heterogeneous image data): mean, $P_{10}$ , and $P_{90}$ across clients.

Panel of Figure 6 — Figure 6: \par Client loss on the MNIST scenario (artificially heterogeneous image data): mean, $P_{10}$ , and $P_{90}$ across clients.

Panel of Figure 9 — Figure 9: \par Client loss on the SpeechCommands scenario (naturally heterogeneous audio data): mean, $P_{10}$ , and $P_{90}$ across clients.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10: \par $\spadesuit$ -KD client-recorded scores over the course of collaborative training in the FEMNIST scenario.

Panel of Figure 11 — Figure 11: \par $\spadesuit$ -KD client-recorded scores over the course of collaborative training in the MNIST scenario.

Panel of Figure 12 — Figure 12: \par $\spadesuit$ -KD client-recorded scores over the course of collaborative training in the SpeechCommands scenario.

Panel of Figure 18 — Figure 18: \par Clients’ loss and accuracy on the FEMNIST scenario (naturally heterogeneous image data).

Panel of Figure 24 — Figure 24: \par Clients’ loss and accuracy on the MNIST scenario (artificially heterogeneous image data).

Panel of Figure 30 — Figure 30: \par Clients’ loss and accuracy on the SpeechCommands scenario (naturally heterogeneous audio data).

	FEMNIST						MNIST						SpeechCommands
	Mean		$P_{10}$		$P_{90}$		Mean		$P_{10}$		$P_{90}$		Mean		$P_{10}$		$P_{90}$
Method	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$	L	$\Delta\%\par$
FedKD	$0.708$	$-14.3$	$0.433$	$-25.2$	$1.063$	$-9.7$	$0.442$	$-2.3$	$0.249$	$-11.6$	$0.679$	$+1.8$	$1.093$	$+0.3$	$0.445$	$-13.1$	$1.828$	$-1.3$
D-KD	$0.620$	$-$	$0.346$	$-$	$0.969$	$-$	$0.432$	$-$	$0.224$	$-$	$0.691$	$-$	$1.095$	$-$	$0.393$	$-$	$1.804$	$-$
D-KD-PS	$0.566$	$+8.7$	$0.314$	$+9.4$	$0.890$	$+8.2$	$0.344$	$+20.4$	$0.185$	$+17.4$	$0.527$	$+23.8$	$1.086$	$+0.9$	$0.383$	$+2.6$	$1.821$	$-0.9$
D-AKD	$0.586$	$+5.4$	$0.300$	$+13.5$	$0.953$	$+1.7$	$0.399$	$+7.7$	$0.204$	$+8.9$	$0.655$	$+5.2$	$1.075$	$+1.9$	$0.352$	$+10.5$	$1.826$	$-1.2$

Appendix

Table 4: \par Final-round client loss (lower is better) and percentage improvement

\Delta\%\par

versus the naive decentralized KD (D-KD) baseline, at the mean,

P_{10}

, and

P_{90}

across clients.

\Delta\%\par>0

means lower loss than D-KD.

	FEMNIST						MNIST						SpeechCommands
	Mean		$P_{10}$		$P_{90}$		Mean		$P_{10}$		$P_{90}$		Mean		$P_{10}$		$P_{90}$
Method	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$	Acc.	$\Delta\%\par$
FedKD	$0.828$	$-1.9$	$0.678$	$-1.9$	$0.956$	$-0.5$	$0.883$	$+0.2$	$0.815$	$+1.7$	$0.944$	$+0.2$	$0.654$	$-1.4$	$0.378$	$-5.7$	$0.918$	$-0.2$
D-KD	$0.844$	$-$	$0.692$	$-$	$0.961$	$-$	$0.881$	$-$	$0.802$	$-$	$0.942$	$-$	$0.663$	$-$	$0.401$	$-$	$0.920$	$-$
D-KD-PS	$0.854$	$+1.2$	$0.727$	$+5.1$	$0.964$	$+0.3$	$0.898$	$+1.9$	$0.846$	$+5.5$	$0.950$	$+0.9$	$0.663$	$+0.0$	$0.387$	$-3.3$	$0.910$	$-1.2$
D-AKD	$0.843$	$-0.1$	$0.699$	$+1.1$	$0.961$	$+0.0$	$0.887$	$+0.7$	$0.822$	$+2.5$	$0.946$	$+0.5$	$0.662$	$-0.2$	$0.399$	$-0.4$	$0.915$	$-0.6$

Appendix

Table 5: \par Final-round client accuracy (higher is better) and percentage improvement

\Delta\%\par

versus the naive decentralized KD (D-KD) baseline, at the mean,

P_{10}

, and

P_{90}

across clients.

\Delta\%\par>0

means higher accuracy than D-KD.

Federated Knowledge Distillation

Federated knowledge distillation transfers predictive behavior among data-isolated clients to train useful models without sharing raw data.

See all 36 papers

Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network
Sep 1, 2026Lucas Qingyang Fang, Tiyao Liu, Jinhao Jing +4Decentralized Learning Cross-Architecture Knowledge Distillation
University of California, Santa Cruz, USA · China University of Petroleum, China · The Chinese University of Hong Kong, Shenzhen, China +2
SPADE-DFL: Communication-Efficient Decentralized Federated Learning via Derivative-Free Linearized ADMM
Sep 24, 2026Mengli Wei, Mengkai Zhu, Jiawen Chen +2Communication-Efficient Distributed Training Black-Box Optimization
School of Mathematics, Southeast University, Nanjing 210096, China
Collaborative Knowledge Distillation via a Learning-by-Education Node Community
Sep 30, 2024Anestis Kaimakamidis, Ioannis Mademlis, Ioannis PitasOnline Knowledge Distillation Continual Learning
Aristotle University of Thessaloniki, Thessaloniki, 54124, Greece

Sep 1, 2026J/K move · Enter open · S save

Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network

Lucas Qingyang Fang, Tiyao Liu, Jinhao Jing, Zeji Li+3

University of California, Santa Cruz, USA · China University of Petroleum, China · The Chinese University of Hong Kong, Shenzhen, China+2

Decentralized Learning Cross-Architecture Knowledge Distillation Knowledge Distillation

Open paper AskPDF arXiv

Decentralized, serverless learning increasingly connects devices running different architectures, where the standard tool, decentralized SGD, is undefined as models with different parameter counts cannot be averaged. Knowledge distillation (KD) exchanges soft predictions rather than weights and sidesteps this obstacle, yet convergence theory for fully decentralized, asynchronous peer-to-peer (P2P) KD is lacking. We provide one, relocating consensus from parameter space to function (output) space: a KD event is a geometric contraction operator in logit space on the peers' predictive distributions, which we analyse in the Hilbert space of predictions on a reference measure. Under standard smoothness/variance assumptions and two realizability assumptions, one bridging parameter SGD to the functional step and one controlling restricted task/KD alignment, the time-averaged functional stationarity and function-space disagreement converge at rate

O(1/(ηT))

to an

O(η)+O(B_f^2)+O(ζ_f^2)

neighbourhood. Here

B_f

is the distance from the task optimum to the peers' reachable classes and

ζ_f

measures persistent local-task heterogeneity. Across homogeneous, width-heterogeneous, and mixed-family networks of the experiments, KD contracts function disagreement by

40-61\times

, while isolated training does not. The sampled stationarity diagnostic has late transient exponents

0.99-1.90

on the shared-skeleton main runs, and the four-point step-size sweep exhibits the predicted transient: neighbourhood tradeoff.

Abstract

Federated Knowledge Distillation

Explore similar work

Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network

SPADE-DFL: Communication-Efficient Decentralized Federated Learning via Derivative-Free Linearized ADMM

Collaborative Knowledge Distillation via a Learning-by-Education Node Community