FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learning

Xinyuan Zhao

cs.CVSep 29, 2026

FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learning

Authors: Xinyuan Zhao

Organizations: Sun Yat-sen University

Abstract

Federated knowledge must remain usable by recipients with different modalities, private architectures, and tasks. We present FedSocket, which makes recipient execution a design requirement of the exchanged model. A shared Q combines recipient-computable inputs, task-owned outputs, and ownership-aware aggregation, connecting heterogeneous private models through a common prediction interface. Private models teach local Q copies; the returned Q supports local learning and Joint inference, with only Q parameters and counts exchanged. Across six datasets, FedSocket improves missing-modality recipient accuracy over Local by 14.44 and 15.51 percentage points on MELD and UCF-51. Under matched inference capacity, Joint exceeds independent ensembles by 11.06 points in UCF-51 accuracy and 4.87 points in mean bidirectional Flickr30k R@1. Joint also improves over Q alone on all four heterogeneous endpoints, demonstrating the value of combining local and exchanged predictions. Teacher controls, sharing-path interventions, and component factorials identify the roles of supervision, sharing, and deployment. FedSocket makes exchanged knowledge directly usable from federated training to recipient inference.

Figures & tables

Dataset	Client inputs	Primary metric
Same task with missing modalities
MELD	T+A $\rightarrow$ T	Accuracy
CREMA-D	V+A $\rightarrow$ V	Accuracy
UCF-51	V+A $\rightarrow$ V	Accuracy
Heterogeneous tasks and architectures
CIFAR-100	3 image clients	Accuracy

Table 1 : Evaluation settings. T/A/V/I: text/audio/video/image. Arrows indicate donor-to-recipient modality access. Flickr uses five-fold 200-image evaluation. Full protocols: Tab. 15 .

	MELD		CREMA-D		UCF-51
Method	All	Rec.	All	Rec.	All	Rec.
Local	$54.13\pm 0.95$	$53.81\pm 6.54$	$74.79\pm 0.32$	$74.65\pm 0.50$	$71.70\pm 0.13$	$71.32\pm 0.21$
Label Q	$61.90\pm 0.78$	$66.97\pm 2.21$	$74.81\pm 0.59$	$74.48\pm 0.64$	$81.22\pm 0.78$	$84.62\pm 1.12$
Single-modal Q	$61.60\pm 0.69$	$67.23\pm 2.01$	$74.49\pm 0.38$	$74.12\pm 0.64$	$81.73\pm 0.22$	$85.32\pm 0.35$
FedSocket	$62.16\pm 0.77$	$68.24\pm 1.99$	$75.45\pm 0.34$	$75.24\pm 0.46$	$82.83\pm 0.93$	$86.83\pm 1.31$

Table 2 : Same-task Joint accuracy (%; mean

\pm

sample SD, five seeds). All: pooled client test samples; Rec.: missing-modality recipients. Red/underline: best/second-best means.

Dataset	$P-L$	$J-P$	$J-L$
MELD	$0.40\pm 1.26$	$14.04\pm 6.60$	$14.44\pm 6.59$
CREMA-D	$0.40\pm 1.17$	$0.19\pm 0.68$	$0.59\pm 0.76$
UCF-51	$0.95\pm 0.46$	$14.56\pm 1.27$	$15.51\pm 1.35$

Table 3: Same-task recipient accuracy decomposition (pp; paired mean

\pm

sample SD, five seeds).

J-L=(P-L)+(J-P)

for Joint, Q-trained Private, and independent Local.

Task / metric	Private	Public Q	Joint
CIFAR Acc.	$45.13\pm 0.39$	$50.32\pm 0.48$	$53.59\pm 1.28$
AG News Acc.	$49.73\pm 0.48$	$75.85\pm 1.11$	$78.50\pm 0.42$
Flickr I2T R@1	$35.09\pm 0.26$	$37.86\pm 0.82$	$42.37\pm 0.51$
Flickr T2I R@1	$30.38\pm 0.45$	$31.13\pm 0.73$	$35.46\pm 0.48$

Table 4 : Task-heterogeneous endpoints (%; mean

\pm

sample SD, five seeds). Joint retains both branches. Flickr: five-fold 200-image R@1. Red/underline: best/second-best means.

Method	UCF-51 Acc.	Flickr R@1
Conditional Joint	$82.83\pm 0.93$	$42.37\pm 0.51$ / $35.46\pm 0.48$
Independent matched	$71.77\pm 0.17$	$36.56\pm 0.21$ / $31.54\pm 0.05$
Independent $2\times$ full	$71.70\pm 0.19$	–
Dual checkpoint	$71.70\pm 0.13$	$21.58\pm 1.03$ / $18.39\pm 0.67$

Table 5 : Matched inference-capacity controls (%; mean

\pm

sample SD, five seeds). Flickr cells are I2T/T2I R@1. “Independent matched” is parameter-matched on UCF and Conditional–Private on Flickr. Red/underline: best/second best.

Method	CIFAR	AG	I2T	T2I
LOCAL [ 21 ]	28.07	48.35	22.33	18.44
FedMD [ 12 ]	22.54	48.18	19.13	15.63
FedGEMS [ 4 ]	22.84	48.30	18.93	16.05
FedET [ 5 ]	31.86	49.38	22.63	18.22
CreamFL [ 28 ]	22.14	42.16	18.38	15.49
FedMKD [ 13 ]	24.99	47.99	22.33	18.37

Table 6 : Protocol-aligned task-heterogeneous reference values (%). Published rows follow FedAFD Tab. 2(b) [ 21 ] ; FedSocket rows are five-seed means. Private uses one model; Joint retains Q. CIFAR/AG: accuracy; Flickr: five-fold R@1.

Dataset	Published ref.	Private	Joint
MELD	MFedMC [ 30 ] 53.31	$61.11\pm 1.13$	$62.16\pm 0.77$
CREMA-D	BMSFed [ 6 ] 59.8	$74.87\pm 0.62$	$75.45\pm 0.34$
UCF-51 †	AMID [ 3 ] 73.8	$72.50\pm 0.29$	$82.83\pm 0.93$

Table 7 : Published same-task references and FedSocket all-client accuracy (%; mean

\pm

SD, five seeds). Reference settings: MELD, natural distribution and 5 MB; CREMA-D, 50% single-modal clients. † AMID is centralized.

Private: KD $\times$ fusion
Effect	CIFAR	AG	I2T	T2I
KD	$4.94\!\pm\!0.19$	$3.23\!\pm\!0.44$	$17.19\!\pm\!0.47$	$15.51\!\pm\!0.42$
Fusion	$1.64\!\pm\!0.25$	$-0.24\!\pm\!0.15$	$-0.32\!\pm\!0.40$	$-0.22\!\pm\!0.14$
Interaction	$3.26\!\pm\!0.57$	$-0.48\!\pm\!0.40$	$0.62\!\pm\!0.68$	$0.71\!\pm\!0.74$
Joint: KD $\times$ fusion
KD	$5.84\!\pm\!1.36$	$17.30\!\pm\!1.65$	$4.24\!\pm\!0.30$	$3.76\!\pm\!0.43$

Table 8 : Component effects (pp; five seeds). Separate

2\times 2

designs: KD/fusion reports paired mean

\pm

SD; gate/PCGrad reports contrasts of cell means ( Tab. 25 ). Interaction: combined minus singles plus reference. Flickr: five-fold R@1.

Dataset	Private	Public Q	Joint
CIFAR-100	$36.88\pm 0.43$	$49.44\pm 0.48$	$49.70\pm 1.23$
AG News	$38.79\pm 0.65$	$74.25\pm 1.57$	$76.90\pm 0.42$

Table 9: Primary classification macro F1 (%; mean

\pm

sample SD, five seeds).

Metric	Private	Public Q	Joint
I2T R@1	$35.09\pm 0.26$	$37.86\pm 0.82$	$42.37\pm 0.51$
I2T R@5	$64.58\pm 0.45$	$65.30\pm 0.49$	$70.21\pm 0.62$
I2T R@10	$75.84\pm 0.46$	$75.76\pm 0.43$	$80.26\pm 0.46$
T2I R@1	$30.38\pm 0.45$	$31.13\pm 0.73$	$35.46\pm 0.48$
T2I R@5	$61.82\pm 0.29$	$61.47\pm 0.56$	$66.93\pm 0.32$
T2I R@10	$74.65\pm 0.22$	$73.58\pm 0.49$	$78.34\pm 0.28$

Table 10: Flickr retrieval (%; mean

\pm

sample SD, five seeds). Each seed aggregates five 200-image folds. † Mean R averages the six recall means; only this point estimate is reported.

Task	Conditional J	Independent J	$\Delta$ mean (pp)
CIFAR	$53.59\pm 1.28$	$50.91\pm 0.34$	$+2.68$
AG News	$78.50\pm 0.42$	$64.64\pm 0.97$	$+13.86$

Table 11 : Exact-private capacity control (%; mean

\pm

SD, five seeds).

\Delta

: difference of endpoint means. Private branches and parameter/FLOP counts match.

Task	Conditional J	Independent J	$\Delta$ mean (pp)
CIFAR	$53.59\pm 1.28$	$50.91\pm 0.34$	$+2.68$
AG News	$78.50\pm 0.42$	$64.64\pm 0.97$	$+13.86$

Table 11 : Exact-private capacity control (%; mean

\pm

SD, five seeds).

\Delta

: difference of endpoint means. Private branches and parameter/FLOP counts match.

Backbone	Private	Q	Joint
MLP	$76.23\pm 0.50$	$83.04\pm 2.06$	$91.25\pm 0.98$
CNN	$61.31\pm 1.95$	$77.69\pm 0.97$	$79.88\pm 0.97$
Transformer	$80.05\pm 1.53$	$75.67\pm 1.75$	$81.87\pm 2.97$

Table 12 : UCF backbone-group accuracy (%; mean

\pm

SD, five seeds). Comparisons are within group; paired gains and intervals appear in Tab. 28 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Common public examples	Sample-indexed upload	Output organization	Recipient branch	Evaluated inference
FedMD [ 12 ]	Required	Logits	Common output space	No	Private
FML [ 19 ]	Not required	None	Adapted client heads	Training meme	Private
FedKD [ 25 ]	Not required	None	Common task outputs	Yes (same task)	Mentor/mentee
MH-pFLID [ 26 ]	Not required	None	Receiver/transmitter modules	Yes	Private
CreamFL [ 28 ]	Required	Representations	Global server space	No	Client/server
FedHTCM [ 27 ]	Required	Representations	Multitask global space	No	Client/global

Appendix

Table 14 : Capability boundary of the closest exchange-object methods. “Sample-indexed upload” denotes logits or representations attached to shared examples; model parameters are not counted. “Recipient branch” requires a returned shared model whose inputs are available to the receiving client and whose outputs are valid for its task. Entries summarize the methods’ stated contracts and evaluated endpoints, not possible extensions.

Dataset	Client/modality setting	Primary metric	Protocol
Same task, different available modalities
MELD	Text–audio donors; text recipients	Accuracy	Four-class selected cohort; group cross-fitting.
CREMA-D	Video–audio donors; video recipients	Accuracy	Six-class emotion; same-actor sentence-group split.
UCF-51	Video–audio donors; video recipients	Accuracy	20 clients; $\alpha=0.5$ ; full supervision.
Heterogeneous tasks and private architectures
CIFAR-100	3 non-IID image clients	Accuracy	Task-heterogeneous training; Private, public Q, and Joint.

Appendix

Table 15 : Experimental protocol and primary endpoints.

		Private			Joint
Dataset	Method	Acc.	Macro F1	UAR	Acc.	Macro F1	UAR
MELD	Local	$53.81\pm 6.54$	$27.78\pm 6.47$	$28.51\pm 5.46$	$53.81\pm 6.54$	$27.78\pm 6.47$	$28.51\pm 5.46$
	Label Q	$53.29\pm 6.22$	$28.02\pm 6.58$	$28.83\pm 5.72$	$66.97\pm 2.21$	$44.68\pm 5.96$	$43.74\pm 4.83$
	Single-modal Q	$53.29\pm 6.22$	$28.02\pm 6.58$	$28.83\pm 5.72$	$67.23\pm 2.01$	$45.79\pm 4.79$	$44.85\pm 4.13$
	FedSocket	$54.21\pm 6.36$	$28.89\pm 7.22$	$29.50\pm 6.12$	$68.24\pm 1.99$	$46.92\pm 4.61$	$45.82\pm 3.94$
CREMA-D	Local	$74.65\pm 0.50$	$71.14\pm 0.60$	$71.36\pm 0.52$	$74.65\pm 0.50$	$71.14\pm 0.60$	$71.36\pm 0.52$

Appendix

Table 16 : Same-task recipient test endpoints (%, mean

\pm

sample standard deviation over five seeds). Accuracy is primary; macro F1 and UAR are supplementary. Local has no public branch; its columns repeat the same predictor. Red/underline: best/second best within each dataset and endpoint.

		CIFAR	Flickr30k
Variant	Endpoint	Accuracy	I2T R@1	T2I R@1
Local	Private	$38.55\pm 0.13$	$18.22\pm 0.74$	$15.08\pm 0.26$
No return	Private	$38.55\pm 0.13$	$18.22\pm 0.74$	$15.08\pm 0.26$
	Q	$48.63\pm 0.30$	$34.80\pm 0.85$	$29.39\pm 0.38$
	Joint	$45.62\pm 0.95$	$37.41\pm 0.58$	$31.39\pm 0.28$
Task-isolated	Private	$41.66\pm 0.22$	$33.57\pm 0.32$	$28.94\pm 0.17$

Appendix

Table 17 : Primary paired controls (set A; %, mean

\pm

SD, five seeds). No return disables Q

\rightarrow

private training; Q denotes the public branch. Flickr: five-fold 200-image R@1.

Setting	CIFAR	AG	I2T	T2I
None / P	$38.55\!\pm\!0.13$	$46.75\!\pm\!0.15$	$18.22\!\pm\!0.74$	$15.08\!\pm\!0.26$
None / J	$45.62\!\pm\!0.95$	$53.28\!\pm\!1.33$	$37.41\!\pm\!0.58$	$31.39\!\pm\!0.28$
KD / P	$41.86\!\pm\!0.14$	$50.21\!\pm\!0.39$	$35.10\!\pm\!0.40$	$30.24\!\pm\!0.34$
KD / J	$49.95\!\pm\!0.97$	$62.82\!\pm\!0.86$	$40.65\!\pm\!0.60$	$34.67\!\pm\!0.20$
Fus. / P	$38.56\!\pm\!0.05$	$46.74\!\pm\!0.07$	$17.59\!\pm\!0.53$	$14.51\!\pm\!0.36$
Fus. / J	$46.23\!\pm\!0.72$	$53.45\!\pm\!1.40$	$37.13\!\pm\!0.75$	$31.24\!\pm\!0.21$

Appendix

Table 18 : KD

\times

fusion endpoints (%; mean

\pm

SD, five seeds). P/J: Private/Joint. CIFAR/AG: accuracy; Flickr: five-fold R@1. Red/underline: best/second-best within each branch.

UCF-51
Control	Class/I2T	T2I
Independent Private (matched)	$0.70\pm 0.45$	–
Independent Private ( $2\times$ )	$0.44\pm 0.52$	–
Private dual checkpoint	$0.00\pm 0.00$	–
Flickr30k
Conditional–Private	$0.53\pm 0.15$	$0.47\pm 0.07$

Appendix

Table 19 : DEV-selected mixture weights for the control alternatives (mean

\pm

sample SD). TEST data are not used for selecting these weights.

Dataset	Method	Acc.	Macro F1	I2T R@1	T2I R@1
UCF-51	Single Private	$71.70\pm 0.13$	$70.99\pm 0.22$	–	–
	Conditional Private	$72.50\pm 0.29$	$71.93\pm 0.40$	–	–
	Independent Private (param.-matched)	$71.77\pm 0.17$	$71.04\pm 0.32$	–	–
	Independent Private ( $2\times$ full)	$71.70\pm 0.19$	$70.99\pm 0.28$	–	–
	Private dual checkpoint	$71.70\pm 0.13$	$70.99\pm 0.22$	–	–
	Conditional Joint	$82.83\pm 0.93$	$80.95\pm 1.25$	–	–

Appendix

Table 20 : Complete inference-capacity controls (%, mean

\pm

sample SD over five seeds). UCF-51 uses pooled accuracy and macro F1; Flickr30k uses five-fold I2T/T2I R@1. Red/underline: best/second best.

Dataset	Capacity control	$\Delta$ (pp)	95% CI	Wins	Exact $p$
UCF-51	Conditional Private	10.33	[9.18, 11.48]	5/5	.0625
	Independent Private (param.-matched)	11.06	[9.99, 12.13]	5/5	.0625
	Independent Private ( $2\times$ full)	11.13	[9.98, 12.28]	5/5	.0625
	Private dual checkpoint	11.13	[9.97, 12.29]	5/5	.0625
Flickr30k	Conditional Private	6.18	[5.76, 6.60]	5/5	.0625
	Independent Conditional–Private	4.87	– †	5/5	– †

Appendix

Table 21 : Seed-index-aligned evidence against a generic extra-predictor explanation. Positive differences favor Conditional Joint. UCF uses accuracy; Flickr uses mean bidirectional five-fold R@1. With five independent nonzero pairs, the exact two-sided sign-flip test has minimum

p=.0625

Explore similar work

FEAST: Federated Shared-Space Training for Resource-Heterogeneous Clients
Aug 10, 2026Bostan Khan, Masoud DaneshtalabBalanced Learning Taco
Mälardalen University, Västerås, Sweden
FedTopo: Relation-Level Topology Sharing for Model-Heterogeneous Federated Learning
Jul 29, 2026Zhaoyang Ma, Zhihao Wu, Xin Gao +3Heterogeneous Federated Learning Federated Learning
Beijing Jiaotong University Beijing, China · Nanyang Technological University Singapore, Singapore
Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation
Sep 15, 2026Polycarpo Souza Neto, José Mairton Barros da Silva Júnior, Charles Casimiro CavalcanteFederated Learning Knowledge Distillation
Universidade Federal do Cear´a, Fortaleza, CE, Brazil · Uppsala University, Uppsala, Sweden

Aug 10, 2026, cs.LGJ/K move · Enter open · S save

FEAST: Federated Shared-Space Training for Resource-Heterogeneous Clients

Bostan Khan, Masoud Daneshtalab

Mälardalen University, Västerås, Sweden

Balanced Learning Taco

Open paper AskPDF arXiv

Federated learning (FL) must serve devices with varying computational capabilities. A fixed model cannot suit all devices, while training one model per deployment limit is costly. Federated supernet training instead learns one elastic model with differently sized subnetworks, then deploys a suitable one to each device. When client inference budgets differ, however, parameters exclusive to high-cost subnetworks are reachable by fewer clients. We propose FEAST, a federated shared-space training framework that counters this imbalance by jointly training multiple subnetworks within each client's limit. Budget-tailored sub-supernet routing sends only the relevant supernet portion, and sparse aggregation merges the returned parameter slices. The trained supernet directly serves the subnetworks used during federation and supports post-hoc extraction of additional subnetworks without federated retraining. We further show that independently assigning clients' training-data volumes and inference budgets can distort accuracy--inference-cost comparisons in heterogeneous FL simulations, and introduce a one-parameter

γ

-allocation protocol to control this coupling. In our experimental setup, the SuperFedNAS and DeepFedNAS supernet training procedures remain near chance at 25M and reach at most

17.09\%

596

M inference MACs; FEAST reaches

71.06\%

596

2.4

points above the strongest model-heterogeneous weight-sharing baseline at its largest tier. Across CIFAR-100, CINIC-10, and TinyImageNet-200, FEAST achieves the highest population-averaged accuracy among the evaluated weight-sharing methods when each client receives its largest affordable subnetwork. Sub-supernet routing reduces aggregate model-parameter traffic by

6.8\times

relative to full-supernet transmission.

Abstract

Explore similar work

FEAST: Federated Shared-Space Training for Resource-Heterogeneous Clients

FedTopo: Relation-Level Topology Sharing for Model-Heterogeneous Federated Learning

Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation