Distributed Subliminal Learning: Replacing Model Updates with Random-Carrier Outputs

Dario Fenoglio; Gabriele Dominici; Martin Gjoreski; Marc Langheinrich

cs.LGOct 4, 2026

Distributed Subliminal Learning: Replacing Model Updates with Random-Carrier Outputs

Authors: Dario Fenoglio, Gabriele Dominici, Martin Gjoreski, Marc Langheinrich

Organizations: Università della Svizzera italiana (USI), Lugano, Switzerland · MIT Media Lab, Massachusetts Institute of Technology, Cambridge, MA, USA

Abstract

Collaborative learning typically exchanges model parameters: federated clients communicate updates, while independently adapted foundation models are combined by exchanging adapters or checkpoints. This makes communication scale with model size and requires local specializations to be reconciled in weight space, where interference is common. We ask whether knowledge can instead be shared through model behavior on task-unrelated inputs. We introduce Distributed Subliminal Learning (DSL), a collaborative learning primitive in which participants adapt a common model locally, probe it with task-unrelated inputs, and transmit only the resulting carrier outputs. A coordinator pools these outputs and distills them into a shared model. The primitive supports one-shot foundation-model composition through carrier completions and iterative federated learning through carrier logits, without transmitting model updates or requiring task-related proxy data. In LLM composition, compared with LoRA averaging, DSL achieves higher preference retention (94.56% vs. 87.76%) and a larger GSM8K gain over the base model (22.0 vs. 0.6 points), while reducing upload by 30.6-49.0 $\times$ . In federated classification, DSL reaches 96.83% on MNIST with 8.9 $\times$ less uplink than FedAvg and provides lower-communication operating points on CIFAR-10 and Tiny ImageNet. These results establish random-carrier outputs as a practical communication primitive for knowledge sharing across distinct collaborative learning paradigms.

Figures & tables

Figure 1: Distributed Subliminal Learning. Participants adapt a common base model on local data and probe it with task-unrelated random carriers. Instead of model updates or adapters, participants transmit only carrier outputs, illustrated here with random-number completions. The coordinator pools these messages and distills a shared model, combining local specializations in function space. The same principle applies to sampled text completions, probabilities, or logits, and can be used either once for model composition or iteratively in federated learning.

Preference	Local	LoRA Avg.	DSL	Upload (MiB)	Reduction
Cat	98.23	95.60	94.90	2.52	$30.6\times$
Pasta	96.07	85.27	95.00	1.85	$41.6\times$
Maple	98.33	82.40	93.77	1.58	$48.8\times$
Average	97.54	87.76	94.56	–	–

Table 1: Controlled preference composition with Qwen2.5-7B-Instruct. Higher retention is better. DSL upload is measured per participant; reduction is relative to the LoRA payload.

Model	Acc. ( $\%$ )	$\Delta$ Base (pp)	Upload (MiB)	Reduction
Base model	10.2	–	–	–
Local models (avg.)	58.9	+48.7	–	–
LoRA Avg.	10.8	+0.6	35.27	$1.0\times$
DSL	32.2	+22.0	0.72	$\mathbf{49.0\times}$

Table 2: GSM8K composition with Qwen2.5-1.5B-Instruct.

\Delta

reports the accuracy gain over the common base model; upload is measured per participant.

Figure 2: MNIST test accuracy versus total uplink. Increasing the carrier budget moves FSL along an explicit accuracy–communication frontier under the same accounting as the baselines.

	MNIST		CIFAR-10		Tiny ImageNet
Method	Acc. (%) $\uparrow$	Uplink (GiB) $\downarrow$	Acc. (%) $\uparrow$	Uplink (GiB) $\downarrow$	Acc. (%) $\uparrow$	Uplink (GiB) $\downarrow$
FedAvg	$97.34\pm 0.32$	$2.37\pm 0.473$	$66.25\pm 0.67$	$14.99\pm 1.47$	$68.12\pm 0.05$	$0.726\pm 0.048$
scaffold	$94.62\pm 0.22$	$8.07\pm 1.12$	$56.00\pm 2.60$	$85.72\pm 18.40$	$65.35\pm 0.73$	$19.22\pm 2.66$
dfrd	$97.24\pm 0.27$	$2.18\pm 0.485$	$66.46\pm 0.61$	$11.94\pm 0.283$	$68.13\pm 0.18$	$0.685\pm 0.189$
FedDF	$96.65\pm 0.13$	$1.61\pm 0.230$	$66.11\pm 0.60$	$10.14\pm 1.58$	$66.92\pm 0.26$	$0.489\pm 0.169$
FedGen	$97.26\pm 0.24$	$2.15\pm 0.408$	$66.39\pm 1.16$	$9.32\pm 3.22$	$61.63\pm 0.52$	$0.182\pm 0.024$

Table 3: Test accuracy and total uplink across datasets, reported as mean

\pm

standard deviation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Federated Subliminal Learning. In each communication round, clients train locally, probe their models with task-unrelated carriers, and upload only the resulting carrier outputs. The server pools these outputs and distills the next global model, without receiving model updates or task-related proxy data.

$M$	Protocol	Carrier	Accuracy (%) $\uparrow$	Total uplink $\downarrow$
4,096	Private	Uniform	$41.60\pm 15.42$	$144.90\pm 112.31$ MiB
4,096	Private	Blurred	$55.44\pm 0.76$	$96.60\pm 13.06$ MiB
4,096	Private	Fourier	$39.38\pm 12.63$	$120.52\pm 145.48$ MiB
4,096	Shared	Uniform	$14.58\pm 0.52$	$27.08\pm 10.22$ MiB
4,096	Shared	Blurred	$51.17\pm 8.64$	$193.65\pm 83.49$ MiB
4,096	Shared	Fourier	$35.87\pm 2.14$	$31.15\pm 15.38$ MiB

Appendix

Table 4: Complete CIFAR-10–ResNet9 carrier sweep. Accuracy and total synthetic uplink are reported as mean

\pm

standard deviation. “Shared-w.” denotes weighted shared-carrier averaging.

$M$	Protocol	Carrier	Accuracy (%) $\uparrow$	Total uplink $\downarrow$
2,048	Private	Uniform	$96.28\pm 1.21$	$175.10\pm 58.39$ MiB
2,048	Private	Blurred	$94.98\pm 0.06$	$153.02\pm 8.46$ MiB
2,048	Private	Fourier	$93.78\pm 0.24$	$109.69\pm 16.63$ MiB
2,048	Shared	Uniform	$92.70\pm 0.34$	$192.97\pm 41.23$ MiB
2,048	Shared	Blurred	$88.34\pm 0.81$	$77.19\pm 19.59$ MiB
2,048	Shared	Fourier	$82.94\pm 4.16$	$52.81\pm 35.42$ MiB

Appendix

Table 5: Complete MNIST–MLP carrier sweep. Accuracy and total synthetic uplink are reported as mean

\pm

standard deviation. “Shared-w.” denotes weighted shared-carrier averaging.

$M$	Protocol	Carrier	Accuracy (%) $\uparrow$	Total uplink $\downarrow$
1,024	Private	Uniform	$75.58\pm 33.80$	$116.12\pm 104.89$ MiB
1,024	Private	Blurred	$95.09\pm 0.87$	$196.35\pm 28.04$ MiB
1,024	Private	Fourier	$94.98\pm 0.51$	$179.09\pm 6.12$ MiB
1,024	Shared	Uniform	$13.95\pm 4.15$	$3.05\pm 1.02$ MiB
1,024	Shared	Blurred	$13.25\pm 1.89$	$3.39$ MiB $\pm 600.44$ KiB
1,024	Shared	Fourier	$36.20\pm 40.92$	$89.04\pm 148.94$ MiB

Appendix

Table 6: Complete MNIST–LeNet5 carrier sweep. Accuracy and total synthetic uplink are reported as mean

\pm

standard deviation. “Shared-w.” denotes weighted shared-carrier averaging.

$M$	Protocol	Carrier	Accuracy (%) $\uparrow$	Total uplink $\downarrow$
64	Private	Uniform	$60.54\pm 0.58$	$33.53\pm 2.03$ MiB
128	Private	Uniform	$61.84\pm 0.35$	$44.27\pm 17.51$ MiB
256	Private	Uniform	$62.74\pm 0.32$	$74.22\pm 14.08$ MiB
512	Private	Blurred	$60.12\pm 0.19$	$101.56\pm 0.00$ MiB
512	Private	Uniform	$63.18\pm 0.32$	$83.33\pm 16.26$ MiB
1,024	Private	Blurred	$60.46\pm 0.33$	$151.04\pm 47.74$ MiB

Appendix

Table 7: Private representation-space carrier sweep on Tiny ImageNet using 50k training samples and a frozen ViT-small feature extractor. Accuracy and total synthetic uplink are reported as mean

\pm

standard deviation. The

M=1{,}024

uniform configuration is the Tiny ImageNet FSL setting reported in Table 3 .

$M$	Uniform (%)	Blurred (%)
512	$\mathbf{61.40\pm 0.45}$	$57.45\pm 0.25$
1,024	$\mathbf{62.04\pm 0.28}$	$58.15\pm 0.10$
2,048	$\mathbf{61.67\pm 0.36}$	$58.28\pm 0.20$
4,096	$\mathbf{61.34\pm 0.36}$	$58.69\pm 0.19$

Appendix

Table 8: Tiny ImageNet representation-space carrier comparison with a reduced 30k-sample training pool. Results are mean

\pm

standard deviation.

Model	Protocol	$M$	Carrier	$H_{\mathrm{query}}$ (%)	$H_{\mathrm{marg}}$ (%)
ResNet9	Private	4,096	Uniform	$31.11\pm 4.79$	$75.95\pm 1.91$
ResNet9	Private	4,096	Blurred	$43.31\pm 2.47$	$84.33\pm 2.20$
ResNet9	Private	4,096	Fourier	$38.65\pm 0.04$	$79.68\pm 4.48$
ResNet9	Shared-w.	32,768	Uniform	$41.42\pm 4.46$	$53.49\pm 5.32$
ResNet9	Shared-w.	32,768	Blurred	$49.40\pm 0.99$	$91.98\pm 1.33$
ResNet9	Shared-w.	32,768	Fourier	$42.82\pm 4.34$	$88.05\pm 1.86$

Appendix

Table 9: Teacher-output diagnostics on CIFAR-10.

H_{\mathrm{query}}

is the normalized mean per-carrier class entropy and

H_{\mathrm{marg}}

is the normalized entropy of the mean class distribution. Results are mean

\pm

standard deviation.

Figure 4: Accuracy–communication trade-off as the number of ghost outputs is varied on MNIST. Marker size is proportional to the number of ghosts. Error bars report the standard error.

Explore similar work

Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
Jul 31, 2026Shengkun Zhu, Jinshan Zeng, Zhihua Allen-Zhao +5Fedavg Parameter-Efficient Fine-Tuning Methods
The Hong Kong Polytechnic University · OceanBase, Ant Group · School of Management, Xi’an Jiaotong University +3
Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs
May 12, 2026Amr Abourayya, Jens Kleesiek, Michael KampLarge Language Model Fine-Tuning Model Fine-Tuning
Lamarr Institute for ML and AI, Technical University Dortmund · Institute for AI in medicine (IKIM),University Hospital Essen
TallyTrain: Communication-Efficient Federated Distillation
Jun 30, 2026Radhakrishna Achanta, Will ReedFederated Learning Dataset Distillation
Cisco Systems Inc.

Jul 31, 2026, cs.LGJ/K move · Enter open · S save

Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients

Shengkun Zhu, Jinshan Zeng, Zhihua Allen-Zhao, Mayi Xu+4

The Hong Kong Polytechnic University · OceanBase, Ant Group · School of Management, Xi’an Jiaotong University+3

Fedavg Parameter-Efficient Fine-Tuning Methods Wireless Foundation Models Model Fine-Tuning Foundation Model Clients

Open paper AskPDF arXiv

Federated learning of foundation models faces a fundamental resource-asymmetry challenge: the institutions holding the most valuable domain-specific data cannot host billion-parameter models. Existing heterogeneous federated approaches attempt to bridge this gap through parameter-efficient tuning, model pruning, or knowledge distillation, yet each trades away a critical property, whether full-model memory reduction, architectural self-containedness, or representational fidelity, leaving the core tension unresolved. We propose FedSLM, a parameter-centric framework for federated fine-tuning with heterogeneous compressed clients. FedSLM uses SVD-based decomposition to produce self-contained client models, whose low-rank subspaces form nested manifolds that are structurally compatible for aggregation. It then applies a two-stage protocol that synchronizes lightweight adapters within compression groups and fuses full-rank reconstructions across groups via structural alignment. Finally, a weak-to-strong elicitation step with auxiliary confidence loss transfers the aggregated knowledge to the full-scale server, while an explicit bias--variance trade-off mitigates compression artifacts. We provide theoretical guarantees for adapter-level aggregation, subspace-alignment bounds for cross-group fusion, and a characterization of how the confidence loss mitigates weak-supervision noise. Experiments on natural language and vision--language benchmarks show that FedSLM outperforms existing federated baselines under both IID and non-IID partitions, while client models operate at roughly 50% of the GPU memory required by the full model.

Abstract

Explore similar work

Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients

Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs

TallyTrain: Communication-Efficient Federated Distillation