Communication-Efficient LLM Adaptation over Decentralized GPU Meshes

Sameera Ramasinghe; Shamane Siriwardhana; Thalaiyasingam Ajanthan; Hadi Mohaghegh Dolatabadi; Chamin P Hewa Koneputugodage; Gil Avraham; Violetta Shevchenko; James Snewin

cs.LGSep 13, 2026

Communication-Efficient LLM Adaptation over Decentralized GPU Meshes

Authors: Sameera Ramasinghe, Shamane Siriwardhana, Thalaiyasingam Ajanthan, Hadi Mohaghegh Dolatabadi, Chamin P Hewa Koneputugodage, Gil Avraham, Violetta Shevchenko, James Snewin,

Organizations: Pluralis Research

Abstract

Decentralized training enables large-model training over low-end GPUs and internet-grade connections, but communication along both data-parallel and pipeline-parallel axes becomes the primary bottleneck. We study post-pretraining adaptation in this setting. We propose an asynchronous two-circuit system: a fast compressed training circuit drives throughput using activation masking for pipeline-parallel (PP) transfer and compressed data-parallel (DP) synchronization, while a slow anchor circuit runs occasional unmasked forward--backward passes off the critical path. Then, we introduce a spectral correction optimizer that uses these delayed anchor priors to denoise masked gradients without blocking the fast stream. Although prior work has found aggressive activation compression unreliable, we show that masking supports post-pretraining adaptation at high compression rates when anchored this way. Pipeline-parallel compression alone yields up to a $9\times$ throughput gain, and combining it with data-parallel compression increases beyond $40\times$ over internet-grade $\sim 200$ Mbps connections, while matching dense uncompressed performance across domain adaptation and continual pretraining.

Figures & tables

Figure 1: System overview. A small unmasked anchor circuit (orange,

1{\times}Y

PP stages) asynchronously supplies clean gradients

G_{\tau}^{\mathrm{anc}}

to a spectral corrector, which denoises the masked gradients

G_{t}^{\mathrm{mask}}

from the fast circuit (blue,

(X{-}1){\times}Y

\times

PP) into corrected updates

G_{t}^{\mathrm{proj}}

. Slow circuit periodically pulls fresh weight snapshots from the fast circuit.

Domain	Dataset	Metric	Baseline	+DP	M90	M90+AP	M95	M95+AP
Medical	MedQA	Acc.	0.280	0.275	0.225	0.325	0.180	0.321
	MedMCQA	Acc.	0.330	0.325	0.275	0.335	0.220	0.320
Code	HumanEval	pass@1	0.177	0.171	0.115	0.215	0.072	0.207
	MBPP	pass@1	0.152	0.148	0.085	0.198	0.040	0.192
SQL	Spider	Exec.	0.423	0.412	0.305	0.418	0.180	0.402
Science	ARC-C	Acc.	0.378	0.374	0.328	0.380	0.295	0.355

Table 1: Per-domain task accuracy and global training efficiency for dense SFT and masked training variants. Baseline : dense PP, uncompressed DP. +DP : dense PP, compressed DP. M90/M95 :

90\%/95\%

activation masking; +AP : Anchor Priors. DP compression is applied to all the masked variants.

(a) Mask ratio $p$ vs. staleness $K$
Dataset	Mask	$K{=}10$	$K{=}20$	$K{=}30$	$K{=}50$
MedQA	$p{=}90\%$	0.330	0.328	0.310	0.265
	$p{=}95\%$	0.328	0.321	0.295	0.250
	$p{=}99\%$	0.245	0.230	0.215	0.180
HumanEval	$p{=}90\%$	0.218	0.215	0.195	0.140
	$p{=}95\%$	0.215	0.207	0.175	0.090

Table 2: Ablations over mask ratio

p

, staleness

K

, spectral-filter threshold

\tau

, and merge coefficient

\alpha

. Default (

p{=}95\%

K{=}20

\tau{=}10^{-3}

\alpha{=}0.3

) in green; (b), (c) hold others at default. Red:

>30\%

relative drop.

Model	Params	Dense	M95	M95+AP
Llama-1B	1.0B	0.177 / 0.152	0.128 / 0.032	0.171 / 0.141
Qwen-1.5B	1.5B	0.213 / 0.188	0.158 / 0.069	0.206 / 0.177
Gemma-2B	2.0B	0.201 / 0.174	0.147 / 0.055	0.194 / 0.162
Phi-2	2.7B	0.238 / 0.205	0.176 / 0.083	0.228 / 0.193
Qwen-3B	3.0B	0.268 / 0.232	0.207 / 0.098	0.260 / 0.222
OLMo-7.5B	7.5B	0.335 / 0.294	0.267 / 0.153	0.324 / 0.280

Table 3: Code-domain generalization across model families and scales. Entries are HumanEval / MBPP pass@1. M95:

95\%

activation masking; AP: Anchor Priors. M95+AP recovers most of the dense performance across all scales tested.

Table 5

Mesh	DP	PP	MedQA	HE	GSM8K	TPS
$4\times 4$	4	4	0.318	0.205	0.338	$32{,}100$
$8\times 4$	8	4	0.325	0.213	0.345	$75{,}000$
$4\times 8$	4	8	0.315	0.200	0.335	$19{,}900$
$8\times 8$	8	8	0.321	0.207	0.341	$46{,}500$
$16\times 8$	16	8	0.328	0.215	0.348	$\mathbf{99{,}600}$
$8\times 16$	8	16	0.312	0.198	0.330	$26{,}400$

Table 6: Ablation over decentralized mesh sizes. Mesh size is DP

\times

PP; one of the DP replicas is dedicated to the unmasked (clean) circuit and the remaining

\text{DP}{-}1

replicas run the masked (fast) circuit. All runs use M95+AP at

200

Mbps with the same training configuration as the main experiments. HE : HumanEval pass@1. TPS : tokens per second.

Method	IFEval	GSM8K	TPS	Speedup
Base	0.13	0.02	—	—
Dense	0.46	0.23	$1.1$ k	$1\times$
M95+AP	0.41	0.21	$46.5$ k	$43\times$

Table 7: Llama-3.2-1B trained through the full SmolLM3-style recipe (reasoning mid-training

\to

SFT

\to

second SFT round). Final model after all three phases;

60

B total tokens. M95+AP is on par with dense quality at

\sim 43\times

throughput.

Method	Trainable	PP fwd	PP bwd	Comp.	MedQA	MedMCQA	HE
Dense SFT	1.23B ( $100\%$ )	Dense	Dense	$1\times/1\times$	0.280	0.330	0.177
LoRA	3.15M ( $0.26\%$ )	Dense	Dense	$1\times/1\times$	0.265	0.318	0.168
M95+AP	1.23B ( $100\%$ )	Sparse	Sparse	$20\times/20\times$	0.321	0.320	0.207
LoRA + M95+AP	3.15M ( $0.26\%$ )	Sparse	Sparse	$20\times/20\times$	0.305	0.310	0.198

Table 8: Orthogonality of M95+AP to parameter-efficient fine-tuning on Llama 3.2 1B. LoRA reduces trainable parameters but still requires dense activation and activation-gradient transfer across PP boundaries. M95+AP reduces PP communication independently and composes with LoRA, giving both axes of efficiency at once. Comp. : forward/backward activation compression ratio. HE : HumanEval pass@1.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

SFT starting point	Pre-RL	Post-RL	$\Delta$
Dense (3-phase)	0.23	0.41	$+0.18$
M95+AP (3-phase)	0.21	0.40	$+0.19$

Appendix

Table 9: GSM8K performance before and after

2

epochs of GSPO on Big-Math, starting from either the dense or M95+AP final checkpoint. RL improvements are comparable from both starting points, indicating that masked fine-tuning leaves the model RL-amenable.

Figure 2: Alignment of $\nabla F(w_{t})$ with the rank- $r$ principal subspace of $M_{t}^{\rm anc}$ , over $1500$ training steps. $\eta_{r}=\|U^{\top}g\,V\|_{F}^{2}/\|g\|_{F}^{2}$ at $61$ snapshots, for $r\in\{16,64,256\}$ (FT solid, PT dashed). FT stays stable around $\eta_{64}^{\rm FT}\approx 0.17$ throughout; PT drifts down to $\sim 0.08$ . The separation persists at every rank, with FT/PT ratio holding in the $1.7$ – $2.7\times$ range at the spectral filter’s operating ranks.

Domain	HuggingFace dataset	# train	Notes
Medical	medalpaca/medical_meadow_medqa	9,160	USMLE-style multiple choice.
Code	iamtarun/python_code_instructions_18k_alpaca	16,750	Python instruction $\to$ code pairs.
Math	openai/gsm8k ( main )	7,473	Official GSM8K test used as validation.
Science	allenai/sciq	10,511	Question $+$ supporting passage.
Commonsense	Rowan/hellaswag $\cup$ allenai/winogrande ( xl )	72,272	Concatenated, shuffled with seed $42$ .
Summarization	cnn_dailymail ( 3.0.0 )	27,000	$30$ k subsample, $90/10$ split.

Appendix

Table 10: Per-domain training datasets after preprocessing and the

90/10

train/validation split.

Domain	Eval dataset	Metric	# items	Source
Medical	MedQA (4-options)	Accuracy	1,273	lm-eval
Medical	MedMCQA	Accuracy	—	lm-eval
Code	HumanEval	pass@1	164	lm-eval
Code	MBPP	pass@1	500	lm-eval
SQL	Spider (dev)	Execution match	1,034	Custom
Science	ARC-Challenge	Acc. (norm)	1,172	lm-eval

Appendix

Table 11: Per-domain evaluation datasets.

Method	Per-step ratio	Composes with $\tau$	Reported convergence cost
FP8/INT8 quantization	$2$ – $4\times$	yes	$\leq 1\%$ loss gap
1-bit Adam / signSGD	$32\times$	yes	$1$ – $3\%$ loss gap
Top- $K$ sparsification	$50$ – $100\times$	yes	$1$ – $5\%$ loss gap; index overhead
PowerSGD (rank 4)	$32$ – $256\times$	yes	$\leq 2\%$ loss gap
Local SGD / DiLoCo	— ( $\tau$ -based)	—	$\leq 1\%$ loss gap up to $\tau{=}500$
Streaming DiLoCo	— ( $\tau$ -based)	—	$\leq$ DiLoCo, hides comm

Appendix

Table 12: DP-compression methods with demonstrated stability for language-model training. Per-step compression refers to bytes-on-wire relative to a full BF16 all-reduce. Frequency-reduction methods do not have a per-step compression ratio because they reduce the number of all-reduces, not the bytes per all-reduce; they are stacked with F1–F3 rather than competing with them.

Activation Compression

See all 11 papers

Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
Jun 2, 2025Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham +2Communication-Efficient Distributed Training Decentralized Learning
Pluralis Research
Unifying Local Communications and Local Updates for LLM Pretraining
Jun 9, 2026Pietro Cagnasso, Eugene Belilovsky, Edouard OyallonLanguage Model Pretraining Communication-Efficient Distributed Training
Concordia University · Mila · CNRS, Sorbonne University
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
Jul 2, 2026Mingkai Zheng, Junlin Chen, Haotian Xie +1Language Model Pretraining Communication-Efficient Distributed Training
Rutgers University

Jun 2, 2025J/K move · Enter open · S save

Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism

Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham, Yan Zuo+1

Pluralis Research

Communication-Efficient Distributed Training Decentralized Learning Activation Compression Distributed Training Neural Network Compression

Open paper AskPDF arXiv

Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data-parallel, they do not extend to model parallelism. Unlike data-parallel training, where weight gradients are exchanged, model-parallel requires compressing activations and activation gradients as they propagate through layers, accumulating compression errors. We propose a novel compression algorithm that compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation with negligible memory/compute overhead. By leveraging a recursive structure in transformer networks, we predefine a low-dimensional subspace to confine the activations and gradients, allowing full reconstruction in subsequent layers. Our method achieves up to 100x improvement in communication efficiency and enables training billion-parameter-scale models over low-end GPUs connected via consumer-grade internet speeds as low as 80Mbps, matching the convergence of centralized datacenter systems with 100Gbps connections with model parallel.

Abstract

Activation Compression

Explore similar work

Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism

Unifying Local Communications and Local Updates for LLM Pretraining

SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication