Unlocking large-scale low-bandwidth decentralized training has the potential to utilize otherwise untapped compute resources. In centralized settings, large-scale multi-node training is primarily enabled by data and pipeline parallelism, two techniques that require ultra-high-bandwidth communication. While efficient methods now exist for decentralized data parallelism, pipeline parallelism remains the primary challenge. Recent efforts, such as Subspace Models (SM), have claimed up to 100x activation compression but rely on complex constrained optimization and diverge from true end-to-end training. In this paper, we propose a different approach, based on an architecture designed from the ground up to be native to low-bandwidth communication environments while still applicable to any standard transformer-based architecture. We call this architecture the Residual Bottleneck Model or ResBM, it introduces a residual encoder-decoder bottleneck module across pipeline boundaries that can be trained end-to-end as part of the model's parameters while preserving an explicit low-rank identity path. We show that ResBMs achieve state-of-the-art 128x activation compression without significant loss in convergence rates and without significant memory or compute overhead.
Figures & tables
Figure 1: Singular value analysis of projection matrices. AdamW exhibits rapid singular value decay (low-rank structure), while Muon maintains larger singular values across more dimensions, preserving a higher-rank activation subspace.
Hidden Dimension
4096
Vocabulary Size
50K
Context Length
1024
Weight Decay
0.01
Peak Learning Rate
2e-4
Warmup Steps
500
Batch Size
32
Table 1: Baseline model and training configuration used across experiments unless otherwise specified.
Figure 2: Comparative performance on C4. We evaluate ResBM and SM with 2B parameters each across both AdamW and Muon optimizers against uncompressed baselines.
Model
Optimizer
Compression
Bottleneck Dim ( h )
Final PPL
Baseline
AdamW
1×
4096
21.75
ResBM (Ours)
Muon
100×
40
21.60
ResBM (Ours)
Muon
128×
32
21.77
Table 2: Final perplexity after 26B training tokens on C4. Compressed models utilize the Muon optimizer to maintain subspace rank.
Figure 3: Pretraining loss on C4 over 26B tokens. Compressed ResBM variants ( 100× and 128× ) optimized with Muon match the final performance of the uncompressed AdamW baseline, demonstrating the effectiveness of the identity-preserving bottleneck.
Configuration
Bandwidth
Compression
TPS
Final Loss
Centralized baseline
10 Gbps
1×
7530
3.86
Decentralized baseline
80 Mbps
1×
609
5.28
ResBM AdamW ( 100× )
80 Mbps
100×
7681
4.10
ResBM Muon ( 100× )
80 Mbps
100×
6795
4.03
ResBM AdamW ( 128× )
80 Mbps
128×
7837
4.05
ResBM Muon ( 128× )
80 Mbps
128×
6795
3.98
Table 3: Training throughput (tokens per second) and final training loss on C4 for the 2B model under pipeline parallelism across 8 A10G GPUs after 12 hours of training.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Training loss versus wall-clock time on C4 over the first 35M tokens for all runs. ResBM at 80 Mbps closely matches the centralized baseline at 10 Gbps, while the uncompressed decentralized configuration converges substantially slower and with greater instability.
Figure A.2: Throughput gain of ResBM ( 128× compression) relative to the uncompressed AdamW baseline as a function of inter-node bandwidth. Gains are negligible at high bandwidths where compute dominates, and rise sharply below ∼800 Mbps where communication becomes the bottleneck, reaching ≈12.8× (AdamW) and ≈11.2× (Muon) at 80 Mbps.
128 × ( b=32 )
Layer
Muon reff
AdamW reff
1
19.95
10.26
2
11.66
14.97
3
16.11
14.24
4
12.41
14.19
5
12.12
11.94
Appendix
Table B.1: SVD effective rank ( reff ) of the bottleneck output projection Wfc5 at each layer, for Muon and AdamW at 128× compression.
Figure B.1: Normalised SVD singular value spectra for Muon vs. AdamW at 128× compression ( Wfc5 , layers 1–7). Each subplot shows one layer; the x -axis is the singular value index and the y -axis is the singular value normalised by σ1 .
Figure C.1: Bottleneck layer placement across a pipeline-parallel communication boundary. The bottleneck follows the feed-forward (FF) layer and is implemented as an autoencoder with an encoder El and decoder Dl+1 placed on opposite sides of the boundary between layers l and l+1 . Only the compressed activation bl=El(⋅) is transmitted, reducing communication from H to h≪H . In the figure, H is set to 2048 and h is referred to as x . Identity pathways are preserved by incorporating the encoder into the residual function of layer l (together with FF), and the decoder into the local residual branch of layer l+1 (preceding attention). Projection operators Plid and Pl+1id align dimensionalities on the identity paths so that residual additions remain well-defined under the generalized update in Equation 2 .
Figure C.2: Muon baseline outperforms AdamW baseline. Consistent with state-of-the-art findings, the Muon-optimized baseline (no compression) outperforms the AdamW baseline. We maintain the AdamW baseline as our primary target performance for comparison, as discussed in Section 3.4 . As observed, the Muon baseline undergoes a lower level of rank collapse; while this yields superior full-rank performance, it arguably makes the model less compressible, as the features do not naturally concentrate into the low-dimensional subspace favored by the bottleneck.
Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data-parallel, they do not extend to model parallelism. Unlike data-parallel training, where weight gradients are exchanged, model-parallel requires compressing activations and activation gradients as they propagate through layers, accumulating compression errors. We propose a novel compression algorithm that compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation with negligible memory/compute overhead. By leveraging a recursive structure in transformer networks, we predefine a low-dimensional subspace to confine the activations and gradients, allowing full reconstruction in subsequent layers. Our method achieves up to 100x improvement in communication efficiency and enables training billion-parameter-scale models over low-end GPUs connected via consumer-grade internet speeds as low as 80Mbps, matching the convergence of centralized datacenter systems with 100Gbps connections with model parallel.
Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham +2
Pretraining language models with extended context windows enhances their ability to leverage rich information during generation. Existing methods split input sequences into chunks, broadcast them across multiple devices, and compute attention block by block which incurs significant communication overhead. While feasible in high-speed clusters, these methods are impractical for decentralized training over low-bandwidth connections. We propose a compression method for communication-efficient context parallelism in decentralized settings, achieving a remarkable compression rate of over 95% with negligible overhead and no loss in convergence. Our key insight is to exploit the intrinsic low-rank structure of activation outputs by dynamically constraining them to learned mixtures of subspaces via efficient reparameterizations. We demonstrate scaling billion-parameter decentralized models to context lengths exceeding 100K tokens on networks as slow as 300Mbps, matching the wall-clock convergence speed of centralized models on 100Gbps interconnects.
Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +5
Pipeline parallelism enables training of large language models that exceed single-device memory, yet inter-stage activation communication becomes the dominant bottleneck when trained on low-bandwidth networks. Recent work in this area has proposed using fixed orthogonal projections to compress activations. However, this still results in a significant performance degradation and requires a number of non-standard adaptations to constrain the optimization. A natural alternative is to learn a low rank projection for each pipeline stage, however maintaining the necessary orthogonality of these projectors during training remains a challenge. We present Manifold Aware Projection Learning (MAPL), a method that treats inter-stage compression as a learnable orthogonal projection under explicit Stiefel manifold (orthogonal matrices) constraints. Rather than prescribing a fixed global subspace, MAPL lets each pipeline stage discover and continuously adapt its own task-optimal compression subspace via manifold-constrained steepest descent. To recover token-specific signals at stage boundaries, we introduce per-stage factorized anchor embeddings that allow for full-rank activation reconstruction with negligible communication overhead. We further show that we can incorporate residual vector quantization after projection with a streaming codebook synchronization protocol that amortizes dictionary communication. Across LLaMA models from 150M to 1B parameters we show that MAPL can be easily applied to the existing pipeline and can achieve high compression with neglibile performance degradation with a drastically improved tradeoffs in performance vs. compression compared to Subspace Networks.
Paul Janson, Edouard Oyallon, Eugene Belilovsky
Concordia University · Mila Quebec AI Institute · CNRS, Sorbonne University