Unlocking large-scale low-bandwidth decentralized training has the potential to utilize otherwise untapped compute resources. In centralized settings, large-scale multi-node training is primarily enabled by data and pipeline parallelism, two techniques that require ultra-high-bandwidth communication. While efficient methods now exist for decentralized data parallelism, pipeline parallelism remains the primary challenge. Recent efforts, such as Subspace Models (SM), have claimed up to 100x activation compression but rely on complex constrained optimization and diverge from true end-to-end training. In this paper, we propose a different approach, based on an architecture designed from the ground up to be native to low-bandwidth communication environments while still applicable to any standard transformer-based architecture. We call this architecture the Residual Bottleneck Model or ResBM, it introduces a residual encoder-decoder bottleneck module across pipeline boundaries that can be trained end-to-end as part of the model's parameters while preserving an explicit low-rank identity path. We show that ResBMs achieve state-of-the-art 128x activation compression without significant loss in convergence rates and without significant memory or compute overhead.
Figures & tables
Figure 1: Singular value analysis of projection matrices. AdamW exhibits rapid singular value decay (low-rank structure), while Muon maintains larger singular values across more dimensions, preserving a higher-rank activation subspace.
Hidden Dimension
4096
Vocabulary Size
50K
Context Length
1024
Weight Decay
0.01
Peak Learning Rate
2e-4
Warmup Steps
500
Batch Size
32
Table 1: Baseline model and training configuration used across experiments unless otherwise specified.
Figure 2: Comparative performance on C4. We evaluate ResBM and SM with 2B parameters each across both AdamW and Muon optimizers against uncompressed baselines.
Model
Optimizer
Compression
Bottleneck Dim ( h )
Final PPL
Baseline
AdamW
1×
4096
21.75
ResBM (Ours)
Muon
100×
40
21.60
ResBM (Ours)
Muon
128×
32
21.77
Table 2: Final perplexity after 26B training tokens on C4. Compressed models utilize the Muon optimizer to maintain subspace rank.
Figure 3: Pretraining loss on C4 over 26B tokens. Compressed ResBM variants ( 100× and 128× ) optimized with Muon match the final performance of the uncompressed AdamW baseline, demonstrating the effectiveness of the identity-preserving bottleneck.
Configuration
Bandwidth
Compression
TPS
Final Loss
Centralized baseline
10 Gbps
1×
7530
3.86
Decentralized baseline
80 Mbps
1×
609
5.28
ResBM AdamW ( 100× )
80 Mbps
100×
7681
4.10
ResBM Muon ( 100× )
80 Mbps
100×
6795
4.03
ResBM AdamW ( 128× )
80 Mbps
128×
7837
4.05
ResBM Muon ( 128× )
80 Mbps
128×
6795
3.98
Table 3: Training throughput (tokens per second) and final training loss on C4 for the 2B model under pipeline parallelism across 8 A10G GPUs after 12 hours of training.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Training loss versus wall-clock time on C4 over the first 35M tokens for all runs. ResBM at 80 Mbps closely matches the centralized baseline at 10 Gbps, while the uncompressed decentralized configuration converges substantially slower and with greater instability.
Figure A.2: Throughput gain of ResBM ( 128× compression) relative to the uncompressed AdamW baseline as a function of inter-node bandwidth. Gains are negligible at high bandwidths where compute dominates, and rise sharply below ∼800 Mbps where communication becomes the bottleneck, reaching ≈12.8× (AdamW) and ≈11.2× (Muon) at 80 Mbps.
128 × ( b=32 )
Layer
Muon reff
AdamW reff
1
19.95
10.26
2
11.66
14.97
3
16.11
14.24
4
12.41
14.19
5
12.12
11.94
Appendix
Table B.1: SVD effective rank ( reff ) of the bottleneck output projection Wfc5 at each layer, for Muon and AdamW at 128× compression.
Figure B.1: Normalised SVD singular value spectra for Muon vs. AdamW at 128× compression ( Wfc5 , layers 1–7). Each subplot shows one layer; the x -axis is the singular value index and the y -axis is the singular value normalised by σ1 .
Figure C.1: Bottleneck layer placement across a pipeline-parallel communication boundary. The bottleneck follows the feed-forward (FF) layer and is implemented as an autoencoder with an encoder El and decoder Dl+1 placed on opposite sides of the boundary between layers l and l+1 . Only the compressed activation bl=El(⋅) is transmitted, reducing communication from H to h≪H . In the figure, H is set to 2048 and h is referred to as x . Identity pathways are preserved by incorporating the encoder into the residual function of layer l (together with FF), and the decoder into the local residual branch of layer l+1 (preceding attention). Projection operators Plid and Pl+1id align dimensionalities on the identity paths so that residual additions remain well-defined under the generalized update in Equation 2 .
Figure C.2: Muon baseline outperforms AdamW baseline. Consistent with state-of-the-art findings, the Muon-optimized baseline (no compression) outperforms the AdamW baseline. We maintain the AdamW baseline as our primary target performance for comparison, as discussed in Section 3.4 . As observed, the Muon baseline undergoes a lower level of rank collapse; while this yields superior full-rank performance, it arguably makes the model less compressible, as the features do not naturally concentrate into the low-dimensional subspace favored by the bottleneck.