Transmitting LoRA adapters to rollout workers can bottleneck distributed on-policy post-training over bandwidth-constrained networks. Lowering the training rank may limit the capacity a workload needs, while adaptive optimizer updates are difficult to compress. Rollout workers can instead reconstruct updates from gradients by maintaining replicated optimizer states. We introduce HiLo, a compressor-agnostic framework that replaces parameter transmission with compressed gradient dissemination and local optimizer replay. By applying the same decoded gradients, the learner and workers maintain consistent policy states at the same version under reproducible execution. We further develop Spar, a budget-aware gradient compressor that combines spectral truncation, adaptive rank allocation and error feedback to decouple training rank from communication cost. Experiments across on-policy distillation and reward-guided post-training show that HiLo with Spar maintains learning quality close to full synchronization while transmitting as little as 1.6% of the adapter bytes, and accelerates end-to-end updates by up to 2.2× under bandwidth constraints.
Figures & tables
Figure 1: HiLo with the Spar compression backend. (a) The learner compresses error-corrected LoRA-factor gradients and broadcasts zt . The learner and rollout workers decode the same gradient and apply the same optimizer transition, maintaining matching parameters and optimizer states under the conditions in § 3.1 . (b) Spar extracts spectra through small Gram eigendecompositions, greedily allocates communication ranks under a global factor-payload budget, and packs low-rank factors for transmission. Spectra and ranks are illustrative; the Gram construction shown is for aj≥bj , with the other case symmetric.
pass@4 (%) ↑
mean@4 (%) ↑
Knights and Knaves ⋅ OPD ⋅ Qwen3-1.7B
Rank k (payload)
1 (1.6%)
2 (3.3%)
4 (6.6%)
8 (13.1%)
1 (1.6%)
2 (3.3%)
4 (6.6%)
8 (13.1%)
Full Sync
83.2 ± 1.2
48.6 ± 1.0
Delta
75.4 ± 1.1
81.8 ± 2.3
80.9 ± 1.5
81.8 ± 2.4
40.6 ± 1.8
46.7 ± 3.0
44.7 ± 0.9
46.0 ± 3.0
Low-rank LoRA ( r=k )
78.0 ± 2.4
79.6 ± 3.0
81.0 ± 0.9
82.0 ± 1.0
42.8 ± 1.8
43.6 ± 1.7
45.3 ± 1.1
47.3 ± 0.3
HiLo + PowerSGD
78.8 ± 1.5
81.7 ± 0.7
80.5 ± 2.7
81.5 ± 1.3
42.9 ± 1.1
46.6 ± 0.9
45.6 ± 2.0
47.8 ± 1.9
Table 1: Held-out accuracy (%) on KK (update 300) and Countdown-4 (update 200), mean ± s.d. over three seeds. Payload is relative to fully synchronizing the rank-64 adapter. Low-rank LoRA reduces the training rank to k , whereas all HiLo variants keep rank 64 and compress only the communicated gradients; a fractional k denotes a global byte budget. Delta uses error feedback to compress post-optimizer changes at the same payload; the learner adopts the compressed weights, so workers keep no optimizer state.
Figure 2: Training reward against cumulative payload per worker for KK OPD with Qwen3-1.7B-Base over the first 300 updates. Each panel fixes the communication rank k ; the x-axis ends at the compressed methods’ 300-update total. Full Sync ships 139.5 MB per update.
Figure 3: Optimizer robustness on Countdown-4 at k=1 . HiLo + Spar is the only compressed method that stays within 3 pass@4 points of Full Sync across all four optimizers, while competing methods suffer substantial degradation under certain optimizers.
pass@4 (%) ↑
Rank k (payload)
1 (1.6%)
2 (3.1%)
Full Sync
63.6 ± 0.9
Exact, k per factor
61.4 ± 2.1
64.1 ± 0.8
PowerSGD ( k per factor)
61.4 ± 0.7
63.1 ± 0.1
Spar w/o EF
62.1 ± 0.8 2/5
62.0 ± 1.7
Spar (ours)
64.1 ± 1.3
63.9 ± 0.4
Table 2: Spar ablation on Countdown-4 at matched payload (AdamW; 5 seeds at k=1 , 3 at k=2 ). c/n : c of n seeds collapsed and are excluded. Bold marks the best compressed variant per column.
Figure 4: Per-update communication payload and pass@4 on Countdown-4. HiLo + Spar ( k=1 ) sends 62 × fewer bytes than full adapter synchronization and 17 × fewer than the strongest lossless delta codec, at comparable accuracy. Payload ratios are relative to HiLo; the lossless methods reproduce Full Sync exactly and share its accuracy.
Lockstep (lag 0) : s/update ↓
One-step-off (lag 1) : s/update ↓
Mbps/worker
Full Sync
HiLo
Speedup
Full Sync
HiLo
Speedup
unpaced
15.3 ± 0.1
15.9 ± 0.1
0.97 ×
–
–
–
125
24.2 ± 0.1
16.2 ± 0.3
1.50 ×
11.9 ± 0.0
8.1 ± 0.0
1.48 ×
62.5
33.1 ± 0.1
16.1 ± 0.2
2.06 ×
17.9 ± 0.0
8.1 ± 0.1
2.22 ×
Table 3: Measured seconds per update for KK OPD with Qwen3-1.7B-Base, one learner and eight rollout workers with the teacher. Full synchronization ships the 139.5 MB adapter while HiLo ships 2.3 MB of rank-1 gradient factors.
Figure 5: Left: lockstep seconds per update as the number of rollout workers N grows behind one learner GPU whose 1 Gbps uplink is shared by unicast, with the global batch fixed at 16 prompts × 8 rollouts. Right: share of each update spent on communication. Points are measured (one seed, updates 6–20); lines are the step-time model of this section, fitted to the measured generation time at N=4 and N=8 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Rank
Retained energy 1−τ1
Update t
mt
Λt
Rt
Gt−1
mt
Rt
1
1
4
3
1.00
1.00
1.00
2
2
48
62
1.00
0.72
0.59
3
3
64
64
1.00
0.49
0.37
10
10
64
64
1.00
0.66
0.20
30
30
64
64
1.00
0.20
0.26
Appendix
Table 4: Synthetic AdamW replay of a rank-one gradient stream on one 1024×64 factor ( scripts/appendix_rank_witness.py , seed 0); not a measurement on the trained model. rank(mt)=t attains Proposition A.2 ; Λt and Rt are numerically full rank from the third update. A rank-one truncation that loses nothing on the gradient discards 60–80% of the update’s energy.
pass@4 (%) ↑
mean@4 (%) ↑
Energy (%) ↑
LoGRA-style, lr 1×10−4
51.6 ± 0.7
32.2 ± 1.2
1.6
LoGRA-style, lr 3×10−4
46.9 ± 6.4 1/3
31.0 ± 7.2
1.6
LoGRA-style, lr 1×10−3
collapsed 3/3
–
–
HiLo + Spar (ours)
64.1 ± 1.3
51.5 ± 1.2
13.8
Appendix
Table 5: A LoGRA-style update applied to the rank-64 LoRA factors on Countdown-4 at the Spar rank-1 payload (recipe of Table 1 ; three seeds per learning rate, five for Spar). Each step sketches every factor gradient along its long side with a fresh seeded random sign matrix, keeps RowAdam state and applies the low-rank update; the sketch rank matches the payload (17 for a 1024×64 factor). Predicted-KL step control is not used, so the learning rate is swept. Energy is the mean fraction of the gradient energy each update retains, in the random sketch for LoGRA and in the shipped directions of the error-corrected gradient for Spar. c/n : c of n seeds collapsed and are excluded.
Figure 6: Fraction of the error-corrected gradient energy ∥ht∥2 that each compressor ships per step, averaged over updates 1–300 of the Qwen3-1.7B KK runs; bars are means over three seeds with sample standard deviations, labels in percent. All compressors send the same number of bytes at each rank.
Relative update error ↓
Rank k
1
2
4
8
Exact, k per factor
0.558
0.511
0.438
0.338
PowerSGD ( k per factor)
0.582
0.549
0.493
0.402
Exact, AdamW metric
0.557
0.510
0.439
0.342
Spar, AdamW metric
0.545
0.498
0.427
0.327
Spar (ours)
0.556
0.531
0.481
0.398
Appendix
Table 6: Relative error ∥U(g^)−U(g)∥/∥U(g)∥ of the AdamW update produced by each compressor’s error-corrected gradient, against the update from the full gradient. All compressors run as shadows on the same Full Sync gradients, and only steps containing every listed diagnostic are retained (200 paired steps). Spar allocates its budget by raw gradient energy; the AdamW-metric variants allocate by the moment-normalized update instead. Bold marks the lowest error per rank.
Figure 7: Countdown-4 pass@4 at update 200 against bytes per update for Full Sync every K updates and HiLo + Spar at every update ( k=1,2,4 ), with the recipe of Table 1 and one seed. Between full synchronizations, actors keep sampling from the last shipped adapter. The dotted line marks the HiLo rank-1 payload, which Full Sync matches at K=62 .
Communication-efficient distributed training reduces the data exchanged among devices so models can be trained faster and over limited-bandwidth networks.
Group-relative policy optimization (GRPO) makes learner-side activations a major memory-computation bottleneck: gradient checkpointing reduces activation memory through recomputation, but fixed schedules can leave roughly 18 GB unused on a 48-GB GPU despite substantial recomputation overhead. Existing activation-management methods set state fidelity from execution cost, tensor properties, or generic compression sensitivity, without explicitly incorporating GRPO's analytic update structure into state-fidelity allocation. We formalize this dependence as policy-update exposure, linking the current GRPO loss coefficients to state-level approximation sensitivity. These coefficients are available before backward without an additional backward pass. We introduce HiLoRe, which allocates graph-attributed recovery units among high-precision storage, low-precision compression, and deterministic recomputation using measured recovery utility and update-conditioned approximation risk. It combines high-precision storage and deterministic recomputation with low-precision recovery under a calibrated risk budget. Across five model-task settings with 2K responses and memory < 1.10 times GC's per-GPU actor-update peak, HiLoRe's actor-update throughput gains reach 13.5% over GC and 7.9% over the fastest evaluated baseline, with paired mean downstream-score differences below 0.6 percentage points.
Xinrui Chen, Mengyang Li, Ou Wu +1
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China · Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, China · University of Southern Queensland
In large-scale reinforcement learning (RL) systems with decoupled Trainer-Rollout execution, the Trainer must regularly synchronize policy weights to the Rollout side to limit policy staleness. When inter-node bandwidth is abundant, such synchronization is usually only a small fraction of end-to-end cost. As model size grows, however, the communication demand rises rapidly. In bandwidth-constrained or network-variable deployments -- for example, cross-datacenter or cross-cluster settings, heterogeneous resource pools, and online RL -- weight synchronization can become a dominant bottleneck for throughput and tail latency. We observe that, in mainstream large-model RL training, the locations where parameters actually change are highly sparse at the element level (often 99%+ sparsity). Building on this observation, we propose and implement SparseRL-Sync, which replaces full-weight transfers with a lossless sparse update payload (indices and values) that can be exactly reconstructed on the inference side, thereby preserving 100% fidelity. Under a simplified cost model, sparse synchronization reduces the per-update communication volume from S to approximately S/X; with 99% sparsity (X ~ 100), this yields about a 100x reduction in transmitted data. Combined with appropriate bucketing, SparseRL-Sync also reduces launch and control-plane overhead, significantly improving scalability and end-to-end efficiency in bandwidth-limited and highly asynchronous RL settings.
The pre-training of Large Language Models (LLMs) is increasingly conducted across multiple data centers. As training scales to a larger number of accelerators, the fraction of time spent on computation decreases, while the fraction spent on communication increases. Therefore, frequent synchronization becomes a growing bottleneck. Local update methods reduce this cost by allowing workers to perform several optimizer steps between synchronizations. Most local update methods set the number of local optimizer steps between synchronizations before training and keep this interval fixed throughout the run. However, the best interval can change during the entire train process. If the interval and optimizer are adapted to the current training state, the communication frequency is reduced while maintaining the training performance. In this work, we introduce AutoLoCo, an adaptive training framework to reduce communication in LLM training. It adapts the local interval using scalar training statistics and corrects each outer update. Our method is motivated by two observations: 1) the appropriate local interval varies across training stages, and 2) changing the number of inner steps per interval creates a mismatch with an unchanged outer optimizer, requiring a correction to the outer update. We optimize this mismatch by correction of the outer optimizer for the momentum and the learning rate using the accumulated inner learning rate. Our experiments under communication constraints demonstrate that AutoLoCo reduces communication frequency by 27% relative to DiLoCo while maintaining training performance.
Pengyu He, Yan Zhang, Ruien Li +1
Department of Electrical Engineering, California Institute of Technology · Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison