Greedy block-wise local learning splits a network into gradient-isolated blocks trained by local auxiliary losses, deleting the backward pass between blocks: inter-stage communication becomes forward-only and every block can step its optimizer independently, properties directly relevant to decentralized model-parallel training. Local learning is competitive with end-to-end backpropagation on image classification, and on small Transformers it is known to trade a worse best loss for parallel speedup. How this loss gap behaves in autoregressive language model (LM) pretraining at larger scale, and which auxiliary designs reduce it, has not been measured. We present a token-budget-matched empirical study at 125M and 400M parameters with K∈{1,2,4} blocks at Chinchilla-optimal budgets, factorizing the auxiliary design into network architecture and training objective. We observe: (i) the gap to end-to-end training more than doubles from K=2 to K=4, but at K=4 shrinks from 125M to 400M; (ii) replacing an MLP auxiliary with a Transformer-based one is a strong network-side intervention, recovering 22-41% of the gap; (iii) a multi-token-prediction (MTP) auxiliary objective helps at the first block boundary, whereas adding it at deeper boundaries hurts, and restricting it to the first block yields the best K=4 configuration (+0.062 vs. +0.075 nats at 400M); and (iv) deployment-style per-block execution reduces activation memory by up to 2.2×. We frame these results as an empirically grounded method direction rather than a finalized method: local objectives should apply future-predictive pressure selectively across boundaries while resisting shortcuts that bypass predictive content.
Figures & tables
MLP-aux
Transformer-aux
Transformer-aux + MTP
125M-class (end-to-end K=1 : 2.771)
K=2 ( Δ )
+0.046
+0.028
+0.017
K=4 ( Δ )
+0.132
+0.078
+0.074
400M-class (end-to-end K=1 : 2.418)
K=2 ( Δ )
+0.041
+0.032
+0.027
K=4 ( Δ )
+0.112
+0.075
+0.062
Table 1: Validation NLL, absolute for end-to-end ( K=1 ) and as Δ against it for K∈{2,4} (nats/token; lower is better). Transformer-aux + MTP denotes first-block MTP; the all-block comparison is in Table 3 .
Figure 1: Per-block local losses at 125M, K=4 , under MLP-aux (left) and Transformer-aux (right). The curves collapse: blocks after the first barely improve on the representation they inherit, and a larger head lowers all curves without separating them.
Table 3
Figure 2: Validation NLL at 400M for K=2 (left) and K=4 (right). At K=4 the first-block placement improves over both Transformer-aux and the all-block MTP weighting throughout training.
125M-class
400M-class
Method
K=2
K=4
K=2
K=4
End-to-end ( K=1 )
14.50/13.98
29.16/27.68
MLP-aux
11.03/10.40
9.52/8.56
19.07/17.43
14.52/12.41
Transformer-aux
12.05/11.07
10.32/9.25
20.49/18.33
15.67/13.36
Transformer-aux + MTP (first block only)
12.37/11.24
10.36/9.26
21.02/18.56
15.76/13.37
Table 4: Peak/activation GPU memory per worker (GiB; single GPU, batch 4×4,096 , bf16). Non-E2E entries hold one block and its auxiliary head and report the largest stage. MTP uses sequential per-offset computation ( Gloeckle et al., 2024 ) .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Per-block local losses at 400M, K=4 , under MLP-aux (left) and Transformer-aux (right). As at 125M, later blocks provide only small improvements over the first block, and the Transformer-based auxiliary lowers the curves without separating them.
Scheme
Forward
Backward
Total
Update locking
Pipeline Parallelism (backprop)
bd
bd
2bd
locked
+ Subspace Compression ( Ramasinghe et al., 2025 )
br
br
2br
locked
Local Learning
bd
0
bd
none
+ Subspace Compression (fwd only)
br
0
br
none
Appendix
Table 5: Analytical inter-stage communication per token per stage boundary ( d : model width; b : bytes per element, 2 for bf16; r≪d : compression rank).
125M-class
400M-class
Per auxiliary head
vocabulary projection alone
16.0%
7.1%
MLP-aux
19.1%
9.0%
Transformer-aux
26.1%
12.9%
MTP head
45.2%
21.9%
Total, K=4 (three auxiliary blocks)
Appendix
Table 6: Analytical training FLOPs of the auxiliary heads as a percentage of end-to-end training FLOPs per token at sequence length 4,096. End-to-end training costs 0.92 and 2.75 GFLOP/token at 125M and 400M. At K=2 there is one auxiliary block, so the total equals the per-head value. At K=4 the total sums over three auxiliary blocks, and first-block MTP places an MTP head at the first and Transformer-aux heads at the other two. Totals are computed before rounding.
Configuration
125M-class
400M-class
Transformer-aux (NTP only, reference)
+0.078
+0.075
first-block MTP
+0.074
+0.062
all-block MTP, linear ramp
+0.082
+0.086
all-block MTP, uniform
+0.085
—
Appendix
Table 7: MTP placement/weighting ablation at K=4 : Δ validation NLL vs. end-to-end (nats/token). The all-block uniform configuration was not run at 400M.
Auxiliary
val NLL
Δ vs. end-to-end (2.771)
CPC only
7.33
collapsed
NTP + CPC hybrid
2.925
+0.154
MLP-aux
2.816
+0.046
Transformer-aux
2.799
+0.028
Transformer-aux + MTP
2.788
+0.017
Appendix
Table 8: Feature-space contrastive auxiliaries at 125M, K=2 (one cut), against the token-space auxiliaries of Table 1 . The pure-CPC run collapsed into a position encoder and was stopped at step 4,000 of 5,100; its auxiliary InfoNCE ( ∼0.28 nats at the stop) sits far below the same-sequence chance level while the final head never learns.
Large language models are trained with backpropagation, whose global gradient coordinates all layers but forces each to hold its activations and wait for the gradient to pass back through every deeper layer. Conventional local learning removes this update locking by training each module to predict the target through its own readout, but has not scaled to billion-parameter pretraining. We identify these private readouts as a key weakness, since they leave each module without information from deeper modules. We propose Shared-Output LOcal learning (SOLO), which replaces them with a shared, read-only copy of the final module's readout, the only one trained on the output of the whole network. Taken from the previous step, the copy transmits information from the final module without passing gradients between modules or reintroducing update locking. SOLO approaches backpropagation on Transformers of 340M to 2B parameters pretrained on 15B tokens, staying within one point in average zero-shot accuracy with a perplexity gap that narrows with scale. Readout ablations attribute SOLO's improvement over private readouts to sharing. Without update locking, each of p pipeline stages holds activations for O(1) micro-batches instead of O(p). The freed memory permits larger micro-batches, which reach up to 1.44x the best measured throughput of pipeline backpropagation on the same partition. To our knowledge, SOLO is the first local learning method to show such memory and throughput gains in billion-parameter language-model pretraining. Local learning thus becomes a practical alternative to backpropagation for large-scale pretraining.
Bojian Yin, Shurong Wang, Yuqi Pan +1
Institute of Automation, Chinese Academy of Sciences
A causal-decoder block is hierarchical: lower layers build the residual basis that upper layers attend over. We identify a failure mode in GPT pretraining: upper layers commit to sharp attention patterns before lower-layer features stabilize. We call this premature upper-layer attention specialization. Temporarily slowing only upper-layer Q/K projections during early training improves final perplexity and downstream accuracy without altering other parameters; it prevents upper attention from collapsing onto an immature residual basis. In LLaMA-style blocks, the same intervention is nearly unnecessary. Through ablations, we isolate multiplicative gated FFNs (not RMSNorm or bias removal) as the component that suppresses the upstream residual writes driving the failure. A pathwise analysis unifies both findings: the learning-rate intervention reduces a step-size factor, while gated FFNs reduce a residual-energy factor on the same growth pathway. Our results identify upper-layer Q/K timing as a concrete interaction point between decoder architecture and optimization.
Jinchang Zhu, Jindong Li, Yuwen Hao +3
The Hong Kong University of Science and Technology (Guangzhou)
We ask whether language-model pre-training can be decomposed into smaller, independently trainable jobs that can later be recomposed into a coherent larger model. We introduce Mixture of Training (MoT), a scaffolded modular pre-training procedure that partitions a target Transformer into contiguous layer blocks, trains each block inside a frozen pretrained aligner scaffold, and then recomposes the trained blocks with an optional short end-to-end adaptation pass. On a 1.3B-parameter Gemma-style model trained on C4, MoT provides a small-scale proof of mechanism: independently trained depth slices can be recomposed into a usable language model, and a quality-parity schedule reaches the same reported perplexity as the monolithic baseline. This parity setting processes more aggregate tokens and has a shorter idealized layer-equivalent critical path after aligner preparation; its effective compute advantage depends on reusing the aligner across runs. We therefore present MoT not as a general replacement for monolithic pre-training, but as a small-scale framework for studying whether scaffolded sub-runs can act as reusable training units.
Mohammed Sabry, Sean Augenstein, Keith Rush +1
Google, Mountain View, CA, USA · School of Computing, Dublin City University, Dublin, Ireland.