Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design

Jihwan Moon; Sheir A. Zaheer; Jinmyoung Lee; Gunhee Kim; Chan Y. Park

cs.LGOct 4, 2026

Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design

Authors: Jihwan Moon, Sheir A. Zaheer, Jinmyoung Lee, Gunhee Kim, Chan Y. Park

Organizations: INFOCZ Inc. · Seoul National University · KC ML2

Abstract

Greedy block-wise local learning splits a network into gradient-isolated blocks trained by local auxiliary losses, deleting the backward pass between blocks: inter-stage communication becomes forward-only and every block can step its optimizer independently, properties directly relevant to decentralized model-parallel training. Local learning is competitive with end-to-end backpropagation on image classification, and on small Transformers it is known to trade a worse best loss for parallel speedup. How this loss gap behaves in autoregressive language model (LM) pretraining at larger scale, and which auxiliary designs reduce it, has not been measured. We present a token-budget-matched empirical study at 125M and 400M parameters with $K \in \{1,2,4\}$ blocks at Chinchilla-optimal budgets, factorizing the auxiliary design into network architecture and training objective. We observe: (i) the gap to end-to-end training more than doubles from $K=2$ to $K=4$ , but at $K=4$ shrinks from 125M to 400M; (ii) replacing an MLP auxiliary with a Transformer-based one is a strong network-side intervention, recovering 22-41% of the gap; (iii) a multi-token-prediction (MTP) auxiliary objective helps at the first block boundary, whereas adding it at deeper boundaries hurts, and restricting it to the first block yields the best $K=4$ configuration ( $+0.062$ vs. $+0.075$ nats at 400M); and (iv) deployment-style per-block execution reduces activation memory by up to $2.2\times$ . We frame these results as an empirically grounded method direction rather than a finalized method: local objectives should apply future-predictive pressure selectively across boundaries while resisting shortcuts that bypass predictive content.

Figures & tables

125M-class (end-to-end $K{=}1$ : 2.771)
	MLP-aux	Transformer-aux	Transformer-aux + MTP
$K{=}2$ ( $\Delta$ )	$+0.046$	$+0.028$	$\mathbf{+0.017}$
$K{=}4$ ( $\Delta$ )	$+0.132$	$+0.078$	$\mathbf{+0.074}$
400M-class (end-to-end $K{=}1$ : 2.418)
$K{=}2$ ( $\Delta$ )	$+0.041$	$+0.032$	$\mathbf{+0.027}$
$K{=}4$ ( $\Delta$ )	$+0.112$	$+0.075$	$\mathbf{+0.062}$

Table 1: Validation NLL, absolute for end-to-end (

K{=}1

) and as

\Delta

against it for

K\in\{2,4\}

(nats/token; lower is better). Transformer-aux + MTP denotes first-block MTP; the all-block comparison is in Table 3 .

Figure 1: Per-block local losses at 125M,

K{=}4

, under MLP-aux (left) and Transformer-aux (right). The curves collapse: blocks after the first barely improve on the representation they inherit, and a larger head lowers all curves without separating them.

Table 3

Figure 2: Validation NLL at 400M for

K{=}2

(left) and

K{=}4

(right). At

K{=}4

the first-block placement improves over both Transformer-aux and the all-block MTP weighting throughout training.

	125M-class		400M-class
Method	$K{=}2$	$K{=}4$	$K{=}2$	$K{=}4$
End-to-end ( $K{=}1$ )	14.50/13.98		29.16/27.68
MLP-aux	11.03/10.40	9.52/8.56	19.07/17.43	14.52/12.41
Transformer-aux	12.05/11.07	10.32/9.25	20.49/18.33	15.67/13.36
Transformer-aux + MTP (first block only)	12.37/11.24	10.36/9.26	21.02/18.56	15.76/13.37

Table 4: Peak/activation GPU memory per worker (GiB; single GPU, batch

4\times 4{,}096

, bf16). Non-E2E entries hold one block and its auxiliary head and report the largest stage. MTP uses sequential per-offset computation ( Gloeckle et al., 2024 ) .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Per-block local losses at 400M,

K{=}4

, under MLP-aux (left) and Transformer-aux (right). As at 125M, later blocks provide only small improvements over the first block, and the Transformer-based auxiliary lowers the curves without separating them.

Scheme	Forward	Backward	Total	Update locking
Pipeline Parallelism (backprop)	$bd$	$bd$	$2bd$	locked
+ Subspace Compression ( Ramasinghe et al., 2025 )	$br$	$br$	$2br$	locked
Local Learning	$bd$	$0$	$bd$	none
+ Subspace Compression (fwd only)	$br$	$0$	$br$	none

Appendix

Table 5: Analytical inter-stage communication per token per stage boundary (

d

: model width;

b

: bytes per element,

2

for bf16;

r\ll d

: compression rank).

Per auxiliary head
	125M-class	400M-class
vocabulary projection alone	16.0%	7.1%
MLP-aux	19.1%	9.0%
Transformer-aux	26.1%	12.9%
MTP head	45.2%	21.9%
Total, $K{=}4$ (three auxiliary blocks)

Appendix

Table 6: Analytical training FLOPs of the auxiliary heads as a percentage of end-to-end training FLOPs per token at sequence length 4,096. End-to-end training costs

0.92

and

2.75

GFLOP/token at 125M and 400M. At

K{=}2

there is one auxiliary block, so the total equals the per-head value. At

K{=}4

the total sums over three auxiliary blocks, and first-block MTP places an MTP head at the first and Transformer-aux heads at the other two. Totals are computed before rounding.

Configuration	125M-class	400M-class
Transformer-aux (NTP only, reference)	$+0.078$	$+0.075$
first-block MTP	$\mathbf{+0.074}$	$\mathbf{+0.062}$
all-block MTP, linear ramp	$+0.082$	$+0.086$
all-block MTP, uniform	$+0.085$	—

Appendix

Table 7: MTP placement/weighting ablation at

K{=}4

\Delta

validation NLL vs. end-to-end (nats/token). The all-block uniform configuration was not run at 400M.

Auxiliary	val NLL	$\Delta$ vs. end-to-end (2.771)
CPC only	7.33	collapsed
NTP + CPC hybrid	2.925	$+0.154$
MLP-aux	2.816	$+0.046$
Transformer-aux	2.799	$+0.028$
Transformer-aux + MTP	2.788	$+0.017$

Appendix

Table 8: Feature-space contrastive auxiliaries at 125M,

K{=}2

(one cut), against the token-space auxiliaries of Table 1 . The pure-CPC run collapsed into a position encoder and was stopped at step 4,000 of 5,100; its auxiliary InfoNCE (

{\sim}0.28

nats at the stop) sits far below the same-sequence chance level while the final head never learns.

Local Learning

Local learning develops neural training rules that use nearby activity or layerwise objectives instead of global backpropagated errors.

See all 52 papers

SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning
Sep 28, 2026Bojian Yin, Shurong Wang, Yuqi Pan +1Language Model Pretraining Efficient Language Model Training
Institute of Automation, Chinese Academy of Sciences
Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
May 11, 2026Jinchang Zhu, Jindong Li, Yuwen Hao +3Language Model Pretraining Self-Attention
The Hong Kong University of Science and Technology (Guangzhou)
Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model
Aug 13, 2026Mohammed Sabry, Sean Augenstein, Keith Rush +1Language Model Pretraining Efficient Language Model Training
Google, Mountain View, CA, USA · School of Computing, Dublin City University, Dublin, Ireland.

Sep 28, 2026J/K move · Enter open · S save

SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning

Bojian Yin, Shurong Wang, Yuqi Pan, Guoqi Li

Institute of Automation, Chinese Academy of Sciences

Language Model Pretraining Efficient Language Model Training Local Learning

Open paper AskPDF arXiv

Large language models are trained with backpropagation, whose global gradient coordinates all layers but forces each to hold its activations and wait for the gradient to pass back through every deeper layer. Conventional local learning removes this update locking by training each module to predict the target through its own readout, but has not scaled to billion-parameter pretraining. We identify these private readouts as a key weakness, since they leave each module without information from deeper modules. We propose Shared-Output LOcal learning (SOLO), which replaces them with a shared, read-only copy of the final module's readout, the only one trained on the output of the whole network. Taken from the previous step, the copy transmits information from the final module without passing gradients between modules or reintroducing update locking. SOLO approaches backpropagation on Transformers of 340M to 2B parameters pretrained on 15B tokens, staying within one point in average zero-shot accuracy with a perplexity gap that narrows with scale. Readout ablations attribute SOLO's improvement over private readouts to sharing. Without update locking, each of p pipeline stages holds activations for O(1) micro-batches instead of O(p). The freed memory permits larger micro-batches, which reach up to 1.44x the best measured throughput of pipeline backpropagation on the same partition. To our knowledge, SOLO is the first local learning method to show such memory and throughput gains in billion-parameter language-model pretraining. Local learning thus becomes a practical alternative to backpropagation for large-scale pretraining.

Abstract

Local Learning

Explore similar work

SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning

Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining

Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model