Learning to Read Out: Unembedding Dynamics in Language Model Pretraining

Matteo He; William F. Shen; Alex Iacob; Andrej Jovanovic; Xinchi Qiu; Nicholas D. Lane

cs.CLOct 14, 2026

Learning to Read Out: Unembedding Dynamics in Language Model Pretraining

Authors: Matteo He, William F. Shen, Alex Iacob, Andrej Jovanovic, Xinchi Qiu, Nicholas D. Lane

Organizations: Department of Computer Science and Technology, University of Cambridge · Flower Labs · Isomorphic Labs

Abstract

When does a language model acquire a capability? Developmental studies usually answer this by tracking when models start producing the correct token. But that prediction passes through the unembedding matrix, or readout, which is itself still being learned during pretraining. We show that a model's hidden states can encode task information before its own readout expresses it, so timelines read from outputs conflate what the model encodes with what its readout expresses. To track the readout directly, we introduce parameter-trajectory crosscoding, which fits one sparse dictionary to every token's readout row at every checkpoint. Unlike crosscoders on activations, it needs only checkpoint weights. We use it to locate a narrow early window, 0.4-1.1% of the way through Pythia's pretraining, in which the readout reorganizes and WordNet word categories become decodable from its rows. By then, linear probes on the hidden states already decode task information that the model's own outputs do not yet show. To test whether such gaps depend on how fast the readout learns, we pretrain small models from scratch that differ only in the readout's learning rate. We find that raising it makes the readout develop earlier and the gap close sooner, with little effect on final loss. Together, these results imply that early in training, where many developmental transitions are dated, a behavioral date can reflect when the readout learns to express task information and not when the model learns it.

Figures & tables

Panel of Figure 1 — Figure 1: Overview. (a) The same Pythia-1B step-1000 hidden state prefers is under its own readout and are under the final readout, while a linear probe measures what the state encodes without passing through any readout. (b) Parameter-trajectory crosscoding fits one sparse dictionary to the trajectories of all readout rows. Feature strengths are schematic and use the profile colors of Figure 2 .

Task		Example prompt	Correct	Foil	$n_{A}$	$n_{L}$
SVA	simple	The keys to the cabinet	are	is	320	320
	across PP	The judge by the street	appears	appear	2,000	–
	subject RC	The author that admires the senators	likes	like	1,200	–
	object RC	The author that the senators admire	likes	like	1,200	–
IOI	availability	Lisa and Jane went to the store. Jane waited. Lisa gave the ring to	Jane	Lisa	2,000	–
	localization	Ryan and Adam went to the store. Ryan gave the pen to	Adam	Ryan	–	462

Table 1: Task examples and item counts for subject–verb agreement (SVA), indirect object identification (IOI), and number comparison. Underlined nouns are attractors, which stand between the subject and the verb.

n_{A}

counts items in the broader task suite used for availability, and

n_{L}

counts the separately built localization items before the selection and scoring split. Simple SVA is the set plotted in Figure 4 a. Templates and word lists are in Section F.1 .

Figure 2: Pythia’s readout reorganizes in one early window, and the mix of feature trajectories differs by model. Top: share of features in each lifecycle profile. Middle: each line is one active feature’s decoder norm divided by its own maximum, so heights do not compare features; thick lines are profile medians. Bottom: reorganization score as a

z

-score against circular-shift nulls, with filled markers at

p\leq 0.05

. Bands mark the selected windows, hatched for OLMo-2-7B (

p=0.12

Figure 3: Pythia-160M readout rows come to separate WordNet categories beyond matched controls, with the largest gains between steps

128

and

1000

, which overlap the start of the reorganization window. (a) Median held-out balanced accuracy over the 35 categories, against a probe on token metadata alone (dashed) and the 95th percentiles (p95) of the matched-random and shuffled-label nulls (gray band). The bracket marks the largest gains and the shaded band the reorganization window. (b) Number of categories above each control.

Figure 4: Pythia-6.9B hidden states encode subject number before the model’s own readout or any intermediate readout expresses it. (a) Accuracy on simple SVA of the logistic probe, the native readout, and the best swept readout over intermediate checkpoints; the arrow marks the gap at step 512. (b) Single-direction probe

A_{1}

and native expression

E

on the four agreement sets of Table 1 at the final checkpoint; annotations give

A_{1}-E

, and bars are 95% Wilson intervals for

E

Figure 5: A faster readout develops earlier and lowers early validation loss, while final losses agree within

0.06

nats. (a) Median feature peak step and peak turnover step against the readout learning rate multiplier; bars span the three fits where they differ, and dashed circles mark the shorter warmup. (b) Validation loss of each 31M run minus that of the baseline run.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: The parameter-trajectory crosscoding forward pass ( Equation 2 ) retains one feature index across checkpoints while allowing its contribution to change through training. One vocabulary row is collected across centered and scaled checkpoints and encoded into a shared preactivation, and a checkpoint-specific gate and decoder reconstruct each view.

Control	Comparison
Step-0 hidden-state probe	Distinctions decodable before training versus gains in decodability during training.
Shuffled/random labels	Probe predictability under targets unrelated to the task assignment.
Prompt-token probe	Hidden-state predictions versus information in token presence/counts.
Gaussian-feature probe	Hidden-state predictions versus fitting an equally sized unrelated feature matrix.
WordNet matched controls	Lexical labels versus frequency, length, row norm, spacing, and token-class metadata or matched token sets.
Answer-row geometry	Differences in contrast scores attributable to candidate norms and token-frequency patterns.

Appendix

Table A.1: Controls and the comparison each supports. Each control addresses one alternative explanation.

Suite	Reason not used
OLMo 7B, April 2024 [ Groeneveld et al., 2024 ]	Sparser coverage before step 1000 than OLMo-2
Olmo 3 [ Team Olmo et al., 2025 ]	No stage-1 checkpoint between steps 0 and 1000
LLM360 Amber [ Liu et al., 2024 ]	Checkpoints indexed ckpt_000 to ckpt_359 instead of by training step
LLM360 Crystal [ Liu et al., 2024 ]	Phase-specific revisions, custom loading, and a code and data mixture
Cerebras-GPT Intermediate [ Dey et al., 2023 ]	No documented early step schedule, and some maximal update parametrization ( $\mu$ P) variants need custom loading
OpenLLaMA previews [ Geng and Liu, 2023 ]	Sparse previews that begin after step 1000 (3B from step 2.5k)

Appendix

Table B.1: Other public checkpoint suites considered for

W_{U}

trajectory analysis, and why each was not used.

Hyperparameter	Pythia-1B	Pythia-160M	Pythia-6.9B $\lambda_{1}=0.6$	Pythia-6.9B $\lambda_{1}=0.3$	OLMo-2-7B
$d_{\mathrm{SAE}}$	24,576	24,576	32,768	32,768	32,768
Expansion factor	12.0	32.0	8.0	8.0	8.0
Sparsity coefficient $\lambda_{1}$	0.3	0.3	0.6	0.3	0.3
Optimizer	Adam	Adam	Adam	Adam	SparseAdam
Learning rate	$2\times 10^{-5}$	$5\times 10^{-5}$	$1\times 10^{-5}$	$1\times 10^{-5}$	$1\times 10^{-5}$
JumpReLU LR factor	0.2	0.1	0.4	0.3	0.3

Appendix

Table B.2: Sparse-dictionary training hyperparameters for every main dictionary setting. Each column is a complete recipe for one model family and matrix setting. All dictionaries train in fp32 on centered-and-scaled vocabulary rows ( center_scale ) for 300 epochs with batches of 1024 rows and no auxiliary loss on dead features (AuxK). Adam-family optimizers use the adaptive-moment method of Kingma and Ba [2015] .

Run family	Worker class	GPU memory	Wall clock	GPU-hours
Pythia-160M $W_{U}/W_{E}$ checkpoint-spanning fits	1–3 A100-class GPUs	40–80 GB/GPU	4–8 h	4–24
Pythia-1B $W_{U}$ capacity sweep	1 A100-80GB GPU	80 GB	up to 12 h	up to 12
Pythia-6.9B $W_{U}$ checkpoint-spanning fit, per $\lambda_{1}$ setting	4 A100/H100-class GPUs	40–80 GB/GPU	4–8 h	16–32
OLMo-2-7B $W_{U}$ checkpoint-spanning fit	4 A100/H100-class GPUs	40–80 GB/GPU	4–8 h	16–32
31M pretraining runs (four compared)	1 A40 GPU per run	48 GB	about 21 h per run	about 85 in total
Probe, null, baseline, and figure jobs	CPU workstation or 1 GPU	24–80 GB if GPU	minutes–4 h	up to 4

Appendix

Table B.3: Compute for reproducing the main dictionary fits and the controlled 31M pretraining runs, by hardware class rather than specific cloud offers. The 31M wall clock is estimated from the

10

B-token budget and the recorded throughput of about

131

k tokens/s. The 31M row covers the four compared runs and leaves out the excluded fifth run ( Section H.1 ).

Figure C.1: The selected dictionaries reconstruct the readout trajectories with aggregate EV

0.808

–

0.920

while activating

1.17

–

2.26\%

of their features per row, and on Pythia-160M a checkpoint-spanning dictionary stays within

0.1

EV of independent per-snapshot SAEs at every checkpoint while activating

1.9

–

3.5\%

of its features against their

11

–

14\%

. (A) Aggregate EV against active density

100\cdot L_{0}/d_{\mathrm{SAE}}

for the four selected dictionaries (large labeled markers), the Pythia-1B width sweep, the denser Pythia-6.9B

\lambda_{1}=0.3

fit, the Pythia-160M

d_{\mathrm{SAE}}=8{,}192

dictionary, and the Pythia-160M single-snapshot SAE baselines ( Table C.1 ). (B) Per-checkpoint EV and (C) per-checkpoint active density. Solid lines compare the Pythia-160M

d_{\mathrm{SAE}}=8{,}192

seed-0 dictionary (“160M cross-snap CC”), not the selected

d_{\mathrm{SAE}}=24{,}576

dictionary, with the 32 per-snapshot SAEs of the same width; dashed lines show the selected Pythia-1B, Pythia-6.9B, and OLMo-2-7B dictionaries for reference. OLMo-2-7B is plotted on its own stage-1 step axis, which panels B and C cut at step

180

k, so its last

11

checkpoints are not shown.

Figure C.2: The mean row of checkpointed

W_{U}

grows and aligns with the raw top right singular vector after early training, inducing a positive row-cosine floor, so later analyses use centered rows. The norm ratio is the mean-row norm relative to the median row norm, and the right panel compares raw and centered mean pairwise row cosine. The shaded band marks steps

64

–

1

k; OLMo-2-7B is plotted on its own stage-1 schedule. Pythia-160M is not shown for readability.

Figure C.3: Next-token fidelity of the reconstructed readouts varies over training, and the Pythia-160M

d_{\mathrm{SAE}}=8{,}192

dictionary keeps logit shape but loses scale calibration late in training. At each of the 32 checkpoints, hidden states from the fixed held-out shard are decoded through both the original (native) and reconstructed readouts. The third panel plots

\mathrm{KL}(\text{native}\,\|\,\text{reconstructed})

, whereas the text reports

\Delta\mathrm{NLL}

Figure C.4: On Pythia-160M, the checkpoint-spanning dictionaries (

d_{\mathrm{SAE}}=8{,}192

and the selected

d_{\mathrm{SAE}}=24{,}576

) reconstruct early and late checkpoints with one feature index, while the SAE baselines either fail early or require far higher

L_{0}

. The final-snapshot SAE (cross-eval) is trained on the final checkpoint and evaluated backward; per-snapshot SAEs are trained independently at each checkpoint.

Figure C.5: Per-snapshot PCA of centered

W_{U}

needs many dense components to approach the EV of the selected sparse dictionaries, and at rank

1024

it falls short of them at every plotted Pythia-1B and Pythia-6.9B checkpoint and at the plotted OLMo-2-7B checkpoints from step

21

k onward. Each checkpoint is approximated independently by its best rank-

r

PCA reconstruction. The dictionary reference line is drawn only for Pythia-160M and Pythia-1B; the selected Pythia-6.9B and OLMo-2-7B dictionaries reach aggregate EV

0.808

and

0.853

( Table C.1 ). Pythia-160M has

d_{\mathrm{model}}=768

, so its PCA is exact at rank

768

Figure C.6: Dominant centered

W_{U}

subspaces in Pythia move most during early training, with the largest top-

10

movement between steps

128

and

512

; the later Pythia-1B step-

116{,}000

anomaly is examined in Section G.3 . For each checkpoint, we center

W_{U}

, estimate top right-singular directions, and track adjacent-checkpoint chordal Grassmann distance (divided by

\sqrt{k}

for a rank-

k

subspace) and top-vector stability, plotted at the later checkpoint of each pair. In the left and right panels, color and line style mark the model. The middle panel restricts the left panel’s distance to steps up to

9

k for subspace ranks

k=5

(dotted),

20

(solid), and

50

(dashed), with colors as in the left panel; gray verticals mark steps

64

256

, and

1

Figure C.7: Interpolated heads of the Pythia-1B

d_{\mathrm{SAE}}{=}24{,}576

seed-0 crosscoder track the in-sample ceiling within

0.01

EV at six held-out steps (

10\text{k}

–

93\text{k}

) and deviate only at

113\text{k}

and

123\text{k}

. These deviations bracket the step-

116\text{k}

anomaly ( Section G.3 ), where the in-sample ceiling dips to

0.755

EV and direct

W_{U}

interpolation drops to

0.36

EV at

113\text{k}

Figure C.8: For the Pythia-160M

d_{\mathrm{SAE}}=8{,}192

dictionary, raising the sparsity weight

\lambda_{1}

from

0.4

1.8

lowers mean

L_{0}

from about

160

to about

23

and EV from

0.75

0.53

along a single tradeoff curve. Red points are the sweep fits ( Table D.3 ). The star (“released anchor”) marks the five-seed Pythia-160M

d_{\mathrm{SAE}}=8{,}192

fits at

\lambda_{1}=0.3

, and the gray and brown markers are the same-width SAE baselines of Table C.1 . The dashed green line marks

L_{0}=50

, near the low end of the sparsities of the Pythia-160M activation crosscoders of Ge et al. [2026] (

L_{0}\approx 47

–

64

in their Figure 2), and “EV @ L0 = 50” is the sweep’s EV at that

L_{0}

, interpolated linearly in

\log L_{0}

between adjacent sweep points.

$\star$	Model	Matrix / Family	$d_{\mathrm{SAE}}$	Exp.	Seed	Params	EV	Mean $L_{0}$	% Act.
	160M	$W_{U}$ cross-snap-32	8 192	10.7 $\times$	0–4	403 M	0.776	203.6–203.9	2.49
	160M	$W_{U}$ cross-snap-32 d	16 384	21.3 $\times$	0	806 M	0.780	104.9	0.64
$\star$	160M	$W_{U}$ cross-snap-32	24 576	32 $\times$	0–2	1.21 B	0.920	286.4–287.2	1.17
	160M	$W_{U}$ final-snap SAE	6 144	8 $\times$	—	9.5 M	0.758	864.2	14.07
	160M	$W_{U}$ final-snap SAE	8 192	10.7 $\times$	—	12.6 M	0.793	1 088	13.28
	160M	$W_{U}$ final-snap SAE	16 384	21.3 $\times$	—	25.2 M	0.867	1 921	11.72

Appendix

Table C.1: Full dictionary inventory used across the validation figures.

\star

marks the selected dictionaries analyzed in Sections 3 and 4 , each the seed-0 fit of its row, and rows with several fit seeds give ranges over seeds. The Exp. column reports the expansion ratio

d_{\mathrm{SAE}}/d_{\mathrm{model}}

(Pythia-160M:

d_{\mathrm{model}}{=}768

; Pythia-1B:

2\,048

; Pythia-6.9B and OLMo-2-7B:

4\,096

). The % Act. column reports

100\cdot L_{0}/d_{\mathrm{SAE}}

, i.e. the fraction of the dictionary active per token row. For the distributed 6.9B and OLMo fits, EV and mean

L_{0}

are recomputed over the full vocabulary after training. a per-snap SAE family aggregates 32 per-checkpoint SAEs; b Pythia-6.9B retrain with

\lambda_{1}=0.6

(selected dictionary; recipe in Table B.2 ); c Pythia-6.9B run trained at

\lambda_{1}=0.3

(denser capacity-point); d fit with saturation scale

4.0

instead of

1.0

Profile	Rule	Meaning
persistent	$(\phi_{f}^{\mathrm{alive}}\geq 0.70\ \mathrm{and}\ r_{f}^{\mathrm{range}}\leq 0.35)$ or $\min(r_{f}^{\mathrm{early}},r_{f}^{\mathrm{mid,mean}},r_{f}^{\mathrm{late}})\geq 0.45$	sustained across training
transitional	$r_{f}^{\mathrm{mid,max}}\geq 0.75$ , $r_{f}^{\mathrm{mid,max}}-\max(r_{f}^{\mathrm{early}},r_{f}^{\mathrm{late}})\geq 0.35$ , $r_{f}^{\mathrm{late}}\leq 0.30$ , $\Delta\tau_{f}^{\mathrm{alive}}\leq 0.35$ , and $0.35<\tau_{f}^{\mathrm{peak}}<0.65$	rare localized mid-training peak
early-decaying	$r_{f}^{\mathrm{early}}\geq 0.50$ , $r_{f}^{\mathrm{early}}-r_{f}^{\mathrm{late}}\geq 0.30$ , and $\tau_{f}^{\mathrm{peak}}\leq 0.45$	strong early, weaker late
late-emerging	$r_{f}^{\mathrm{late}}\geq 0.50$ , $r_{f}^{\mathrm{late}}-r_{f}^{\mathrm{early}}\geq 0.30$ , and $\tau_{f}^{\mathrm{peak}}\geq 0.55$	weak early, stronger late
mixed/ambiguous	no previous rule matched	residual category

Appendix

Table D.1: Rule-based lifecycle profile assignment used in Figure D.1 . Rules are applied top to bottom, so each active feature receives exactly one descriptive label. The labels “persistent”, “early-decaying”, “late-emerging”, “transitional”, and “mixed/ambiguous” are thresholded trajectory descriptions.

Figure D.1: The four selected cross-snapshot sparse dictionaries contain persistent and late-emerging features, with substantial early-decaying shares in Pythia-160M and Pythia-1B. Sharply transitional mid-training peaks are rare, so readout features do not follow a single monotone growth pattern. Bars give the share of active features in each rule-based profile, and lines give each profile’s median trajectory with interquartile bands.

Model	Selected window	$p$	Same window	Other windows (resamples)
Pythia-160M	512–1.6k	0.005	370	1k–3.2k (30)
Pythia-1B	512–1.6k	0.005	283	1k–3.2k (117)
Pythia-6.9B	512–1.6k	0.04	222	9k–28k (134) , 14k–44k (33) , 1k–3.2k (10) , 8k–25k (1)
OLMo-2-7B	9k–28k	0.12	399	900–2.8k (1)

Appendix

Table D.2: Windows selected by the reorganization score, in training steps.

p

is the one-sided circular-shift probability with a plus-one correction over 200 draws, so

0.005

is the smallest attainable value. The last two columns count how many of 400 resamples of active features select the same window and each other window.

Figure D.2: The three Pythia dictionaries select the same reorganization window, steps 512–1.6k, where several feature change metrics co-occur more than expected under shifted-component nulls. OLMo-2-7B’s highest-scoring window, 9k–28k stage-1 steps, is not separated from the nulls ( Table D.2 ). Each curve is plotted at its window’s center and min–max scaled within each model. Black curves (“enriched score”) show the null-standardized reorganization score, gray bands mark the selected windows, and dotted verticals mark their centers. In the legend, “direction velocity” is movement toward the terminal decoder direction and “rank instability” is instability of the top feature set; the other four components match the text. Mass growth is plotted as a control and is excluded from the score.

Model	Width	$\lambda_{1}$	Seeds	Early-decaying	Persistent	Late-emerging	Window	$p$
Pythia-160M	8,192	0.3	5	45.6–46.1	0.5–0.9	52.5–53.0	512–1.6k	0.005
Pythia-160M †	16,384	0.3	1	7.8	80.9	11.2	512–1.6k	0.005
Pythia-160M ∗	24,576	0.3	3	42.9–43.1	6.9	49.5–49.7	512–1.6k	0.005
Pythia-160M	8,192	0.4	1	43.2	1.2	55.0	512–1.6k	0.005
Pythia-160M	8,192	1.0	1	27.8	43.4	28.3	512–1.6k	0.005
Pythia-160M	8,192	1.2	1	22.3	57.5	19.7	512–1.6k	0.005

Appendix

Table D.3: Lifecycle statistics across dictionary fits. Profile shares are percentages of active features; rows with several fit seeds give the range over seeds. The window is the one selected by the reorganization score, and

p

is its circular-shift probability (200 draws). ∗ Selected dictionaries of the main text. † Fit with penalty saturation scale

c=4

instead of

c=1

. The rows with

\lambda_{1}

from

0.4

1.8

come from a separate sweep that also used an auxiliary loss on dead features.

Figure E.1: In Pythia-160M, all 35 WordNet supersense categories reach sustained decodability in the readout rows between steps $128$ and $1000$ , and 33 have their largest adjacent-checkpoint gain in that interval. Rows are the 35 lexicographer-file categories with at least 20 clean token ids, in alphabetical order, and color is the held-out balanced accuracy of a one-vs-rest linear probe on centered $W_{U}$ rows under the row split of Section E.1 . Columns are the 32 Pythia checkpoints, equally spaced by index. The white box marks steps $512$ and $1000$ , the checkpoints inside the $512$ – $1.6$ k reorganization window.

Figure F.1: In the 32-checkpoint Pythia-6.9B trajectories for SVA (

n=320

) and role-balanced IOI (

n=2{,}000

), the logistic hidden state probe reaches high accuracy before native contrast accuracy rises, and on SVA the best swept readout tracks the native one. The best swept readout is the post hoc envelope

B_{\mathrm{env}}

. IOI probe accuracy is already

0.928

at initialization, so the entire early probe–expression gap cannot be attributed to learning. All three curves in panel A use the same task inventory, and panel B compares the probe with the strongest prompt-token or Gaussian-feature control.

Task	Template	Answers	Set	$n$
Simple SVA	The {noun} {prepositional phrase}	are / is	A, L	320
Across-PP SVA	The {head} {preposition} the {attractor}	agreeing verb / other form	A	2000
Subject-RC SVA	The {head} that {clause verb} the {attractor}	agreeing main verb / other form	A	1200
Object-RC SVA	The {head} that the {attractor} {clause verb}	agreeing main verb / other form	A	1200
IOI	{A} and {B} went to the store. {R} waited. {S} gave the {object} to	{R} / {S}	A	2000
IOI	{A} and {B} went to the store. {A} gave the {object} to	{B} / {A}	L	462

Appendix

Table F.1: Prompt templates. Braces mark slots filled from the word lists in the text, and answers are the correct token and the foil, each with a leading space. In role-balanced IOI, {R} is the recipient and {S} the giver, one each of {A} and {B} . Set A is the availability task suite and set L the localization items.

family	$n$	$A_{1}@0$	$A_{1}$	$E$	$B$	$A_{1}\!-\!E$	unreach	$A_{1}\!-\!E$ (1B)	regime
SVA simple	320	0.60	1.00	1.00	1.00	0.00	–	0.00	transient
SVA across-PP	2000	0.43	1.00	0.97	0.98	0.03	63%	0.03	transient
SVA subject-RC	1200	0.73	1.00	0.75	0.79	0.25	85%	0.21	persistent
SVA object-RC	1200	0.73	1.00	0.91	0.93	0.10	72%	0.13	persistent
IOI	2000	0.92	1.00	1.00	1.00	0.00	–	0.00	transient
numeric $>$	2000	0.86	1.00	0.63	0.82	0.37	47%	0.55	persistent

Appendix

Table F.2: Binary families at the final checkpoint (Pythia-6.9B), with the 1B gap and regime.

A_{1}@0

is the step-0 initialization baseline, and the across-PP value of

0.43

lies below chance for a reason the stored runs do not record.

B

B_{\mathrm{ho}}

, and “unreach”

=(A_{1}-B)/(A_{1}-E)

is the fraction of the gap that the evaluated intermediate readout does not reach, computed from unrounded values. The Pythia-1B numeric probes use stratified folds without number pair grouping. A gap is persistent when

A_{1}-E

exceeds

0.05

at the final checkpoint and transient when it ends at or below

0.05

after exceeding

0.30

while

E

was below

0.7

family	example prompt (truncated)	$y^{+}$ / $y^{-}$	$n$	$E$	$B$	rank1
BLiMP anaphor	Katherine can’t help	herself / himself	1455	0.99	0.99	0.02
BLiMP det-noun	Raymond is selling this	sketch / sketches	838	0.92	0.94	0.03
BLiMP NPI	Teresa had	not / probably	656	0.98	0.97	0.15
copy control	valley valley anchor anchor zebra	zebra / jacket	2000	0.98	0.98	0.01
fixed-token control	mirror thunder cookie music island	the / ,	2000	0.98	0.90	0.00
FV antonym	selfish:altruistic / note:silence / known…	perfect / socialist	2000	1.00	1.00	0.00

Appendix

Table F.3: Open-vocabulary and control families (Pythia-6.9B), with a truncated example prompt and the contrast tokens

y^{+}/y^{-}

(leading space shown). The BLiMP families come from Warstadt et al. [2020] and the function vector (FV) families, which are in-context prompts, from Todd et al. [2024] ; BLiMP probe folds are grouped by antecedent name for anaphor agreement and by singular noun lemma for determiner–noun agreement. Apart from these two BLiMP agreement families, these carry no binary feature, so

A_{1}

is undefined; expression is reported by contrast accuracy

E

, the swept accuracy

B=B_{\mathrm{ho}}

, and the full vocabulary rank-1 rate of

y^{+}

. NPI is negative polarity item licensing, and for BLiMP anaphor

n

counts the gender-labeled items of Section F.2 . The control families are built from single-token nouns drawn at random. Fixed-token control prompts list two to five distinct nouns, and the answer is always the against , , so the context carries no information about the answer. Copy control prompts have the form

a\,a\,b\,b\,c

, with answer

c

and a foil absent from the prompt. Induction prompts have the form

a\,b\,a\,b\,a

, with answer

b

and a foil absent from the prompt. Relational (bal.) gives each of

24

countries five foils drawn at random from the other capitals, whereas relational facts uses the fixed foil list of the localization items ( Section F.1 ).

Figure F.2: Across the eight binary families in Pythia-6.9B, agreement and numeric tasks differ in both initialization baselines and final checkpoint gaps. Panel titles are colored by the regime of Table F.2 , orange for a transient gap and red for a persistent one.

B

(best-other in the legend) is

B_{\mathrm{ho}}

. Shaded bands are

95\%

percentile bootstrap intervals for

A_{1}

over its five per-fold accuracies, and dotted curves show the label-shuffle control (blue) and chance (black). Native expression on numeric

<

rises to about

0.75

between steps

128

and

2000

and falls to

0.45

at the final checkpoint, so this family’s final gap is larger than its gap in mid-training.

				step $512$			step $2000$		final
model	family	$n$	$A_{1}@0$	$A_{1}$	$E$	$B$	$A_{1}$	$E$	$E$
6.9B	SVA simple	320	0.60	0.93	0.50	0.49	1.00	1.00	1.00
6.9B	SVA across-PP	2000	0.43	0.95	0.49	0.48	1.00	0.90	0.97
6.9B	BLiMP anaphor	1455	0.60	0.63	0.48	0.48	0.85	0.90	0.99
1B	SVA simple	320	0.45	0.97	0.50	0.49	1.00	1.00	1.00
1B	SVA across-PP	2000	0.53	0.98	0.48	0.48	1.00	0.93	0.97

Appendix

Table F.4: Families whose probe starts within

0.1

of chance, from the broader task suite (

24

checkpoints;

A_{1}

is the shrinkage-LDA probe and

B

B_{\mathrm{ho}}

). For BLiMP anaphor,

n

counts the items with a gender label used by

A_{1}

, and

E

and

B

use all

2{,}000

items.

			$t=256$		$t=512$		$t=1000$
Language	Targets	Share (%)	$s^{\ast}$	Gain	$s^{\ast}$	Gain	$s^{\ast}$	Gain
Aggregate	173,229	100.0	1000	1.503 [1.455, 1.548]	1000	0.147 [0.141, 0.153]	1000	0
English	3,939	2.3	512	0.048 [0.026, 0.070]	1000	0.064 [0.039, 0.089]	2000 †	0.002 [ $-$ 0.021, 0.026]
Russian	10,639	6.1	512	1.033 [1.007, 1.060]	1000	0.114 [0.090, 0.139]	1000 †	0
Chinese	25,330	14.6	1000	1.251 [1.176, 1.305]	1000	0.134 [0.123, 0.146]	1000	0
Japanese	20,698	11.9	2000	1.148 [1.090, 1.195]	1000	0.192 [0.178, 0.207]	2000	0.013 [0.002, 0.024]

Appendix

Table G.1: Readout swap gains by language for Pythia-1B. For hidden states from checkpoint

t

s^{\ast}

is the readout checkpoint with the lowest NLL, and the gain is native NLL minus the NLL under

W_{U}^{(s^{\ast})}

in nats, so a gain of

0

means that the native readout is best. Targets are scored next-token positions and share is their percentage of all targets; the aggregate is the token-weighted mean over all targets, including

8

targets whose context token belongs to the preceding language slice. Brackets give

95\%

paired bootstrap intervals from

2{,}000

resamples of

256

-token blocks within each slice, which do not measure variation across documents, and † marks a bootstrap probability below

0.95

that

s^{\ast}

is the minimum.

Figure G.1: In the token-weighted aggregate for Pythia-1B, the step- $1000$ readout is the NLL optimum for hidden states from steps $256$ , $512$ , and $1000$ , while later hidden states prefer nearby or same-checkpoint readouts and become more readout-specific. Per language, step $1000$ is the shared optimum only for step- $512$ states ( Section G.2 ). Rows are hidden state checkpoints $t$ (the axis label $h_{t}$ denotes $h^{(t)}$ ), and columns are readout checkpoints $s$ . White diamonds mark same-checkpoint readouts, red circles mark the row-wise minimum-NLL readout, and the dashed vertical line marks the step- $1000$ readout column. Panel B shows $\mathrm{KL}(\mathrm{softmax}(h_{p}^{(t)}\left(W_{U}^{(s)}\right)^{\top})\,\|\,\mathrm{softmax}(h_{p}^{(t)}\left(W_{U}^{(t)}\right)^{\top}))$ , so dark cells are close to the same-checkpoint predictive distribution for that hidden-state row. The bright column near $116\text{k}$ is the released-checkpoint anomaly of Section G.3 .

Figure G.2: In Pythia-1B, the readout swap NLL gain and the best readout checkpoint for step-

256

hidden states vary by language, while every language prefers an intermediate readout to its native and terminal readouts; for step-

512

states, all nine languages share the step-

1000

optimum. The gain is native NLL minus swapped NLL, so positive values favor the swapped readout. Open circles mark each curve’s best readout, bands give

95\%

block-bootstrap intervals, and values below

-2.5

nats are clipped.

Figure G.3: In Pythia-1B, each hidden-state checkpoint keeps its minimum-NLL readout across four alignments of the swapped readout (step

512

for

t=256

and step

1000

for

t=512

), except for a one-checkpoint shift under scale alignment for hidden states from checkpoint

1000

with a small NLL gap. Panels are hidden-state checkpoints, the horizontal axis is the readout checkpoint

s

, and the thin vertical line marks

s=1000

. “None” is the raw swap. Its curve (dashed) lies under the “mean” curve, which is identical by construction.

			ablate		preserve-only		hidden projection
Family	orig.	recon	top-8	sign	top-8	sign	native	proj.
Subject–verb agreement	0.925	0.900	0.487	0.931	0.938	0.463	0.781	0.500
Greater-than completion	0.529 ∗	0.512	0.084	0.500	0.894	0.507	0.601	0.497
Indirect object id. (IOI)	0.455 ∗	0.429	0.394	0.424	0.481	0.420	0.377	0.338
Relational facts	0.269 ∗	0.231	0.154	0.231	0.923	0.308	0.308	0.154

Appendix

Table G.2: Per-family top eight intervention accuracies at the fixed evaluation cell (Pythia-1B, held-out half; relational facts, scored on

13

items, are exploratory). orig. is the original terminal readout and recon the full reconstructed terminal readout; ablate and preserve-only act on the top eight positive attribution features, and sign is the same edit applied to eight random features matched on decoder norm, firing rate, and attribution sign. The hidden projection columns score step-

1000

states under the native

W_{U}^{(1000)}

before ( native ) and after ( proj. ) the top eight decoder directions are projected out. ∗ Scored on all localization items (

2{,}000

462

, and

26

), not the held-out half.

Condition	EV	$L_{0}$	val. loss	persist.	early-dec.	late-em.	mixed	med. peak	turnover step
wu_lr_0p25	0.905	193.9	3.405	0.875	0.000	0.093	0.032	128	2048
baseline	0.891	177.2	3.381	0.403	0.219	0.125	0.252	64	1024
warmup_short	0.890	175.5	3.350	0.011	0.442	0.153	0.394	64	1024
wu_lr_4x	0.874	168.5	3.369	0.003	0.675	0.131	0.191	32	512

Appendix

Table H.1: Per-condition crosscoder quality and lifecycle summary for the recipe control (

d_{\mathrm{SAE}}=8192

K=16

8192

active features each, seed

0

). EV is explained variance;

L_{0}

is mean active features per row; “val. loss” is the condition’s final validation loss in nats; the profile columns are fractions of active features; “med. peak” is the median peak step; “turnover step” is the peak turnover step. The validation loss at the peak turnover step is

3.72

nats for

0.25\times

(step

2048

) and

5.13

nats for

4\times

(step

512

). At

1\times

, turnover at steps

1024

and

2048

differs by less than

0.002

in every fit, and seed

1

selects step

2048

. The two timing statistics are the primary measures; the rule-based profile fractions are discrete summaries sensitive to the profile rule boundaries ( Section H.3 ).

Figure H.1: Higher readout learning rate shifts the peak of decoder norm turnover (left) earlier, and warmup_short tracks baseline . Decoder rotation (right) peaks at

1024{\to}2048

only for

4\times

. Its late spike, and the late rise in norm turnover for

0.25\times

, come from the pair from the slot checkpoint near step

2479

to step

4096

, which spans about

1{,}600

steps, the longest interval of the schedule, against fewer than

450

steps for the pair before it. Each point averages the change over active features of the seed-

0

crosscoder for one consecutive checkpoint pair, plotted at the pair midpoint.

Figure H.2: By the end of training, higher readout learning rate gives larger

W_{U}

row norms, a less concentrated spectrum, and a smaller final LayerNorm gain, and warmup_short tracks baseline . The stable rank of

W_{U}

\|W_{U}\|_{F}^{2}/\sigma_{1}^{2}

, where

\sigma_{1}

is its largest singular value. The top-1 singular fraction is

\sigma_{1}^{2}/\|W_{U}\|_{F}^{2}

, the share of squared Frobenius norm in the top singular direction. The final LayerNorm gain is the

\ell_{2}

norm of its gain vector

\gamma

, and the warmup_short curves end at step

4096

because its final geometry measurement is missing.

Figure H.3: The swap optimum

s^{\ast}

, the readout checkpoint with the lowest NLL for hidden states from checkpoint

t

, tracks the readout learning rate. Points above the diagonal are hidden states best decoded by a later readout. wu_lr_4x follows the diagonal and wu_lr_0p25 lies above it, while for baseline and warmup_short the gain over the native readout is at most

0.05

nats.

Figure H.4: The frequency class gap (probe accuracy minus native accuracy) in the early window is ordered

0.25\times>1\times>4\times

and closes by step

1024

in every run except

0.25\times

, which closes by step

2048

. Accuracy is on four next-token frequency classes of equal corpus occurrence mass. From step

2048

on the native readout exceeds the probe in every condition, so the probe is not a ceiling late in training.

Family	Analyzed object	Input data	Temporal correspondence	What the comparison isolates
Lens methods	Hidden states in vocabulary space	Text and forward passes	Vocabulary coordinates; native or learned readout	Predictions under the chosen readout
Vocabulary geometry	Input/output embedding rows	Weights; lexical annotations	Token identities or geometric statistics	Changes in geometry and lexical association
Independent SAEs	Activations per checkpoint	Activation corpus	Separate indices; post-hoc matching	Features within each fit
Activation crosscoders	Activations across checkpoints	Activation corpus	Shared sparse indices	Representation change; feature use via interventions
Parameter methods	Decomposed or sparse weights	Weights and inputs	Typically one model state	Parameter components and circuits
This work	Readout row trajectories	Weights; text for tests	Token identities and shared sparse indices	Readout change; swaps and ablations at fixed states

Appendix

Table I.1: Analysis families and the comparisons they support. Temporal correspondence describes the unit compared across checkpoints. The last column distinguishes dictionary fitting from the interventions used to assess behavior.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 440 papers

Apr 9, 2026

What do Language Models Learn and When? The Implicit Curriculum Hypothesis

Large language models (LLMs) can perform remarkably complex tasks, yet the fine-grained details of how these capabilities emerge during pretraining remain poorly understood. Scaling laws on validation loss tell us how much a model improves with additional compute, but not what skills it acquires in which order. To remedy this, we propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models and data mixtures. We test this by designing a suite of simple, composable tasks spanning retrieval, morphological transformations, coreference, logical reasoning, and mathematics. Using these tasks, we track emergence points across four model families spanning sizes from 410M-13B parameters. We find that emergence orderings of when models reach fixed accuracy thresholds are strikingly consistent (

ρ= .81

across 45 model pairs), and that composite tasks most often emerge after their component tasks. Furthermore, we find that this structure is encoded in model representations: tasks with similar function vector representations also tend to follow similar trajectories in training. By using the space of representations derived from our task set, we can effectively predict the training trajectories of simple held-out compositional tasks throughout the course of pretraining (

R^2 = .68

.84

across models) without previously evaluating them. Together, these results suggest that pretraining is more structured than loss curves reveal: skills emerge in a compositional order that is consistent across models and readable from their internals.

Emmy Liu, Kaiser Sun, Millicent Li +4

Language Technologies Institute, Carnegie Mellon University · Department of Computer Science, Data Science and AI Institute, Johns Hopkins University · Khoury College of Computer Science, Northeastern University +1

Emergent Abilities in Language Models Language Model Pretraining

Ask PDF

Oct 20, 2026

How Language Models Organize Task Information

Dense linear probes assess task readability from language model (LM) representations, but do not directly reveal how many features suffice for prediction. We measure readout sparsity with

k^*

, the smallest feature count found by a fixed search protocol that recovers 95% of dense-probe AUROC above chance. We evaluate 19 LMs from five labs on 14 core tasks. Ten base models give the nine reliably readable tasks similar feature counts and nearly the same order. This agreement extends to the selected features themselves, which on three tasks show cross-model response correspondence beyond class-mean differences and a label-correlation-matched null. Instruction-tuned deployment selectively reduces

k^*

, with both weights and input format contributing. In controlled synthetic experiments across 20 models, successfully learned targets admit single-feature readouts, whereas readable controls require tens of features. Reference-model

k^*

is associated with label requirements across eight students, following an empirical scaling of

N^*\propto(k^*)^{0.65}

. After continual learning, saved heads can fail while linear readability remains high, and recovery requires fewer labels than initial task learning under the tested procedures. Pruning reduces linear readability, with wider pre-pruning readouts generally associated with larger losses. These findings distinguish task readability from readout sparsity and establish their complementary value for predicting learning requirements and diagnosing LM changes.

Kaiyang Wan, Yuxia Wang

INSAIT, Sofia University “St. Kliment Ohridski” Sofia, Bulgaria

Language Model Probing Language Model Performance Prediction Representation Geometry in Language Models

Ask PDF

Apr 20, 2026

Probing for Reading Times

Probing has shown that language model representations encode rich linguistic information, but it remains unclear whether they also capture cognitive signals about human processing. In this work, we probe language model representations for human reading times. Using regularized linear regression on two eye-tracking corpora spanning five languages (English, Greek, Hebrew, Russian, and Turkish), we compare the representations from every model layer against scalar predictors -- surprisal, information value, and logit-lens surprisal. We find that the representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration. The concentration of predictive power in the early layers suggests that human-like processing signatures are captured by low-level structural or lexical representations, pointing to a functional alignment between model depth and the temporal stages of human reading. In contrast, for late-pass measures such as total reading time, scalar surprisal remains superior, despite its being a much more compressed representation. We also observe performance gains when using both surprisal and early-layer representations. Overall, we find that the best-performing predictor varies strongly depending on the language and eye-tracking measure.

Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re +4

ETH Zürich · Toyota Technological Institute at Chicago · University College London

Computational Psycholinguistics Language Model Probing

Ask PDF

Abstract

Language Model Pretraining

Explore similar work

What do Language Models Learn and When? The Implicit Curriculum Hypothesis

How Language Models Organize Task Information

Probing for Reading Times