How Language Models Organize Task Information

Kaiyang Wan; Yuxia Wang

cs.AIOct 20, 2026

How Language Models Organize Task Information

Authors: Kaiyang Wan, Yuxia Wang

Organizations: INSAIT, Sofia University “St. Kliment Ohridski” Sofia, Bulgaria

Abstract

Dense linear probes assess task readability from language model (LM) representations, but do not directly reveal how many features suffice for prediction. We measure readout sparsity with $k^*$ , the smallest feature count found by a fixed search protocol that recovers 95% of dense-probe AUROC above chance. We evaluate 19 LMs from five labs on 14 core tasks. Ten base models give the nine reliably readable tasks similar feature counts and nearly the same order. This agreement extends to the selected features themselves, which on three tasks show cross-model response correspondence beyond class-mean differences and a label-correlation-matched null. Instruction-tuned deployment selectively reduces $k^*$ , with both weights and input format contributing. In controlled synthetic experiments across 20 models, successfully learned targets admit single-feature readouts, whereas readable controls require tens of features. Reference-model $k^*$ is associated with label requirements across eight students, following an empirical scaling of $N^*\propto(k^*)^{0.65}$ . After continual learning, saved heads can fail while linear readability remains high, and recovery requires fewer labels than initial task learning under the tested procedures. Pruning reduces linear readability, with wider pre-pruning readouts generally associated with larger losses. These findings distinguish task readability from readout sparsity and establish their complementary value for predicting learning requirements and diagnosing LM changes.

Figures & tables

Figure 1: Task readout measurements reveal cross-model structure, its dependence on training, and implications for task learning and model modification.

Figure 2: (a) Task readout counts and rankings are similar across base models, and (b) selected SAE features exhibit cross-model response correspondence beyond class-mean differences.

Figure 3: Instruction-tuned deployment reduces feature counts unevenly across tasks through weight and format effects, while supervised targets admit single-feature readouts in controlled experiments.

Figure 4: Target queries receive correct training answers and control queries receive independent random answers, while probes predict the true aggregate values for both.

Figure 5: Feature counts anticipate label requirements and pruning vulnerability, while dense probes reveal retained linear readability after saved-head failure.

Appendix figures & tables63 assets

Supplementary material from the paper’s appendix.

Appendix

Statement in Section 2	Formal object	Supporting quantity
$k^{*}$ is the smallest searched count recovering 95% of the dense AUROC above chance	Equations 4 and 5	Definition, with the threshold example in Section A.1.1
A finite $k^{*}$ certifies a qualifying readout, and $\infty$ records that no candidate qualified	Lemma 1 (i)	Exact statement about the searched family
Raising the threshold can only raise $k^{*}$	Lemma 1 (ii) and (iv)	Holds for local, confirmed, and task-level counts
$k^{*}$ refers to the model’s own coordinates	Proposition 1	Random-rotation control in Table 15
Two search procedures supply candidates	Algorithm 1	Ranked path alone keeps the task order at 0.99 (Table 19 )
A single-seed sweep places a window of four grid layers	Section A.1.6	240 of 266 core cells use the four-layer window

Appendix

Table 1: Claim-to-evidence map for Section 2 . Each row pairs one statement of the main text with its formal object in this appendix and with the quantity that supports it.

Component	Setting	Defined in
Input length	768 tokens, 1024 for mathematical-step correctness	A.1.2
Input format	Raw text for base models, one chat-template user turn for instruction-tuned models	A.1.2
Representation	Residual stream after a block, mean over all attended positions	A.1.1
Numerics	Forward pass in bfloat16, stored features in float32	A.1.2
Standardization	Training-split mean and standard deviation per feature	A.1.4
Probe	Logistic regression with intercept, L2 penalty, class-balanced weights	A.1.4

Appendix

Table 2: The main protocol at a glance. Each setting holds for all 19 models and 24 tasks unless the middle column states otherwise, and the last column names the section that defines and justifies it.

Task	Source	Decisions	Train	Test	Positive rate	Split
Language identification	( Keung et al., 2020 ; Conneau et al., 2018 ; Cer et al., 2017 )	10	8000	3000	0.09 to 0.10	test
Tool selection	( Glaive AI, 2023 )	4	3300	840	0.05 to 0.12	hold-out
Routing	GSM8K vs. MATH ( Cobbe et al., 2021 ; Hendrycks et al., 2021 )	1	6000	2000	0.50	test
Intent (CLINC150)	( Larson et al., 2019 )	20	8000	3000	0.01	test
Intent (Banking77)	( Casanueva et al., 2020 )	10	3420	780	0.05	test
PII detection	( ai4privacy, 2023 )	1	6000	2000	0.49	hold-out

Appendix

Table 3: Task statistics. Each row is one task, and each class decision is probed separately. Train and test give the sizes of the pools shared by every model, and the positive rate is the fraction of positives in the training pool, as a range for tasks with several class decisions. The split column states whether the test pool is the source’s test split, its development split, or a 20% hold-out formed as in Section A.1.3 . The lower block lists the ten additional tasks evaluated only on Gemma-2-9B.

Figure 6: Reading a local count off one probe cell. Each panel shows Gemma-2-9B on one task at the window layer where seed 1 attains its smallest local count. The dark curve is the prefix curve

A_{\mathrm{rank}}(k)

of seed 1, the thin curves are the other four seeds at the same layer, and the hollow diamonds are the L1 path candidates of seed 1 placed at their support sizes, with supports above 1024 lying outside the axis. The solid horizontal line marks the threshold

\tau_{c}

of Equation 4 , the dotted horizontal line marks the dense ceiling

C_{c}

over all window layers and seeds, and the dashed vertical line marks the local count of Equation 5 , the smallest qualifying size on either path. The gray vertical line and each title give the task-level count of Table 11 , the median over five seeds after neighbor confirmation (Appendix A.1.6 ). (a) Intent classification, class account_blocked, layer 13, ceiling 0.9997,

\tau_{c}=0.975

. The prefix curve qualifies at

k^{*}=2

, layer 17 ties at 2, and the task-level count of 4 is the median over 20 class decisions. (b) Jailbreak detection, layer 17, ceiling 0.9999,

\tau_{c}=0.975

. An L1 support of 6 qualifies before the prefix grid point 8, and neighbor confirmation with layer 20 returns 8. (c) Sentiment, layer 20, ceiling 0.980,

\tau_{c}=0.956

. An L1 support of 71 qualifies before the prefix grid point 128, layers 17 and 24 need 128 for this seed, and the task-level median over seeds is 72. (d) Harmfulness, layer 20, ceiling 0.979,

\tau_{c}=0.955

. An L1 support of 137 qualifies, while layers 17 and 24 need 341 and 342, so neighbor confirmation returns 341.

Figure 7: Local counts across depth on Gemma-2-9B and the five-seed window inside them. Each panel shows one task in the mean-pooled representation. The blue line traces the preliminary single-seed sweep on the full training split over the ten grid layers, giving at each layer the median local count over class decisions under the sweep’s ceiling for each decision. An open circle at the

\infty

tick marks a layer where no candidate qualified, which happens at layer 5 for sentiment and for harmfulness detection. The shaded band covers the layers of the five-seed measurement, which are 24 to 35 for language identification, 9 to 20 for intent classification and banking intent, 13 to 24 for jailbreak detection, 17 to 28 for harmfulness detection, and the five-layer follow-up 13 to 28 for sentiment. Inside the band, small gold dots give the median local count over class decisions for each seed under the common ceiling of the window, gold diamonds give the median of the five seeds at each layer, and the black line gives the reported count after neighbor confirmation and the medians over class decisions and seeds. The reported counts read 1, 4, 8, 16, 72, and 341, and the median dense ceilings over class decisions read 1.000, 1.000, 1.000, 0.997, 0.980, and 0.979, in panel order. The five seeds spread by at most one count at the window layers of language identification, intent classification, and jailbreak detection, and they spread from 61 to 128 at layer 17 for sentiment. Neighbor confirmation keeps single-layer minima out of the reported count, as at layer 17 for jailbreak detection, where the seed median of 7 sits below the reported 8, and at layer 20 for harmfulness detection, where the seed median of 141 sits below the reported 341. The sweep and the five-seed medians differ most for harmfulness detection at window layers 17 and 20, where the single-seed sweep reads 149 and 164 and the five-seed medians read 341 and 141.

	Dense AUROC by seed					Local count by seed
Layer	1	2	3	4	5	1	2	3	4	5
17	0.9748	0.9751	0.9763	0.9759	0.9750	341	333	361	351	339
20	0.9787	0.9779	0.9782	0.9790	0.9788	137	137	141	142	141
24	0.9768	0.9756	0.9776	0.9769	0.9772	342	332	342	344	340
28	0.9719	0.9709	0.9720	0.9715	0.9724	369	369	341	352	347
Confirmed						341	332	342	344	339

Appendix

Table 4: Every local count behind one reported cell. Harmfulness detection on Gemma-2-9B. The left block gives the dense-probe test AUROC at each window layer and seed, and the right block gives the local count of Equation 5 at the common threshold. The last row applies neighbor confirmation of Equation 14 within each seed. Every local count in this cell comes from the L1 path.

Quantity	Median	90th percentile	Maximum
Standard error $\widehat{\mathrm{se}}(C_{c})$ , all 1025 decisions	0.0015	0.0090	0.0208
Standard error $\widehat{\mathrm{se}}(C_{c})$ , 941 readable decisions	0.0011	0.0071
Worst-case radius $\varepsilon_{0.05}$	0.36		0.41
Gap $0.05\,(C_{c}-0.5)$	0.025		0.025
Decisions with $1.96\,\widehat{\mathrm{se}}(C_{c})$ below the gap, all	92.1%
Decisions with $1.96\,\widehat{\mathrm{se}}(C_{c})$ below the gap, readable	99.4%

Appendix

Table 5: Test-split precision of the 1025 class-decision ceilings. The standard error is Equation 20 at the ceiling with the test class counts of the class decision, the worst-case radius is Equation 18 at

\delta=0.05

, and the gap is

0.05\,(C_{c}-0.5)

. Readable class decisions are those with

C_{c}>0.90

Base model	Blocks	$d$	Probed layers	IT	SAE suite	Features
Gemma-3-270M	18	640	1, 3, 5, 6, 8, 9, 10, 12, 15, 17	yes	Gemma Scope 2	16384
Gemma-3-1B	26	1152	1, 4, 7, 8, 11, 13, 15, 17, 18, 22, 25	yes	Gemma Scope 2	16384
Gemma-3-4B	34	2560	1, 6, 9, 10, 15, 17, 19, 22, 24, 28, 29, 33	yes	Gemma Scope 2	16384
Gemma-2-9B	42	3584	5, 9, 13, 17, 20, 24, 28, 31, 35, 39	yes	Gemma Scope	16384
Gemma-3-12B	48	3840	1, 8, 12, 14, 21, 24, 27, 31, 34, 40, 41, 47	yes	Gemma Scope 2	16384
Gemma-3-27B	62	5376	1, 10, 16, 18, 27, 31, 35, 40, 44, 52, 53, 61	yes	Gemma Scope 2	16384

Appendix

Table 6: Models and probed layers. Blocks is the number of transformer blocks and

d

the residual width. Layer

\ell

denotes the residual stream after block

\ell

, with blocks indexed from 0, and the listed layers form the grid of the preliminary sweep. IT marks base models whose instruction-tuned counterpart is also in the study and probed on the same layers. The SAE suite and Features columns name the suite used for the SAE representation and give its dictionary size.

Choice	Reason	Measured effect of changing it
Relative threshold at 95%	A fraction of the above-chance ceiling applies the same relative requirement to tasks with different ceilings	90% and 97.5% keep the task order at 0.97 (Table 19 )
Two search paths	The ranked path has one-octave resolution and scores features one at a time, and the L1 path selects them jointly	The ranked path alone keeps the order at 0.99 (Table 19 )
Neighbor confirmation	A single layer can reach a small count by chance on a fixed test set	The best single layer keeps the order at 0.98 (Table 19 ), and Table 4 shows a cell where the two readings differ by a factor of 2.4
Linear probe	$k^{*}$ measures linear readout, which the dense ceiling also measures	A one-hidden-layer MLP gives the same count on 5 of 7 tasks and one grid step apart on 2 (Table 18 )
Model coordinates	The residual stream is the basis the model computes in, and Proposition 1 fixes the invariances	Random rotations raise counts by factors of 1.6 and 2.0 on two models, and by at most 20 in one cell (Table 15 )
Original wording	The pools are fixed across models, so wording is held constant	Paraphrased test inputs raise counts by 1.5 to 10.8 on six cells (Table 16 )

Appendix

Table 7: Why each design choice was made, and what changing it does. Task order is the Spearman correlation of Equation 23 between task-level

k^{*}

under a variant and under the main protocol, as a median over the 19 models. Appendix A.2.2 reports every control in full.

Reported quantity	Unit	Meaning
Count seed range	Training subsample and fit seed	Minimum and maximum on the same evaluation pool, not a confidence interval
Ceiling standard error	Test texts of one class decision	Equation 20 at the ceiling, conditional on the training data, with the worst case in Lemma 2
Cross-model $\rho$	Model pair over shared tasks	Descriptive rank agreement, with pairs that share models
Matching pass rate	Class decision and model pair	Fraction above a resampled-null threshold, with tests that share data and models
Slope interval	Resampled task block	Percentile interval with all student rows repeated per sampled task
Held-out error	Student and task residual	Empirical error after holding out each complete task

Appendix

Table 8: Uncertainty and missingness conventions. Each quantity is interpreted at its actual unit of analysis, and the paper never combines quantities with different units.

Experiment	Representation	Ceiling and aggregation	Details
Main counts and input-format factorial	Mean-pooled residual coordinates	Common dense maximum per class decision, neighbor confirmation, median over class decisions and then seeds	A.1.6 , A.3.2
SAE correspondence supports	Last-position SAE features	Qualifying support at the selected SAE layer, with a sweep ceiling that may come from other representations	A.1.10 , A.2.4
Controlled synthetic models	Query-position residual or its principal components	Layer-specific dense reference, minimum across readable layers	A.3.4
Continual learning	Mean-pooled residual coordinates	Dense maximum per class decision and neighbor-confirmed count, compared with the same chain’s baseline	A.4.7
Pruning	Mean-pooled residual coordinates	Median over class decisions per layer, then maximum over layers, compared with the main count	A.4.8
Label requirements	Student test AUROC	Median at the actual training size, first threshold crossing on a logarithmic budget grid	A.4.3

Appendix

Table 9: Which estimator belongs to which experiment. The main estimator is defined in Algorithms 1 and 2 , and the appendix in the last column specifies each variant.

Claim	Evidence	Statistic and reference
Counts differ across tasks but agree within a task	A.2.1 , Figure 2 a, Table 11	Cross-model spread ratio of Equation 24 , below 3.2 on 8 of 9 readable tasks
Models rank the tasks alike	A.2.1 , Table 12	Spearman $\rho$ of Equation 25 , median 0.93 over 30 cross-lab pairs, against a within-model permutation null with 99th percentile 0.21
The order survives protocol changes	A.2.2 , Tables 14 to 19	Order correlation with the main protocol, 0.97 to 0.99 for estimator variants and 0.61 to 0.72 for rotations
Absolute counts depend on basis and wording	A.2.2 , Tables 15 and 16	Multiplicative factors of 1.6 and 2.0 under rotation and 1.5 to 10.8 under paraphrase, with ceilings within 0.03
The agreement is not a shared text-derived order	A.2.1 , Table 13	Median cross-lab partial $\rho$ of Equation 26 , 0.86 to 0.92 under six text-side controls, against 0.93 without control
Selected SAE features correspond across models	A.2.4 , Tables 21 to 24 , Figure 8	Pass rate of Equation 34 against a nominal 1% under the label-correlation-matched null

Appendix

Table 10: Claims of Section 3 and the evidence behind each. The last column names the statistic and the reference value against which the main text reads each claim.

Model	Fact checking	Relevance	Math answer	Math step	Code bug
Gemma 3 270M	235 0.840	303 0.636	$\infty$ 0.646	94 0.751	$\infty$ 0.622
Gemma 3 1B	358 0.879	$\infty$ 0.647	$\infty$ 0.634	140 0.756	$\infty$ 0.643
Gemma 3 4B	206 0.888	128 0.639	256 0.654	223 0.763	$\infty$ 0.656
Gemma 2 9B	253 0.910	128 0.649	144 0.661	295 0.782	937 0.650
Gemma 3 12B	222 0.913	128 0.645	128 0.666	255 0.781	$\infty$ 0.670
Gemma 3 27B	239 0.887	128 0.644	8 0.675	128 0.789	$\infty$ 0.660

Appendix

Table 11: Tasks outside the common high-ceiling subset. Each cell shows the unrounded

k^{*}

followed by the dense-probe ceiling in small type. Muted counts have a ceiling

C\leq 0.90

and are not interpreted as reliable task sizes, and

\infty

means that no tested support qualifies. Together with Figure 2 a, these rows cover all 14 core tasks on the ten base models.

Model set and treatment	Same lab	Different labs	All pairs
Ten base models, all	$0.886$ [ $0.807$ , $0.967$ ], $n=15$	$0.911$ [ $0.818$ , $0.969$ ], $n=30$	$0.907$ [ $0.807$ , $0.969$ ], $n=45$
Ten base models, finite	$0.926$ [ $0.846$ , $0.981$ ], $n=15$	$0.904$ [ $0.771$ , $0.981$ ], $n=30$	$0.906$ [ $0.771$ , $0.981$ ], $n=45$
Ten base models, gated	$0.931$ [ $0.897$ , $1.000$ ], $n=15$	$0.929$ [ $0.817$ , $1.000$ ], $n=30$	$0.931$ [ $0.817$ , $1.000$ ], $n=45$
All 19 models, all	$0.892$ [ $0.750$ , $0.993$ ], $n=69$	$0.905$ [ $0.767$ , $0.969$ ], $n=102$	$0.901$ [ $0.750$ , $0.993$ ], $n=171$
All 19 models, finite	$0.889$ [ $0.718$ , $0.991$ ], $n=69$	$0.890$ [ $0.750$ , $0.981$ ], $n=102$	$0.890$ [ $0.718$ , $0.991$ ], $n=171$
All 19 models, gated	$0.929$ [ $0.785$ , $1.000$ ], $n=69$	$0.915$ [ $0.797$ , $1.000$ ], $n=102$	$0.920$ [ $0.785$ , $1.000$ ], $n=171$

Appendix

Table 12: Spearman correlation between two models’

k^{*}

orderings over the 14 core tasks. Each entry is the median over pairs with the minimum and maximum in brackets, and

n

is the number of pairs. The three treatments differ only in how they handle cells with no finite

k^{*}

or a ceiling below 0.90.

Task	$s(4)$	$s(8)$	$s(16)$	$s(64)$	$s(256)$	Text ceiling	Median $k^{*}$
Language ID	0.911	0.940	0.967	0.996	1.000	1.000	1
PII	0.789	0.842	0.884	0.994	0.998	1.000	1
Tool selection	0.995	1.000	1.000	1.000	1.000	1.000	1
Routing	0.780	0.826	0.899	0.969	0.985	0.995	2
Intent (CLINC150)	1.000	1.000	1.000	1.000	1.000	1.000	4
Jailbreak	0.679	0.832	0.890	0.938	0.973	0.997	12

Appendix

Table 13: Text-side readings of the nine readable tasks and the cross-model order they leave unexplained. Columns

s(k)

give the prefix score of Equation 30 on TF-IDF features of the raw text, the fraction of the text-only ceiling reached within

k

terms, and the text ceiling is that representation’s dense-probe AUROC. Rows are ordered by the median

k^{*}

over the ten base models, rounded for display. The first summary row is the median over models of the Spearman correlation between each model’s

k^{*}

and the column, and the second is the median over the 30 cross-lab pairs of the partial rank correlation of Equation 26 with the column as the control, against 0.929 without control.

Change	Measured on	$k^{*}$	Ceiling	Task order
Feature basis, random rotation	2 models, 22 cells	$\times 1.6$ and $\times 2.0$ , at most $\times 20$	$\leq 0.008$	$0.72$ and $0.61$
Input format, chat template	1 model, 2 tasks	$\div 1.2$ to $\div 5.7$	$\leq 0.02$	not measured
Wording, paraphrase	1 model, 6 cells	$\times 1.5$ to $\times 10.8$	$\leq 0.03$	kept at task level
Threshold, 90% or 97.5% of ceiling	19 models	moves by construction	unaffected	$0.97$ and $0.97$
Search, ranked path only	19 models	moves by construction	unaffected	$0.99$
Layers, best single layer	19 models	moves by construction	not reported	$0.98$

Appendix

Table 14: What each protocol choice moves. The upper block changes the representation or the inputs and the lower block changes the estimator. Effects on

k^{*}

are multiplicative factors, effects on the ceiling are absolute AUROC changes, and effects on task order are Spearman correlations against the main protocol, computed over 22 cells for the rotation and over the 14 tasks for the estimator variants. Two tasks do not define an order, so the chat-template row reports none.

Model	Task	Original $k^{*}$	Rotated $k^{*}$	Factor
Gemma-2-9B	Intent, 20 class decisions	4	8	$1.44$
	Step check	68	244	$3.59$
	Harmfulness	150	251	$1.67$
Gemma-3-4B	Intent, 20 class decisions	4	12	$2.38$
	Step check	128	214	$1.67$
	Harmfulness	139	142	$1.02$

Appendix

Table 15: Effect of a random rotation on

k^{*}

. Entries are medians over the three rotations, and the intent row is additionally a median over the 20 class decisions. The factor column is the median over cells of each cell’s own ratio, which is why it does not equal the ratio of the two preceding columns.

Task	Original	Paraphrased	Factor
Intent, account blocked	$1.000$ / 2	$1.000$ / 8	$4.0$
Intent, smart home	$1.000$ / 4	$1.000$ / 6	$1.5$
Intent, weather	$1.000$ / 4	$0.990$ / 16	$4.0$
Jailbreak	$1.000$ / 8	$0.982$ / 86	$10.8$
Sentiment	$0.980$ / 85	$0.970$ / 208	$2.4$
Harmfulness	$0.977$ / 166	$0.947$ / 389	$2.3$

Appendix

Table 16: Effect of paraphrasing the test inputs. Each entry is a ceiling followed by

k^{*}

, with Gemma-2-9B as the instrument. The factor is the ratio of the two

k^{*}

values.

Task	$k^{*}_{o\to o}$	$k^{*}_{o^{\prime}\to o^{\prime}}$	$k^{*}_{p\to p}$	$k^{*}_{o\to p}$	Sample size	Intrinsic width	Distribution shift
Intent, account blocked	2	2	8	8	$1.0$	$4.0$	$1.0$
Intent, smart home	4	8	8	6	$2.0$	$1.0$	$0.8$
Intent, weather	4	8	15	16	$2.0$	$1.9$	$1.1$
Jailbreak	8	8	45	86	$1.0$	$5.6$	$1.9$
Sentiment	85	86	128	208	$1.0$	$1.5$	$1.6$
Harmfulness	166	222	264	389	$1.3$	$1.2$	$1.5$

Appendix

Table 17: Decomposition of the paraphrase expansion by Equation 28 . The count columns give

k^{*}

for each train and test combination, and the factor columns multiply to the factor of Table 16 .

Task	Layer	Linear ceiling	Linear $k^{*}$	MLP ceiling	MLP $k^{*}$	Shuffled labels
Language id	35	$1.0000$	1	$1.0000$	1	$0.535$
Intent	5	$1.0000$	4	$0.9999$	4	$0.569$
Jailbreak	17	$0.9999$	8	$0.9983$	8	$0.552$
Banking intent	20	$0.9999$	8	$0.9990$	8	$0.577$
Sentiment	20	$0.9798$	128	$0.9825$	128	$0.521$
Fact check	24	$0.9093$	256	$0.9094$	512	$0.504$

Appendix

Table 18: Probe-family and capacity controls on Gemma-2-9B. The MLP column reports

k^{*}

at the linear arm’s threshold. The last column is the highest test AUROC reached at any

k

on the grid after permuting the training labels. This control uses one seed, the full training split, and one layer per task, so its absolute values differ from the five-seed values of Table 11 .

Variant	Median	Range
Threshold at 90% of the ceiling	$0.97$	$0.93$ to $0.99$
Threshold at 97.5% of the ceiling	$0.97$	$0.91$ to $0.99$
Ranked-feature path only, no L1 path	$0.99$	$0.91$ to $1.00$
Best single layer, no neighbor confirmation	$0.98$	$0.93$ to $0.99$

Appendix

Table 19: Task order under estimator variants. Each entry is a Spearman correlation against the main protocol over the 14 core tasks, summarized over the 19 models.

Features	Bag of words	Model	Model higher	Wilcoxon $p$
64	$0.919$	$0.991$	16 of 24	$0.029$
256	$0.969$	$0.998$	13 of 24	$0.099$

Appendix

Table 20: Fraction of its own ceiling each representation reaches within

k

features. Entries are medians over the 24 tasks of the prefix score of Equation 30 , the model is Gemma-2-9B, and the

p

value comes from a two-sided Wilcoxon signed-rank test over tasks.

	Cross-model matching				Dictionary
Task	Models	Tests	Match	Median $z$	Recovery	Null
Intent classification	8	447	56.2%	$+3.08$	$0.984$	$0.630$
Language identification	8	280	48.9%	$+3.71$	$0.996$	$0.022$
Tool selection	8	105	34.3%	$+1.68$	$0.533^{\ddagger}$	$0.020$
Core tasks the matching test cannot cover
Routing	7	21	supports of 1 to 2 features, every test degenerate

Appendix

Table 21: Where feature correspondence can be tested and what it shows. The upper block reports the three tasks the matching test covers. Matching runs over the eight base models with an SAE suite, and one test is one model pair on one class decision. The dictionary columns report the median over three models of a held-out class decision’s recovery and of its same-size random-dictionary null. The double dagger marks tool selection, whose four class decisions give a leave-one-out dictionary of a median 10 features, so its recovery reflects the power of the protocol rather than a negative result. The lower block lists the remaining core tasks with the number of the eight models that hold a support. Sections A.2.4 and A.2.5 give the per-pair and per-model readings.

Figure 8: Match rate for every model pair on intent classification. Each bar is the fraction of that pair’s shared intents whose one-to-one feature match exceeds the label-correlation-matched null’s 99th percentile. The dashed line marks the 1% rate the null allows, and every pair clears it. Dots mark cross-lab pairs. Pairs containing Gemma-3-270M occupy most of the lower half.

Intent	Match	Median $z$	Support	Intent	Match	Median $z$	Support
smart_home	$1.00$	$+5.56$	38	update_playlist	$0.46$	$+2.52$	8
ingredients_list	$0.96$	$+5.19$	12	spelling	$0.40$	$+1.59$	72
sync_device	$0.93$	$+4.23$	12	weather	$0.39$	$+2.26$	10
traffic	$0.79$	$+5.05$	16	yes	$0.33$	$-1.94$	50
account_blocked	$0.75$	$+4.55$	14	confirm_reservation	$0.33$	$-5.73$	155
goodbye	$0.71$	$+3.69$	28	change_volume	$0.33$	$-1.14$	63

Appendix

Table 22: Per-intent match rates on intent classification. The match rate is the fraction of the pairs holding a support for that intent whose observed match exceeds the null’s 99th percentile. Support is the median over models of the two supports’ mean size in features.

Figure 9: Cross-model SAE feature correspondence, test by test. Each test pairs two of the eight base models on one class decision and compares the observed one-to-one feature match with a null of equally sized feature sets matched in label correlation. (a) Histograms of the standardized score

z

over the 447 intent classification tests, the 280 language identification tests, and the 105 tool selection tests. The median

z

+3.08

+3.71

, and

+1.68

, positive scores make up 84%, 89%, and 78% of each task, and 56.2%, 48.9%, and 34.3% of tests exceed the null 99th percentile where the null allows 1%. Four intent tests and one language test with

z

above 20 fall in the last bin. (b) The same 832 tests plotted against the mean of the two support sizes on a logarithmic axis. Tests with equal support sizes are jittered horizontally, and the five tests with

z

above 20 appear as triangles at the top edge. Over the 447 intent tests the Spearman correlation between support size and

z

-0.09

with

p=0.07

Intent	Support	Full-size $z$	Reduced $z$
smart_home	38	$+5.56$	$+3.1$
income	31	$+4.51$	$+3.8$
goodbye	28	$+3.69$	$+2.7$
spelling	72	$+1.59$	$-0.6$
change_volume	63	$-1.14$	$-0.7$
yes	50	$-1.94$	$-1.4$

Appendix

Table 23: Median

z

before and after reducing every support to 20 features. The upper group matches at full size and the lower group, the four intents with supports of 50 or more, does not.

Group	Pairs	Intent	Language id	Tool selection
Contains the 270M model	7	$0.36$	$0.40$	$0.33$
Contains the 1B model	6	$0.56$	$0.65$	$0.25$
Both models at 4B or above	15	$0.63$	$0.40$	$0.50$

Appendix

Table 24: Median match rate by the scale of the pair. The three groups partition the 28 pairs. A pair enters a task’s column only when that task has at least one test for it.

Task	Model	Decisions	Dictionary	$k^{*}$	Recovery	Minimum	Null	Own overlap
Intent	Llama-3.1-8B	20	835	17	$0.984$	$0.790$	$0.630$	$0.16$
	Gemma-2-9B	19	499	12	$0.981$	$0.897$	$0.766$	$0.11$
	Qwen3-8B	18	1428	22	$0.989$	$0.889$	$0.562$	$0.24$
Language id	Llama-3.1-8B	10	15	1	$0.999$	$0.784$	$0.012$	$0.00$
	Gemma-2-9B	10	33	1	$0.990$	$0.955$	$0.079$	$0.00$
	Qwen3-8B	10	39	1	$0.996$	$0.654$	$0.022$	$0.08$

Appendix

Table 25: Leave-one-out dictionary recovery. Recovery, the null, and own overlap are medians over class decisions, and the minimum is the lowest recovery of any class decision.

Model	Language	$k^{*}$	Baseline	Treatment	Collapse	Corrected treatment	Separates
Gemma-2-9B-IT	Japanese	1	$+0.990$	$+0.015$	98%	not needed	yes
	Greek	4	$+0.890$	$+0.000$	100%	not needed	yes
	Hindi	4	$+0.880$	$+0.005$	99%	not needed	yes
	Bulgarian	1	$+0.915$	$+0.760$	17%	$+0.560$	yes
Gemma-3-27B-IT	Japanese	1	$+1.000$	$+0.960$	4%	not needed	yes
	Greek	1	$+1.000$	$+1.000$	0%	not needed	no

Appendix

Table 26: Ablating a language’s

k^{*}

support at the original magnitude. Baseline and treatment are gap values under the script readout of Equation 38 . The corrected column restates the treatment after discounting replies in a same-script neighbor language, for the cells where we recorded that diagnostic. The last column reports whether the treatment’s 95% bootstrap interval separates from all four controls, three frequency-matched and one equal-norm, under the script readout.

Model	Language	Baseline	$\times 1$	$\times 2$	$\times 4$	$\times 16$	Equal-norm noise
Gemma-3-27B-IT	Japanese	$1.000$	$0.960$	$0.000$	$0.000$	$0.000$	$1.000$ , $1.000$ , $0.960$ , $0.010$
	Greek	$1.000$	$1.000$	$0.870$	$0.000$	$0.000$	$1.000$ , $1.000$ , $1.000$ , $0.090$
	Hindi	$0.995$	$0.990$	$0.575$	$0.240$	$0.000$	$0.995$ , $0.995$ , $0.995$ , $0.160$
	Bulgarian	$1.000$	$0.105$	$0.000$	$0.000$	$0.000$	$1.000$ , $1.000$ , $1.000$ , $0.210$
			$\times 0.25$	$\times 0.5$	$\times 1$
Gemma-2-9B-IT	Japanese	$0.990$	$0.990$	$0.940$	$0.000$		$0.990$ , $0.990$ , $0.990$

Appendix

Table 27: Corrected in-language reply rates under scaled ablation. Each treatment column has an equal-norm random direction at the same scale, reported in the last column in the same order. The

16\times

column enters no conclusion because its noise control also destroys the behavior.

	Model	Intervention	Fitting	Treated gap	Control collapse	Separation
1	Gemma-2-9B-IT	SAE top-64 ablation	layer 20	$+0.275$	$-0.053$	separated
2	Gemma-2-9B-IT	Rank-1 LEACE	default	$+0.450$	$+0.015$	overlaps
3	Gemma-2-9B-IT	Rank-1 LEACE	layer 20, last token	$+0.420$	$-0.057$	separated
4	Gemma-2-9B-IT	Rank-1 LEACE	layer 20, last 3 tokens	$+0.415$	$+0.003$	separated
5	OLMo-2-7B-IT	Rank-1 LEACE	default	$+0.205$	$+0.007$	separated
6	OLMo-2-7B-IT	Rank-1 LEACE	layer 15, last token	$+0.255$	$+0.262$	overlaps

Appendix

Table 28: All seven fitted refusal configurations. Control collapse is the mean reduction in gap across that arm’s random controls. Configurations vary in edit type, layer, and activation pooling. A configuration is separated when its treated interval overlaps no control interval.

$n$	Gemma-2-9B-IT	OLMo-2-7B-IT	Mistral-7B-Instruct
16	$0.9341$ / $0.9826$	$0.9462$ / $0.9880$	$0.8836$ / $0.9779$
51	$0.9646$ / $0.9825$	$0.9674$ / $0.9880$	$0.9276$ / $0.9773$
64	$0.9676$ / $0.9827$	$0.9733$ / $0.9879$	$0.9310$ / $0.9772$
128	$0.9752$ / $0.9824$	$0.9817$ / $0.9875$	$0.9490$ / $0.9768$
256	$0.9783$ / $0.9822$	$0.9824$ / $0.9873$	$0.9631$ / $0.9768$
None pinned	$0.9825$	$0.9878$	$0.9778$

Appendix

Table 29: Sufficiency without exclusivity on harmfulness detection. Each cell gives the test AUROC of a probe on the top

n

coordinates alone, followed by the refitted dense AUROC after pinning those coordinates to their training means. The last row gives the dense AUROC with nothing pinned.

Claim	Evidence	Supporting statistic
Deployment reductions are task-selective on Gemma-2-9B	Table 32	Harmfulness 341 to 47 and fact checking 253 to 40, while five tasks with $k^{*}\leq 8$ keep a ratio of 1.00
Harmfulness reductions depend on the model	Tables 31 and 33	Ratios from 1.09 on Gemma-3-270M to 8.53 on Gemma-3-12B, with non-overlapping seed ranges in all six Gemma pairs
Weights and format both reduce counts	Table 34	Every single-factor change lowers the median count on both tasks
Harmfulness is approximately separable	Equation 43	Interaction ratio $I\approx 1.08$
Fact checking needs both changes	Equation 43	Interaction ratio $I\approx 4.87$
Factorial medians do not depend on the layer window	Table 34	Shared-layer medians reproduce all eight cells

Appendix

Table 30: Claims of Section 4 and their evidence. Each row names the claim, the place in this appendix that reports it, and the statistic that carries it. Seed ranges in the last column are minima and maxima over five seed-level task counts, not confidence intervals.

Model	Tasks	$\rho$	$k^{*}_{\rm base}$	$k^{*}_{\rm IT}$	Ratio
Gemma 3 270M	9	0.626	174	159	1.09
Gemma 3 1B	9	0.814	234	91	2.57
Gemma 3 4B	9	0.995	293	64	4.58
Gemma 2 9B	10	0.826	341	47	7.26
Gemma 3 12B	10	0.760	324	38	8.53
Gemma 3 27B	9	0.597	374	50	7.48

Appendix

Table 31: Selectivity across all nine base/IT pairs. Rank correlations compare base counts with base/IT count ratios over tasks that pass the ceiling gate on both sides. Harmfulness columns show the deployment comparison, which combines weights and input format. Ratios are descriptive and share the base count with the predictor, as Proposition 3 shows.

Task	Base / plain	IT / chat	Ratio
Harmfulness detection	341 [332, 344]	47 [44, 48]	7.26
Fact checking	253 [233, 256]	40 [39, 51]	6.33
Intent (Banking77)	16 [15.5, 16]	4 [4, 6]	4.00
Intent (CLINC150)	4 [3, 4]	2	2.00
Sentiment	72 [68, 128]	64 [60, 64]	1.12
Tool selection	1 [1, 1.5]	1	1.00

Appendix

Table 32: Every gated task on Gemma-2-9B. Counts are medians over five seeds with the seed range in brackets where the five seed-level counts differ. The ratio is the base count divided by the instruction-tuned count, as in Equation 40 , and 1.00 marks an unchanged count.

Model	Base / plain	IT / chat	Ratio
Gemma-3-270M	174 [162, 179]	159 [154, 161]	1.1
Gemma-3-1B	234 [231, 243]	91 [91, 92]	2.6
Gemma-3-4B	293 [256, 310]	64 [43, 64]	4.6
Gemma-2-9B	341 [332, 344]	47 [44, 48]	7.3
Gemma-3-12B	324 [308, 329]	38 [37, 40]	8.5
Gemma-3-27B	374 [365, 389]	50 [48, 54]	7.5

Appendix

Table 33: Harmfulness detection across Gemma sizes. Counts are medians over five seeds, and brackets give the seed range. The ratios repeat Table 31 at one decimal.

Task	Weights / input	$k^{*}$	Seed range	Ceiling
Harmfulness	Base / plain	341	[332, 344]	0.979
Harmfulness	Base / chat	135	[130, 141]	0.981
Harmfulness	IT / plain	128	[128, 128]	0.984
Harmfulness	IT / chat	47	[44, 48]	0.986
Fact checking	Base / plain	253	[233, 256]	0.910
Fact checking	Base / chat	217	[212, 222]	0.909

Appendix

Table 34: Seed ranges and ceilings for the four-cell design. Brackets give the minimum and maximum over five seed-level task counts, not confidence intervals. Restricting to the shared layers reproduces every median, with layers

17,20,24,28

for harmfulness and

20,24,28

for fact checking.

Family	Target subset	Control subset
thr2	0, 1	2, 3
thr4	4, 5, 6, 7	8, 9, 10, 11
thr8	12 to 19	16 to 23
thr16	0 to 15	16 to 31
and2	24, 25	26, 27
and4	28, 29, 30, 31	20, 21, 22, 23

Appendix

Table 35: Factor subsets of the 18 aggregate queries. Factors are numbered from 0 to 31. Target and control instances of a family use subsets of equal size. The eight-factor threshold pair shares factors 16 to 19, which forms the overlap control of Section A.3.4 .

Component	Setting
Transformer grid	Depth $\{2,4\}$ , width $\{64,128,256\}$ , seeds $\{0,1,2\}$ , giving 18 models
Transformer block	Pre-layer normalization, 4 attention heads, causal mask, GELU MLP of width $4d$ , learned positions, no dropout
Transformer output	Final layer normalization and an untied vocabulary projection without bias
Initialization	Transformer linear and embedding weights normal with standard deviation 0.02 and zero biases, with attention input projections, layer normalization, and the GRU at their library defaults
Recurrent grid	Two-layer GRU, width 256, seeds $\{0,1\}$ , giving 2 models
Optimizer	AdamW ( Loshchilov & Hutter, 2017 ) , learning rate $3\times 10^{-4}$ , weight decay 0.01, batch size 256

Appendix

Table 36: Architecture and optimization of the controlled synthetic models. The main 20-model grid uses these settings. Section A.3.4 covers the auxiliary width, duration, and trajectory runs.

Component	Main estimator	Synthetic readout
Position	Mean over attended positions	Query position
Seeds	Five probe seeds on 85% subsamples	One probe seed on all 4,000 training examples
Ceiling	Common maximum over window layers and seeds	Layer-specific dense AUROC
Candidates of one size	Any qualifying candidate	Median AUROC over candidates of that size
Layers	Neighbor confirmation over a four-layer window	Minimum over layers that pass the gate
Gate	Ceiling below 0.90 flagged	Layers at or below 0.90 excluded

Appendix

Table 37: Synthetic readout versus the main estimator. The two estimators share standardization, feature ranking, the probe objective, the regularization grid, and the L1 path of Appendix A.1.5 .

Family	Raw target	Raw control	PCA target	PCA control
thr2	1 (20/20)	37 (20/20)	1 (20/20)	31.5 (20/20)
thr4	1 (20/20)	53.5 (20/20)	1 (20/20)	48 (20/20)
thr8	1 (20/20)	60.5 (20/20)	1 (20/20)	64 (20/20)
thr16	1 (20/20)	52.5 (20/20)	1 (20/20)	32 (20/20)
and2	1 (20/20)	30 (20/20)	1 (20/20)	30 (20/20)
and4	1 (20/20)	29 (20/20)	1 (20/20)	16 (20/20)

Appendix

Table 38: Target and control readouts in the complete main toy grid. Each entry is the median count over readable models, followed by readable coverage. A dash indicates no model above the ceiling gate, not a large finite count. Raw and PCA bases are separate measurements.

Figure 10: Synthetic readout

k^{\mathrm{syn}}

for every target and control in the 20 main-grid models. Each marker is one model. Within each query family, gold diamonds show the supervised target, blue circles show the control that shares the factor count but answers a coin, and black bars mark the median over models. (a) Raw basis. Every target reads at

k^{\mathrm{syn}}=1

in all 20 models, and the control medians are 37, 53.5, 60.5, 52.5, 30, and 29 for thr2, thr4, thr8, thr16, and2, and and4. (b) PCA basis. Every target again reads at 1 in all 20 models, and the control medians are 31.5, 48, 64, 32, 30, and 16. The xor2 control passes the readability gate in none of the 20 models in either basis and appears as open markers in the band at the top. No control reaches a count of one in any model.

Step	thr8 target	xor2 target	xor4 target	xor8 target	thr8 control
500	1	—	—	—	69
1000	1	—	—	—	67
2000	1	1	—	—	53
4000	1	1	—	—	78
8000	1	1	—	—	65
16000	1	1	—	—	83

Appendix

Table 39: Readouts at saved training checkpoints. Four-layer width-256 transformer, training seed 0, query-token raw basis. Dashes mark failed readability gates. The checkpoints locate changes only at sampled times, and they cannot exclude a transition between observations.

Claim	Estimator	Display	Supporting statistic
Reference $k^{*}$ anticipates student label requirements	A.4.3	Eq. 57 , Table 42	Shared exponent 0.65 on 109 uncensored observations
The association carries task-level uncertainty	A.4.4	Table 44	Task-block interval $[0.44,0.91]$ , held-out factor 5.3
The association remains under transferability and text controls	A.4.5	Table 45	Partial $\rho$ of $k^{*}$ of $-0.67$ to $-0.79$ given LogME or the H-score and $-0.55$ to $-0.67$ given the bag-of-words ceiling
Saved heads fail while linear readability persists	A.4.7	Eq. 64	Head losses of 0.47 to 0.68 over all 18 chain seeds against ceiling losses of at most 0.029
Recovery needs fewer labels than initial learning	A.4.7	Tables 48 and 49	Labels to a common target, 101 to 244 for recovery against 194 to 602 for initial learning
Wider readouts lose more readability under pruning	A.4.8	Table 50	$\rho\geq 0.886$ with $q<0.05$ in 30 of 32 conditions

Appendix

Table 40: Claim-to-evidence map for Section 5 . Each row names a main-text claim, the subsection that defines its estimator, the display that reports it, and the statistic that carries it. The last two rows are intervention claims whose tested rules did not outperform their uniform and random baselines, and we report them with the same level of detail as the three predictive claims.

Setting	Initial learning	Sequential updates	Recovery
Trainable weights	LoRA and head	All backbone weights and head	LoRA and new head
Learning rate	$2\times 10^{-4}$	$10^{-5}$	$2\times 10^{-4}$
LoRA	$r=16$ , $\alpha=32$ , dropout 0.05	Not used	Same as initial learning
Batch size	16	16	16
Token limit	768, and 512 for the Gemma-3 students on the 14 core tasks	512	512
Label grid	16, 32, 64, 256, 512, 1024, 2048, 4096	Up to 4096 per step	16, 64, 256, 1024

Appendix

Table 41: Adaptation settings of the three procedures. Sequential updates train all backbone weights, whereas initial learning and recovery train rank-16 LoRA adapters and a head. The recovery label grid of four budgets from 16 to 1024 is coarser than the eight-point initial-learning grid, and Section A.4.7 states how this difference enters the comparison.

Task	$k^{*}$	G270M	G1B	G4B	L1B	L3B	Q0.6B	Q1.7B	O1B
Language ID	1	$\leq 16$	$\leq 16$	$\leq 16$	$\leq 16$	$\leq 16$	$\leq 16$	$\leq 16$	29
Tool selection	1	$\leq 16$	25	21	$\leq 16$	$\leq 16$	19	$\leq 16$	44
Routing	2	$\leq 16$	$\leq 16$	$\leq 16$	24	$\leq 16$	$\leq 16$	$\leq 16$	$\leq 16$
Intent (CLINC150)	4	316	28	35	164	387	239	24	282
PII	4	58	80	55	73	58	54	63	90
Jailbreak	8	59	61	41	62	52	30	29	94

Appendix

Table 42: Label requirements underlying the shared-exponent fit. Columns identify the eight students (G for Gemma 3, L for Llama 3.2, Q for Qwen 3, O for OLMo 2). Counts are rounded for display, and

\leq 16

denotes a left-censored requirement that the regression excludes. Rows contain all 16 contributing tasks, and the reference model is Gemma 2 9B.

Student	Tasks	Separate slope
G270M	13	0.694
G1B	14	0.696
G4B	14	0.641
L1B	14	0.640
L3B	12	0.454
Q0.6B	14	0.700

Appendix

Table 43: Student-specific descriptive fits. Each slope comes from a separate regression on the uncensored tasks of that student. By Proposition 8 the shared exponent of Equation 57 is a weighted average of these slopes, and the shared fit remains the primary analysis.

Figure 11: Learning curves behind the label requirement and the complete fit panel. Panel (a) shows the Gemma 3 1B student on six tasks whose reference counts in Gemma 2 9B are

k^{*}=1

(language identification), 8 (jailbreak), 16 (Banking77), 72 (sentiment), 253 (fact checking), and 341 (harmfulness). Each curve is the median test AUROC over seeds and class decisions at every realized training size from 16 to 4096 labels. The short horizontal tick marks the threshold

\tau=0.5+0.95\,[\bar{A}(n_{J})-0.5]

of Equation 55 , and the dot marks

N^{*}

from the geometric interpolation of Equation 56 . The curves give

N^{*}=61

for jailbreak, 232 for Banking77, 215 for sentiment, 1181 for fact checking, and 864 for harmfulness. Language identification already exceeds its threshold at 16 labels, so its requirement is left-censored at

N^{*}\leq 16

and appears as an open triangle. Panel (b) plots all 109 uncensored observations over 16 tasks and 8 students with one color per student and the fitted lines of Equation 57 , which share the slope

b=0.646

and differ only in their student intercepts. Open triangles at

N^{*}=16

mark the 19 left-censored cells, with the number of cells below each triangle (7 for language identification, 4 for tool selection, 7 for routing, and 1 for subjectivity), and the regression excludes them. Rings mark the five uncensored cells of panel (a).

Quantity	Value
Observations / tasks / students	109 / 16 / 8
Shared exponent	0.645848
95% task-block interval	[0.438945, 0.907697]
90th-percentile held-out log error	0.727170
Corresponding multiplicative error	5.335

Appendix

Table 44: Shared fit and task-level uncertainty. The percentile interval uses 10,000 task-block bootstrap draws with multiplicities retained. Held-out error leaves whole task blocks out.

Student	Predictor / control	$\rho$	$p$
Gemma 3 270M	$k^{*}\mid$ LogME	-0.668	0.0343
Gemma 3 270M	LogME $\mid k^{*}$	+0.140	0.7071
Gemma 3 270M	$k^{*}\mid$ H-score	-0.708	0.0228
Gemma 3 270M	H-score $\mid k^{*}$	+0.813	0.0028
Gemma 3 270M	$k^{*}\mid$ bag of words	-0.547	0.0957
Gemma 3 270M	Bag of words $\mid k^{*}$	+0.551	0.0970

Appendix

Table 45: Fixed-budget incremental validity on ten high-ceiling tasks. Partial rank correlations with student AUROC at 256 labels, where

x\mid z

denotes predictor

x

with control

z

and bag of words denotes the text-only ceiling of Appendix A.2.3 . The residual-permutation

p

values are not adjusted for multiplicity. These controls use the single-seed counts of the preliminary sweep and the requested 256-label runs, as specified in Appendix A.4.5 , which also reports the

N^{*}

-level control.

Control	Comparison	Interpretation
Reference-model replacement	Refit using another LM’s measured counts	Instrument sensitivity on the same student curves
LogME and H-score	Partial rank correlations at fixed budget	Incremental or complementary predictive information
Difficulty proxies	Add measured proxies individually	Sensitivity to measured difficulty, not exhaustive adjustment
Base and IT hard teachers	Same task, different teacher-probe labels	Label transfer can stay similar despite different teacher counts
Soft teachers	Train on teacher probabilities	Probability information beyond hard labels

Appendix

Table 46: What each learning control tests. Distinct controls address distinct alternative explanations, and no single control identifies the cause of the relation between counts and requirements.

Rule	Median ratio	Range	Cheaper
Pilot error	0.66	[0.56, 0.82]	18/18
Predicted $N^{*}$	1.03	[0.65, 1.98]	6/18
Reference $k^{*}$	4.03	[2.29, 13.02]	0/18

Appendix

Table 47: Budget allocation replay relative to uniform allocation. Ratios below one use fewer labels under the replay accounting. Only settings in which the uniform, pilot, count, and predicted-requirement policies all reach the target enter the 18-setting summary, and the oracle policy is excluded because its exact requirements are undefined for left-censored tasks.

Model	Target	Initial	Recovery	Recovery, own endpoint	Initial, 16 to 4096
Gemma-3-1B	0.931	602	101 [77, 129]	182 [133, 208]	864
Gemma-3-4B	0.951	520	155 [141, 229]	163 [163, 224]	478
Llama-3.1-8B	0.949	524	244 [237, 544]	236 [220, 456]	497
Qwen3-8B	0.947	194	126 [43, 164]	176 [61, 200]	269

Appendix

Table 48: Recovering harmfulness detection after the same three-task sequence requires fewer labels than initial learning. Both curves are read on the recovery grid of 16 to 1024 labels. The target is the AUROC at which initial learning attains 95% of its gain above chance at 1024 labels, and the first requirement columns give the labels that each procedure needs to reach it. Recovery values give the median and, in brackets, the range over three chain seeds. The last two columns give recovery

N^{*}

against its own endpoint and initial

N^{*}

on the full grid of 16 to 4096 labels.

Model	Labels	Median AUROC	Chain range
Gemma 3 1B	16	0.754	[0.749, 0.862]
Gemma 3 1B	64	0.920	[0.908, 0.926]
Gemma 3 1B	256	0.955	[0.955, 0.965]
Gemma 3 1B	1024	0.970	[0.970, 0.971]
Gemma 3 4B	16	0.763	[0.720, 0.784]
Gemma 3 4B	64	0.915	[0.827, 0.931]

Appendix

Table 49: Harmfulness recovery curves after the common three-task chain. We first aggregate the three recovery seeds within each final backbone and then report the median and range over three chain seeds. These endpoints supplement the label requirements in Table 48 .

Figure 12: Saved heads fail while new probes keep reading harmfulness along the common chain. Each panel follows one model through the chain that trains harmfulness, then sentiment, then jailbreak detection, with three chain seeds. Gold circles give the AUROC of the harmfulness head saved after step 1 and evaluated on each later backbone without refitting. Blue squares give the dense-probe ceiling of new probes fitted on the frozen backbone after each step, and step 0 denotes the pretrained model. Markers show the median over the three chain seeds and whiskers span the seed range. The dashed line marks chance. The arrow and the labels report the final losses of Equation 64 from the seed medians. Head losses are 0.55, 0.56, 0.59 and 0.56 for Gemma-3-1B, Gemma-3-4B, Llama-3.1-8B and Qwen3-8B, and the ceiling losses are

-0.009

-0.001

0.002

and

-0.001

, where a negative value means that the final ceiling exceeds the pretrained ceiling. Over every step and seed of these twelve chains the largest harmfulness ceiling loss stays below 0.010.

Model	Method	30%	50%	60%	70%	80%	90%
Gemma 3 1B	Magnitude	0.429	0.943	0.886 0.0178	0.886 0.0178	0.600 0.1208	0.886 0.0178
Gemma 3 1B	Wanda	0.943	0.600	0.886 0.0178	0.886 0.0178	0.600 0.1208	0.886 0.0178
Gemma 2 9B	Magnitude	0.257	0.943	1.000 0.0074	0.943 0.0127	0.943 0.0127	0.886 0.0178
Gemma 2 9B	Wanda	0.257	0.943	1.000 0.0074	0.943 0.0127	0.943 0.0127	0.943 0.0127
OLMo 2 7B	Magnitude	0.714	0.771	0.943 0.0127	0.943 0.0127	0.943 0.0127	0.943 0.0127
OLMo 2 7B	Wanda	0.257	1.000	1.000 0.0074	0.943 0.0127	0.886 0.0178	0.886 0.0178

Appendix

Table 50: Correlation of pre-pruning counts with ceiling losses. Cells give Spearman

\rho

between the six baseline counts and the unrounded ceiling losses of Equation 67 . At 60% to 90% sparsity, the small number beside each correlation gives the Benjamini–Hochberg

q

over those 32 conditions. Each one-sided test enumerates all

6!=720

permutations. Low-sparsity correlations are descriptive, and at 30% sparsity every maximum ceiling loss is below 0.007.

Task	Sparsity	Ceiling	Recovery	Gap
Harmfulness	50%	0.929	0.910	0.020
Sentiment	50%	0.909	0.895	0.014
Harmfulness	70%	0.914	0.876	0.038
Sentiment	70%	0.877	0.836	0.041
Harmfulness	80%	0.905	0.862	0.042
Sentiment	80%	0.874	0.831	0.043

Appendix

Table 51: Pruning ceilings and recovery at 1,024 labels. Gemma 3 1B with Wanda, median LoRA recovery AUROC across three seeds. The gap is the ceiling minus the recovery AUROC, computed before rounding. These are finite-budget observations, not asymptotic limits.

Model	Task	50%	70%	80%	90%
Gemma 3 1B	Banking77	0.989	0.996	0.994	0.992
Gemma 3 1B	Jailbreak	0.996	0.995	0.996	0.993
Gemma 3 1B	Sentiment	0.895	0.836	0.831	0.837
Gemma 3 1B	Harmfulness	0.910	0.876	0.862	0.873
Gemma 2 9B	Banking77	0.993	0.994	0.987	0.991
Gemma 2 9B	Jailbreak	0.998	0.997	0.993	0.993

Appendix

Table 52: Wanda recovery endpoints across models. Median test AUROC at 1,024 labels over the available LoRA recovery seeds. A dash denotes an unavailable endpoint. These arms are separate from the single-model Gemma 3 1B ceiling comparison in Table 51 .

Model	Task	50%	70%	80%	90%
Gemma 3 1B	Banking77	0.988	0.940	0.887	0.882
Gemma 3 1B	Jailbreak	0.996	0.986	0.977	0.967
Gemma 3 1B	Sentiment	0.897	0.705	0.594	0.582
Gemma 3 1B	Harmfulness	0.905	0.775	0.762	0.755

Appendix

Table 53: Magnitude recovery endpoints across models. Median test AUROC at 1,024 labels over the available LoRA recovery seeds. A dash denotes an unavailable endpoint. These arms are separate from the single-model Gemma 3 1B ceiling comparison in Table 51 .

Task	Sparsity	LoRA	Full parameters
Sentiment	50%	0.895	0.912
Sentiment	70%	0.836	0.835
Sentiment	80%	0.831	0.807
Sentiment	90%	0.837	0.838
Harmfulness	50%	0.910	0.904
Harmfulness	70%	0.876	0.881

Appendix

Table 54: Adapter versus full-parameter recovery. Gemma 3 1B with Wanda at 1,024 labels, median AUROC across seeds. Full-parameter and LoRA runs use their respective default learning rates, so the comparison is between procedures and not a capacity-only ablation.

Requested	Arm	Protected dims	Actual	Sentiment	Harmfulness
70%	Ordinary	0	—	0.947	0.948
70%	Jailbreak support	48	—	0.944	0.947
70%	Single-layer support	385	0.675	0.931	0.940
70%	Selected union	969	—	0.891	0.925
70%	Random control	975	0.636	0.888	0.925
80%	Ordinary	0	—	0.904	0.932

Appendix

Table 55: Complete support-protection comparison. Gemma 2 9B with Wanda. Protected dimensions count the distinct residual coordinates in

P

, actual sparsity is

s_{\mathrm{act}}

of Equation 70 , and a dash marks a value that was not recorded. Task columns give post-pruning dense-probe ceilings, maximized over the evaluation layers. All three requested sparsities and all five arms are shown.

Result	Procedure	Randomness	Estimator
Student learning curves	LoRA and head, Table 41 , Eqs. 51 and 53	Seeds 0, 1, 2 per budget	Eq. 54
Label requirements	Definition 1 on realized sizes	Inherited from the curves	Eqs. 55 and 56
Shared exponent	Least squares with student intercepts	None	Eq. 57 , Proposition 8
Exponent interval	Task-block bootstrap	10,000 draws, seed 0	Eq. 59
Held-out error	Leave one task out	None	Eq. 60
Partial correlations	Rank residuals at 256 labels	10,000 permutations, seed 0	Eq. 61

Appendix

Table 56: Ingredients needed to reproduce each application result. Task pools are those of Table 3 , reference counts and ceilings come from Algorithm 2 on Gemma-2-9B unless a subsection states otherwise, and every run evaluates on the fixed test split of its task.

Language Model Probing

Language model probing measures what information and capabilities models encode and how those signals relate to their behavior.

See all 275 papers

Learning to Read Out: Unembedding Dynamics in Language Model Pretraining
Oct 14, 2026Matteo He, William F. Shen, Alex Iacob +3Language Model Pretraining
Department of Computer Science and Technology, University of Cambridge · Flower Labs · Isomorphic Labs
Probing for Reading Times
Apr 20, 2026Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re +4Computational Psycholinguistics Language Model Probing
ETH Zürich · Toyota Technological Institute at Chicago · University College London
Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States
Jun 1, 2026Subramanyam Sahoo, Vinija Jain, Aman Chadha +1LLM Interpretability Representation Geometry in Language Models
Horizon Research · Meta · Apple +1

Oct 14, 2026J/K move · Enter open · S save

Learning to Read Out: Unembedding Dynamics in Language Model Pretraining

Matteo He, William F. Shen, Alex Iacob, Andrej Jovanovic+2

Department of Computer Science and Technology, University of Cambridge · Flower Labs · Isomorphic Labs

Language Model Pretraining

Open paper AskPDF arXiv

When does a language model acquire a capability? Developmental studies usually answer this by tracking when models start producing the correct token. But that prediction passes through the unembedding matrix, or readout, which is itself still being learned during pretraining. We show that a model's hidden states can encode task information before its own readout expresses it, so timelines read from outputs conflate what the model encodes with what its readout expresses. To track the readout directly, we introduce parameter-trajectory crosscoding, which fits one sparse dictionary to every token's readout row at every checkpoint. Unlike crosscoders on activations, it needs only checkpoint weights. We use it to locate a narrow early window, 0.4-1.1% of the way through Pythia's pretraining, in which the readout reorganizes and WordNet word categories become decodable from its rows. By then, linear probes on the hidden states already decode task information that the model's own outputs do not yet show. To test whether such gaps depend on how fast the readout learns, we pretrain small models from scratch that differ only in the readout's learning rate. We find that raising it makes the readout develop earlier and the gap close sooner, with little effect on final loss. Together, these results imply that early in training, where many developmental transitions are dated, a behavioral date can reflect when the readout learns to express task information and not when the model learns it.

Abstract

Language Model Probing

Explore similar work

Learning to Read Out: Unembedding Dynamics in Language Model Pretraining

Probing for Reading Times

Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States