Target-Token Asymmetry Can Masquerade As Depth Specialization

Danlin Li; Miao Li; Xu Han; Jiecheng Lu

cs.CLOct 14, 2026

Target-Token Asymmetry Can Masquerade As Depth Specialization

Authors: Danlin Li, Miao Li, Xu Han, Jiecheng Lu

Organizations: Georgia Institute of Technology · AWS

Abstract

Layer-ablation studies often perturb early or late layers of a language model, compare the damage to prediction on two corpora, and read a larger early-layer effect on one corpus as early-layer specialization for its task. Such a comparison averages over the tokens that each corpus asks the model to predict, and these target tokens differ: math solutions, for example, ask for far more whitespace and numbers than encyclopedia prose. We show that this difference can create the appearance of depth specialization. We split a cross-corpus early--late damage interaction by target-token type and recompute it under a target mixture shared by both corpora. Between GSM8K and Wikitext, a positive interaction replicates across five model instances. In the three fully decomposed models, only whitespace has a positive interaction. Removing whitespace reverses the pooled effect in all three, and matching the full target mixture reverses it in two. The diagnosis holds under position matching, adversarial reassignment of token types, and a continuous readout. The risk can also be anticipated from the tokenizer alone. A screen computed before any model runs places every held-out pair it scored in the region its label predicts, and on the pairs it rates at high or medium risk, removing whitespace reverses every positive pooled estimate. A smaller content-word effect one token ahead follows the share of content targets that the unablated model already predicts. A cross-corpus depth effect therefore supports an interpretation of task specialization only after the target-token mixture is reported, the effect is decomposed by token type, and a mixture-matched summary keeps its sign.

Figures & tables

Figure 1: The argument on one model instance: Qwen2.5-7B, GSM8K against Wikitext, three tokens ahead, mean-blend, top-1 accuracy. Panel a quotes two real evaluated positions, one chip per token with the scored target filled, selected for readability and target type (Appendix B ), and marks the two blended layer bands. Panel b gives each corpus’s H=3 target-type shares in all three models. Panel c fills in the four damage terms of Eq. ( 2 ) and gives each model’s pooled CAI. Panel d gives the conditional CAI of each target type, Eq. ( 3 ), and panel e sets the pooled estimate beside two summaries

M(q)

and the directly measured non-whitespace CAI. Under the two corpora’s combined mixture

q

, with whitespace share

5.3\%

M(q)=0.053\times 0.518-0.023=+0.004

. The pooled

+0.0253

weights each corpus by its own mixture:

M(\pi_{\mathcal{R}})=+0.0207

, the mixture shift adds

+0.0057

, and per-sequence averaging adds

-0.0011

. Intervals are 95%, and Table 1 lists the values.

Estimate	Qwen2.5-7B	Llama-3.2-3B	gemma-2-2b-it	$n_{\mathcal{N}}$
	ws $8.1\%$ / $2.5\%$	ws $9.5\%$ / $2.8\%$	ws $8.2\%$ / $2.7\%$
pooled	$+0.0253$	$+0.0428$	$+0.0421$	300
	[ $0.0211,0.0292$ ]	[ $0.0389,0.0468$ ]	[ $0.0376,0.0460$ ]
whitespace	$+0.5181$	$+0.8215$	$+0.7155$	217/217/218
	[ $0.4786,0.5560$ ]	[ $0.7920,0.8538$ ]	[ $0.6879,0.7420$ ]
content word	$-0.0077$	$-0.0076$	$-0.0019$	300

Table 1: Canonical GSM8K/Wikitext H=3 mean-blend top-1 audit on 300 sequences per corpus: the pooled estimate, every conditional cell, and three summaries (whitespace excluded, all non-whitespace targets measured as one cell, and the reference mixture over all five cells), each with its 95% sequence-bootstrap interval. Under each model, the whitespace share of the evaluated targets in GSM8K and in Wikitext.

n_{\mathcal{N}}

counts the Wikitext sequences that supply a target of that type (Qwen/Llama/Gemma), and every cell uses all 300 GSM8K sequences. The whitespace cell rests on 3,348/3,526/3,281 GSM8K and 1,093/1,171/1,150 Wikitext target tokens, and Table 6 gives all counts.

Figure 2: The positive contrast arises from late-band mean-blend on GSM8K whitespace. It persists across instances, positions, and strengths from

\alpha=0.1

0.4

, and its whitespace concentration is largest under mean-blend three tokens ahead. Panel a gives H=3 whitespace accuracy, averaged within sequences, and the share of positions whose top-1 prediction is whitespace, under control and the two blended bands. The chip is the modal late-band top-1 token. Panel b gives the diagnostic interaction

I

of Eq. ( 4 ) in the five same-sequence audit instances. Panel c gives the whitespace CAI of the position-control sample by relative-position quartile, after position matching, and by distance from the final answer span. The dashed line is each model’s all-positions estimate. Panel d gives the operator

\times

horizon grid, and panel e the whitespace cell, the pooled CAI and the directly measured non-whitespace CAI at five mean-blend strengths, with

\alpha=0.2

shaded. Intervals are 95%. Appendix Tables 9 , 10 , 16 , 51 and 52 list the values.

Label	Input rule (tokenizer only)	Observed region and interval criterion
boundary	focal corpus is source code, and either $\Delta_{\mathrm{ws}}>0.04$ or newline and indentation tokens exceed $25\%$ of its whitespace targets	not scored
high	$\Delta_{\mathrm{ws}}\geq 0.045$ , $\Delta_{\mathrm{chars}}\geq 0.50$ , and JS $\geq 0.01$	$I\geq 0.35$ with whitespace largest; $I$ interval above zero
low	$\|\Delta_{\mathrm{ws}}\|\leq 0.020$ and $\|\Delta_{\mathrm{chars}}\|\leq 0.35$	$\|I\|\leq 0.20$ and $\|\operatorname{CAI}_{\rm pooled}\|\leq 0.08$ ; $I$ interval contains zero
medium	$\Delta_{\mathrm{ws}}\geq 0.030$ or $\Delta_{\mathrm{chars}}\geq 0.35$	no held-out pair
ambiguous	otherwise	not scored

Table 2: The tokenizer-only screen. The input rule assigns the first label whose condition holds, and each label is scored against the observed region, which is computed from model outputs. “Whitespace largest” means that the H=3 mean-blend whitespace cell exceeds both zero and the content-word cell.

Figure 3: The tokenizer-only screen and every pair it labelled in advance. Panels a and b plot the diagnostic interaction

I

, measured from model outputs, against the two tokenizer-only gaps for the eleven calibration rows: characters per token,

\Delta_{\mathrm{chars}}

, in a and target whitespace rate,

\Delta_{\mathrm{ws}}

, in b , each with its least-squares line. Panel c gives both Pearson correlations over all eleven rows and with one tokenizer family or one corpus pair left out. Panel d lists the five held-out pairs and the two DoLa pairs with the screen’s two inputs, the label fixed before any intervention, and

I

with its 95% bootstrap interval in each primary model (Tables 26 and 29 ).

Pair	screen	$n$	Qwen2.5-7B	Llama-3.2-3B	gemma-2-2b-it	reversed
			pooled / whitespace excluded
GSM8K/Wikitext	high (cal.)	300	$+0.025$ / $-0.025$	$+0.043$ / $-0.033$	$+0.042$ / $-0.013$	3/3
GSM8K/TinyStories	high	200/195/200	$+0.025$ / $-0.029$	$+0.041$ / $-0.028$	$+0.028$ / $-0.024$	3/3
GSM8K/FACTOR-News	high	200	$+0.034$ / $-0.018$	$+0.047$ / $-0.027$	$+0.035$ / $-0.021$	3/3
MATH/Wikitext	medium	200	$-0.009$ / $-0.024$	$+0.006$ / $-0.031$	$+0.012$ / $-0.008$	2/2
DROP/Wikitext	medium	168/185/159	$+0.008$ / $-0.021$	$+0.018$ / $-0.028$	$+0.012$ / $-0.014$	3/3

Table 3: H=3 mean-blend top-1 CAI for every decomposed corpus pair, as pooled and whitespace-excluded estimates per model. “reversed” counts rows whose whitespace-excluded estimate is negative among rows with a positive pooled estimate.

n

is the number of sequences per corpus for Qwen/Llama/Gemma, and a value that rounds to zero is printed to its first significant digit. Table 1 and Table 23 give the 95% intervals, and Appendix I lists two further screen-only pairs and the condition count.

Appendix figures & tables53 assets

Supplementary material from the paper’s appendix.

Appendix

Case	Target type	Context excerpt
GSM8K H=3	whitespace	. That means the price for her ticket was 15 - 6 = $9. The answer is[[[SPACE]]]9
Wikitext H=3	content	12 Olympics . In July 2014 , she was featured in the ESPN ’s The Body[[ Issue]] .
Code H=3	whitespace	.X = vec1.X * vec2.X new_vec.Y = vec1.Y * vec2.Y[[\n]] return new_vec
GSM8K H=1 content cell	content	supplies remaining in the ship is now 240-144 = 96 pounds. The[[ answer]] is 96

Appendix

Table 4: Prompt-level cases illustrating the target-token confound and the code boundary. The bracketed token is the evaluated target.

Target type	Qwen2.5-7B	Llama-3.2-3B	gemma-2-2b-it
whitespace	$+0.5181$ [ $0.4786,0.5560$ ]	$+0.8215$ [ $0.7920,0.8538$ ]	$+0.7155$ [ $0.6879,0.7420$ ]
content	$-0.0077$ [ $-0.0097,-0.0056$ ]	$-0.0076$ [ $-0.0099,-0.0051$ ]	$-0.0019$ [ $-0.0040,0.0000$ ]
number	$-0.0181$ [ $-0.0233,-0.0129$ ]	$-0.0055$ [ $-0.0082,-0.0032$ ]	$-0.0246$ [ $-0.0313,-0.0191$ ]
punctuation	$-0.0981$ [ $-0.1099,-0.0867$ ]	$-0.1975$ [ $-0.2125,-0.1821$ ]	$-0.0350$ [ $-0.0482,-0.0202$ ]
function word	$-0.0257$ [ $-0.0324,-0.0194$ ]	$-0.0138$ [ $-0.0191,-0.0089$ ]	$-0.0154$ [ $-0.0211,-0.0094$ ]

Appendix

Table 5: Canonical H=3 mean-blend target-token decomposition on GSM8K versus Wikitext. Entries report CAI with 95% bootstrap intervals in brackets.

Figure 4: The pooled depth interaction for GSM8K versus Wikitext is carried by one target type. Panel a reports ablation damage by layer band, corpus, and model for whitespace targets, on a separate 200-sequence sample. Its interaction subtracts the late row of the last column from the early row. Panels b – d come from the canonical 300-sequence sample. Panel b shows each corpus’s target-token mixture at a three-token-ahead horizon (H=3), panel c gives the interaction conditional on target type with 95% intervals, and panel d places the pooled estimate beside

M(q)

at the combined target shares, split into its whitespace term and its other four terms, and beside the two mixture summaries. Table 1 lists the values of panel c and the pooled and summary values of panel d with their support, and Appendix A gives the three

M(q)

values.

Model	Corpus	content	number	punct.	whitespace	function	number	whitespace
		targets					sequences
Qwen2.5-7B	GSM8K	15,425	9,259	3,561	3,348	9,632	300	300
Qwen2.5-7B	Wikitext	21,193	2,884	5,696	1,093	12,245	219	217
Llama-3.2-3B	GSM8K	15,427	5,093	3,618	3,526	9,512	300	300
Llama-3.2-3B	Wikitext	20,786	1,609	5,603	1,171	12,051	218	217
gemma-2-2b	GSM8K	14,624	8,997	3,731	3,281	9,482	300	300

Appendix

Table 6: H=3 target counts in the canonical sample, 300 sequences per corpus. The whitespace-excluded row of Table 1 weights the four non-whitespace cells by the summed counts of the two corpora, and the reference-mixture row weights all five cells by the Wikitext (

\mathcal{N}

) counts. The last two columns count the sequences with at least one number or whitespace target. Every other type appears in all 300 sequences of each corpus.

Model	Domain	content	number	punctuation	whitespace	function-word
Qwen2.5-7B-Instruct	GSM8K	0.3742	0.2246	0.0864	0.0812	0.2336
Qwen2.5-7B-Instruct	Wikitext	0.4916	0.0669	0.1321	0.0254	0.2840
Llama-3.2-3B-Instruct	GSM8K	0.4150	0.1370	0.0973	0.0948	0.2559
Llama-3.2-3B-Instruct	Wikitext	0.5043	0.0390	0.1359	0.0284	0.2924
gemma-2-2b-it	GSM8K	0.3646	0.2243	0.0930	0.0818	0.2364
gemma-2-2b-it	Wikitext	0.4793	0.0708	0.1303	0.0267	0.2929

Appendix

Table 7: Target-type distributions for GSM8K (

\mathcal{R}

) and Wikitext (

\mathcal{N}

) at H=3.

Estimand	Qwen2.5-7B-Instruct	Llama-3.2-3B-Instruct	gemma-2-2b-it
raw pooled CAI	0.0253	0.0428	0.0421
$\Delta_{\mathrm{ws}}$ -rate	0.0559	0.0664	0.0551
ws-excluded, combined weights	−0.0247	−0.0330	−0.0131
ws-excluded, focal weights	−0.0233	−0.0294	−0.0143
reference-reweighted	−0.0121	−0.0116	0.0074
$M(\pi_{\mathcal{R}})$	0.0207	0.0513	0.0454

Appendix

Table 8: Mixture audit estimands for the canonical H=3 mean-blend interaction. The first block gives the estimands that test whether the broad H=3 interpretation survives target-token exclusion and reference-mixture reweighting.

\Delta_{\mathrm{ws}}

is the target whitespace-rate gap, a tokenizer-only quantity. The second block splits the pooled CAI by the identity of Appendix A , on the per-sequence records of Appendix C , which reproduce the first row to four decimals. The third block splits

M(q)

at the combined target shares, as drawn in Figure 4 d.

Model	Corpus	$n$	control	early	late	$D_{E}$	$D_{L}$	early contr.	late contr.
Qwen2.5-7B	GSM8K	300	0.0166	0.1273	0.7936	$-0.1107$	$-0.7770$	$-0.1013$	$+0.6194$
	Wikitext	217	0.0418	0.0512	0.1994	$-0.0094$	$-0.1576$
Llama-3.2-3B	GSM8K	300	0.0179	0.1747	0.9457	$-0.1567$	$-0.9277$	$-0.0453$	$+0.8667$
	Wikitext	217	0.0451	0.1566	0.1061	$-0.1114$	$-0.0610$
gemma-2-2b	GSM8K	300	0.0196	0.1438	0.8269	$-0.1242$	$-0.8073$	$-0.1040$	$+0.8194$
	Wikitext	218	0.0504	0.0706	0.0383	$-0.0202$	$+0.0121$

Appendix

Table 9: Whitespace accuracy at H=3 under control, early-band, and late-band mean-blend on the canonical sample, with the damage terms

D_{E}

and

D_{L}

and the number

n

of sequences with a whitespace target. The last two columns give the early and late contributions to the whitespace cell of Table 1 .

Model	Corpus	control	late	token	share	non-ws targets	of targets
		ws top-1		modal late top-1		ws top-1 at	ws share
Qwen2.5-7B	GSM8K	0.088	0.705	␣	0.705	0.700	0.081
	Wikitext	0.035	0.119	␣and	0.215	0.118	0.025
Llama-3.2-3B	GSM8K	0.102	0.906	␣	0.906	0.900	0.095
	Wikitext	0.036	0.066	␣,	0.672	0.066	0.028
gemma-2-2b	GSM8K	0.097	0.726	␣	0.726	0.717	0.082

Appendix

Table 10: Top-1 predictions behind Table 9 , from control and late-band mean-blend passes at

\alpha=0.2

on the same 300 sequences per corpus. The columns give the share of all positions whose top-1 token is whitespace under control and under the late band, the most frequent late-band top-1 token ( ␣ marks a space) and its share of positions, the share of non-whitespace H=3 targets at which the late band predicts whitespace, and the whitespace share of the H=3 targets.

Figure 5: The interaction persists across model instances and whitespace positions. Panel a gives the diagnostic interaction per instance with 95% intervals. Panel b shows whitespace and content-word CAI under each operator and horizon for all five same-sequence audit instances. Panel c gives the whitespace interaction by relative-position quartile and after position reweighting. Panel d groups the same estimate by distance from the final answer span, and Table 13 gives the event support.

Model	quartile	CAI	95% CI	events $\mathcal{R}/\mathcal{N}$
Qwen	Q1	+0.422	[0.327, 0.513]	349/215
	Q2	+0.492	[0.403, 0.577]	407/205
	Q3	+0.460	[0.380, 0.537]	723/197
	Q4	+0.431	[0.341, 0.518]	778/184
Llama	Q1	+0.612	[0.522, 0.696]	371/225
	Q2	+0.919	[0.845, 0.992]	370/194

Appendix

Table 11: Whitespace CAI by relative-position quartile. Event support is reported as GSM8K/Wikitext. Sequence support varies because a sequence need not contain a whitespace event in every quartile.

Model	position-matched CAI	95% CI
Qwen2.5-7B	+0.447	[0.395, 0.496]
Llama-3.2-3B	+0.791	[0.754, 0.828]
gemma-2-2b	+0.626	[0.589, 0.663]

Appendix

Table 12: Position-matched whitespace estimate. The GSM8K depth term is reweighted to Wikitext’s relative-position decile weights. The Wikitext term retains its empirical distribution.

Model	bin	CAI	95% CI	sequences/events
Qwen	inside answer	+0.485	[0.334, 0.635]	200/200
	1–4 tokens	+0.323	[0.126, 0.520]	46/46
	5–16 tokens	+0.386	[0.275, 0.492]	169/331
	17+ tokens	+0.457	[0.407, 0.507]	196/1680
Llama	inside answer	+0.936	[0.848, 1.017]	200/200
	1–4 tokens	+0.649	[0.512, 0.782]	68/68

Appendix

Table 13: GSM8K whitespace CAI by distance from the final answer span, each compared with position-matched Wikitext. Support is GSM8K sequences/events.

Model	total layers	early band	late band
Qwen2.5-7B	28	0–8	18–27
Llama-3.2-3B	28	0–8	18–27
gemma-2-2b	26	0–7	16–25

Appendix

Table 14: Layer bands used for the position control. Qwen and Llama use early layers 0–8 and late layers 18–27 of 28 total layers.

Model	$\mathrm{mb}_{H=3,\mathrm{ws}}$	$\mathrm{mb}_{H=3,\mathrm{cont}}$	$\mathrm{rs}_{H=1,\mathrm{ws}}$	$\mathrm{rs}_{H=1,\mathrm{cont}}$	$I$	95% CI
Qwen2.5-7B-Instruct	0.5066	−0.0074	0.0020	0.0072	0.5193	[0.4705, 0.5621]
gemma-2-2b-it	0.6820	−0.0022	0.0087	0.0094	0.6850	[0.6534, 0.7192]
Llama-3.2-3B-Instruct	0.7954	−0.0072	−0.0005	0.0102	0.8133	[0.7811, 0.8436]
gemma-2-9b-it	0.6015	−0.0019	0.0093	0.0093	0.6034	[0.5729, 0.6325]
Hermes-3-Llama-3.1-8B	0.7879	−0.0069	0.0069	0.0089	0.7968	[0.7597, 0.8312]

Appendix

Table 15: Same-sequence 2

\times

2 audit cells for

I=(\mathrm{mb}_{H=3,\mathrm{ws}}-\mathrm{mb}_{H=3,\mathrm{cont}})-(\mathrm{rs}_{H=1,\mathrm{ws}}-\mathrm{rs}_{H=1,\mathrm{cont}})

. The estimates use the audit sample and bootstrap analysis.

Model	Operator	H=1 ws	H=1 content	H=3 ws	H=3 content
Qwen2.5-7B	mean-blend	−0.149	−0.093	+0.518	−0.008
Qwen2.5-7B	mild-resample	−0.001	+0.011	−0.013	−0.001
Llama-3.2-3B	mean-blend	+0.208	−0.206	+0.821	−0.008
Llama-3.2-3B	mild-resample	−0.004	+0.012	−0.009	−0.0001
gemma-2-2b	mean-blend	+0.231	−0.116	+0.715	−0.002
gemma-2-2b	mild-resample	−0.004	+0.009	+0.014	−0.001

Appendix

Table 16: Operator

\times

horizon

\times

target-type grid on GSM8K/Wikitext. Values are top-1 CAI. Mean-blend cells use the target-type decomposition, whereas mild-resample cells use the effect-size analysis.

Figure 6: Whitespace concentration is largest under one operator and horizon, and checks outside that cell yield small positive content-token effects. Panel a shows whitespace and content-word CAI over the operator

\times

horizon grid. Panel b shows two measurements of the H=1 mild-resample content-word CAI: 250 sequences per corpus with 95% intervals, and 200 per corpus with Cohen’s

d

. Panel c gives the activation-patching check at top-1 by target type, with the same-corpus donor above and the other-corpus donor below. Panel d reports unablated logit-lens top-1 accuracy at H=1, GSM8K minus Wikitext, averaged within each layer third. Table 16 records the grid cell by cell, and Appendix G.2 gives the patching and logit-lens protocols.

Model	Readout	pooled	ws	content	nonws	ws-excl.	ref-rw.
Qwen2.5-7B	top-1	+0.026	+0.464	−0.008	−0.023	−0.021	−0.010
Qwen2.5-7B	logprob	−1.031	+0.444	−1.062	−1.298	−1.222	−1.381
Llama-3.2-3B	top-1	+0.042	+0.797	−0.008	−0.033	−0.030	−0.011
Llama-3.2-3B	logprob	+0.141	+1.448	−0.266	−0.114	+0.056	−0.166
gemma-2-2b	top-1	+0.041	+0.682	−0.002	−0.016	−0.015	+0.007
gemma-2-2b	logprob	+0.249	+0.961	−0.046	+0.097	−0.012	−0.147

Appendix

Table 17: H=3 mean-blend CAI under top-1 and log-probability readouts, split by target type on GSM8K/Wikitext. The top-1 rows provide a matched comparison for log probability. “ws-excl.” renormalizes over non-whitespace types, and “ref-rw.” applies Wikitext target-type proportions.

Model	H=1 whitespace	H=1 content	H=3 whitespace	H=3 content
Qwen2.5-7B	+1.661 [1.338, 1.974]	−0.535 [−0.758, −0.313]	+1.721 [1.413, 2.035]	−1.188 [−1.354, −1.013]
Llama-3.2-3B	+2.198 [1.971, 2.420]	−0.819 [−1.039, −0.617]	+3.374 [3.101, 3.644]	−0.388 [−0.530, −0.246]
gemma-2-2b	+1.229 [1.010, 1.444]	+0.172 [−0.018, 0.368]	+1.593 [1.374, 1.803]	−0.279 [−0.420, −0.145]

Appendix

Table 18: Target-margin CAI on GSM8K/Wikitext. Brackets are 95% bootstrap intervals.

Model	H=1 pooled	H=1 ws	H=1 cont.	H=1 nonws	H=3 pooled	H=3 ws	H=3 cont.	H=3 nonws
Qwen2.5-7B	+0.1332	+0.0778	+0.1918	+0.1365	+0.0004	−0.0047	+0.0012	+0.0005
Llama-3.2-3B	+0.1908	+0.1961	+0.2460	+0.1928	+0.0021	−0.0041	+0.0051	+0.0025
gemma-2-2b	+0.1678	+0.1004	+0.1984	+0.1657	+0.0026	−0.0167	+0.0020	+0.0030

Appendix

Table 19: Non-CAI logit-lens target-type audit. Values are late-band top-1 domain differences on GSM8K/Wikitext, averaged over the final third of layers.

Model	content same	content cross	nonws same	nonws cross
Qwen2.5-7B	+9.725 [8.891, 10.576]	−4.723 [−5.510, −3.842]	+8.632 [7.798, 9.365]	−4.763 [−5.515, −3.949]
Llama-3.2-3B	+1.616 [1.093, 2.186]	+0.048 [−0.483, 0.581]	+2.225 [1.772, 2.715]	+0.694 [0.149, 1.172]
gemma-2-2b	+1.654 [1.108, 2.178]	−0.218 [−0.784, 0.325]	+1.641 [1.183, 2.105]	−0.170 [−0.692, 0.299]

Appendix

Table 20: Activation-patching target-type check. Values are H=1 log-probability effects with 95% bootstrap intervals.

Pattern	Risk	Required check / result here
Layerwise readout	Pooling next-token scores across naturalistic corpora	Target-type cells and mixture matching
Layer contrast / DoLa	Corpus-level layer buckets may covary with format	GSM8K/FACTOR high-risk in 3/3. StrategyQA/FACTOR low-region in 3/3
Causal tracing / patching	Localization score depends on target population as well as intervention	Log target id and condition on task-appropriate targets
Ablation CAI	Replicated pooled interaction may inherit a stable mixture gap	Canonical pooled effect becomes negative after whitespace removal in 3/3

Appendix

Table 21: At-risk reporting patterns and the evidence needed before a broad depth claim. DoLa motivates the two corpus pairs measured here with our operators, and the other rows identify settings in which the diagnostic applies.

Experimental pattern	Why it is at risk	What the audit changes
Layerwise top-1 next-token readouts over math/prose or code/prose corpora, as in logit-lens, tuned-lens, or patchscope-style studies when summarized by naturalistic-domain accuracy	The readout pools different target-token populations before assigning a depth label	Report target-type cells and reweighted estimates before interpreting a replicated layer peak as depth specialization
Layer-contrast or intervention scores aggregated over naturalistic domain pairs, including DoLa-style layer contrasts or activation-patching variants when used as domain-level next-token metrics	The layer or intervention contrast can inherit tokenizer and target-mixture asymmetries from the benchmark	Surviving non-whitespace cells and mixture-matched estimates support the broad claim. Their collapse identifies a stable measurement confound
Fixed-target factual tracing or prompt-local causal restoration, such as ROME-style factual association localization	The target token is usually part of the constructed prompt or fact, so domain mixture is a secondary threat there	The audit becomes relevant when the same tools are scaled to naturalistic domain aggregates
Scenario- or option-level evaluations with matched answer formats	Target-token composition is partly controlled by the benchmark design	Report the format control. Target-type auditing becomes important if results are converted into free-form next-token aggregates

Appendix

Table 22: Reporting patterns covered by the target-type audit.

Pair	model	$n$	pooled	95% interval	ws-excl.	95% interval
GSM8K/TinyStories	Qwen	200	$+0.0254$	[0.0209, 0.0298]	$-0.0291$	[−0.0321, −0.0262]
	Llama	195	$+0.0407$	[0.0358, 0.0453]	$-0.0280$	[−0.0307, −0.0255]
	Gemma	200	$+0.0276$	[0.0211, 0.0336]	$-0.0238$	[−0.0273, −0.0201]
GSM8K/FACTOR-News	Qwen	200	$+0.0340$	[0.0297, 0.0386]	$-0.0182$	[−0.0213, −0.0151]
	Llama	200	$+0.0467$	[0.0420, 0.0514]	$-0.0266$	[−0.0296, −0.0236]
	Gemma	200	$+0.0351$	[0.0286, 0.0413]	$-0.0207$	[−0.0247, −0.0167]

Appendix

Table 23: H=3 mean-blend top-1 CAI for the ten expansion pairs of Table 3 , one row per pair and model (30 rows), pooled and with whitespace excluded, each with its 95% sequence-bootstrap interval. Each of 2,000 draws resamples sequences within each corpus and recomputes the cells, with the weights held at the screen’s target-type shares. The intervals come from an evaluation pass that repeats each audit’s sequences, operator, bands, and horizon and reproduces every pooled value, conditional cell, and mixture summary of the audit to within

10^{-6}

. The number cells of GSM8K/TinyStories and TinyStories/AGNews, with 4 and 1 sequence pairs, fall below the support rule of 10 and stay out of the whitespace-excluded summary, as in the audit, and Table 24 gives every row’s support. Measured directly over all non-whitespace targets, numbers included, the CAI of these six rows has the sign of their whitespace-excluded values, from

-0.0314

-0.0289

on GSM8K/TinyStories and from

+0.0064

+0.0222

on TinyStories/AGNews. A draw in which a retained cell falls below the rule is dropped, which leaves 1,750 to 1,997 draws in five whitespace-excluded rows, the three ARC rows and the Qwen and Llama SQuAD rows. A value that rounds to zero is printed to its first significant digit.

Pair	model	$n$	pairs	targets	pairs	targets	targets
			whitespace		number		non-whitespace
GSM8K/TinyStories	Qwen	200	89	2,600 / 208	4 ∗	7,002 / 4	28,837 / 33,433
	Llama	195	84	2,722 / 192	4 ∗	3,977 / 4	27,058 / 32,332
	Gemma	200	193	2,631 / 770	4 ∗	7,036 / 4	29,125 / 32,955
GSM8K/FACTOR-News	Qwen	200	195	2,339 / 733	196	6,201 / 2,090	25,533 / 28,488
	Llama	200	195	2,447 / 745	196	3,632 / 1,180	23,729 / 27,605

Appendix

Table 24: Support behind Table 23 . A sequence pair is the focal and the reference sequence at one index. Each sequence receives three passes, the unablated control pass and the early-band and late-band mean-blend passes, and a pair enters a cell when both sequences have an H=3 target of that type in all three. The content, punctuation, and function-word cells use every pair in every row except the code/docstrings function-word cells, at 193, 193, and 192 of 200 for Qwen, Llama, and Gemma. Target counts are the H=3 targets of each corpus in the control pass, focal / reference, and the non-whitespace column sums the four other types. ∗ marks a cell below the support rule of 10 pairs, which is not estimated. The six starred number cells stay out of their rows’ whitespace-excluded summaries. The starred whitespace cell, Llama on SQuAD/AGNews, enters neither printed value of Table 23 , since the pooled value uses all targets and the whitespace-excluded value drops whitespace.

Pair	Pred.	$\Delta_{\rm ws}$	$\Delta_{\rm chars}$	JS	Selection reason
GSM8K/TinyStories	high	+0.0739	+0.8052	0.0952	Large positive whitespace-rate and chars/token gaps in a prose pair
SQuAD/AGNews	low	+0.0077	+0.3276	0.0074	New QA/news pair with a small target-mixture gap
AGNews/Wikitext	low	−0.0099	+0.0055	0.0036	Non-reasoning prose/prose baseline with small gaps
CodeSearchNet code/docs	boundary	+0.1414	+0.0771	0.0537	Structural whitespace flag with newline/indentation-heavy whitespace
TinyStories/AGNews	ambiguous	−0.0202	−0.0383	0.0374	Weak or inconsistent tokenizer-side signal

Appendix

Table 25: Prospective selection of held-out pairs based on tokenizer-side features.

\Delta_{\rm ws}

\Delta_{\rm chars}

, and target-type Jensen–Shannon divergence (JS) are means over the three tokenizers.

Pair	Model	predicted region	$n$	H=3 pooled	ws-excl.	audit $I$ [95% CI]
GSM8K/TinyStories	Qwen	high	200	+0.0254	−0.0291	+0.6579 [0.6252, 0.6874]
	Llama	high	195	+0.0407	−0.0280	+0.7522 [0.7223, 0.7805]
	Gemma	high	200	+0.0276	−0.0238	+0.7088 [0.6844, 0.7360]
SQuAD/AGNews	Qwen	low	118	+0.0245	+0.0221	+0.0573 [−0.1302, 0.2287]
	Llama	low	119	−0.0017	−0.0120	−0.0727 [−0.1763, 0.0225]
	Gemma	low	138	+0.000019	−0.0082	+0.0421 [−0.0615, 0.1420]

Appendix

Table 26: Held-out audit outcomes across three model families. Each

n

is the number of matched sequences per side. Rows labelled boundary or ambiguous in advance are reported unscored. The audit counts support within each corpus. Llama’s SQuAD/AGNews

I

uses an H=3 whitespace cell from 28 SQuAD and 38 AGNews sequences with a whitespace target, and 8 matched pairs have one on both sides, below the paired rule of Table 24 .

Pair	Model	$n$ /side	pooled	ws	ws-excl.	ref-rw.	$I$	95% CI
MATH/Wikitext	Qwen	200	−0.0087	+0.4324	−0.0242	−0.0058	+0.5183	[0.4504, 0.5841]
	Llama	200	+0.0062	+0.6431	−0.0314	+0.0027	+0.6895	[0.6249, 0.7527]
	Gemma	200	+0.0120	+0.5188	−0.0079	+0.0124	+0.5878	[0.5285, 0.6471]
DROP/Wikitext	Qwen	168	+0.0078	+0.4327	−0.0211	−0.0105	+0.4265	[0.3723, 0.4794]
	Llama	185	+0.0179	+0.7584	−0.0283	−0.0079	+0.7605	[0.7217, 0.7985]
	Gemma	159	+0.0123	+0.4588	−0.0143	−0.0019	+0.4925	[0.4568, 0.5277]

Appendix

Table 27: MATH, DROP, and ARC extension against Wikitext across three model families. Values are H=3 mean-blend top-1 CAI except

I

, which is the interaction in Eq. 4 . Intervals use 2,000 sequence-level bootstrap resamples. Across these nine rows and the three canonical rows of Table 1 , 12 model–pair rows, the whitespace-rate gap and pooled CAI have Spearman

\rho=0.9371

, with agreement in 60 of 66 pairwise orderings.

Pair	$\Delta_{\mathrm{ws}}$	$\Delta_{\mathrm{chars}}$	JS	Label
MATH/Wikitext	+0.0080	+1.8287	0.0252	medium
DROP/Wikitext	+0.0301	+0.5212	0.0126	medium
ARC/Wikitext	−0.0167	+0.0923	0.0127	low
GSM8K/FACTOR-News	+0.0619	+1.1306	0.0372	high
StrategyQA/FACTOR-News	−0.0068	+0.0865	0.0040	low

Appendix

Table 28: Screen inputs of the reasoning and DoLa-motivated pairs, as means over the three tokenizers, and the label each pair receives under Table 2 .

Pair	Model	$n$ /side	pooled	ws	ws-excl.	ref-rw.	$I$	95% CI
GSM8K/FACTOR-News	Qwen	200	+0.0340	+0.5374	−0.0182	−0.0052	+0.5289	[0.4832, 0.5733]
	Llama	200	+0.0467	+0.7878	−0.0266	−0.0060	+0.7883	[0.7494, 0.8270]
	Gemma	200	+0.0351	+0.7598	−0.0207	−0.0043	+0.7566	[0.7196, 0.7932]
StrategyQA/FACTOR-News	Qwen	150	−0.0048	−0.0141	−0.0055	−0.0080	−0.0507	[−0.1023, 0.0034]
	Llama	135	−0.0070	−0.0118	−0.0075	−0.0070	−0.0409	[−0.0982, 0.0119]
	Gemma	149	+0.0009	+0.0335	+0.0011	+0.0017	+0.0337	[−0.0078, 0.0738]

Appendix

Table 29: DoLa-motivated corpus audit. The first six rows are prospective. The last three are GSM8K/Wikitext calibration replications and are excluded from the 41-condition coverage count. Intervals use 10,000 sequence-level bootstrap resamples.

Set	pooled-positive rows	ws-excl.	ref.-rw.
High/medium risk	14	14	11
Other decomposed rows	10	2	n/a

Appendix

Table 30: Aggregate sign reversals among rows with positive pooled estimates. The two right-hand columns count reversals under whitespace removal (“ws-excl.”) and under the reference mixture (“ref.-rw.”). The high- and medium-risk set comprises the canonical, GSM8K/TinyStories, pooled-positive MATH/DROP, and GSM8K/FACTOR-News rows.

Domain pair	Model	$\Delta_{\rm chars}$	$\Delta_{\rm ws}$	audit $I$	$p$
GSM8K/Wikitext	Qwen2.5-7B	1.1960	0.0559	0.5193	$<10^{-4}$
GSM8K/Wikitext	Llama-3.2-3B	0.8890	0.0664	0.8133	$<10^{-4}$
GSM8K/Wikitext	gemma-2-2b	1.3120	0.0551	0.6850	$<10^{-4}$
GSM8K/Wikitext	gemma-2-9b	1.2610	0.0527	0.6034	$<10^{-4}$
GSM8K/Wikitext	Hermes-3-Llama-3.1-8B	0.8292	0.0642	0.7968	$<10^{-4}$
Code/Wikitext	Qwen2.5-7B	−0.0595	0.1043	−0.0272	1.0

Appendix

Table 31: Calibration-screen risk rows.

\Delta_{\rm chars}

is the mean number of characters per token in

\mathcal{N}

minus

\mathcal{R}

\Delta_{\rm ws}

is the target whitespace rate in

\mathcal{R}

minus

\mathcal{N}

Figure 7: The full screen figure. Panels a and b plot the diagnostic interaction

I

for the eleven calibration rows against the characters-per-token gap and against the target whitespace-rate gap. Panel c gives both Pearson correlations over all eleven rows and after leaving out one tokenizer family or one corpus pair. Panel d shows

I

with its 95% bootstrap interval for the five held-out pairs on Qwen2.5-7B, grouped by label, and panel e compares three H=3 mean-blend quantities for each held-out pair: the whitespace cell, the whitespace-excluded summary, and the pooled value. Figure 3 repeats panels a to c , and its panel d extends d to all three primary models and the two DoLa pairs.

Domain pair	$n$	$\Delta_{\rm chars}$	$\Delta_{\rm ws}$	audit $I$
GSM8K/Wikitext	5	1.097 [0.829, 1.312]	0.0589 [0.0527, 0.0664]	0.684 [0.519, 0.813]
Code/Wikitext	3	0.283 [−0.060, 0.921]	0.1191 [0.1026, 0.1505]	0.109 [−0.027, 0.285]
HotpotQA/AGNews	3	0.382 [0.220, 0.493]	0.0212 [0.0111, 0.0392]	−0.018 [−0.191, 0.146]

Appendix

Table 32: Domain-pair-level aggregation for the calibration screen. Each cell reports mean [min, max] over model rows in that domain pair.

Region	Observed predictor range	Interpretation
GSM8K/Wikitext	$\Delta_{\rm ws}=0.053$ – $0.066$ , $\Delta_{\rm chars}=0.83$ – $1.31$ , $I=0.52$ – $0.81$	High-risk prose pair. Both predictors agree, and all five rows have zero non-positive bootstrap draws
Code/Wikitext	$\Delta_{\rm ws}=0.103$ – $0.150$ , $\Delta_{\rm chars}=-0.06$ – $0.92$ , $I=-0.03$ – $0.28$	Boundary case. Whitespace targets are frequent and mostly newlines or indentation
HotpotQA/AGNews	$\Delta_{\rm ws}=0.011$ – $0.039$ , $\Delta_{\rm chars}=0.22$ – $0.49$ , $I=-0.19$ – $0.15$	Lower-risk prose pair. Two rows are null or reversed, and one Gemma row is weakly positive

Appendix

Table 33: Calibration-screen risk regions. Held-out domain-pair outcomes appear in Table 26 .

Predictor	Pearson $r$	Spearman $\rho$
Target whitespace-rate gap	0.093	0.209
Chars-per-token gap	0.783	0.718

Appendix

Table 34: Predictor correlations over the 11 model rows. The rows form three domain-pair clusters.

Dropped lineage	$n$	Pearson ws	Spearman ws	Pearson chars	Spearman chars
Gemma	7	0.192	0.179	0.771	0.643
Llama	7	0.064	0.214	0.901	0.964
Qwen	8	0.006	0.310	0.778	0.619

Appendix

Table 35: Leave-one-model-lineage sensitivity. The characters-per-token correlations stay positive with any one lineage left out.

Dropped pair	$n$	Pearson ws	Spearman ws	Pearson chars	Spearman chars
GSM8K/Wikitext	6	0.666	0.486	0.538	0.600
HotpotQA/AGNews	8	−0.689	−0.524	0.784	0.429
Code/Wikitext	8	0.961	0.905	0.771	0.619

Appendix

Table 36: Leave-one-domain-pair sensitivity. Removing the code pair restores the direct relationship with target whitespace rate because whitespace serves a different function in code and prose.

Model	$R$ ws rate	$N$ ws rate	$R$ targets	$N$ targets	$R$ chars/token	$N$ chars/token	$\Delta_{\rm chars}$
Qwen2.5-7B	0.1319	0.0277	23307	26118	4.4728	4.4132	−0.0595
Llama-3.2-3B	0.1332	0.0306	23297	25991	4.4909	4.4788	−0.0121
gemma-2-2b	0.1786	0.0281	25321	26528	3.6877	4.6089	0.9211

Appendix

Table 37: Target rates for Code/Wikitext.

Model	$n_{R}/n_{N}$	$R$ ws rate	$N$ ws rate	$R$ targets	$N$ targets	$R$ chars/token	$N$ chars/token	$\Delta_{\rm chars}$
Qwen2.5-7B	146/146	0.0301	0.0167	5789	6403	4.2404	4.6743	0.4339
Llama-3.2-3B	135/135	0.0292	0.0181	5336	5856	4.4513	4.6715	0.2201
gemma-2-2b	161/161	0.0565	0.0173	6179	6950	4.2427	4.7354	0.4926

Appendix

Table 38: Target rates for HotpotQA/AGNews.

Model	$R$ ws rate	$N$ ws rate	$R$ targets	$N$ targets	$R$ chars/token	$N$ chars/token	$\Delta_{\rm chars}$
gemma-2-9b-it	0.0813	0.0286	26947	28403	3.3460	4.6070	1.2610
Hermes-3-Llama-3.1-8B	0.0945	0.0303	24743	26392	3.6596	4.4888	0.8292

Appendix

Table 39: Tokenizer statistics for the additional GSM8K/Wikitext audit models.

Pair	Domain	ws rate	plain	newline	multi/indent	other	lead cont.
GSM8K/Wikitext	GSM8K	0.0888	1.000	0.000	0.000	0.000	0.9086
GSM8K/Wikitext	Wikitext	0.0276	1.000	0.000	0.000	0.000	0.8598
Code/Wikitext	Code	0.1454	0.039	0.266	0.687	0.007	0.4644
Code/Wikitext	Wikitext	0.0281	1.000	0.000	0.000	0.000	0.8647

Appendix

Table 40: Whitespace subtype robustness at H=3, averaged over the Qwen2.5-7B, Llama-3.2-3B, and gemma-2-2b tokenizers. “ws rate” is the all-whitespace target rate. Subtype columns are proportions within whitespace targets. “lead cont.” is the proportion of content-word targets whose decoded token begins with whitespace and is classified after stripping. These tokenizer-only rates use their own 200 length-matched sequences per corpus, so they differ slightly from the canonical sample’s in Table 7 and the screen’s in Table 37 .

Model	policy	whitespace	content	number	punctuation	function
Qwen	canonical	+0.4525	−0.0084	−0.0107	−0.1026	−0.0216
	lead-only	+0.4525	−0.0090	−0.0107	−0.1026	−0.0182
	nolead	+0.4525	−0.0018	−0.0107	−0.1026	−0.0137
	adversarial	+0.0552	−0.0018	−0.0107	−0.0624	n/a
Llama	canonical	+0.8137	−0.0081	−0.0058	−0.1973	−0.0140
	lead-only	+0.8137	−0.0082	−0.0058	−0.1973	−0.0133

Appendix

Table 41: H=3 mean-blend top-1 CAI under four fixed classifier policies, using 200 sequences per side and 10,000 resamples. “n/a” denotes an empty or undefined function-word cell under the adversarial map. Under that map, whitespace CAI shrinks by factors of 8.2, 8.7, and 8.4 for Qwen, Llama, and Gemma. The shared positive Gemma nolead/adversarial content cell is

+0.003

, with intervals

[-0.001,0.010]

and

[-0.0007,0.010]

Model	policy	whitespace	content	number	punctuation	function
Qwen	canonical	441	146438	28	4106	630
	lead-only	441	52484	28	4106	94584
	nolead	441	93954	28	4106	53114
	adversarial	54409	93954	28	2985	267
Llama	canonical	530	121495	1206	4136	633
	lead-only	530	57295	1206	4136	64833

Appendix

Table 42: Vocabulary class sizes for Table 41 . Each entry counts vocabulary types. Under the adversarial map, whitespace accounts for approximately 36%, 46%, and 49% in Qwen, Llama, and Gemma.

Model	CAI	$d$	rel.%	ctrl $R$	ctrl $N$	early $R$	early $N$	late $R$	late $N$	$R/N$ chars
Qwen2.5-7B	0.0107	0.3507	1.22	0.8768	0.3904	0.0499	0.0350	0.0735	0.0479	3.260/4.456
Llama-3.2-3B	0.0117	0.3898	1.89	0.6194	0.3658	0.0604	0.0398	0.0858	0.0535	3.656/4.545
gemma-2-2b	0.0091	0.3407	1.51	0.6004	0.3277	0.0480	0.0289	0.0673	0.0392	3.335/4.647
Hermes-3-Llama-3.1-8B	0.0103	0.3370	1.64	0.6297	0.4262	0.0658	0.0490	0.0909	0.0638	3.656/4.545

Appendix

Table 43: Scale diagnostics for the four canonical H=1 content-token mild-resample rows. CAI is the absolute top-1 interaction. Rel.% is

100\times

CAI divided by the reasoning-domain control top-1 level.

Model	CAI	95% CI	$n_{R}/n_{N}$	Positive cell?
Qwen2.5-7B-Instruct	0.0103	[0.0050, 0.0149]	250/250	yes
Llama-3.2-3B-Instruct	0.0142	[0.0089, 0.0193]	250/250	yes
gemma-2-2b-it	0.0086	[0.0044, 0.0135]	250/250	yes
OLMo-2-1124-7B-Instruct	0.0148	[0.0092, 0.0207]	250/250	yes
Qwen3-1.7B	0.0125	[0.0076, 0.0172]	250/250	yes

Appendix

Table 44: Supporting H=1 content-token mild-resample results under the same protocol. “Positive cell?” means that the reported 95% bootstrap interval is strictly positive. The intervals use 1,000 sequence-bootstrap resamples. Table 43 gives the four-row Cohen’s

d

result with its paired per-sequence diagnostics.

Target type	Model	CAI	95% CI	$n_{\mathcal{R}}/n_{\mathcal{N}}$	Positive cell?
whitespace	Qwen2.5-7B-Instruct	−0.0235	[−0.0465, −0.0030]	250/181	no
whitespace	Llama-3.2-3B-Instruct	−0.0028	[−0.0208, 0.0155]	250/179	no
whitespace	gemma-2-2b-it	−0.0076	[−0.0208, 0.0046]	250/182	no
whitespace	OLMo-2-1124-7B-Instruct	−0.0073	[−0.0303, 0.0107]	250/179	no
whitespace	Qwen3-1.7B	0.0038	[−0.0081, 0.0166]	250/181	no
number	Qwen2.5-7B-Instruct	0.0038	[−0.0054, 0.0134]	250/182	no

Appendix

Table 45: H=1 mild-resample top-1 CAI on GSM8K/Wikitext for the four target types outside Table 44 , under the same protocol and the same 250 sequences per corpus.

n_{\mathcal{R}}/n_{\mathcal{N}}

counts the sequences supplying a target of the row’s type, so whitespace and number rows draw on fewer Wikitext sequences. “Positive cell?” means that the reported 95% bootstrap interval, from 1,000 sequence-bootstrap resamples, is strictly positive.

Model	$p_{\mathcal{R}}/p_{\mathcal{N}}$	CAI	Correct	Wrong	Reweighted	95% CI	Diff.
content word
Qwen2.5-7B	0.88/0.39	$+0.0103$	$-0.0037$	$-0.0021$	$-0.0028$	$[-0.0073,0.0018]$	$+0.0131$
Llama-3.2-3B	0.62/0.37	$+0.0142$	$+0.0106$	$-0.0014$	$+0.0031$	$[-0.0008,0.0069]$	$+0.0111$
gemma-2-2b	0.61/0.33	$+0.0086$	$-0.0021$	$+0.0010$	$-0.00003$	$[-0.0036,0.0034]$	$+0.0086$
OLMo-2-7B	0.69/0.43	$+0.0148$	$+0.0019$	$+0.0002$	$+0.0009$	$[-0.0034,0.0055]$	$+0.0139$
Qwen3-1.7B	0.62/0.31	$+0.0125$	$+0.0008$	$-0.0001$	$+0.0002$	$[-0.0029,0.0034]$	$+0.0123$

Appendix

Table 46: The four non-whitespace types of the H=1 mild-resample series, split by whether the control pass predicts the target at top-1.

p_{\mathcal{R}}/p_{\mathcal{N}}

is the control-correct share of the type’s GSM8K and Wikitext targets, and CAI is the type cell of Table 44 and Table 45 , which give its interval. Correct and wrong give CAI within each stratum. Reweighted is

p_{\mathcal{N}}\operatorname{CAI}_{\rm correct}+(1-p_{\mathcal{N}})\operatorname{CAI}_{\rm wrong}

, and Diff. is CAI minus reweighted, paired within each draw. Bold marks a correct, wrong, reweighted, or Diff. value whose 95% interval excludes zero. The other intervals use 2,000 sequence-bootstrap draws. A value that rounds to zero is printed to its first significant digit.

Content stratum	Qwen2.5-7B	Llama-3.2-3B	gemma-2-2b	OLMo-2-7B	Qwen3-1.7B
all content targets	$+0.0103$	$+0.0142$	$+0.0086$	$+0.0148$	$+0.0125$
repeated in the sequence	$+0.0099$	$+0.0186$	$+0.0106$	$+0.0060$	$+0.0113$
first occurrence	$+0.0051$	$+0.0028$	$+0.0012$	$+0.0076$	$+0.0014$
leading space	$+0.0089$	$+0.0158$	$+0.0090$	$+0.0168$	$+0.0124$
no leading space	$+0.0214$	$+0.0003$	$-0.0021$	$-0.0021$	$+0.0191$
repeated, leading space	$+0.0052$	$+0.0196$	$+0.0103$	$+0.0081$	$+0.0116$

Appendix

Table 47: Content-word CAI of the H=1 mild-resample series within strata that depend on the input alone, in bold when the 95% interval of 2,000 sequence-bootstrap draws excludes zero. A target is repeated when its token occurs earlier in the same sequence, and a leading space marks a word start. Frequency tertiles rank each token by its count among the content targets of both corpora. The last two rows combine the four repeated-by-leading-space strata with the GSM8K and with the Wikitext stratum shares.

Control	Result	Interpretation
Easy/hard GSM8K split at H=3 top-1	3/3 artifact pattern in each split	Both splits show the artifact pattern, with no monotonic difficulty gradient.
Shuffled-rationale H=3 top-1	0/3 positive content pattern	With the rationale shuffled and its token distribution preserved, no model shows a positive H=3 content-token pattern, the same sign as the canonical H=3 content cells of Table 1 .
Shuffled-rationale H=3 logprob	0/3 positive content pattern	The null also appears under the log-probability readout.
Norm-equalized H=3 top-1	3/3 pooled positive	After equalizing perturbation norms across domains, pooled CAI is +0.027/+0.040/+0.040 in Qwen/Llama/Gemma.

Appendix

Table 48: Controls supporting the audit interpretation. Numerators report how many tested primary models satisfy the named criterion.

Base checkpoint	$n$	pooled H=3 CAI	95% CI
OLMo-2-1124-7B	298	+0.0342	[0.0298, 0.0388]
Qwen2.5-7B	290	+0.0237	[0.0192, 0.0279]
gemma-2-2b	285	+0.0107	[0.0058, 0.0152]

Appendix

Table 49: Pooled H=3 top-1 CAI of base checkpoints on GSM8K/Wikitext under mean-blend at

\alpha=0.2

, with 95% bootstrap intervals and

n

length-matched sequences per corpus. The rows name base releases, which carry no -Instruct or -it suffix.

Qwen2.5-7B
$\alpha$	H=1 CAI	H=1 95% CI	H=3 CAI	H=3 95% CI
0.05	+0.08698	[0.07763, 0.09647]	−0.00148	[−0.00334, 0.00021]
0.10	+0.10093	[0.08539, 0.11567]	−0.00649	[−0.00911, −0.00373]
0.15	+0.00970	[0.00411, 0.01469]	−0.00132	[−0.00260, −0.00002]
0.20	+0.00531	[0.00147, 0.00933]	−0.00094	[−0.00201, 0.00014]
0.25	+0.00296	[−0.00048, 0.00639]	−0.00045	[−0.00136, 0.00044]

Appendix

Table 50: Pooled top-1 CAI under mild-resample on GSM8K/Wikitext across perturbation strengths, with 200 sequences per corpus and 1,000 sequence-level bootstrap resamples. Values carry five decimals because several H=3 interval endpoints fall below

10^{-4}

. The

\alpha=0.20

rows are the setting used elsewhere in the paper, measured on this 200-sequence subsample.

Qwen2.5-7B
$\alpha$	pooled	95% CI	ws	95% CI	ws-excl.	95% CI	nonws direct	95% CI
0.05	−0.0008	[−0.0022, 0.0005]	−0.0157	[−0.0390, 0.0025]	−0.0017	[−0.0032, −0.0004]	−0.0015	[−0.0028, −0.0003]
0.10	+0.0025	[0.0005, 0.0043]	+0.0514	[0.0240, 0.0753]	−0.0050	[−0.0067, −0.0034]	−0.0047	[−0.0063, −0.0031]
0.20	+0.0253	[0.0213, 0.0292]	+0.5181	[0.4770, 0.5562]	−0.0247	[−0.0272, −0.0220]	−0.0253	[−0.0280, −0.0227]
0.30	+0.0262	[0.0213, 0.0310]	+0.8573	[0.8323, 0.8797]	−0.0529	[−0.0557, −0.0499]	−0.0511	[−0.0541, −0.0481]
0.40	+0.0284	[0.0239, 0.0328]	+0.8999	[0.8791, 0.9192]	−0.0543	[−0.0574, −0.0509]	−0.0516	[−0.0544, −0.0489]

Appendix

Table 51: H=3 mean-blend top-1 CAI on the canonical GSM8K/Wikitext sample across perturbation strengths, with 300 sequences per corpus and 2,000 sequence-level bootstrap resamples. The columns are the pooled value, the whitespace cell, the whitespace-excluded summary with combined-count weights, and the non-whitespace CAI measured directly. At

\alpha=0.20

the estimates equal those of Table 1 , and the intervals come from this sweep’s resamples. A value that rounds to zero is printed to its first significant digit.

Model	Q1	Q2	Q3	Q4	position-matched
Qwen2.5-7B	$+0.422$	$+0.492$	$+0.460$	$+0.431$	$+0.447$
Llama-3.2-3B	$+0.612$	$+0.919$	$+0.842$	$+0.810$	$+0.791$
gemma-2-2b-it	$+0.351$	$+0.663$	$+0.786$	$+0.762$	$+0.626$

Appendix

Table 52: Whitespace H=3 CAI by relative-position quartile and after position matching, as shown in Figure 2 c and Figure 5 c. Values are point estimates, and all 15 intervals exclude zero. The full intervals, event counts, and answer-distance bins are in Appendix E .

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 781 papers

Apr 14, 2026

Do Transformers Use their Depth Adaptively? Evidence from a Relational Reasoning Task

We investigate whether transformers use their depth adaptively across tasks of increasing difficulty. Using a controlled multi-hop relational reasoning task based on family stories, where difficulty is determined by the number of relationship hops that must be composed, we monitor (i) how predictions evolve across layers via early readouts (the logit lens) and (ii) how task-relevant information is integrated across tokens via causal patching. For pretrained models, we find some limited evidence for adaptive depth use: some larger models need fewer layers to arrive at plausible answers for easier tasks, and models generally use more layers to integrate information across tokens as chain length increases. For models finetuned on the task, we find clearer and more consistent evidence of adaptive depth use, with the effect being stronger for less constrained finetuning regimes that do not preserve general language modeling abilities.

Alicia Curth, Rachel Lawrence, Sushrut Karmalkar +1

Microsoft Research Cambridge

Adaptive Computation Transformer Interpretability

Ask PDF

Sep 4, 2026

Some Tokens Behave like Magnets: Revealing Linguistic Organization in the Layers of Language Models

We identify a special group of token vectors inside large language models (LLMs), which we term magnetic vectors, that organize the surrounding tokens by either attracting or repelling them. Particularly, tokens pointing the same way as an attracting magnet are elongated; tokens pointing the same way as a repelling magnet are compressed. Just as physical magnets pull or push away the iron filings around them, these vectors organize their surroundings through two opposing polarities. Moreover, we identify a statistically significant pattern in linguistic category where function words consistently act as repelling magnets in early layers, and we also find magnets consistently reorganize their polarities in unique ways deeper in the model. In a further case study we find this observation may unveil a deliberate, layer-wise organization in how LLMs process language. This pattern is consistent across different LLM architectures, sizes, and layer configurations. It is also causally relevant. When the LLM is fine-tuned for a downstream task, the task-functional tokens emerge as magnets. E.g., in question answering, the answer-span tokens become uniquely repelling magnets in the final layer, geometrically carving the answer out of the surrounding context. Furthermore, removing early-layer repelling magnets devastates syntactic tasks (POS tagging accuracy drops from 91% to below 10%) while sparing semantic ones, and removing late-layer attracting magnets does the reverse. We believe this phenomenon warrants further investigation, as it opens the first probe-free path to understanding how language models geometrically organize linguistic computation across their layers.

Andrew Liu, Devan Srinivasan, Gerald Penn

University of Toronto Department of Computer Science

LLM Interpretability Representation Geometry in Language Models Language Modeling

Ask PDF

Jun 22, 2026

Tapered Language Models

Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly across depth. This is a default inherited from the original transformer and largely unchanged since, yet a growing body of evidence suggests that layers contribute non-uniformly to the final output, with later layers refining the residual stream rather than transforming it. We ask whether parameter capacity should reflect this asymmetry. Our controlled experiment shows that, under a fixed budget, allocating more capacity to earlier layers and less to later layers improves perplexity over a uniform-width baseline, while the reverse allocation hurts. Building on this result, we introduce Tapered Language Models (TLMs), an architectural principle in which a parameter-bearing component is monotonically tapered across depth under a fixed total budget. MLPs are the natural site for this instantiation: they dominate parameter count across all modern LM families and expose width as a single, clean axis of variation. Across three model scales and four architectures (Transformer, Gated Attention, Hope-attention, and Titans), tapering MLP width via a smooth cosine schedule consistently improves perplexity and downstream benchmark performance over uniform baselines, at no additional parameter or compute cost. These findings establish depth-aware capacity allocation as a simple, architecture-agnostic axis of language model design, a free lever hidden in plain sight.

Reza Bayat, Ali Behrouz, Aaron Courville

1Mila · 2Cornell University · 3Université de Montréal +1

Autoregressive Language Modeling

Ask PDF

Abstract

LLM Interpretability

Explore similar work

Do Transformers Use their Depth Adaptively? Evidence from a Relational Reasoning Task

Some Tokens Behave like Magnets: Revealing Linguistic Organization in the Layers of Language Models

Tapered Language Models