Cross-Prompt Reuse of Entity-Replacement Responses

Jinkun Chen; Xuemin Yu; Sijia Han; Vlado Keselj

cs.CLOct 30, 2026

Cross-Prompt Reuse of Entity-Replacement Responses

Authors: Jinkun Chen, Xuemin Yu, Sijia Han, Vlado Keselj

Organizations: Dalhousie University Halifax, Canada · Meta

Abstract

With model weights fixed, we measure how much of an entity-replacement response transfers between English factual questions, beyond whether answers change. We add the hidden-state change from two source questions to a target question about the same entity, comparing the predicted log-probability shift between two relations' published answers, given known answer prefixes, with the measured one. Hidden reuse injects the aligned change at 50% depth before the target's remaining layers; direct reuse adds source output-score changes. With other-relation sources, hidden reuse removes 45% and 49% of no-change error on 0.5-1B Qwen and Llama. Gemma-2-2B removes 22% and passes one of three comparisons in plain text without its default beginning-of-sequence (BOS) token. A later protocol restoring it yields 58-68%, passing every comparison on three cohorts. Larger checkpoints, chat format and new entities show replication and format- or cohort-dependent failures. On small plain-format checkpoints, other-relation hidden reuse mainly reproduces the shift both relations' questions share, without reliably lowering relation-selective error. With target-relation sources, hidden reuse's post hoc advantage over direct reuse is 0.2-0.9 nats outside chat format, unconfirmed and not replacement-specific. The same relation-neutral change at both questions' entity tokens lowers relation-selective error more than random or other-replacement changes, indicating receiving-question dependence.

Figures & tables

Figure 1: Response reuse; model parameters stay fixed. Both routes reuse each source’s 50%-depth change and differ in which read-out, the remaining layers, transforms it: the source’s for direct reuse, the target’s for hidden reuse. (a) The three answer comparisons, pre-specified block sign tests, ask whether hidden reuse from

S_{O}

has lower error than each of zero prediction, hidden reuse from

S^{\prime}_{O}

and direct reuse from

S_{O}

. In the answer studies, analyses with read-out-matched sources (

S_{T}

S^{\prime}_{T}

) are descriptive, exploratory or post hoc except one failed pre-specified family (Section 4.1 ). (b) In the source and target shapes, depth increases upward; fills as in (a). The target’s read-out runs twice, with the hidden change (left) and without it (right).

\Delta_{s,e}

is the source’s 50%-depth change from

a

e

, recaptured at each known answer prefix; hidden reuse aligns its token rows to target positions (Equation 2 ). Weights

w_{s}

use only unedited states.

\widehat{c}^{\,\mathrm{hid}}

\widehat{c}^{\,\mathrm{dir}}

c

from each route’s predicted scores. The block and example (plain format, cue \nAnswer:\n ; Appendix A ) are from the earlier crossing cohort, not the 35-block answer cohort, chosen retrospectively for readable wording, not outcomes (Appendix B.1 ). (c)

M_{x}(A)

: log probability mass of group

A

’s complete answers on input

x

; shares measure contrast reproduction, not answer correctness or free generation.

Study	Material	Blocks	Tests	Pass	Lost	Main result
Answer confirmation	new	35	9	7	3	45/49/22% of no-change error removed
Larger checkpoints	reused	35	9	7	0	51/46/14% removed
Chat format	reused	35	18	18	4	All comparisons pass, Gemma’s included
First fresh cohort	new	33	20	14	4	7–8B selectivity tests pass in family; Gemma passes none
BOS token	both	99	18	18	0	Gemma passes every comparison
Second fresh cohort	new	31	38	14	3	Matched-route test fails on all six

Table 1: Studies and their pre-specified test families. Tests: number of tests; Pass: passes under Holm correction within each family; Lost: passes lost when all 223 confirmatory tests form one Holm family (Table 25 ). Each family was fixed by an internal written protocol before its outcomes, although for Qwen some testbed arms include blocks already observed (Appendix B.5 ). Material: blocks new to the study, reused from earlier studies, or both (earlier: these studies’ own materials, which precede the answer studies). Blocks hold four targets (components for two earlier next-token studies; Table 6 ). Protocols named one family primary without making the others conditional on it. Two steering-completion tests repeat testbed tests (Appendix F ). BOS: beginning-of-sequence token.

Source relation:		Other			Target	Other
Metric:		Overall % (passes)			Gap/nats	Sel. %
Model	Input	Orig.	Fresh 1	Fresh 2	$n=99$ chat: 35	Orig. $n=35$
Qwen 0.5B	plain	45 (3)	42 (3)	38 (2)	0.32	8.7
	chat	43 (3)	–	–	0.63	21.8
Llama 1B	plain	49 (3)	50 (3)	40 (1)	0.24	$-$ 11.4
	+BOS	57 (–)	66 (–)	57 (–)	0.20	8.3

Table 2: Answer reuse at 1/2 depth. Overall (other-relation sources): % of the no-change error removed by same-replacement hidden reuse in each cohort (35/33/31 blocks; chat: original blocks), with passes of the three pre-specified comparisons in parentheses (–: untested). Gap (target-relation sources): direct minus hidden error, nats, over 99 blocks (chat: 35); positive favors hidden reuse (Tables 18 and 16 b). Sel. (other-relation sources): selective-error reduction from zero prediction (%), original 35 blocks (Table 24 ). Gap and Sel. are descriptive. Robust shares: Table 14 .

Checkpoint	Best	Span	Gain [95% interval]
Qwen 0.5B	13.3	47.3	0.85 ∗ [0.33, 1.62]
Llama 1B	38.3	65.3	0.66 ∗ [0.35, 1.05]
Gemma 2B	42.1	65.2	0.82 ∗ [0.41, 1.25]
Qwen 7B	29.9	48.5	1.38 ∗† [0.18, 2.50]
Llama 8B	45.1	55.5	0.48 [ $-$ 0.11, 1.10]
Gemma 9B	48.1	69.8	1.09 ∗ [0.50, 1.77]

Table 3: Best single-vector construction at the 1/2 interface, default encoding. The best of 90 single-vector constructions is selected on the original cohort; Best and Span (span-aligned addition) are evaluated on the same 64 already-studied new-entity blocks. Best/Span: zero-prediction error removed (%). Gain: mean vector minus span error (nats), with 95% label-resampled intervals. ∗ : passes its pre-specified six-test Holm family (Appendix F ); † : fails the 223-test pooled correction. The Qwen 0.5B and Gemma 9B comparisons repeat earlier tests on the same blocks (Section 4.3 ).

Figure 2: Answer-error reduction relative to zero prediction over 35 blocks, default encoding. Solid: span-aligned addition, with other-relation sources unless labelled target-relation; dashed: target-own edited-minus-clean patches, except the horizontal line, other-relation direct reuse; the legend gives row coverage. Vertical: the confirmatory interface; the curves use span-aligned addition, not the suffix interpolation of Table 2 . The terminal interface

d=1

precedes final normalization. Appendix E gives numerical summaries and row definitions, and explains why Qwen2.5-7B uses its saved plain-format scan.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Term	Meaning
Replacement $a\to e$	Swapping the original entity $a$ for an entity $e$ with the same token length in a question
Replacement response	Hidden-state or output-score difference between the replaced and original question (Equation 1 )
Source, target	Questions whose recorded responses are reused; the question whose response is predicted
Hidden reuse	Aligned, weighted source hidden responses added at 50% depth and continued through the target’s remaining layers
Direct (output) reuse	Weighted source output-score changes added to the target’s unedited scores
Zero prediction	Predicts no change; its answer error, the zero or no-change error, is the size of the observed change

Appendix

Table 4: Notation and terms. Source sets

S_{T}

and

S_{O}

ask about the target’s relation and the other relation; primed sets (

S^{\prime}_{T}

S^{\prime}_{O}

) are recorded for the block’s other replacement under the same source templates.

\widehat{c}^{\,\mathrm{hid}}_{t,e}

and

\widehat{c}^{\,\mathrm{dir}}_{t,e}

are the contrasts predicted by hidden and direct reuse.

Relation label	Role	Published question
developer	Source 1	By whom was $x$ developed?
developer	Source 2	Which company developed $x$ ?
developer	Target	What is the name of $x$ ’s developer?
programming language	Source 1	What is the programming language for $x$ ?
programming language	Source 2	What programming language was used to write $x$ ?
programming language	Target	Which programming language is $x$ written in?

Appendix

Table 5: Six published question templates in one complete block. Substitute

x

with each endpoint of SLIB to JDeveloper and Apache Derby to OpenLayers. The entity slot is the only replacement in the displayed wording.

Study	Role	Main result	Targets	Units
Single-token comparisons	Pre-spec.	Identity contrasts recur; derivative contrasts inconclusive	64	10 components
Multi-token comparisons	Pre-spec.	Finite sources beat local derivatives	184	45 components
Replacement/relation crossing	Pre-spec.	Same beats different replacement in 40 of 40 blocks	160	40 blocks
Native amplitude	Pre-spec.	Route gain largely not replacement-specific	144	36 blocks

Appendix

Table 6: Earlier next-token studies in run order: role, main result, and targets and inference units per checkpoint. Blocks contain four targets; components group relations sharing templates.

(a) Small checkpoints
Prediction	Qwen	Llama	Gemma
Zero	2.869	2.821	$2.961^{\dagger}$
Same, direct	2.923	1.887	2.858
Same, hidden	1.572	1.427	2.298
Different, hidden	3.269	3.059	$3.382^{\dagger}$
Share removed, direct	$-2$ %	33%	3%

Appendix

Table 7: Mean answer-contrast error with other-relation sources in the plain format (nats; lower is better) on 140 targets in 35 equally weighted blocks; zero predicts no change, and same/different denotes replacement identity. Share removed is the fraction of the zero-prediction error removed.

\dagger

: Holm-adjusted

p>0.05

for same-replacement hidden reuse against this baseline.

Model	Baseline	Mean	95% interval	Signs	$p_{\mathrm{Holm}}$
Qwen	Output	1.351	[0.882, 1.840]	30/5/0	$1.01\times 10^{-4}$
Qwen	Zero	1.297	[0.655, 2.073]	28/7/0	$1.52\times 10^{-3}$
Qwen	Different hidden	1.696	[1.107, 2.448]	29/6/0	$4.09\times 10^{-4}$
Llama	Output	0.459	[0.163, 0.767]	27/8/0	$4.70\times 10^{-3}$
Llama	Zero	1.393	[0.649, 2.295]	26/9/0	$1.20\times 10^{-2}$
Llama	Different hidden	1.632	[0.875, 2.570]	30/5/0	$1.01\times 10^{-4}$

Appendix

Table 8: Pre-specified answer-contrast error improvement over each baseline, in nats. Positive means lower same-replacement/other-relation hidden error. Signs give positive/negative/zero block counts. Intervals describe means pointwise; the corrected sign tests assess prevalence of positive block effects.

Figure 3: Both source relations on shared axes, small checkpoints in the original 35-block cohort and plain format; all 140 targets per checkpoint and three reuse conditions. Matching lines are calibration fits of observed on predicted changes; the gray dash-dot line is exact reproduction. Target-relation sources bring the output route closer to hidden reuse, without making the different-replacement control track the target response.

Figure 4: All 35 block gains behind the nine original answer comparisons (other-relation sources). Each point is one block’s mean error reduction by same-replacement hidden reuse relative to direct reuse, zero prediction or different-replacement hidden reuse; open circles are positive and filled diamonds negative. Squares and bars give block means and pointwise 95% intervals; numbers give positive/negative block counts.

Source	Model	Route	Slope [95%]	Intercept [95%]	Bias [95%]
Other	Qwen	Hidden	0.900 [0.622, 1.046]	0.421 [0.047, 0.899]	$-$ 0.440 [ $-$ 0.975, $-$ 0.048]
Other	Qwen	Output	0.460 [0.130, 0.700]	0.509 [ $-$ 0.131, 1.160]	$-$ 0.812 [ $-$ 1.905, 0.111]
Other	Llama	Hidden	0.945 [0.787, 1.016]	0.273 [ $-$ 0.012, 0.553]	$-$ 0.271 [ $-$ 0.556, 0.026]
Other	Llama	Output	0.870 [0.667, 0.958]	0.552 [0.078, 1.043]	$-$ 0.589 [ $-$ 1.102, $-$ 0.092]
Other	Gemma	Hidden	0.728 [0.560, 0.823]	0.243 [ $-$ 0.183, 0.701]	$-$ 0.253 [ $-$ 0.781, 0.249]
Other	Gemma	Output	0.548 [0.218, 0.779]	0.253 [ $-$ 0.285, 0.748]	$-$ 0.283 [ $-$ 0.936, 0.365]

Appendix

Table 9: Calibration of same-replacement predictions: observed change on predicted change. Brackets give pointwise 95% block-bootstrap intervals. Intercept and mean prediction minus observed change (bias) are in nats. Other/target denotes the source relation. Zero prediction has no calibration slope.

Source	Model	$r_{\mathrm{hidden}}-r_{\mathrm{output}}$ [95%]
Other	Qwen	0.249 [0.117, 0.415]
Other	Llama	0.063 [0.017, 0.155]
Other	Gemma	0.249 [0.139, 0.410]
Target	Qwen	0.053 [0.013, 0.127]
Target	Llama	0.023 [ $-$ 0.008, 0.070]
Target	Gemma	0.150 [0.055, 0.277]

Appendix

Table 10: Paired Pearson-correlation differences for same-replacement hidden minus output reuse. Intervals are pointwise descriptive, not new corrected tests. With target-relation sources, Llama’s interval includes zero.

Checkpoint	Qwen 0.5B	Llama 1B	Gemma 2B	Qwen 7B	Llama 8B	Gemma 9B
Selective share	20.6	8.7	19.1	38.3	23.5	33.3

Appendix

Table 11: Selective share of observed contrast variance on the original 35 blocks, plain format (%). Balanced relation-swapped pairing uses

\operatorname{Var}(\delta c)=\operatorname{Var}(s)+\mathbb{E}[h^{2}]

; the share is

\operatorname{Var}(s)/\operatorname{Var}(\delta c)

, not an attribution of the primary absolute-error gain. The complementary share is shared between questions. Each checkpoint is evaluated on the same original cohort.

Source	Model	Prediction	$r_{s}$	$r_{h}$	MAE s	MAE h
Other	Qwen	Same/hidden	0.300	0.917	1.181	1.076
Other	Qwen	Same/output	$-$ 0.499	0.885	2.538	1.508
Other	Qwen	Different/hidden	0.036	0.138	1.499	2.785
Other	Qwen	Different/output	0.017	0.039	2.057	3.154
Other	Qwen	Zero	—	—	1.293	2.571
Other	Llama	Same/hidden	0.539	0.938	1.021	1.046

Appendix

Table 12: Paired contrast components: selective

s

and shared

h

, 70 relation-swapped pairs per checkpoint, retained in 35 four-target blocks. Errors are in nats; zero has undefined correlation. Correlations use the fixed pair orientation described above.

Endpoint or prediction	Qwen	Llama	Gemma
Unedited target	$-$ 5.689 [ $-$ 6.980, $-$ 4.429]	$-$ 4.607 [ $-$ 5.380, $-$ 3.568]	$-$ 8.390 [ $-$ 11.532, $-$ 6.821]
Replaced target	$-$ 5.744 [ $-$ 7.164, $-$ 4.195]	$-$ 4.570 [ $-$ 5.620, $-$ 3.421]	$-$ 8.397 [ $-$ 11.195, $-$ 6.965]
Same/hidden, other	$-$ 5.625 [ $-$ 7.149, $-$ 4.371]	$-$ 4.601 [ $-$ 5.411, $-$ 3.350]	$-$ 8.387 [ $-$ 11.252, $-$ 6.895]
Same/output, other	$-$ 5.866 [ $-$ 7.399, $-$ 4.494]	$-$ 4.609 [ $-$ 5.528, $-$ 3.354]	$-$ 8.692 [ $-$ 11.158, $-$ 6.849]
Different/hidden, other	$-$ 5.954 [ $-$ 7.217, $-$ 4.466]	$-$ 4.609 [ $-$ 5.553, $-$ 3.515]	$-$ 8.542 [ $-$ 11.021, $-$ 6.656]
Different/output, other	$-$ 6.195 [ $-$ 7.879, $-$ 4.892]	$-$ 4.875 [ $-$ 5.983, $-$ 3.864]	$-$ 8.873 [ $-$ 11.430, $-$ 6.872]

Appendix

Table 13: Absolute complete-candidate mass: median

\log_{10}

mass [25th, 75th percentiles] over all 140 targets per small checkpoint in the original 35-block cohort and plain format. Brackets describe the target distribution, not confidence intervals. Other/target denotes source relation. All four nonzero predictions per source relation are retained.

		Original		First fresh		Second fresh		All 99 blocks
Model	Input	Med.	No ov.	Med.	No ov.	Med.	No ov.	Share	Med.	No ov.
Qwen 0.5B	plain	35.1	35.3	32.3	40.0	30.2	35.7	41.9 [31.8, 50.5]	32.3 [22.5, 47.4]	37.1
	chat	44.8	42.1	–	–	–	–	–	–	–
Llama 1B	plain	39.9	46.1	47.2	48.5	11.7	32.7	46.9 [38.0, 54.6]	39.9 [29.8, 46.8]	43.4
	+BOS	46.2	52.4	64.8	63.5	52.7	56.6	59.7 [52.9, 65.6]	55.7 [46.2, 62.6]	57.7
	chat	52.9	53.8	–	–	–	–	–	–	–

Appendix

Table 14: Robust summaries of the shares in Table 2 (post hoc; no tests). For same-replacement hidden reuse with other-relation sources: the median over blocks of each block’s share of its no-change error removed (Med.), the share after excluding targets whose answers share a word with an entity name (No ov.; 22, 8 and 14 targets in 8, 3 and 5 blocks of the three cohorts), and, over all 99 blocks, the share of Table 2 with an interval resampling blocks within cohorts. Brackets on medians are distribution-free intervals from inverting the sign test (coverage at least 0.95). Chat rows cover the original blocks.

Sources	Model	Input	Unscaled	Rescaled	Interaction
Other relation	Qwen 0.5B	plain	1.14 [0.80, 1.52]	0.74 [0.47, 1.04]	0.45 [0.07, 0.86]
	Llama 1B	plain	0.61 [0.41, 0.82]	0.50 [0.32, 0.69]	$-$ 0.34 [ $-$ 0.77, 0.05]
	Gemma 2B	plain	0.20 [ $-$ 0.10, 0.51]	0.13 [ $-$ 0.07, 0.32]	$-$ 0.22 [ $-$ 0.56, 0.10]
	Qwen 7B	plain	4.80 [3.82, 5.79]	2.93 [2.26, 3.59]	1.89 [0.69, 3.05]
	Llama 8B	plain	2.84 [2.23, 3.48]	1.75 [1.37, 2.15]	0.57 [ $-$ 0.10, 1.22]
	Gemma 9B	plain	0.40 [0.09, 0.74]	0.09 [ $-$ 0.13, 0.32]	$-$ 0.21 [ $-$ 0.67, 0.23]

Appendix

Table 15: Route comparisons that allow for prediction magnitude and replacement identity (nats; post hoc). Unscaled and Rescaled: mean block error of direct minus hidden reuse (positive favors hidden reuse) on the two fresh cohorts (64 blocks), without and with each route’s predictions multiplied by its least-squares factor fitted on the original cohort. Interaction: over all 99 blocks, the same-replacement route gain minus the different-replacement route gain; positive values mean a larger route gain for the same replacement. Brackets are 95% intervals resampling blocks within cohorts.

Checkpoint	Versus direct	Versus zero	Versus different hidden
Qwen 0.5B	22/5; $6.1\times 10^{-3}$	20/7; 0.038	22/5; $6.1\times 10^{-3}$
Llama 1B	22/5; $6.1\times 10^{-3}$	21/6; 0.015	24/3; $2.2\times 10^{-4}$
Gemma 2B	19/8; 0.078	13/14; 0.701	15/12; 0.701
Qwen 7B	24/3; $1.7\times 10^{-4}$	26/1; $1.9\times 10^{-6}$	24/3; $1.7\times 10^{-4}$
Llama 8B	25/2; $2.3\times 10^{-5}$	22/5; $3.0\times 10^{-3}$	24/3; $1.7\times 10^{-4}$
Gemma 9B	16/11; 0.248	17/10; 0.248	22/5; $3.0\times 10^{-3}$

Appendix

Table 16: Additional comparisons for Sections 4.1 and 4.3 , all at 1/2 depth. (a) Original plain-format overlap sensitivity: positive/negative signs and Holm-adjusted

p

after removing the eight name-overlap blocks, leaving 27; the three baselines on three checkpoints retain one nine-test Holm family per scale. This reduced analysis is post hoc. (b) Original 35-block chat cohort: the share of no-change error removed by other-relation hidden or direct reuse (%); negative shares mean increased error. The last two columns are target-relation direct error minus other-relation or target-relation hidden error (nats; pointwise 95% block-bootstrap interval; positive/negative blocks), so negative values favor matched-source direct reuse; the last column gives the chat-format gaps of Table 2 . Neither route comparison was pre-specified. (c) Descriptive paired error advantages (nats; pointwise 95% intervals resampling blocks within cohorts) over all 99 blocks in default encoding, with other-relation sources and coefficient 1 for the entity-row vector. The last column gives the native-scale vector’s share of no-change error removed when broadcast from the entity onward; negative shares mean larger mean error than zero prediction. E denotes entity rows.

(a) Other-relation versus neutral sources, plain format
Checkpoint	Mean gain in nats [95%]	Fav. [95%]
Qwen 0.5B	0.39 [0.10, 0.69]	0.70 [0.52, 0.85]
Llama 1B	0.45 [0.12, 0.84]	0.66 [0.47, 0.82]
Gemma 2B	0.06 [ $-$ 0.32, 0.48]	0.52 [0.33, 0.70]
Qwen 7B	0.44 [ $-$ 0.28, 1.18]	0.57 [0.40, 0.72]
Llama 8B	0.24 [ $-$ 0.11, 0.57]	0.63 [0.45, 0.78]

Appendix

Table 17: Label-resampling sensitivity for two distinct 99-block comparisons, with equal block weights in the point estimates (35 original, 33 first fresh, 31 second fresh). Brackets are saved 95% intervals from 20,000 relation-label resamples, seed 20261013; Fav. is the share of positive block effects, not a probability of significance. (a) Plain format: neutral-source hidden error minus other-relation hidden error. These pooled summaries are distinct from the 31-block tests in Table 22 . (b) Default encoding: zero-prediction selective error minus error from the same replacement’s identical neutral entity-row content (D2a). These intervals are post hoc sensitivity summaries, not adjusted sign tests.

Checkpoint	Input	Other gap	Target gap [95%]	Target Fav. [95%]
Qwen 0.5B	plain	1.21	0.32 [0.10, 0.56]	0.64 [0.48, 0.79]
Llama 1B	plain	0.55	0.24 [0.03, 0.45]	0.69 [0.51, 0.84]
Gemma 2B	plain	0.33	0.29 [ $-$ 0.09, 0.66]	0.58 [0.41, 0.73]
Qwen 7B	plain	5.05	0.90 [0.18, 1.56]	0.70 [0.53, 0.84]
Llama 8B	plain	2.70	0.40 [0.17, 0.65]	0.73 [0.56, 0.87]
Gemma 9B	plain	0.57	0.45 [0.09, 0.87]	0.61 [0.44, 0.77]

Appendix

Table 18: Source-relation matching at 1/2 depth with suffix interpolation on the same 99 blocks (35 original, 33 first fresh, 31 second fresh), equally weighted within each encoding. Gap: direct minus hidden mean answer error (nats), using other- or target-relation sources for the same replacement. Brackets: 95% label-resampling sensitivity intervals for the target mean gap and its share of favoring blocks (Fav.), not family-wise tests. All six pre-specified matched-route tests on the 31-block second cohort failed (Table 22 ).

Checkpoint	Baseline	Signs	Gain	$p_{\mathrm{Holm}}$
Qwen 7B	Direct reuse	32/3	5.523	$1.7\times 10^{-6}$
Qwen 7B	Zero	33/2	3.573	$1.7\times 10^{-7}$
Qwen 7B	Different	32/3	3.577	$1.7\times 10^{-6}$
Llama 8B	Direct reuse	32/3	2.434	$1.7\times 10^{-6}$
Llama 8B	Zero	30/5	1.803	$4.5\times 10^{-5}$
Llama 8B	Different	32/3	1.728	$1.7\times 10^{-6}$

Appendix

Table 19: Larger-checkpoint confirmation: same-replacement other-relation hidden reuse at the 1/2 interface against three baselines on the 35 answer blocks. Gain is the mean block error reduction (nats); signs count positive/negative blocks. Baselines are direct reuse, zero prediction and different-replacement hidden reuse. One-sided exact sign tests are Holm-adjusted within this nine-test family.

	Over direct reuse			Over zero			Over different
Checkpoint	Signs	Gain	$p_{\mathrm{Holm}}$	Signs	Gain	$p_{\mathrm{Holm}}$	Signs	Gain	$p_{\mathrm{Holm}}$
Chat format, small checkpoints
Qwen 0.5B	29/6	1.92	$1.2\times 10^{-4}$	30/5	1.83	$4.5\times 10^{-5}$	27/8	2.03	$9.4\times 10^{-4}$
Llama 1B	30/5	2.14	$4.5\times 10^{-5}$	32/3	2.44	$1.3\times 10^{-6}$	34/1	3.43	$9.4\times 10^{-9}$
Gemma 2B	32/3	4.24	$1.3\times 10^{-6}$	33/2	2.97	$1.5\times 10^{-7}$	33/2	3.54	$1.5\times 10^{-7}$
Chat format, larger checkpoints

Appendix

Table 20: The chat-format and first-fresh-cohort nine-comparison families: same-replacement other-relation hidden reuse at the 1/2 interface against direct reuse, zero prediction and different-replacement hidden reuse. Signs count positive/negative blocks (35 blocks in the chat format, 33 in the first fresh cohort); gains are mean block error reductions (nats); one-sided exact sign tests are Holm-adjusted within each family.

\dagger

p_{\mathrm{Holm}}>0.05

	Over direct reuse			Over zero			Over different
Checkpoint	Signs	Gain	$p_{\mathrm{Holm}}$	Signs	Gain	$p_{\mathrm{Holm}}$	Signs	Gain	$p_{\mathrm{Holm}}$
Plain+BOS, original blocks (35 blocks)
Gemma 2B	32/3	1.63	$4.2\times 10^{-7}$	35/0	2.92	$1.7\times 10^{-10}$	32/3	2.93	$4.2\times 10^{-7}$
Gemma 9B	34/1	4.08	$4.2\times 10^{-9}$	34/1	3.00	$4.2\times 10^{-9}$	35/0	3.20	$1.7\times 10^{-10}$
Plain+BOS, first fresh cohort (33 blocks)
Gemma 2B	30/3	2.21	$1.4\times 10^{-6}$	31/2	2.35	$2.0\times 10^{-7}$	29/4	2.16	$5.5\times 10^{-6}$

Appendix

Table 21: Answer-comparison families of Appendix D : same-replacement other-relation hidden reuse at the 1/2 interface against direct reuse, zero prediction and different-replacement hidden reuse. The plain+BOS families cover the two Gemma checkpoints, one six-test family per cohort; the second fresh cohort has one nine-test family per scale. Signs count positive/negative blocks; gains are mean block error reductions (nats); one-sided exact sign tests are Holm-adjusted within each family.

\dagger

p_{\mathrm{Holm}}>0.05

Checkpoint	Signs	Gain	$p_{\mathrm{Holm}}$	Signs	Gain	$p_{\mathrm{Holm}}$	Signs	Gain	$p_{\mathrm{Holm}}$
	Other over neutral			Neutral over zero			Matched route
Qwen 0.5B	21/10	0.40	0.142 †	19/12	0.54	0.422 †	21/10	0.38	0.212 †
Llama 1B	22/9	0.45	0.088 †	19/12	0.43	0.422 †	19/12	0.18	0.422 †
Gemma 2B	22/9	0.47	0.088 †	19/12	$-$ 0.03	0.422 †	14/17	$-$ 0.07	0.763 †
Qwen 7B	18/13	0.35	0.710 †	26/5	3.30	$5.8\times 10^{-4}$	21/10	0.41	0.212 †
Llama 8B	19/12	0.22	0.562 †	25/6	1.91	0.002	21/10	0.34	0.212 †

Appendix

Table 22: Neutral-source and read-out-matched route families on the second fresh cohort (31 blocks, plain format). Other over neutral compares same-replacement hidden reuse from other-relation sources with hidden reuse from the two relation-neutral sources; neutral over zero compares the neutral sources with zero prediction (one family of six per scale). The matched route compares hidden with direct reuse when both use target-relation sources (one family of six). Signs, gains (nats) and Holm adjustment as in Table 21 .

\dagger

p_{\mathrm{Holm}}>0.05

Checkpoint	Cohort	Zero error	Mass	Share	Selective	New first	Signs, plain+BOS
Llama 1B	Original	2.82 / 2.81	$-$ 4.61 / $-$ 4.61	49.4 / 56.5	$-$ 11.4 / 8.3	82 / 76	27/8, 31/4, 31/4
	First fresh	2.63 / 2.62	$-$ 4.86 / $-$ 4.76	50.0 / 66.0	21.2 / 41.4	65 / 66	29/4, 30/3, 31/2
	Second fresh	2.22 / 2.25	$-$ 4.39 / $-$ 4.44	39.6 / 56.5	0.4 / 18.4	52 / 60	28/3, 28/3, 29/2
Gemma 2B	Original	2.96 / 4.31	$-$ 8.39 / $-$ 4.53	22.4 / 67.8	$-$ 9.8 / 35.9	42 / 91	32/3, 35/0, 32/3
	First fresh	2.53 / 3.63	$-$ 8.97 / $-$ 4.31	2.1 / 64.6	$-$ 32.9 / 52.0	36 / 83	30/3, 31/2, 29/4
	Second fresh	2.49 / 3.45	$-$ 8.25 / $-$ 4.16	17.8 / 57.6	$-$ 8.8 / 40.0	38 / 74	29/2, 28/3, 30/1

Appendix

Table 23: Plain format against plain+BOS on the three cohorts (descriptive except the Gemma signs, which are the tests of Table 21 ). Each pair gives plain / plain+BOS values of the mean zero-prediction error (nats), the median log 10 candidate mass, the share of the zero error removed by same-replacement other-relation hidden reuse (%), the reduction in selective absolute error relative to zero (%), and the number of targets whose highest-scoring edited candidate is a published target-relation answer of the replacement (of 140, 132 and 124). The last column gives plain+BOS block signs over direct reuse, zero prediction and different-replacement hidden reuse; Llama’s are not tested.

Checkpoint	$s>0$	Push	Hidden	Same	Diff.	Same over different
Original blocks, plain format
Qwen 0.5B	52.9	11/9	19/16	8.7 [ $-$ 13.5, 28.9]	$-$ 15.9 [ $-$ 32.9, 1.7]	0.32 [0.01, 0.63] 21/14
Llama 1B	58.6	15/9	14/21	$-$ 11.4 [ $-$ 42.7, 9.9]	$-$ 22.6 [ $-$ 46.9, $-$ 3.2]	0.10 [ $-$ 0.10, 0.32] 17/18
Gemma 2B	50.0	8/8	14/21	$-$ 9.8 [ $-$ 33.9, 10.3]	$-$ 11.2 [ $-$ 28.1, 3.8]	0.02 [ $-$ 0.31, 0.36] 19/16
Qwen 7B	62.9	15/6	27/8	50.2 [36.8, 60.8]	3.5 [ $-$ 16.8, 20.2]	2.16 [1.28, 3.06] 27/8
Llama 8B	74.3	22/5	23/12	19.7 [1.5, 34.2]	$-$ 10.5 [ $-$ 28.3, 3.0]	0.65 [0.31, 1.00] 28/7

Appendix

Table 24: Selectivity checks of same-replacement other-relation hidden reuse (post hoc; no tests). For each listed cohort, encoding and checkpoint: the share of relation-swapped pairs whose observed selective component

s

is positive (%); blocks favoring / opposing a vanishingly small constant

\widehat{s}>0

under the selectivity statistic

\lvert s\rvert-\lvert\widehat{s}-s\rvert

(Push; blocks whose two pairs differ in sign tie); the block signs of that statistic for hidden reuse (Hidden); the reduction in selective absolute error relative to zero prediction (%) of same-replacement (Same) and different-replacement (Diff.) hidden reuse, with 95% block-bootstrap intervals; and the mean block advantage in selective absolute error of the same over the different replacement (nats), with its interval and favoring / opposing blocks. In the pre-specified two-test selectivity families (plain format, Qwen 7B and Llama 8B), the constant push would give Holm-adjusted

p

of 0.011 and 0.002 on the first fresh cohort and 0.029 and

<

0.001 on the second; hidden reuse gives 0.001 and 0.018, and 0.075 and

<

0.001. The final panel reports the original 35-block chat selective-error reductions with pointwise 95% block-bootstrap intervals; these are descriptive, not multiplicity-adjusted tests.

Test	Family $p_{\mathrm{Holm}}$	Pooled (223)
Single-token: Qwen same over different replacement	0.012	0.081
Single-token: Llama same over different replacement	0.012	0.081
Single-token: Gemma same over different replacement	0.012	0.081
Answer study, small, plain: Llama 1B over direct reuse	0.005	0.081
Answer study, small, plain: Llama 1B over zero	0.012	0.225
Answer study, small, plain: Gemma 2B over direct reuse	0.025	0.592

Appendix

Table 25: All 26 conclusions that change under the final 223-test pooled Holm correction: original family-wise and pooled adjusted

p

values. Every other test keeps its conclusion; Table 1 gives the study inventory.

Checkpoint	Construction	0	1/8	1/4	3/8	1/2	5/8	3/4	7/8	1	Direct
Qwen 0.5B	Exact, all rows	100.0	77.2	66.2	56.2	52.7	45.2	35.1	21.9	1.9	$-$ 1.9
	Exact, entity rows	100.0	75.5	47.3	37.3	35.9	26.9	9.6	5.0	0.0
	Exact, post-entity rows	0.0	$-$ 1.4	3.1	3.1	$-$ 2.4	$-$ 4.5	17.8	19.7	1.9
	Target relation, all rows	100.0	81.2	72.1	69.8	62.4	57.3	52.3	47.6	45.1
	Own entity-row change	100.0	85.0	54.9	48.2	40.9	25.7	8.6	4.2	0.0
	Own post-entity change	0.0	1.0	11.1	10.3	7.0	15.7	41.3	63.7	100.0

Appendix

Table 26: Answer reuse across depth in each checkpoint’s default encoding: percentage of the zero-prediction error removed on the 35 original blocks by exact span alignment with other-relation sources (all, entity or post-entity rows), by target-relation sources (all rows), and by the target’s own edited-minus-clean change on entity or post-entity rows. Qwen 7B uses its saved plain-format records from the earlier scan, since its depth re-run failed the consistency check. The last column is direct reuse, which does not depend on depth. The table includes the plotted series in Figure 2 and additional row subsets.

Checkpoint	$d$	Oth. all	Oth. EQ	Oth. E	Oth. QA	Tgt. all	Tgt. E	Tgt. QA	Neu. all	Neu. E	Own E	All over E
Qwen 0.5B	1/4	66.2	49.8	47.3	3.1	72.1	49.8	6.6	60.9	47.9	54.9	0.64 [0.34, 1.02]
	1/2	52.7	36.5	35.9	$-$ 2.4	62.4	35.3	$-$ 0.6	45.1	35.1	40.9	0.78 [0.39, 1.20]
Llama 1B	1/4	72.9	59.3	59.3	7.8	80.5	63.1	8.3	72.8	60.7	67.7	0.49 [0.28, 0.74]
	1/2	63.3	43.3	40.7	10.4	75.4	40.2	15.4	54.1	39.4	42.4	0.99 [0.61, 1.41]
Gemma 2B	1/4	80.8	76.9	75.8	3.8	86.2	76.0	4.8	81.3	75.4	79.5	0.44 [0.27, 0.62]
	1/2	69.8	58.7	52.9	14.6	79.1	53.5	17.4	70.4	51.4	57.4	1.10 [0.78, 1.45]

Appendix

Table 27: Row subsets by source relation at

d=1/4

and

d=1/2

in the default encoding (35 original blocks): percentage of the zero-prediction error removed by exact alignment on all rows, entity, question and cue rows (EQ), entity rows (E) or post-entity rows (QA) with other-relation, target-relation or relation-neutral sources (neutral entity rows with equal weights), and by the target’s own entity-row change. The last column is the target-relation all-row gain over entity rows (nats, pointwise 95% interval). Qwen 7B uses its saved plain-format records from the earlier scan, which lack the target-relation subsets and neutral sources.

KL recovery
	Interp.	Span all	Span E	Repl. E	Vec. E	Vec. later	Tgt. all	Dir. oth.	Dir. tgt.	Clean
Qwen 0.5B	88	91	89	89	42	$-$ 42	92	44	80	–
Llama 1B	97	98	97	97	25	6	98	87	93	–
Gemma 2B	94	95	93	93	45	14	97	67	88	–
Qwen 7B	94	95	94	94	32	33	96	76	88	–
Llama 8B	94	95	91	92	38	36	97	72	90	–

Appendix

Table 28: Full-vocabulary readouts at the answer cue, whose next-token distribution does not depend on the candidate set: KL recovery, one minus the mean block KL divergence from the edited target over that of the clean target (%), and agreement of the most probable token with the edited target’s (%), over the 99 blocks of three cohorts in each checkpoint’s default encoding. Constructions use other-relation sources unless marked target (Tgt.); one vector (Vec.) at native scale. Interp.: suffix interpolation; Span: span-aligned addition; Repl.: entity-state replacement; Dir.: direct output reuse; E: entity rows; later: every row from the entity onward. The clean target’s top-token agreement is the baseline.

Checkpoint	Changed	Span oth.	Span tgt.	Repl.	Vec.	Clean	Edited	Edited	Span oth.	Vec.	Clean
		Exact				In set		Contains
Qwen 0.5B	96	18	18	17	2	7	6	11 (42)	81	29	31
Llama 1B	100	5	5	5	0	0	0	15 (60)	28	32	27
Gemma 2B	96	17	21	14	3	6	7	21 (82)	66	29	21
Qwen 7B	98	6	7	5	1	4	4	18 (72)	43	28	14
Llama 8B	98	10	11	8	4	7	9	39 (154)	46	38	19

Appendix

Table 29: Greedy generation of up to 16 tokens after the answer cue, over 396 targets of three cohorts in each checkpoint’s default encoding. Changes are added at the 1/2 interface: span-aligned entity, question and cue rows from one capture (Span, other or target relation), replaced entity states (Repl.), or one vector at every row from the entity on, generated rows included (Vec.). Changed: the edited target’s answer string, the text before the first line break, differs from the clean target’s (%). Exact: answer string identical to the edited target’s (%). In set: greedy answer equal to a scored candidate (%). Contains (post hoc): answer contains a published target-relation answer of the replacement, for the edited target (% of targets, count) and, among those targets, for the Span oth. and Vec. predictions and for the clean target (%).

Checkpoint	D1a	D1b	D1c	D2a	D2b
Qwen 0.5B	99/0, 7.61, $<$ 0.001	55/44, $-$ 0.06, 1.000	50/14, 0.85, $<$ 0.001	71/28, 0.40, $<$ 0.001	41/58, 0.10, 1.000
Llama 1B	97/2, 6.76, $<$ 0.001	50/49, $-$ 0.01, 1.000	51/13, 0.67, $<$ 0.001	60/39, 0.16, 0.022 †	72/27, 0.92, $<$ 0.001
Gemma 2B	99/0, 6.50, $<$ 0.001	43/56, $-$ 0.06, 1.000	51/13, 0.77, $<$ 0.001	78/21, 0.75, $<$ 0.001	46/53, $-$ 0.03, 1.000
Qwen 7B	97/2, 7.26, $<$ 0.001	51/48, 0.01, 1.000	42/22, 1.06, 0.034 †	68/31, 1.69, $<$ 0.001	45/54, $-$ 0.11, 1.000
Llama 8B	95/4, 6.04, $<$ 0.001	43/56, $-$ 0.05, 1.000	42/22, 0.87, 0.034 †	69/30, 0.74, $<$ 0.001	76/23, 1.39, $<$ 0.001
Gemma 9B	96/3, 5.26, $<$ 0.001	56/43, 0.01, 1.000	52/12, 1.09, $<$ 0.001	83/16, 1.29, $<$ 0.001	34/65, $-$ 0.62, 1.000

Appendix

Table 30: Confirmatory families of the testbed and read-out protocol, each six one-sided (D1a, D2a, D2b) or two-sided (D1b, D1c) exact sign tests with Holm correction within the family, in each checkpoint’s default encoding. Cells give positive / negative block counts, the mean signed error difference in nats and the Holm-adjusted

p

. D1a: later-row vector error minus span-aligned error; D1b: state-replacement error minus additive entity-row error on the same rows; D1c: tuned-vector error minus span-aligned error, with tuning on the original cohort (fresh cohorts, 64 blocks); D2a: zero minus neutral selective error for identical entity-row changes; D2b: error with answer-position attention to entity rows blocked minus error with attention to other prompt rows blocked, sparing the first token and final cue row in the latter. Other families use 99 blocks. A dagger marks tests that do not pass the final 223-test pooled Holm correction (Table 25 ). For 80% power the true share of favoring blocks must reach 0.63–0.66 in the one-sided families, 0.64–0.68 in D1b and 0.68–0.71 in D1c (last to first Holm rank).

Checkpoint	E1	E2	E3	E4
Qwen 0.5B	38/26, 0.13, 0.253	50/14, 0.85, $<$ 0.001	42/57, $-$ 0.08, 0.954	48/51, 0.20, 0.656
Llama 1B	44/20, 0.45, 0.007 †	51/13, 0.66, $<$ 0.001	47/52, $-$ 0.04, 1.000	66/33, 0.54, 0.001 †
Gemma 2B	37/27, 0.53, 0.253	51/13, 0.82, $<$ 0.001	56/43, 0.04, 1.000	76/23, 1.65, $<$ 0.001
Qwen 7B	38/26, 0.97, 0.253	45/19, 1.38, 0.003 †	51/48, 0.07, 1.000	72/27, 2.12, $<$ 0.001
Llama 8B	45/19, 1.47, 0.004 †	38/26, 0.48, 0.169	46/53, $-$ 0.03, 1.000	81/18, 1.79, $<$ 0.001
Gemma 9B	49/15, 2.05, $<$ 0.001	52/12, 1.09, $<$ 0.001	47/52, $-$ 0.06, 1.000	81/18, 2.21, $<$ 0.001

Appendix

Table 31: Confirmatory families of the steering-completion protocol, each six exact sign tests with Holm correction within the family, in each checkpoint’s default encoding. Cells give positive / negative block counts, the mean signed error difference in nats and the Holm-adjusted

p

. E1 (one-sided, fresh cohorts, 64 blocks): the single-pair vector at every later row, at the coefficient chosen on the original cohort (Qwen 0.5B 1/8, Llama 1B 1/4, Gemma 2B 1/2, Qwen 7B 1/2, Llama 8B 1/2, Gemma 9B 1/2), has lower error than zero prediction. E2 (two-sided, 64 blocks): best-vector error minus span-aligned error, for the best of the 90 single-vector constructions chosen on the original cohort (Qwen 0.5B, single-pair vector at each entity row, coefficient 1; Llama 1B, norm-matched single-pair direction at the entity rows; Gemma 2B, norm-matched single-pair direction at every later row; Qwen 7B, norm-matched sixteen-context direction at every later row; Llama 8B, norm-matched sixteen-context direction at every later row; Gemma 9B, single-pair vector at each entity row, coefficient 1); for Qwen 0.5B and Gemma 9B this is the vector tuned for D1c, so their E2 tests repeat D1c on the same blocks (Table 30 ). E3 (two-sided, 99 blocks): single-pair error minus sixteen-context entity-vector error, both at each entity row at coefficient 1. E4 (one-sided, 99 blocks): the norm-matched single-pair direction at every later row has lower error than zero prediction. A dagger marks tests that do not pass the final 223-test pooled Holm correction (Table 25 ). For 80% power the true share of favoring blocks must reach 0.67–0.71 in E1, 0.68–0.71 in E2, 0.64–0.68 in E3 and 0.63–0.66 in E4 (last to first Holm rank).

	E1		E2		E3		E4
Checkpoint	Within	Labels	Within	Labels	Within	Labels	Within	Labels
Qwen 0.5B	[0.00, 0.27]	[ $-$ 0.10, 0.37]	[0.53, 1.23]	[0.33, 1.62]	[ $-$ 0.15, $-$ 0.02]	[ $-$ 0.19, 0.03]	[ $-$ 0.08, 0.50]	[ $-$ 0.31, 0.79]
Llama 1B	[0.21, 0.69]	[0.08, 0.87]	[0.45, 0.88]	[0.35, 1.05]	[ $-$ 0.10, 0.01]	[ $-$ 0.14, 0.04]	[0.27, 0.81]	[0.12, 0.99]
Gemma 2B	[0.01, 1.05]	[ $-$ 0.34, 1.41]	[0.57, 1.07]	[0.41, 1.25]	[ $-$ 0.03, 0.12]	[ $-$ 0.09, 0.18]	[1.24, 2.08]	[0.93, 2.49]
Qwen 7B	[0.13, 1.80]	[ $-$ 0.53, 2.45]	[0.69, 2.05]	[0.18, 2.50]	[ $-$ 0.11, 0.24]	[ $-$ 0.24, 0.37]	[1.55, 2.70]	[1.15, 3.16]
Llama 8B	[0.88, 2.10]	[0.46, 2.57]	[0.16, 0.80]	[ $-$ 0.11, 1.10]	[ $-$ 0.09, 0.02]	[ $-$ 0.13, 0.05]	[1.35, 2.24]	[1.00, 2.66]

Appendix

Table 32: Mean signed error differences defined in Table 31 (nats) with 95% intervals resampling blocks within cohorts (Within) or relation labels, each block weighted by the product of its two labels’ counts (Labels; Appendix C ). The lower panel gives the share of blocks with positive differences and its label-resampled interval;

\dagger

marks intervals that include one half.

Checkpoint	KO-E	KO-E first half	KO-E second half	KO-Q	KO-P	KO-PQ	Shift
Qwen 0.5B	0.03/0.03/0.03	0.50/0.51/0.45	0.60/0.62/0.59	0.77/0.78/0.77	0.65/0.60/0.64	0.14/ $-$ 0.09/0.15	1.72
Llama 1B	0.12/0.12/0.11	0.42/0.44/0.42	0.73/0.71/0.72	0.97/0.96/0.96	0.94/0.93/0.94	0.91/0.90/0.90	1.21
Gemma 2B	0.55/0.54/0.55	0.69/0.69/0.67	0.90/0.90/0.90	0.61/0.61/0.60	1.00/1.00/0.99	0.54/0.53/0.53	1.07
Qwen 7B	0.15/0.12/0.11	0.66/0.62/0.58	0.56/0.59/0.56	0.58/0.59/0.59	0.74/0.75/0.74	0.11/0.16/0.23	3.89
Llama 8B	0.11/0.12/0.12	0.14/0.16/0.15	0.95/0.93/0.93	1.01/1.00/1.00	0.97/0.96/0.97	0.93/0.92/0.93	1.38
Gemma 9B	0.70/0.71/0.69	0.78/0.78/0.78	0.95/0.95/0.94	0.69/0.67/0.67	0.99/1.00/0.99	0.47/0.46/0.48	0.98

Appendix

Table 33: Attention knockouts above the interface: share of the zero-prediction error removed under each knockout divided by the share without it, for other-relation entity rows / identical neutral entity rows / the target’s own entity-row change, over the 99 blocks of three cohorts. Knockouts block attention from the answer positions (final cue row and candidate prefix) to entity rows (E), to the question and cue rows between entity and final cue row (Q), to rows before the entity except the first (P), or to both (PQ), in every remaining layer or in its first or second half; each prediction is measured from the clean target under the same knockout. Blocking attention from every post-entity row to the entity rows removes the entity-row effect entirely, as the causal structure requires; this check is not tabulated. Shift: mean absolute change of the clean target’s answer contrast under KO-E (nats).

Checkpoint	G1	G2
Qwen 0.5B	74/25, 0.42, $<$ 0.001	71/28, 0.46, $<$ 0.001
Llama 1B	61/38, 0.15, 0.013 †	64/35, 0.21, 0.002 †
Gemma 2B	81/18, 0.70, $<$ 0.001	75/24, 0.77, $<$ 0.001
Qwen 7B	75/24, 1.54, $<$ 0.001	68/31, 1.47, $<$ 0.001
Llama 8B	70/29, 0.67, $<$ 0.001	68/31, 0.64, $<$ 0.001
Gemma 9B	81/18, 1.25, $<$ 0.001	84/15, 1.19, $<$ 0.001

Appendix

Table 34: Confirmatory families of the identical-content controls, each six one-sided exact sign tests over the 99 blocks with Holm correction within the family, in each checkpoint’s default encoding. Cells give blocks favoring / opposing the hypothesis, the mean block difference of selective error in nats (positive favors it) and the Holm-adjusted

p

. G1: the same replacement’s identical neutral entity-row change has lower selective error than identical random entity-row perturbations with its row norms, averaged over eight draws. G2: it has lower selective error than the identical neutral change of the block’s other replacement mapped onto the target’s entity span. A dagger marks tests that do not pass when all 223 confirmatory tests form one Holm family (pooled

p

: G1 Llama 1B 0.822; G2 Llama 1B 0.176). For 80% power the true share of favoring blocks must reach 0.63–0.66 (last to first Holm rank).

	G1			G2
Checkpoint	Within	Labels	Fav.	Within	Labels	Fav.
Qwen 0.5B	[0.29, 0.57]	[0.17, 0.70]	0.75 [0.59, 0.88]	[0.31, 0.62]	[0.17, 0.76]	0.72 [0.52, 0.88]
Llama 1B	[0.05, 0.25]	[ $-$ 0.02, 0.34]	0.62 [0.44, 0.77] †	[0.10, 0.33]	[0.01, 0.42]	0.65 [0.48, 0.80] †
Gemma 2B	[0.51, 0.90]	[0.36, 1.09]	0.82 [0.66, 0.94]	[0.56, 1.00]	[0.40, 1.20]	0.76 [0.59, 0.89]
Qwen 7B	[1.12, 1.96]	[0.76, 2.37]	0.76 [0.58, 0.90]	[1.02, 1.93]	[0.67, 2.35]	0.69 [0.51, 0.84]
Llama 8B	[0.44, 0.91]	[0.30, 1.10]	0.71 [0.55, 0.85]	[0.43, 0.86]	[0.27, 1.06]	0.69 [0.49, 0.86] †

Appendix

Table 35: Mean block differences of the two control families (nats; positive favors the hypothesis) with 95% intervals resampling blocks within cohorts (Within) or relation labels, each block weighted by the product of its two labels’ counts (Labels; Appendix C ), and the share of favoring blocks with its label-resampled interval (Fav.);

\dagger

marks share intervals that include one half.

Checkpoint	NEU	DR	DRN	RND	RND labels	RND draws
Qwen 0.5B	0.40 [0.26, 0.56]	$-$ 0.06 [ $-$ 0.17, 0.04]	$-$ 0.05 [ $-$ 0.17, 0.06]	$-$ 0.02 [ $-$ 0.09, 0.04]	[ $-$ 0.15, 0.11]	$-$ 0.10 to 0.05
Llama 1B	0.16 [0.06, 0.27]	$-$ 0.05 [ $-$ 0.12, 0.02]	$-$ 0.06 [ $-$ 0.14, 0.02]	0.01 [ $-$ 0.02, 0.04]	[ $-$ 0.04, 0.07]	$-$ 0.01 to 0.03
Gemma 2B	0.75 [0.54, 0.96]	$-$ 0.03 [ $-$ 0.15, 0.09]	$-$ 0.02 [ $-$ 0.13, 0.09]	0.04 [0.00, 0.09]	[ $-$ 0.04, 0.14]	0.01 to 0.10
Qwen 7B	1.69 [1.21, 2.19]	0.22 [ $-$ 0.08, 0.54]	0.31 [0.01, 0.63]	0.15 [ $-$ 0.08, 0.39]	[ $-$ 0.21, 0.59]	0.03 to 0.33
Llama 8B	0.74 [0.50, 1.00]	0.11 [ $-$ 0.02, 0.23]	0.12 [ $-$ 0.01, 0.24]	0.07 [0.03, 0.12]	[0.01, 0.14]	0.02 to 0.13
Gemma 9B	1.29 [0.98, 1.62]	0.11 [ $-$ 0.05, 0.27]	0.11 [ $-$ 0.06, 0.28]	0.05 [0.00, 0.10] ‡	[ $-$ 0.03, 0.14]	0.02 to 0.07

Appendix

Table 36: Selective-error reduction from zero prediction,

|s|-|\hat{s}-s|

(nats), of each identical entity-row construction over the 99 blocks, with 95% intervals resampling blocks within cohorts. NEU: the same replacement’s neutral change, the identical-content construction of Appendix G ; DR: the other replacement’s neutral change mapped onto the target’s span; DRN: DR with each row rescaled to NEU’s norm; RND: random rows with NEU’s norms, averaged over eight draws, with its interval resampling relation labels (RND labels; Appendix C ) and the range of the eight draws’ means.

\ddagger

: the interval excludes zero although a bound rounds to 0.00.

		$\|\hat{s}\|$ (% of $\|s\|$ )				Share removed (%)
Checkpoint	$\|s\|$	NEU	DR	DRN	RND	NEU	DR	DRN	RND
Qwen 0.5B	1.37	74.1	53.0	54.3	45.1	32.1	$-$ 5.8	$-$ 6.6	$-$ 6.1
Llama 1B	0.89	80.5	48.9	52.6	31.1	44.5	0.3	$-$ 0.8	3.3
Gemma 2B	1.76	74.7	38.2	39.7	21.4	50.7	0.3	$-$ 0.1	3.5
Qwen 7B	4.48	76.4	43.0	43.3	37.6	37.6	5.1	5.2	4.9
Llama 8B	2.53	57.3	26.6	27.3	13.6	36.1	5.9	6.2	3.3

Appendix

Table 37: Size of the predicted selective component and the full-metric share of the zero-prediction error removed by each identical entity-row construction over the 99 blocks: the mean observed

|s|

(nats), each construction’s mean

|\hat{s}|

as a percentage of it (for RND, the mean over draws of each draw’s mean

|\hat{s}|

), and the share removed on the complete answer contrast (%). Constructions as in Table 36 .

LLM Prompting

Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.

See all 437 papers

The Key Handoff: Retrieval in Hybrid Language Models
Sep 26, 2026Kaan Kale, Oguzhan Baser, Sriram VishwanathKnowledge Retrieval in Language Models Hybrid Language Models
Georgia Institute of Technology, USA · University of Texas at Austin, USA
Shared relations, specific choices: conditional sensitivity and competing pathways in language models
Nov 1, 2026Kaixiang ShuLLM Interpretability Circuits in Language Models
Independent Researcher
Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Context
Sep 28, 2026Muyu He, Yuchen Liu, Ran Tao +1Memorization in Language Models LLM Interpretability
Independent · University of Pennsylvania · Drexel University

Sep 26, 2026J/K move · Enter open · S save

The Key Handoff: Retrieval in Hybrid Language Models

Kaan Kale, Oguzhan Baser, Sriram Vishwanath

Georgia Institute of Technology, USA · University of Texas at Austin, USA

Knowledge Retrieval in Language Models Hybrid Language Models Hybrid Transformer-State Space Model Architectures Multi-Hop QA Hybrid Attention Mechanistic Interpretability

Open paper AskPDF arXiv

A two-hop question makes a language model retrieve twice: once to produce a bridge entity, and once to retrieve with it. Transformers resolve that entity in their early layers. Hybrid models replace most of the attention with a recurrent state, so where the key becomes usable, and where it is spent on an answer, is not known. Answering "Where is the ball?" from "the ball belongs to Alice" and "Alice is in the garden" turns on Alice, a name the question does not mention. A model could reach garden through Alice, the key it computed, or through where the fact sits in the prompt. Across twelve models, dense and hybrid, we move a hidden state from one story into another where the two routes lead to different places, and read off which place the model gives. An attention layer converts the key in every model we tested, and crossing that layer removes its usable effect, all of it where no attention follows. In sequential hybrids this makes the answer a handoff: recurrent layers carry the key forward, and attention spends it. Retrieval does not always stop there. Writing a different fact into the memory of the recurrent layers that follow the last attention layer can move the answer toward that fact, multiplying its odds by 1.3 to 2.7, and which hybrids do this is not settled by their architecture or training. That read is addressed by the key, not by position: recurrent state in a hybrid is not only a carrier, but a memory that later layers can query.

Abstract

LLM Prompting

Explore similar work

The Key Handoff: Retrieval in Hybrid Language Models

Shared relations, specific choices: conditional sensitivity and competing pathways in language models

Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Context