Learning Cross-Model Activation Alignments with Explicit Many-to-Many Layer Maps

Alina Sudakov; Guy Bar-Shalom; Fabrizio Frasca; Haggai Maron

cs.LGOct 6, 2026

Learning Cross-Model Activation Alignments with Explicit Many-to-Many Layer Maps

Authors: Alina Sudakov, Guy Bar-Shalom, Fabrizio Frasca, Haggai Maron

Organizations: Technion · Technion, NVIDIA Research

Abstract

LLMs are released at a rapid pace, raising a natural question: how do two independently trained models relate, both in which layers correspond and in how features transform between them? We study this by learning an activation alignment, a map from a source model's layerwise activations to a target's. Our method, MATCHA, factors this map into a layer map, whose output is an explicit target-by-source matrix that can be extracted and inspected, and a layer-shared feature map between hidden spaces. Most of prior work fixes the layer correspondence in advance, pairing layers at roughly the same relative depth; in contrast, we learn both factors jointly from prompts. Across 42 pairs of seven models spanning three different families, MATCHA reconstructs the target's activations more faithfully and improves retrieval-based metrics substantially, w.r.t. previous approaches. The recovered maps are broadly monotone in depth but, in contrast with most previous approaches, are consistently many-to-many: each target layer draws on a band of source layers. Our alignments also enable transfer of activation-space interventions, allowing steering vectors and probes developed for one model to transfer to another.

Figures & tables

Panel of Figure 1 — Figure 1 : MATCHA aligns paired activations $\mathbf{X}^{s},\mathbf{X}^{s}$ using an (interpretable) layer map, and a feature map. MATCHA can be used for downstream applications.

Panel of Figure 2 — Figure 2 : Layer maps from source ( $L_{s}{=}6$ ) to target ( $L_{t}{=}4$ ), with shallow layers on top. Prior maps are fixed; ours is learned and input-dependent.

Panel of Figure 3 — Figure 3 : Both models encode the same prompt. Highlighted are the last-token representations, $\mathbf{X}^{s}$ and $\mathbf{X}^{t}$ , with $\mathcal{A}^{s\to t}$ mapping $\mathbf{X}^{s}$ to $\mathbf{X}^{t}$ .

Panel of Figure 4 — Figure 4 : Visualization of our alignment map $\mathcal{A}^{s\to t}$ .

	comparable-size				Bigger $\to$ Smaller				Smaller $\to$ Bigger
Method	TJ@5	R@1	P@1	J@5	TJ@5	R@1	P@1	J@5	TJ@5	R@1	P@1	J@5
Terminal	0.449 $\scriptscriptstyle\pm 0.001$	0.716 $\scriptscriptstyle\pm 0.001$	0.915 $\scriptscriptstyle\pm 0.001$	0.376 $\scriptscriptstyle\pm 0.000$	0.407 $\scriptscriptstyle\pm 0.001$	0.759 $\scriptscriptstyle\pm 0.000$	0.874 $\scriptscriptstyle\pm 0.000$	0.381 $\scriptscriptstyle\pm 0.000$	–	–	–	–
Relative Depth	0.447 $\scriptscriptstyle\pm 0.002$	0.727 $\scriptscriptstyle\pm 0.001$	0.917 $\scriptscriptstyle\pm 0.001$	0.377 $\scriptscriptstyle\pm 0.000$	0.411 $\scriptscriptstyle\pm 0.001$	0.757 $\scriptscriptstyle\pm 0.000$	0.918 $\scriptscriptstyle\pm 0.000$	0.395 $\scriptscriptstyle\pm 0.000$	0.435 $\scriptscriptstyle\pm 0.001$	0.686 $\scriptscriptstyle\pm 0.000$	0.856 $\scriptscriptstyle\pm 0.000$	0.381 $\scriptscriptstyle\pm 0.000$
CKA	0.361 $\scriptscriptstyle\pm 0.004$	0.544 $\scriptscriptstyle\pm 0.002$	0.457 $\scriptscriptstyle\pm 0.002$	0.204 $\scriptscriptstyle\pm 0.001$	0.283 $\scriptscriptstyle\pm 0.002$	0.634 $\scriptscriptstyle\pm 0.001$	0.561 $\scriptscriptstyle\pm 0.001$	0.256 $\scriptscriptstyle\pm 0.001$	0.311 $\scriptscriptstyle\pm 0.002$	0.567 $\scriptscriptstyle\pm 0.001$	0.464 $\scriptscriptstyle\pm 0.002$	0.217 $\scriptscriptstyle\pm 0.001$
Command-V	0.316	0.541	0.395	0.321	0.294	0.617	0.517	0.347	0.281	0.627	0.465	0.325
HOT	0.091	0.030	0.343	0.253	0.093	0.041	0.434	0.259	0.101	0.037	0.401	0.263

Table 1: Alignment quality across comparable-size, larger-to-smaller, and smaller-to-larger model pairs. Best results are in bold , and second-best results are underlined. per-pair breakdowns are provided in Section B.3.1 .

Figure 5 : Per-pair Precision@1 difference ( MATCHA $-$ relative depth ).

Figure 6 : The final, learned $\alpha$ obtained by our method.

Panel of Figure 7 — Figure 7 : Transfer of activation-space interventions. Dashed arrows denote source-side operators, which are approximated by the solid path through the target model.

Figure 8 : Steering results. We report per-dataset mean test ASR, averaged over all pairs.

Method	Same-size	Bigger $\to$ Smaller	Smaller $\to$ Bigger
Terminal	0.647 $\scriptscriptstyle\pm 0.006$	0.634 $\scriptscriptstyle\pm 0.004$	–
Relative Depth	0.646 $\scriptscriptstyle\pm 0.007$	0.654 $\scriptscriptstyle\pm 0.004$	0.664 $\scriptscriptstyle\pm 0.002$
CKA	0.549 $\scriptscriptstyle\pm 0.009$	0.593 $\scriptscriptstyle\pm 0.005$	0.597 $\scriptscriptstyle\pm 0.003$
Command-V	0.589	0.605	0.601
HOT	0.642	0.639	0.652
MATCHA (Ours)	0.683 $\scriptscriptstyle\pm 0.007$	0.658 $\scriptscriptstyle\pm 0.007$	0.693 $\scriptscriptstyle\pm 0.004$

Table 2: Probing transfer (TriviaQA test AUC

\uparrow

) by size group — the receiver’s own probe applied to the transferred representation.

Figure 9 : Retrieval performance versus top- $k$ source layers, averaged across all LLM pairs.

Variant	R@1	P@1	J@5	TJ@5
MATCHA-K	0.715 $\scriptscriptstyle\pm 0.002$	0.834 $\scriptscriptstyle\pm 0.001$	0.368 $\scriptscriptstyle\pm 0.001$	0.416 $\scriptscriptstyle\pm 0.001$
MATCHA-FL	0.707 $\scriptscriptstyle\pm 0.002$	0.794 $\scriptscriptstyle\pm 0.002$	0.340 $\scriptscriptstyle\pm 0.001$	0.412 $\scriptscriptstyle\pm 0.001$
MLP	0.460 $\scriptscriptstyle\pm 0.017$	0.817 $\scriptscriptstyle\pm 0.009$	0.318 $\scriptscriptstyle\pm 0.003$	0.364 $\scriptscriptstyle\pm 0.003$
MATCHA (Ours)	0.829 $\scriptscriptstyle\pm 0.001$	0.961 $\scriptscriptstyle\pm 0.001$	0.422 $\scriptscriptstyle\pm 0.001$	0.438 $\scriptscriptstyle\pm 0.001$

Table 3: Architecture ablation . The results are averaged over all 42 LLM pairs.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10 : Recall@1 on three samples. Each predicted representation $\hat{\mathbf{X}}^{s}_{i}$ (orange) queries its nearest neighbour in the target space (teal). A sample is a hit (✓) when $\mathbf{X}^{t}_{i}$ is that neighbour, and a miss ( $\times$ ) otherwise.

Panel of Figure 11 — Figure 11 : Precision@1 on three samples. Each target representation $\mathbf{X}^{t}_{i}$ (teal) queries its nearest neighbour in the predicted space (orange). A sample is a hit (✓) when $\hat{\mathbf{X}}^{s}_{i}$ is that neighbour, and a miss ( $\times$ ) otherwise.

Panel of Figure 12 — Figure 12 : Jaccard@1 on three samples. For each hit (✓), $|\cap|{=}|\cup|{=}1$ , giving score 1. For the miss (×), $|\cap|{=}0$ and $|\cup|{=}2$ —yielding score 0. The union penalty distinguishes Jaccard from Overlap when $K{>}1$ .

Hyperparameter	#	Values swept
MATCHA
learning_rate	2	0.0003, 0.001
attention_temperature	4	0.5, 1, 4, 8
bottleneck_dim	3	256, 512, 1024
num_epochs	3	10, 30, 60
MATCHA-K

Appendix

Table 4 : Hyperparameter grid per method.

Target	LR	Temp.	Bottleneck	Epochs	Val loss
Source: Llama-3.2-1B
Llama-3.2-3B	0.0003	1	512	30	0.1736
Llama-3.1-8B	0.0003	1	256	30	0.1450
Qwen2.5-1.5B	0.001	1	1024	30	0.0391
Qwen2.5-3B	0.001	0.5	256	30	0.0426
Qwen2.5-7B	0.001	0.5	512	60	0.0703

Appendix

Table 5: Best MATCHA hyperparameters per pair (min. validation loss).

Target	LR	Temp.	Bottleneck	Epochs	Val loss
Source: Llama-3.2-1B
Llama-3.2-3B	0.001	0.5	1024	60	0.2092
Llama-3.1-8B	0.001	0.5	1024	60	0.1731
Qwen2.5-1.5B	0.001	0.5	1024	60	0.0445
Qwen2.5-3B	0.001	0.5	1024	60	0.0514
Qwen2.5-7B	0.001	0.5	1024	60	0.0879

Appendix

Table 6: Best MATCHA-K hyperparameters per pair (min. validation loss). MATCHA-K replaces the dynamic key with an input-independent one.

Target	LR	Temp.	Bottleneck	Epochs	Val loss
Source: Llama-3.2-1B
Llama-3.2-3B	0.001	0.5	512	30	0.1878
Llama-3.1-8B	0.0003	0.5	512	30	0.1584
Qwen2.5-1.5B	0.001	0.5	512	30	0.0431
Qwen2.5-3B	0.001	0.5	512	30	0.0480
Qwen2.5-7B	0.001	0.5	512	30	0.0778

Appendix

Table 7: Best MATCHA-FL hyperparameters per pair (min. validation loss). MATCHA-FL uses a single fully-learned logit map.

Target	LR	Temp.	Bottleneck	Epochs	Val loss
Source: Llama-3.2-1B
Llama-3.2-3B	0.001	1	512	10	0.2186
Llama-3.1-8B	0.001	0.5	256	10	0.1975
Qwen2.5-1.5B	0.001	8	256	30	0.0696
Qwen2.5-3B	0.001	0.5	256	30	0.0807
Qwen2.5-7B	0.001	0.5	512	30	0.1073

Appendix

Table 8: Best MATCHA (no FFN) hyperparameters per pair (min. validation loss). MATCHA (no FFN) omits the FFN value path; trained from scratch, not derived from MATCHA’s checkpoint.

	Relative depth			Terminal
Target	LR	Ep.	Val	LR	Ep.	Val
Source: Llama-3.2-1B
Llama-3.2-3B	0.0003	30	0.2169	0.0003	30	0.3659
Llama-3.1-8B	0.0003	30	0.2023	0.0003	30	0.3629
Qwen2.5-1.5B	0.001	30	0.0698	0.001	30	0.1403
Qwen2.5-3B	0.001	30	0.0952	0.001	30	0.2019

Appendix

Table 9: Best Relative depth and Terminal hyperparameters per pair (min. validation loss); both share the LR

\times

epochs grid.

Target	LR	Epochs	Val loss
Source: Llama-3.2-1B
Llama-3.2-3B	0.0003	60	0.2725
Llama-3.1-8B	0.0003	30	0.2567
Qwen2.5-1.5B	0.001	30	0.1350
Qwen2.5-3B	0.001	30	0.1276
Qwen2.5-7B	0.0003	60	0.1587

Appendix

Table 10: Best CKA hyperparameters per pair (min. validation loss). CKA shares the fixed-map methods’ grid (learning rate

\times

epochs).

Target	LR	Epochs	Val loss
Source: Llama-3.2-1B
Llama-3.2-3B	0.0003	60	0.1740
Llama-3.1-8B	0.0003	60	0.1448
Qwen2.5-1.5B	0.0003	60	0.0394
Qwen2.5-3B	0.0003	60	0.0428
Qwen2.5-7B	0.0003	60	0.0720

Appendix

Table 11 : Best MLP hyperparameters per pair (min. validation loss). Learning rate and number of epochs are searched (LR

\in\{0.0003,0.001\}

, epochs

\in\{10,30,60\}

); layers and hidden width are fixed (2, 10,000).

Target	$k$ ( routing_top_k )	Val loss
Source: Llama-3.2-1B
Llama-3.2-3B	3	0.1384
Llama-3.1-8B	3	0.1721
Qwen2.5-1.5B	10	5.4011
Qwen2.5-3B	10	6.6036
Qwen2.5-7B	5	5.3680

Appendix

Table 12: Best HOT routing-

k

per pair (min. validation loss);

k

caps how many donor layers each receiver layer routes from, swept over

\{3,5,10\}

Model	Family	Layers	Hidden
Llama-3.2-1B	Llama	16	2048
Gemma-2-2B	Gemma	26	2304
Qwen2.5-1.5B	Qwen	28	1536
Llama-3.2-3B	Llama	28	3072
Qwen2.5-7B	Qwen	28	3584
Llama-3.1-8B	Llama	32	4096

Appendix

Table 13 : The seven instruction-tuned LLMs used, with their features.

Target	Term	Rel-dep	CKA	MATCHA	MATCHA-K	MATCHA-FL	MATCHA-avg	MATCHA-no FF	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	0.453 $\scriptscriptstyle\pm 0.008$	0.455 $\scriptscriptstyle\pm 0.002$	0.327 $\scriptscriptstyle\pm 0.002$	0.455 $\scriptscriptstyle\pm 0.002$	0.427 $\scriptscriptstyle\pm 0.005$	0.411 $\scriptscriptstyle\pm 0.006$	0.457 $\scriptscriptstyle\pm 0.003$	0.433 $\scriptscriptstyle\pm 0.001$	0.431	0.188	0.418 $\scriptscriptstyle\pm 0.003$
Llama-3.1-8B	0.461 $\scriptscriptstyle\pm 0.003$	0.442 $\scriptscriptstyle\pm 0.008$	0.259 $\scriptscriptstyle\pm 0.004$	0.461 $\scriptscriptstyle\pm 0.002$	0.413 $\scriptscriptstyle\pm 0.007$	0.410 $\scriptscriptstyle\pm 0.012$	0.464 $\scriptscriptstyle\pm 0.002$	0.413 $\scriptscriptstyle\pm 0.005$	0.393	0.182	0.400 $\scriptscriptstyle\pm 0.001$
Qwen2.5-1.5B	0.474 $\scriptscriptstyle\pm 0.008$	0.468 $\scriptscriptstyle\pm 0.004$	0.378 $\scriptscriptstyle\pm 0.003$	0.493 $\scriptscriptstyle\pm 0.009$	0.490 $\scriptscriptstyle\pm 0.004$	0.484 $\scriptscriptstyle\pm 0.003$	0.495 $\scriptscriptstyle\pm 0.002$	0.443 $\scriptscriptstyle\pm 0.005$	0.315	0.020	0.466 $\scriptscriptstyle\pm 0.009$
Qwen2.5-3B	0.431 $\scriptscriptstyle\pm 0.007$	0.435 $\scriptscriptstyle\pm 0.001$	0.339 $\scriptscriptstyle\pm 0.006$	0.455 $\scriptscriptstyle\pm 0.002$	0.428 $\scriptscriptstyle\pm 0.002$	0.431 $\scriptscriptstyle\pm 0.007$	0.455 $\scriptscriptstyle\pm 0.002$	0.421 $\scriptscriptstyle\pm 0.009$	0.317	0.052	0.402 $\scriptscriptstyle\pm 0.001$
Qwen2.5-7B	0.436 $\scriptscriptstyle\pm 0.003$	0.435 $\scriptscriptstyle\pm 0.004$	0.322 $\scriptscriptstyle\pm 0.005$	0.428 $\scriptscriptstyle\pm 0.004$	0.422 $\scriptscriptstyle\pm 0.004$	0.425 $\scriptscriptstyle\pm 0.007$	0.434 $\scriptscriptstyle\pm 0.003$	0.412 $\scriptscriptstyle\pm 0.008$	0.297	0.104	0.416 $\scriptscriptstyle\pm 0.008$

Appendix

Table 14 : Token-Jaccard@5 (

\uparrow

): full per-pair results, seed mean

\pm

std. Best bold , second underlined ; grouped by source (donor) LLM.

Target	Term	Rel-dep	CKA	MATCHA	MATCHA-K	MATCHA-FL	MATCHA-avg	MATCHA-no FF	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	0.397 $\scriptscriptstyle\pm 0.001$	0.783 $\scriptscriptstyle\pm 0.001$	0.737 $\scriptscriptstyle\pm 0.002$	0.857 $\scriptscriptstyle\pm 0.003$	0.745 $\scriptscriptstyle\pm 0.019$	0.758 $\scriptscriptstyle\pm 0.005$	0.853 $\scriptscriptstyle\pm 0.002$	0.744 $\scriptscriptstyle\pm 0.009$	0.904	0.063	0.728 $\scriptscriptstyle\pm 0.006$
Llama-3.1-8B	0.338 $\scriptscriptstyle\pm 0.001$	0.800 $\scriptscriptstyle\pm 0.001$	0.735 $\scriptscriptstyle\pm 0.002$	0.862 $\scriptscriptstyle\pm 0.002$	0.777 $\scriptscriptstyle\pm 0.002$	0.779 $\scriptscriptstyle\pm 0.002$	0.863 $\scriptscriptstyle\pm 0.004$	0.730 $\scriptscriptstyle\pm 0.028$	0.898	0.053	0.745 $\scriptscriptstyle\pm 0.008$
Qwen2.5-1.5B	0.445 $\scriptscriptstyle\pm 0.001$	0.612 $\scriptscriptstyle\pm 0.001$	0.498 $\scriptscriptstyle\pm 0.004$	0.762 $\scriptscriptstyle\pm 0.006$	0.735 $\scriptscriptstyle\pm 0.005$	0.679 $\scriptscriptstyle\pm 0.014$	0.770 $\scriptscriptstyle\pm 0.002$	0.593 $\scriptscriptstyle\pm 0.026$	0.500	0.037	0.610 $\scriptscriptstyle\pm 0.007$
Qwen2.5-3B	0.330 $\scriptscriptstyle\pm 0.001$	0.591 $\scriptscriptstyle\pm 0.000$	0.435 $\scriptscriptstyle\pm 0.001$	0.765 $\scriptscriptstyle\pm 0.007$	0.640 $\scriptscriptstyle\pm 0.017$	0.630 $\scriptscriptstyle\pm 0.023$	0.758 $\scriptscriptstyle\pm 0.002$	0.488 $\scriptscriptstyle\pm 0.039$	0.451	0.040	0.554 $\scriptscriptstyle\pm 0.013$
Qwen2.5-7B	0.412 $\scriptscriptstyle\pm 0.001$	0.668 $\scriptscriptstyle\pm 0.001$	0.573 $\scriptscriptstyle\pm 0.003$	0.856 $\scriptscriptstyle\pm 0.003$	0.619 $\scriptscriptstyle\pm 0.025$	0.693 $\scriptscriptstyle\pm 0.009$	0.862 $\scriptscriptstyle\pm 0.002$	0.620 $\scriptscriptstyle\pm 0.012$	0.557	0.046	0.632 $\scriptscriptstyle\pm 0.008$

Appendix

Table 15 : Recall@1 (

\uparrow

): full per-pair results, seed mean

\pm

std. Best bold , second underlined ; grouped by source (donor) LLM.

Target	Term	Rel-dep	CKA	MATCHA	MATCHA-K	MATCHA-FL	MATCHA-avg	MATCHA-no FF	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	0.606 $\scriptscriptstyle\pm 0.002$	0.928 $\scriptscriptstyle\pm 0.002$	0.702 $\scriptscriptstyle\pm 0.006$	0.976 $\scriptscriptstyle\pm 0.001$	0.868 $\scriptscriptstyle\pm 0.010$	0.867 $\scriptscriptstyle\pm 0.009$	0.977 $\scriptscriptstyle\pm 0.001$	0.948 $\scriptscriptstyle\pm 0.005$	0.669	0.702	0.922 $\scriptscriptstyle\pm 0.006$
Llama-3.1-8B	0.594 $\scriptscriptstyle\pm 0.002$	0.948 $\scriptscriptstyle\pm 0.001$	0.651 $\scriptscriptstyle\pm 0.003$	0.989 $\scriptscriptstyle\pm 0.000$	0.896 $\scriptscriptstyle\pm 0.001$	0.853 $\scriptscriptstyle\pm 0.012$	0.989 $\scriptscriptstyle\pm 0.001$	0.960 $\scriptscriptstyle\pm 0.005$	0.608	0.739	0.947 $\scriptscriptstyle\pm 0.005$
Qwen2.5-1.5B	0.611 $\scriptscriptstyle\pm 0.003$	0.748 $\scriptscriptstyle\pm 0.003$	0.358 $\scriptscriptstyle\pm 0.007$	0.937 $\scriptscriptstyle\pm 0.002$	0.825 $\scriptscriptstyle\pm 0.002$	0.761 $\scriptscriptstyle\pm 0.017$	0.943 $\scriptscriptstyle\pm 0.001$	0.746 $\scriptscriptstyle\pm 0.024$	0.336	0.210	0.864 $\scriptscriptstyle\pm 0.008$
Qwen2.5-3B	0.529 $\scriptscriptstyle\pm 0.001$	0.821 $\scriptscriptstyle\pm 0.001$	0.347 $\scriptscriptstyle\pm 0.004$	0.951 $\scriptscriptstyle\pm 0.002$	0.718 $\scriptscriptstyle\pm 0.006$	0.712 $\scriptscriptstyle\pm 0.021$	0.956 $\scriptscriptstyle\pm 0.001$	0.699 $\scriptscriptstyle\pm 0.010$	0.273	0.199	0.846 $\scriptscriptstyle\pm 0.005$
Qwen2.5-7B	0.614 $\scriptscriptstyle\pm 0.002$	0.854 $\scriptscriptstyle\pm 0.003$	0.462 $\scriptscriptstyle\pm 0.005$	0.976 $\scriptscriptstyle\pm 0.002$	0.765 $\scriptscriptstyle\pm 0.018$	0.797 $\scriptscriptstyle\pm 0.013$	0.977 $\scriptscriptstyle\pm 0.001$	0.857 $\scriptscriptstyle\pm 0.023$	0.297	0.372	0.897 $\scriptscriptstyle\pm 0.003$

Appendix

Table 16 : Precision@1 (

\uparrow

): full per-pair results, seed mean

\pm

std. Best bold , second underlined ; grouped by source (donor) LLM.

Target	Term	Rel-dep	CKA	MATCHA	MATCHA-K	MATCHA-FL	MATCHA-avg	MATCHA-no FF	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	0.271 $\scriptscriptstyle\pm 0.000$	0.443 $\scriptscriptstyle\pm 0.001$	0.320 $\scriptscriptstyle\pm 0.002$	0.464 $\scriptscriptstyle\pm 0.001$	0.415 $\scriptscriptstyle\pm 0.005$	0.390 $\scriptscriptstyle\pm 0.003$	0.467 $\scriptscriptstyle\pm 0.000$	0.473 $\scriptscriptstyle\pm 0.006$	0.385	0.324	0.380 $\scriptscriptstyle\pm 0.003$
Llama-3.1-8B	0.251 $\scriptscriptstyle\pm 0.001$	0.435 $\scriptscriptstyle\pm 0.001$	0.306 $\scriptscriptstyle\pm 0.001$	0.455 $\scriptscriptstyle\pm 0.002$	0.414 $\scriptscriptstyle\pm 0.001$	0.371 $\scriptscriptstyle\pm 0.002$	0.460 $\scriptscriptstyle\pm 0.002$	0.460 $\scriptscriptstyle\pm 0.011$	0.369	0.326	0.385 $\scriptscriptstyle\pm 0.004$
Qwen2.5-1.5B	0.299 $\scriptscriptstyle\pm 0.001$	0.364 $\scriptscriptstyle\pm 0.001$	0.177 $\scriptscriptstyle\pm 0.002$	0.380 $\scriptscriptstyle\pm 0.001$	0.345 $\scriptscriptstyle\pm 0.000$	0.312 $\scriptscriptstyle\pm 0.006$	0.384 $\scriptscriptstyle\pm 0.001$	0.341 $\scriptscriptstyle\pm 0.013$	0.292	0.242	0.313 $\scriptscriptstyle\pm 0.002$
Qwen2.5-3B	0.262 $\scriptscriptstyle\pm 0.001$	0.352 $\scriptscriptstyle\pm 0.000$	0.178 $\scriptscriptstyle\pm 0.001$	0.372 $\scriptscriptstyle\pm 0.002$	0.300 $\scriptscriptstyle\pm 0.003$	0.292 $\scriptscriptstyle\pm 0.005$	0.373 $\scriptscriptstyle\pm 0.002$	0.311 $\scriptscriptstyle\pm 0.013$	0.268	0.236	0.298 $\scriptscriptstyle\pm 0.008$
Qwen2.5-7B	0.292 $\scriptscriptstyle\pm 0.002$	0.381 $\scriptscriptstyle\pm 0.000$	0.217 $\scriptscriptstyle\pm 0.001$	0.398 $\scriptscriptstyle\pm 0.002$	0.317 $\scriptscriptstyle\pm 0.007$	0.320 $\scriptscriptstyle\pm 0.003$	0.399 $\scriptscriptstyle\pm 0.001$	0.371 $\scriptscriptstyle\pm 0.008$	0.293	0.261	0.326 $\scriptscriptstyle\pm 0.001$

Appendix

Table 17 : Jaccard@5 (

\uparrow

): full per-pair results, seed mean

\pm

std. Best bold , second underlined ; grouped by source (donor) LLM.

Target	Terminal	Relative depth	CKA	MATCHA	MATCHA-K	MATCHA-FL	MATCHA-avg	MATCHA-no FF	CommandV	HOT
Source: Llama-3.2-1B
Llama-3.2-3B	0.077 $\scriptscriptstyle\pm 0.000$	0.073 $\scriptscriptstyle\pm 0.000$	0.120 $\scriptscriptstyle\pm 0.000$	0.071 $\scriptscriptstyle\pm 0.000$	0.079 $\scriptscriptstyle\pm 0.001$	0.072 $\scriptscriptstyle\pm 0.001$	0.070 $\scriptscriptstyle\pm 0.000$	0.077 $\scriptscriptstyle\pm 0.000$	0.516	0.145
Llama-3.1-8B	0.090 $\scriptscriptstyle\pm 0.000$	0.113 $\scriptscriptstyle\pm 0.000$	0.195 $\scriptscriptstyle\pm 0.002$	0.085 $\scriptscriptstyle\pm 0.000$	0.090 $\scriptscriptstyle\pm 0.000$	0.088 $\scriptscriptstyle\pm 0.001$	0.083 $\scriptscriptstyle\pm 0.000$	0.098 $\scriptscriptstyle\pm 0.000$	0.627	0.180
Qwen2.5-1.5B	0.576 $\scriptscriptstyle\pm 0.000$	0.564 $\scriptscriptstyle\pm 0.001$	1.576 $\scriptscriptstyle\pm 0.020$	0.451 $\scriptscriptstyle\pm 0.002$	0.501 $\scriptscriptstyle\pm 0.002$	0.463 $\scriptscriptstyle\pm 0.003$	0.446 $\scriptscriptstyle\pm 0.002$	0.611 $\scriptscriptstyle\pm 0.021$	16.772	5.424
Qwen2.5-3B	0.932 $\scriptscriptstyle\pm 0.001$	1.481 $\scriptscriptstyle\pm 0.000$	1.655 $\scriptscriptstyle\pm 0.016$	0.724 $\scriptscriptstyle\pm 0.002$	0.797 $\scriptscriptstyle\pm 0.005$	0.738 $\scriptscriptstyle\pm 0.001$	0.718 $\scriptscriptstyle\pm 0.002$	0.953 $\scriptscriptstyle\pm 0.027$	49.502	6.687
Qwen2.5-7B	1.214 $\scriptscriptstyle\pm 0.000$	1.278 $\scriptscriptstyle\pm 0.001$	3.237 $\scriptscriptstyle\pm 0.055$	1.113 $\scriptscriptstyle\pm 0.002$	1.269 $\scriptscriptstyle\pm 0.011$	1.149 $\scriptscriptstyle\pm 0.004$	1.104 $\scriptscriptstyle\pm 0.002$	1.370 $\scriptscriptstyle\pm 0.008$	35.587	5.432

Appendix

Table 18 : MSE loss (

\downarrow

): full per-pair results, seed mean

\pm

std. Lowest bold , second underlined ; grouped by source (donor) LLM.

Target	Term	Rel-dep	CKA	MATCHA	MATCHA-K	MATCHA-FL	MATCHA-avg	MATCHA-no FF	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	–	0.731 $\scriptscriptstyle\pm 0.004$	0.696 $\scriptscriptstyle\pm 0.005$	0.723 $\scriptscriptstyle\pm 0.003$	0.701 $\scriptscriptstyle\pm 0.009$	0.668 $\scriptscriptstyle\pm 0.014$	0.726 $\scriptscriptstyle\pm 0.003$	0.733 $\scriptscriptstyle\pm 0.003$	0.653	0.734	0.696 $\scriptscriptstyle\pm 0.010$
Llama-3.1-8B	–	0.730 $\scriptscriptstyle\pm 0.002$	0.705 $\scriptscriptstyle\pm 0.004$	0.722 $\scriptscriptstyle\pm 0.005$	0.704 $\scriptscriptstyle\pm 0.006$	0.703 $\scriptscriptstyle\pm 0.012$	0.724 $\scriptscriptstyle\pm 0.002$	0.731 $\scriptscriptstyle\pm 0.003$	0.663	0.743	0.693 $\scriptscriptstyle\pm 0.007$
Qwen2.5-1.5B	–	0.662 $\scriptscriptstyle\pm 0.010$	0.648 $\scriptscriptstyle\pm 0.021$	0.671 $\scriptscriptstyle\pm 0.009$	0.669 $\scriptscriptstyle\pm 0.019$	0.656 $\scriptscriptstyle\pm 0.010$	0.668 $\scriptscriptstyle\pm 0.005$	0.644 $\scriptscriptstyle\pm 0.011$	0.641	0.630	0.659 $\scriptscriptstyle\pm 0.009$
Qwen2.5-3B	–	0.662 $\scriptscriptstyle\pm 0.012$	0.607 $\scriptscriptstyle\pm 0.018$	0.691 $\scriptscriptstyle\pm 0.011$	0.635 $\scriptscriptstyle\pm 0.006$	0.620 $\scriptscriptstyle\pm 0.019$	0.683 $\scriptscriptstyle\pm 0.006$	0.659 $\scriptscriptstyle\pm 0.019$	0.645	0.638	0.662 $\scriptscriptstyle\pm 0.017$
Qwen2.5-7B	–	0.679 $\scriptscriptstyle\pm 0.007$	0.638 $\scriptscriptstyle\pm 0.006$	0.680 $\scriptscriptstyle\pm 0.008$	0.619 $\scriptscriptstyle\pm 0.013$	0.653 $\scriptscriptstyle\pm 0.010$	0.663 $\scriptscriptstyle\pm 0.011$	0.642 $\scriptscriptstyle\pm 0.009$	0.602	0.631	0.660 $\scriptscriptstyle\pm 0.014$

Appendix

Table 19 : Probing — best probe by validation (test AUC

\uparrow

): the receiver probe re-selected by validation on the transferred features. Best bold , second underlined ; grouped by source (donor) LLM.

Target	Term	Rel-dep	CKA	MATCHA	MATCHA-K	MATCHA-FL	MATCHA-avg	MATCHA-no FF	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	–	0.700 $\scriptscriptstyle\pm 0.004$	0.696 $\scriptscriptstyle\pm 0.005$	0.723 $\scriptscriptstyle\pm 0.004$	0.683 $\scriptscriptstyle\pm 0.010$	0.646 $\scriptscriptstyle\pm 0.030$	0.716 $\scriptscriptstyle\pm 0.008$	0.728 $\scriptscriptstyle\pm 0.002$	0.592	0.695	0.678 $\scriptscriptstyle\pm 0.011$
Llama-3.1-8B	–	0.705 $\scriptscriptstyle\pm 0.003$	0.663 $\scriptscriptstyle\pm 0.002$	0.715 $\scriptscriptstyle\pm 0.001$	0.692 $\scriptscriptstyle\pm 0.005$	0.677 $\scriptscriptstyle\pm 0.008$	0.714 $\scriptscriptstyle\pm 0.005$	0.716 $\scriptscriptstyle\pm 0.001$	0.607	0.722	0.672 $\scriptscriptstyle\pm 0.005$
Qwen2.5-1.5B	–	0.555 $\scriptscriptstyle\pm 0.005$	0.592 $\scriptscriptstyle\pm 0.007$	0.619 $\scriptscriptstyle\pm 0.013$	0.589 $\scriptscriptstyle\pm 0.029$	0.579 $\scriptscriptstyle\pm 0.052$	0.623 $\scriptscriptstyle\pm 0.003$	0.596 $\scriptscriptstyle\pm 0.028$	0.606	0.545	0.654 $\scriptscriptstyle\pm 0.017$
Qwen2.5-3B	–	0.633 $\scriptscriptstyle\pm 0.004$	0.599 $\scriptscriptstyle\pm 0.016$	0.681 $\scriptscriptstyle\pm 0.012$	0.633 $\scriptscriptstyle\pm 0.020$	0.605 $\scriptscriptstyle\pm 0.031$	0.665 $\scriptscriptstyle\pm 0.012$	0.657 $\scriptscriptstyle\pm 0.012$	0.571	0.598	0.649 $\scriptscriptstyle\pm 0.019$
Qwen2.5-7B	–	0.640 $\scriptscriptstyle\pm 0.018$	0.622 $\scriptscriptstyle\pm 0.003$	0.662 $\scriptscriptstyle\pm 0.009$	0.580 $\scriptscriptstyle\pm 0.018$	0.616 $\scriptscriptstyle\pm 0.013$	0.638 $\scriptscriptstyle\pm 0.012$	0.647 $\scriptscriptstyle\pm 0.008$	0.562	0.547	0.628 $\scriptscriptstyle\pm 0.003$

Appendix

Table 20 : Probing — target’s own probe (test AUC

\uparrow

): the receiver’s globally-best probe applied to the transferred features (transfer-free ceiling). Best bold , second underlined ; grouped by source (donor) LLM.

Target	Term	Rel-dep	CKA	MATCHA	MATCHA-K	MATCHA-FL	MATCHA-avg	MATCHA-no FF	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	–	20	13	13	21	–	–	–	21	19	21
Llama-3.1-8B	–	21	16	18	17	–	–	–	29	18	21
Qwen2.5-1.5B	–	16	18	20	18	–	–	–	21	8	21
Qwen2.5-3B	–	18	27	25	18	–	–	–	27	10	24
Qwen2.5-7B	–	18	5	18	5	–	–	–	21	2	8

Appendix

Table 21 : Winning probe layer (receiver index) per method and pair, i.e. the layer whose test AUC is in Table 19 . Grouped by source (donor) LLM.

Panel of Figure 13 — Figure 13 : Refusal-suppression steering. A direction and native coefficient are selected once for each target model (top). At each decoding step, an unmodified source-model pass provides the complete last-token activation stack. The stack is mapped into target space, edited at the selected target layer, and mapped back both with and without the edit; subtracting the two outputs yields one delta for each source layer (middle). The same token is then processed a second time by the source model, with these deltas added at the corresponding source layers (bottom). The alignment maps are trained beforehand and remain frozen throughout steering.

Figure 14 : Best-by-val transfer, aggregated: mean test ASR per method and dataset (42 pairs), against the no-steering floor and native-adapter ceiling — the same quantity reported in the main results.

Target	Ours	Rel-dep	Term	CKA	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	89.5 $\pm$ 1.1	0.3 $\pm$ 0.0	–	31.5 $\pm$ 4.0	84.0	62.9	80.4 $\pm$ 7.9
Llama-3.1-8B	88.3 $\pm$ 0.4	0.3 $\pm$ 0.0	–	55.0 $\pm$ 21.2	85.1	42.5	82.7 $\pm$ 6.9
Qwen2.5-1.5B	33.8 $\pm$ 5.1	57.6 $\pm$ 3.0	–	67.4 $\pm$ 13.3	72.7	0.5	38.6 $\pm$ 22.8
Qwen2.5-3B	49.1 $\pm$ 7.2	0.3 $\pm$ 0.0	–	34.0 $\pm$ 5.5	33.5	0.5	30.6 $\pm$ 13.3
Qwen2.5-7B	90.5 $\pm$ 0.8	44.2 $\pm$ 12.5	–	35.7 $\pm$ 12.8	77.3	51.3	80.9 $\pm$ 10.1

Appendix

Table 22 : Refusal-steering transfer on AdvBench — best-by-val : per-pair test ASR (%,

\uparrow

), steering coefficient chosen by the highest average held-out validation ASR ( refusal_val ) of the transfer. Each cell is the mean

\pm

std over the pair’s seeds/runs (std blank for the deterministic CommandV/HOT). Methods: Ours

=

our method, Rel-dep

=

Relative depth, Term

=

Terminal, CmdV

=

CommandV. Best method per row in bold , second underlined ; the last two rows count, per column, how many pairs each method won (1st) or came 2nd on. Rows grouped by source (donor) LLM; –

=

run not yet available (and, for Terminal, every smaller

\to

bigger pair, where the terminal map is degenerate).

Target	Ours	Rel-dep	Term	CKA	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	89.1 $\pm$ 1.1	2.3 $\pm$ 0.4	–	49.6 $\pm$ 4.8	87.5	63.0	79.4 $\pm$ 9.0
Llama-3.1-8B	89.2 $\pm$ 1.7	2.4 $\pm$ 0.2	–	65.4 $\pm$ 15.3	82.0	51.0	83.8 $\pm$ 6.4
Qwen2.5-1.5B	37.3 $\pm$ 4.8	68.8 $\pm$ 6.2	–	57.8 $\pm$ 7.4	61.0	3.0	54.0 $\pm$ 12.3
Qwen2.5-3B	51.1 $\pm$ 7.5	2.5 $\pm$ 0.2	–	44.0 $\pm$ 7.2	40.0	3.5	49.1 $\pm$ 11.7
Qwen2.5-7B	89.9 $\pm$ 1.1	61.3 $\pm$ 4.4	–	48.1 $\pm$ 7.4	73.5	65.0	80.4 $\pm$ 11.5

Appendix

Table 23 : Refusal-steering transfer on HarmBench — best-by-val : per-pair test ASR (%,

\uparrow

), steering coefficient chosen by the highest average held-out validation ASR ( refusal_val ) of the transfer. Each cell is the mean

\pm

std over the pair’s seeds/runs (std blank for the deterministic CommandV/HOT). Methods: Ours

=

our method, Rel-dep

=

Relative depth, Term

=

Terminal, CmdV

=

CommandV. Best method per row in bold , second underlined ; the last two rows count, per column, how many pairs each method won (1st) or came 2nd on. Rows grouped by source (donor) LLM; –

=

run not yet available (and, for Terminal, every smaller

\to

bigger pair, where the terminal map is degenerate).

Target	Ours	Rel-dep	Term	CKA	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	83.6 $\pm$ 1.6	1.7 $\pm$ 0.5	–	39.0 $\pm$ 3.8	86.0	55.0	75.2 $\pm$ 9.2
Llama-3.1-8B	84.7 $\pm$ 1.2	1.3 $\pm$ 0.5	–	57.7 $\pm$ 15.6	82.0	45.0	77.9 $\pm$ 5.9
Qwen2.5-1.5B	36.3 $\pm$ 5.5	63.0 $\pm$ 4.3	–	57.3 $\pm$ 11.0	61.0	1.0	44.4 $\pm$ 14.4
Qwen2.5-3B	47.1 $\pm$ 8.8	1.4 $\pm$ 0.5	–	40.1 $\pm$ 6.0	33.0	1.0	40.2 $\pm$ 12.4
Qwen2.5-7B	84.0 $\pm$ 1.3	49.6 $\pm$ 6.6	–	43.3 $\pm$ 7.6	64.0	61.0	74.3 $\pm$ 10.4

Appendix

Table 24 : Refusal-steering transfer on JailbreakBench — best-by-val : per-pair test ASR (%,

\uparrow

), steering coefficient chosen by the highest average held-out validation ASR ( refusal_val ) of the transfer. Each cell is the mean

\pm

std over the pair’s seeds/runs (std blank for the deterministic CommandV/HOT). Methods: Ours

=

our method, Rel-dep

=

Relative depth, Term

=

Terminal, CmdV

=

CommandV. Best method per row in bold , second underlined ; the last two rows count, per column, how many pairs each method won (1st) or came 2nd on. Rows grouped by source (donor) LLM; –

=

run not yet available (and, for Terminal, every smaller

\to

bigger pair, where the terminal map is degenerate).

Target	Ours	Rel-dep	Term	CKA	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	89.5 $\pm$ 1.1	0.3 $\pm$ 0.0	–	42.3 $\pm$ 16.3	82.2	62.9	80.4 $\pm$ 7.9
Llama-3.1-8B	87.0 $\pm$ 1.3	0.3 $\pm$ 0.0	–	40.7 $\pm$ 23.3	76.0	4.1	79.6 $\pm$ 7.9
Qwen2.5-1.5B	15.3 $\pm$ 13.6	38.5 $\pm$ 5.6	–	41.4 $\pm$ 7.7	18.8	0.0	31.1 $\pm$ 18.7
Qwen2.5-3B	17.4 $\pm$ 10.8	0.3 $\pm$ 0.0	–	32.9 $\pm$ 8.5	12.1	0.3	21.9 $\pm$ 13.3
Qwen2.5-7B	19.0 $\pm$ 5.9	44.2 $\pm$ 12.5	–	3.7 $\pm$ 1.5	77.3	0.8	66.8 $\pm$ 13.3

Appendix

Table 25 : Refusal-steering transfer on AdvBench — best-target : per-pair test ASR (%,

\uparrow

), steering coefficient the target model’s own native steering coefficient (chosen on the target’s native validation, no transfer-side tuning). Each cell is the mean

\pm

std over the pair’s seeds/runs (std blank for the deterministic CommandV/HOT). Methods: Ours

=

our method, Rel-dep

=

Relative depth, Term

=

Terminal, CmdV

=

CommandV. Best method per row in bold , second underlined ; the last two rows count, per column, how many pairs each method won (1st) or came 2nd on. Rows grouped by source (donor) LLM; –

=

run not yet available (and, for Terminal, every smaller

\to

bigger pair, where the terminal map is degenerate).

Target	Ours	Rel-dep	Term	CKA	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	89.1 $\pm$ 1.1	2.3 $\pm$ 0.4	–	50.5 $\pm$ 6.6	83.0	63.0	79.4 $\pm$ 9.0
Llama-3.1-8B	84.0 $\pm$ 1.3	2.4 $\pm$ 0.2	–	57.9 $\pm$ 14.9	71.0	12.0	79.9 $\pm$ 6.8
Qwen2.5-1.5B	20.4 $\pm$ 15.1	51.8 $\pm$ 4.6	–	42.4 $\pm$ 6.2	28.5	2.5	43.8 $\pm$ 12.2
Qwen2.5-3B	28.6 $\pm$ 11.4	2.5 $\pm$ 0.2	–	38.7 $\pm$ 8.4	23.5	0.0	35.4 $\pm$ 12.8
Qwen2.5-7B	31.7 $\pm$ 6.1	61.3 $\pm$ 4.4	–	14.4 $\pm$ 3.4	73.5	4.5	65.2 $\pm$ 10.2

Appendix

Table 26 : Refusal-steering transfer on HarmBench — best-target : per-pair test ASR (%,

\uparrow

), steering coefficient the target model’s own native steering coefficient (chosen on the target’s native validation, no transfer-side tuning). Each cell is the mean

\pm

std over the pair’s seeds/runs (std blank for the deterministic CommandV/HOT). Methods: Ours

=

our method, Rel-dep

=

Relative depth, Term

=

Terminal, CmdV

=

CommandV. Best method per row in bold , second underlined ; the last two rows count, per column, how many pairs each method won (1st) or came 2nd on. Rows grouped by source (donor) LLM; –

=

run not yet available (and, for Terminal, every smaller

\to

bigger pair, where the terminal map is degenerate).

Target	Ours	Rel-dep	Term	CKA	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	83.6 $\pm$ 1.6	1.7 $\pm$ 0.5	–	45.3 $\pm$ 11.2	86.0	55.0	75.2 $\pm$ 9.2
Llama-3.1-8B	80.2 $\pm$ 3.2	1.3 $\pm$ 0.5	–	47.7 $\pm$ 15.3	65.0	9.0	74.7 $\pm$ 8.1
Qwen2.5-1.5B	20.0 $\pm$ 14.1	48.9 $\pm$ 6.8	–	40.4 $\pm$ 6.8	25.0	1.0	35.2 $\pm$ 15.9
Qwen2.5-3B	24.7 $\pm$ 15.2	1.4 $\pm$ 0.5	–	40.3 $\pm$ 5.8	12.0	1.0	26.4 $\pm$ 14.7
Qwen2.5-7B	30.0 $\pm$ 7.0	49.6 $\pm$ 6.6	–	9.2 $\pm$ 2.4	64.0	1.0	59.0 $\pm$ 9.2

Appendix

Table 27 : Refusal-steering transfer on JailbreakBench — best-target : per-pair test ASR (%,

\uparrow

), steering coefficient the target model’s own native steering coefficient (chosen on the target’s native validation, no transfer-side tuning). Each cell is the mean

\pm

std over the pair’s seeds/runs (std blank for the deterministic CommandV/HOT). Methods: Ours

=

our method, Rel-dep

=

Relative depth, Term

=

Terminal, CmdV

=

CommandV. Best method per row in bold , second underlined ; the last two rows count, per column, how many pairs each method won (1st) or came 2nd on. Rows grouped by source (donor) LLM; –

=

run not yet available (and, for Terminal, every smaller

\to

bigger pair, where the terminal map is degenerate).

Target	Ours	Rel-dep	Term	CKA	CmdV	HOT	MLP
Source: Llama-3.2-1B
Llama-3.2-3B	-10	-0.5	–	-4	-6	-10	-10
Llama-3.1-8B	-8	-0.5	–	-6	-2	-8	-8
Qwen2.5-1.5B	6	8	–	-8	1.5	-0.5	8
Qwen2.5-3B	-8	-0.5	–	6	2	-0.5	-8
Qwen2.5-7B	-4	-1	–	-6	-1	-8	-8

Appendix

Table 28 : Best-by-val chosen steering coefficient per (pair, method): the coefficient with the highest average validation ASR ( refusal_val ), used for the val table (Table 22 ). Negative coefficients suppress the refusal direction. Rows grouped by source (donor) LLM; – = not yet available (and, for Terminal, every smaller

\to

bigger pair, where the terminal map is degenerate).

Figure 17 : Per-pair Token-Jaccard@5 difference, MATCHA $-$ Relative depth. Axes as in Figure 16 .

Panel of Figure 18 — Figure 18 : Assumption 1 as a condition on the keys. Entry $(j,r)$ of $XW_{k}$ is the score $\langle w_{r},x_{j}(X)\rangle$ that detector $r$ assigns to source layer $j$ . Reading down any of the leading $L_{s}$ columns, the entry at $j=r$ (dark) must lead every other by $\gamma$ , uniformly over $X\in\mathcal{X}$ ; off-diagonal magnitudes and the remaining $d-L_{s}$ columns are unconstrained.

Panel of Figure 19 — Figure 19 : Proposition 2 on a concrete instance: $L_{t}=3$ , $L_{s}=4$ , and $\pi(1)=3$ , $\pi(2)=2$ , $\pi(3)=4$ . Each query is one-hot at $\pi(\ell)$ and vanishes on the last $d-L_{s}$ coordinates, so the product annihilates the unconstrained rows of $(XW_{k})^{\!\top}$ and copies its row $\pi(\ell)$ into row $\ell$ of $S$ (orange). By Assumption 1 each such row is largest at its own layer (dark), so after the softmax every row of $\alpha$ concentrates on $\pi(\ell)$ . With $\gamma=1.0$ and $c=3$ , Lemma 1 gives $\|\alpha-P_{\pi}\|_{1,\infty}\leq 2(L_{s}-1)e^{-3}\approx 0.30$ ; larger $c$ drives $\alpha\to P_{\pi}$ exponentially with $W_{k}$ unchanged. Note that detector $1$ is selected by no target layer, and that $\pi$ need not be injective.

Panel of Figure 20 — Figure 20 : Assumption 2 as a condition on the source stacks. Reading each layer’s activation with each of the $L_{s}$ decoding directions must return exactly one on the matching layer and exactly zero elsewhere — that is, $XU=I_{L_{s}}$ , with the same identity matrix for every input. This is strictly stronger than Assumption 1 ( Figure 18 ), which constrains only the ordering within each column and leaves all magnitudes free.

Panel of Figure 21 — Figure 21 : Proposition 4 on a concrete instance, with $L_{t}=3$ , $L_{s}=4$ and $d=L_{s}$ , for which the construction takes the transparent form $W_{k}=U$ and $Q=\sqrt{d}\,\Lambda$ . Assumption 2 makes the key matrix $XW_{k}=XU$ equal to the identity for every source stack, so the input drops out of the product entirely and the logits are $\Lambda=\log\alpha^{\star}$ regardless of $X$ ; the softmax then returns $\alpha^{\star}$ itself. Unlike the one-to-one case, the target here is soft — no row concentrates on a single source layer — and is reached exactly rather than in the limit of a large logit scale. For general $d\geq\min(L_{t},L_{s})$ the same argument runs with $Q$ and $W_{k}$ obtained from a rank factorization of $\sqrt{d}\,M$ , as in Step 2 of the proof.

Explore similar work

A Negative Result on Cross-Model Activation Transfer in a Pythia Multi-Hop Setting
Jun 2, 2026Peiyan Zhang, Jason XinCausal Interventions in Language Models Representation Alignment
Independent Researcher
One Adapter Pair per Model: A Universal Activation Interface for Language Models
Aug 10, 2026Su-Hyeon Kim, Jiwan Mun, Yo-Sub HanRepresentation Geometry in Language Models Language Model Probing
Department of Artificial Intelligence, Yonsei University · Department of Computer Science, Yonsei University
Cross-Family Universality of Behavioral Axes via Anchor-Projected Representations
May 11, 2026Su-Hyeon Kim, Yo-Sub HanLLM Interpretability Transfer Learning
Department of Artificial Intelligence Yonsei University Seoul, South Korea · Department of Computer Science Yonsei University Seoul, South Korea

Jun 2, 2026, cs.AIJ/K move · Enter open · S save

A Negative Result on Cross-Model Activation Transfer in a Pythia Multi-Hop Setting

Peiyan Zhang, Jason Xin

Independent Researcher

Causal Interventions in Language Models Representation Alignment

Open paper AskPDF arXiv

Recent work shows that language models can transmit behavioural traits through hidden signals in generated data during training. We ask whether a different activation-mediated channel is viable: can one language model communicate a useful intermediate reasoning state to another at inference time through a post-hoc linear activation bridge, rather than through a textual or structured-token relay? We test this question in a controlled Pythia-160M to Pythia-410M multi-hop reasoning setting. A linear translation layer learns a strong normalized-space map between sender and receiver hidden states, with normalized cosine similarity near 0.97 across seeds. However, when the translated activations are injected into the receiver at inference time, they do not improve downstream answering. Low-strength additive injection remains near the no-injection baseline, with confidence intervals that cross zero. Replacement-style injection is consistently destructive, and rescaling translated vectors to the receiver hidden-state norm does not rescue performance. The result is therefore a scoped negative result: in this setting, offline representational alignment is not sufficient for useful causal communication inside the receiver.

Abstract

Explore similar work

A Negative Result on Cross-Model Activation Transfer in a Pythia Multi-Hop Setting

One Adapter Pair per Model: A Universal Activation Interface for Language Models

Cross-Family Universality of Behavioral Axes via Anchor-Projected Representations