Architecture-Dependent Causal Transfer of Activation States Across Large Language Models

Fernando Cardenas Piepereit

cs.CLAug 17, 2026

Architecture-Dependent Causal Transfer of Activation States Across Large Language Models

Authors: Fernando Cardenas Piepereit

Organizations: Velez-Malaga, Spain

Abstract

Direct communication between AI systems relies on natural language as an intermediate layer, incurring encoding/decoding overhead, token cost, and latency. We ask whether internal activation states can instead be transferred causally between different large language model (LLM) architectures via a learned projection, evaluated at three levels: representational similarity, cross-model retrieval from projected states, and end-to-end causal transfer via activation injection during generation. Using four architecturally diverse open-weight models (Qwen2-0.5B, Phi-3-mini, Mistral-7B, FLAN-T5-base), we find that representational alignment in trained models exceeds a random-initialization null baseline and is best captured by a rank-based metric (mutual k-nearest-neighbour alignment), more robust to activation-magnitude outliers than centered kernel alignment (CKA) or Procrustes analysis. A learned projection network retrieves the correct target-model representation from a held-out set well above chance for the three causal decoder-only model pairs (45-50% top-1 accuracy vs. 5% chance) but at chance level for the encoder-based FLAN-T5. Injecting projected activations into a target model during generation produces a statistically significant, pre-registered causal effect on retrieval-based output similarity for only one of the three decoder-only pairs (Qwen2-0.5B to Phi-3-mini: 23.3% vs. 0.0% under negative control, p=0.047, FDR-corrected); the two pairs targeting Mistral-7B show no such effect despite comparable representational alignment at the hidden-state level. We interpret these results as evidence for causal transfer of the representational vehicle, not of meaning, and conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.

Figures & tables

Metric	Trained	Untrained (null)
CKA	0.013–0.015	0.81–0.82
Procrustes	0.11–0.12	0.95
Mutual $k$ -NN	0.18–0.30	0.05–0.06

Table 1: Trained vs. untrained (randomly initialised) alignment metrics, Qwen2

\sim

FLAN-T5, middle layers (fraction 0.25–0.75). This is the only model pair for which a full null baseline was computed for all three metrics (see Limitations, Section 6 ).

Projection	Accuracy	95% CI	$p$ (FDR)	Untrained baseline	Chance
Qwen2-0.5B $\to$ Phi-3-mini	45.0%	[25%, 65%]	$<0.0001$	5.0%	5.0%
Qwen2-0.5B $\to$ Mistral-7B	50.0%	[30%, 70%]	$<0.0001$	15.0%	5.0%
Phi-3-mini $\to$ Mistral-7B	45.0%	[25%, 65%]	$<0.0001$	5.0%	5.0%
Qwen2-0.5B $\to$ FLAN-T5 (encoder)	5.0%	[0%, 15%]	0.657 (n.s.)	5.0%	5.0%

Table 2: Cross-model projection network: top-1 retrieval accuracy (20-item candidate pool, chance = 5%), 95% confidence interval, FDR-corrected

p

-value, and untrained-MLP baseline, per model pair.

Model pair	Acc. (real)	Acc. (negative control)	$p$ (McNemar)	$p$ (FDR)
Qwen2-0.5B $\to$ Phi-3-mini	23.3%	0.0%	0.0156	0.0469
Qwen2-0.5B $\to$ Mistral-7B	0.0%	3.3%	1.0000	1.0000
Phi-3-mini $\to$ Mistral-7B	0.0%	3.3%	1.0000	1.0000

Table 3: End-to-end causal state transfer: top-1 retrieval accuracy under real injection vs. negative-control injection,

n=30

trials per pair (10 questions

\times

3 seeds), McNemar test, FDR-corrected across the three pairs.

Causal Interventions in Language Models

Causal interventions in language models test and change how internal representations and computational components cause outputs.

See all 484 papers

Jun 2, 2026

A Negative Result on Cross-Model Activation Transfer in a Pythia Multi-Hop Setting

Recent work shows that language models can transmit behavioural traits through hidden signals in generated data during training. We ask whether a different activation-mediated channel is viable: can one language model communicate a useful intermediate reasoning state to another at inference time through a post-hoc linear activation bridge, rather than through a textual or structured-token relay? We test this question in a controlled Pythia-160M to Pythia-410M multi-hop reasoning setting. A linear translation layer learns a strong normalized-space map between sender and receiver hidden states, with normalized cosine similarity near 0.97 across seeds. However, when the translated activations are injected into the receiver at inference time, they do not improve downstream answering. Low-strength additive injection remains near the no-injection baseline, with confidence intervals that cross zero. Replacement-style injection is consistently destructive, and rescaling translated vectors to the receiver hidden-state norm does not rescue performance. The result is therefore a scoped negative result: in this setting, offline representational alignment is not sufficient for useful causal communication inside the receiver.

Peiyan Zhang, Jason Xin

Independent Researcher

Causal Interventions in Language Models Representation Alignment

Ask PDF

Oct 6, 2026

Learning Cross-Model Activation Alignments with Explicit Many-to-Many Layer Maps

LLMs are released at a rapid pace, raising a natural question: how do two independently trained models relate, both in which layers correspond and in how features transform between them? We study this by learning an activation alignment, a map from a source model's layerwise activations to a target's. Our method, MATCHA, factors this map into a layer map, whose output is an explicit target-by-source matrix that can be extracted and inspected, and a layer-shared feature map between hidden spaces. Most of prior work fixes the layer correspondence in advance, pairing layers at roughly the same relative depth; in contrast, we learn both factors jointly from prompts. Across 42 pairs of seven models spanning three different families, MATCHA reconstructs the target's activations more faithfully and improves retrieval-based metrics substantially, w.r.t. previous approaches. The recovered maps are broadly monotone in depth but, in contrast with most previous approaches, are consistently many-to-many: each target layer draws on a band of source layers. Our alignments also enable transfer of activation-space interventions, allowing steering vectors and probes developed for one model to transfer to another.

Alina Sudakov, Guy Bar-Shalom, Fabrizio Frasca +1

Technion · Technion, NVIDIA Research

Latent Space Alignment Transfer Learning

Ask PDF

Mar 19, 2026

Ventriloquist LLMs: Linear Alignment of Late-Stage Representations

Independently trained language models often learn compatible late-stage representations, despite differences in training objectives, architectures, and data modalities. We ask how far this compatibility extends: can a simple affine map let one model's hidden states be read directly by another model's output head? In this work, we learn affine transformations between the final hidden states of independent models and evaluate them across embedding classification, out-of-distribution detection, and autoregressive text generation. Across model pairs, we find that downstream performance is largely preserved under alignment, with linearly mapped source representations retaining both the decision boundaries and the confidence structure of a target model's classifier. Additionally, we show for the first time that linear alignment sometimes enables text generation across independently trained models, decoding one model's hidden states through another's frozen output head without fine-tuning. This capability is far from universal, and characterizing when it holds is a contribution of our work. We find that success is governed by two factors: tokenizer overlap, which correlates strongly with generation quality, and source-model scale, below which quality degrades sharply. Transfer is also asymmetric, with strong-to-weak mappings substantially outperforming weak-to-strong ones, indicating that source representational capacity, rather than the target head, is the limiting factor. Finally, we characterize where in the network this compatibility arises. A layer-wise analysis localizes this compatibility to the final one or two layers, indicating a late-stage, output-space phenomenon rather than evidence of shared reasoning.

Matt Gorbett, Suman Jana

Capital One New York, NY 10003, USA · Department of Computer Science Columbia University New York, NY 10027, USA

Latent Space Alignment LLM Alignment Representation Geometry in Language Models

Ask PDF

Abstract

Causal Interventions in Language Models

Explore similar work

A Negative Result on Cross-Model Activation Transfer in a Pythia Multi-Hop Setting

Learning Cross-Model Activation Alignments with Explicit Many-to-Many Layer Maps

Ventriloquist LLMs: Linear Alignment of Late-Stage Representations