The Communication Map of a Transformer

Richard Zhe Wang

cs.LGAug 22, 2026

The Communication Map of a Transformer

Authors: Richard Zhe Wang

Organizations: St. John Fisher University

Abstract

The components of a transformer communicate by writing to and reading from a shared residual stream, and the mechanistic interpretability literature has mapped these connections by hand, one circuit at a time. We present the communication map, which charts every potential communication channel from the geometry of the model's weights alone, generalizing the composition score of Elhage et al. (2021) into a single coupling coefficient covering all 18 connection classes, from head-to-head to neuron-to-neuron and everything in between. We provide an account of the properties of the coupling coefficient, including its geometric interpretation and its exact chance level. The census finds that 70-89% of head pairs are oriented far from chance, some coupled strongly and others actively avoiding each other. We demonstrate the communication map in two novel applications. In Application 1, we recover the known induction circuits blind from the strongest head-to-head couplings and group the heads into communities, and ablating one such community destroys the model's in-context copying. In Application 2, we pool the coupling coefficients of every head to identify a distinct two-dimensional residual stream subspace, whose deletion abolishes the induction capability in six models up to Pythia-6.9B. We show that this subspace is different from those identified by either activation PCA or outlier dimensions. We release the map, the statistical machinery, and the intervention suite.

Figures & tables

Panel of Figure 1 — Figure 1: The communication map’s unit of analysis. Left: components interact only by adding writes to, and linearly reading from, a shared residual stream (Eq. 1 ). Right: an edge is a writer–reader pair whose write and read subspaces align far beyond chance.

Model	head–head	ifc $\leftrightarrow$ head	head $\leftrightarrow$ neuron	ifc $\leftrightarrow$ neuron	neuron–neuron	Total	Wires
GPT-2	$28{,}512$	$1{,}008$	$1.02\times 10^{7}$	$110{,}592$	$6.23\times 10^{8}$	$6.33\times 10^{8}$	$2{,}066$
GPT-2-medium	$211{,}968$	$2{,}688$	$7.39\times 10^{7}$	$294{,}912$	$4.63\times 10^{9}$	$4.70\times 10^{9}$	$3{,}124$
GPT-2-large	$756{,}000$	$5{,}040$	$2.62\times 10^{8}$	$552{,}960$	$1.65\times 10^{10}$	$1.68\times 10^{10}$	$6{,}001$
GPT-Neo-125M	$28{,}512$	$1{,}008$	$1.02\times 10^{7}$	$110{,}592$	$6.23\times 10^{8}$	$6.33\times 10^{8}$	$376$
Pythia-160m	$28{,}512$	$576$	$1.02\times 10^{7}$	$73{,}728$	$6.23\times 10^{8}$	$6.33\times 10^{8}$	$725$
Pythia-2.8B	$1{,}523{,}712$	$4{,}096$	$6.61\times 10^{8}$	$655{,}360$	$5.20\times 10^{10}$	$5.27\times 10^{10}$	$2{,}389$

Table 1: Candidate-edge summary of 7 mapped models . Counts are causally eligible ordered pairs, and wires are defined as neuron

\to

neuron pairs with

|\cos|\geq 0.5

. The rotary Pythia models lack the 4 classes where the positional embedding matrix is the writer. (ifc = interface matrices).

	K-composition	Q-composition	V-composition
Model	above / below	above / below	above / below
GPT-2	$55\%\,(+16)$ / $34\%\,(-8)$	$61\%\,(+18)$ / $27\%\,(-7)$	$25\%\,(+15)$ / $65\%\,(-10)$
GPT-2-medium	$54\%\,(+12)$ / $28\%\,(-7)$	$62\%\,(+15)$ / $20\%\,(-6)$	$25\%\,(+13)$ / $58\%\,(-9)$
GPT-2-large	$63\%\,(+13)$ / $23\%\,(-6)$	$70\%\,(+14)$ / $15\%\,(-5)$	$32\%\,(+12)$ / $51\%\,(-7)$
GPT-Neo-125M	$40\%\,(+8)$ / $39\%\,(-6)$	$48\%\,(+9)$ / $33\%\,(-7)$	$24\%\,(+8)$ / $57\%\,(-8)$
Pythia-160m	$50\%\,(+7)$ / $27\%\,(-7)$	$65\%\,(+9)$ / $16\%\,(-5)$	$21\%\,(+9)$ / $63\%\,(-10)$

Table 2: Head-to-head pairs against the theoretical null distribution , for every mapped model. Share of head pairs significantly above (

z\geq 2

) and below (

z\leq-2

) chance orientation, per composition channel, with the median

z

of each group in parentheses. Here

z=\big(C^{2}-1/d\big)/\sqrt{\operatorname{Var}[C^{2}]}

, and both moments are closed-form (Appendix A.4 ).

Figure 2: The head-to-head subgraph of the communication map (GPT-2). Left: adjacency of the $1{,}051$ selected head $\to$ head channel edges under $C$ , one cell per ordered pair, darker where more of the three channels are selected, layer 0 at the lower left. Right: the seed- $0$ Louvain partition of the same graph (assignment archived), one cell per head, with the five induction heads ( $\star$ ), the IOI name-movers ( $\bullet$ ), and the S-inhibition heads ( $\blacksquare$ ) marked. White cells are heads outside the giant component.

Model	RW-PCA (ours)	Activation PCA	Outlier dims	Random	Act. PCA	Outliers
					Cosines: ours vs
GPT-2	$92.3\%$ / $+1.6$	$87.4\%$ / $+7.0$	$57.5\%$ / $+4.2$	$0.2\%$ / $+0.1$	$.29,\,.07$	$.08,\,.00$
GPT-2-medium	$82.5\%$ / $+3.4$	$88.5\%$ / $+7.7$	$71.2\%$ / $+5.1$	$1.3\%$ / $+0.1$	$.25,\,.01$	$.16,\,.00$
GPT-2-large	$81.8\%$ / $+1.3$	$86.6\%$ / $+3.4$	$-19.0\%$ / $+1.8$	$0.1\%$ / $+0.0$	$.50,\,.05$	$.05,\,.01$
Pythia-160m (rotary)	$95.8\%$ / $+8.1$	$42.0\%$ / $+3.8$	$35.4\%$ / $+2.4$	$3.0\%$ / $+0.2$	$.66,\,.12$	$.59,\,.00$
Pythia-2.8B (rotary)	$85.9\%$ / $+9.7$	$78.0\%$ / $+7.2$	$83.5\%$ / $+6.9$	$0.2\%$ / $+0.0$	$.26,\,.02$	$.21,\,.01$

Table 3: The deletion protocol and subspace comparison at six scales. Condition cells are percentages of induction gain destroyed / Pile

\Delta

NLL in nats, for RW-PCA (ours), activation PCA ( Mu and Viswanath, 2018 ; Ethayarajh, 2019 ) , and outlier dimensions ( Timkey and van Schijndel, 2021 ; Kovaleva et al., 2021 ; Dettmers et al., 2022 ) . The last two columns are the two principal cosines between the RW-PCA subspace and each baseline’s subspace (

1

= identical,

0

= orthogonal). The control deletes a random

2

-dimensional subspace (median of five draws). GPT-Neo is excluded because random deletion already removes

60

–

98.1\%

of its induction gain.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Matrix	Shape (general)	Shape (GPT-2)	Rank	Role
$W_{Q}$	$d\times d_{\mathrm{head}}$	$768\times 64$	$\leq d_{\mathrm{head}}$	query factor: $q=W_{Q}^{\top}x$
$W_{K}$	$d\times d_{\mathrm{head}}$	$768\times 64$	$\leq d_{\mathrm{head}}$	key factor: $k=W_{K}^{\top}x$
$W_{V}$	$d\times d_{\mathrm{head}}$	$768\times 64$	$\leq d_{\mathrm{head}}$	value factor: $v=W_{V}^{\top}x$
$W_{O}$	$d\times d_{\mathrm{head}}$	$768\times 64$	$\leq d_{\mathrm{head}}$	output factor: head space $\to$ stream, $\Delta=W_{O}z$
$W_{QK}=W_{Q}W_{K}^{\top}$	$d\times d$	$768\times 768$	$\leq d_{\mathrm{head}}$	QK circuit: bilinear form on position pairs
$W_{OV}=W_{O}W_{V}^{\top}$	$d\times d$	$768\times 768$	$\leq d_{\mathrm{head}}$	OV circuit: stream $\to$ stream operator

Appendix

Table 4: Shapes, ranks, and roles of the per-component matrices (column convention). Ranks are generic, meaning that with trained weights all

d_{\mathrm{head}}

singular values are numerically nonzero, though effective ranks are often far smaller.

Figure 3: The neuron $\to$ neuron coupling distribution. Read–write $|\cos|$ over all $622{,}854{,}144$ candidate neuron pairs (blue, log count scale) against the exact chance density for random directions in $d=768$ (red), scaled to the same number of pairs. Chance and observation agree through the bulk. The dotted line marks $0.23$ , beyond which the chance law expects $0.07$ pairs. The observed distribution continues to $0.89$ . The $2{,}066$ pairs beyond $0.5$ are the wires .

Figure 4: The sampled rotation null distribution of one head pair (writer L4H7 $\to$ reader L8H10 of GPT-2, $4{,}000$ Haar rotations), standardized by the closed-form moments of Equation 11 . The rank- $64$ null is a near-symmetric bump with mass in both tails (sampled mean $0.02$ , SD $1.00$ , skew $+0.20$ ), so the below-chance regime of Table 2 is reachable. The perfectly orthogonal pair sits at $z=-36$ , far off-scale, in contrast with the rank-one law of Figure 3 , whose floor of $-0.7$ makes $z\leq-2$ unreachable for neuron pairs. The trained pair scores $z=15.7$ , far outside its own null distribution.

$t$	$P(\|\cos\|>t)$	Expected $N\cdot P$	Observed
$0.15$	$3.0\times 10^{-5}$	$1.8\times 10^{4}$	$511{,}605$
$0.20$	$2.2\times 10^{-8}$	$13.8$	$140{,}927$
$0.23$	$1.1\times 10^{-10}$	$0.07$	$78{,}861$
$0.25$	$2.0\times 10^{-12}$	$1.3\times 10^{-3}$	$55{,}991$
$0.30$	$1.9\times 10^{-17}$	$1.2\times 10^{-8}$	$26{,}100$
$0.50$	$7.0\times 10^{-50}$	$4.4\times 10^{-41}$	$2{,}066$

Appendix

Table 5: Exceedance counts over the neuron

\to

neuron chance law (GPT-2). For each threshold

t

: the single-pair tail

P(|\cos|>t)

under the exact chance law (Lemma 6 ), the resulting expected number of pairs beyond

t

among all

N=622{,}854{,}144

candidate pairs, and the observed count in the trained weights.

Component	Matrix	Gram	Normalizer	Rank
Head, reader side (one row per channel)
K-composition	$R=W_{QK,r}$	—	$\lVert W_{QK}\rVert_{F}^{2}=\operatorname{tr}(g_{Q}g_{K})$	$\leq d_{\mathrm{head}}$
Q-composition	$R=W_{QK,r}^{\top}$	—	same as K-composition	$\leq d_{\mathrm{head}}$
V-composition	$R=W_{OV,r}$	—	$\lVert W_{OV}\rVert_{F}^{2}=\operatorname{tr}(g_{O}g_{V})$	$\leq d_{\mathrm{head}}$
Head, writer side (one row serves all three channels)
head write	$W=W_{OV,w}$	—	$\lVert W_{OV}\rVert_{F}^{2}=\operatorname{tr}(g_{O}g_{V})$	$\leq d_{\mathrm{head}}$

Appendix

Table 6: Every reader and writer of the map , as instantiations of

G=R^{\top}R

and

H=WW^{\top}

. Only the three interface-matrix Grams are built as arrays (the rotation-null sampling conjugates them). A dash marks a Gram never formed, collapsed through the inner-Gram identities of Appendix B.3 ; neuron normalizers are folded into the unit vectors up front.

Channel	Reader $R$	Cross factor $X$	$\lVert R\,W_{OV,w}\rVert_{F}^{2}$
K-composition	$W_{QK,r}$	$W_{K,r}^{\top}W_{O,w}$	$\operatorname{tr}(X^{\top}g_{Q,r}\,X\,g_{V,w})$
Q-composition	$W_{QK,r}^{\top}$	$W_{Q,r}^{\top}W_{O,w}$	$\operatorname{tr}(X^{\top}g_{K,r}\,X\,g_{V,w})$
V-composition	$W_{OV,r}$	$W_{V,r}^{\top}W_{O,w}$	$\operatorname{tr}(X^{\top}g_{O,r}\,X\,g_{V,w})$

Appendix

Table 7: The factored numerator for each head-to-head channel. The writer always contributes

W_{O,w}

and

g_{V,w}

; the channels differ in which reader factor forms the cross factor

X

and which reader inner Gram weights it.

Connection class	Mask	Count formula
head $\to$ head {K,Q,V}	$l_{w}<l_{r}$	$3\times 12^{2}\times 66$
embedding $\to$ head {K,Q,V}	always	$3\times 144$
positional $\to$ head {K,Q,V}	always	$3\times 144$
head $\to$ unembedding	always	$144$
head $\to$ neuron	$l_{w}\leq l_{r}$	$(12\times 3072)\times 78$
neuron $\to$ head {K,Q,V}	$l_{w}<l_{r}$	$3\times(3072\times 12)\times 66$

Appendix

Table 8: The

18

connection classes. Reader–writer pairing, causal mask, and candidate-count formula, instantiated for GPT-2 small. The head {K,Q,V} rows aggregate three separately scored channel classes over the same pairs.

Model	Community (holds)	Five heads	Comm. minus five	Whole comm.	Ctrl median
GPT-2	$n{=}30$ (5/5)	$60.8\%$	$72.9\%$	$94.7\%$	$2.9\%$
GPT-2-medium	$n{=}47$ (5/5)	$4.7\%$	$75.4\%$	$93.4\%$	$2.0\%$
GPT-2-large	$n{=}184$ (4/5)	$3.6\%$	$102.0\%$	$108.5\%$	$16.2\%$
GPT-Neo-125M	$n{=}32$ (5/5)	$63.7\%$	$100.5\%$	$101.3\%$	$86.8\%$
Pythia-160m	$n{=}25$ (3/5)	$60.0\%$	$92.1\%$	$86.2\%$	$27.4\%$
Pythia-2.8B	$n{=}443$ (4/5)	$41.3\%$	$96.9\%$	$97.1\%$	$1.0\%$

Appendix

Table 9: The community ablation replicates across models. Percent of the clean induction gain destroyed by mean-ablating each head set. “Holds” is how many of the model’s top-five behavioral induction heads fall in the community. Ctrl median is over the whole-community row’s five matched random sets. Communities come from each model’s own selected head graph (protocol below). Pythia-6.9B is absent due to our hardware’s RAM constraint.

Ablated set	$n$	Gain destroyed	Ctrl median	Ctrl max	Pile $\Delta$ NLL
five induction heads	5	$60.8\%$	$0.5\%$	$1.4\%$	$+0.09$
previous-token head L4H11	1	$1.9\%$	$0.1\%$	$0.6\%$	$+0.03$
remaining community heads	24	$16.1\%$	$3.2\%$	$21.7\%$	$+0.22$
community minus induction heads	25	$77.3\%$	$3.7\%$	$12.1\%$	$+0.35$
whole community	30	$93.8\%$	$2.5\%$	$5.1\%$	$+0.53$

Appendix

Table 10: Decomposing the induction-community knockout in GPT-2. Fraction of the clean induction gain (

12.40

nats) destroyed by mean-ablating each head set. Each ablated set is compared with its own five random control sets, matched to it in size and per-layer head counts. Pile

\Delta

NLL is the natural-text loss increase in nats. All numbers are stable across corpus seeds (largest change

1.1

points).

layer separation	1	2	3	4	5	6	7	8	9	10	11
eligible pairs	1584	1440	1296	1152	1008	864	720	576	432	288	144
wired pairs	102	88	77	65	59	69	83	68	48	40	18
wired (%)	6.4	6.1	5.9	5.6	5.9	8.0	11.5	11.8	11.1	13.9	12.5

Appendix

Table 11: Wired head pairs by writer-to-reader layer separation. Eligible pairs are all ordered pairs with the writer in a strictly earlier layer. A pair is wired if at least one of its three channels is selected under

C

(FDR

q=0.05

Figure 5: Eigenvalues of the pooled coupling matrix (GPT-2 small). Smooth decay with no sharp cutoff. The ten leading bands are the candidate pool for the selection rule of Section 5 .

Term	Definition
residual stream	the $d$ -dimensional per-position vector every component reads from and adds to (Eq. 1 )
QK / OV circuit	a head’s pair-scoring form $W_{QK,h}$ / content-transforming operator $W_{OV,h}$
K/Q/V-composition	the three channels by which one head’s writes enter another’s keys / queries / values
potential connectivity	a wired channel: write/read subspaces aligned in the weights, regardless of use
edge; wire	a head pair selected by Application 1 (FDR $q=0.05$ ); a neuron $\to$ neuron pair with $\|\cos\|>0.5$
bands	leading eigendirections of the pooled coupling matrix, the directions most components read and write

Appendix

Table 17

Transformer Interpretability

Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.

See all 339 papers

Align, Unify, Suppress, Route: A Coherentist View of Transformer Computation
Aug 22, 2026Nura Aljaafari, Andre FreitasTransformer Interpretability Mechanistic Interpretability
University of Manchester, United Kingdom · Idiap Research Institute, Switzerland
Reader--Writer Coupling in Transformer MLPs
Oct 21, 2026Giyeong Oh, Sunghee Ahn, Youngjae YuTransformer FFNs Transformer Interpretability
Seoul National University
The Geometry of Inference in Transformer Residual Streams
Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateTransformer Interpretability Transformer Inference
University of Luxembourg, Luxembourg · London Institute for Mathematical Sciences, London, UK

Aug 22, 2026J/K move · Enter open · S save

Align, Unify, Suppress, Route: A Coherentist View of Transformer Computation

Nura Aljaafari, Andre Freitas

University of Manchester, United Kingdom · Idiap Research Institute, Switzerland

Transformer Interpretability Mechanistic Interpretability

Open paper AskPDF arXiv

Mechanistic interpretability has identified transformer circuits, but lacks a shared vocabulary for describing how their functions compose across tasks and architectures. We introduce Coherentist Probabilistic Compositionalism (CPC), an interpretive framework that grounds transformer computation in coherentist theories of interpretation and describes it through four operator roles. Alignment identifies candidate relations, unification integrates supporting information, suppression reduces incompatible alternatives, and routing carries selected information to the output. Across 15 models from five architecture families, the suppression, unification, and routing weight-space signatures correlate with held-out activation-level role measures above random baselines. Suppression is more stable across tasks than unification. Ablating alignment heads reduces downstream suppressive activity beyond a random-head control in 10 models, but similar effects on no-conflict prompts indicate a general upstream dependency, not contradiction-specific coupling. Explicit contradictions significantly shift a layerwise coherence proxy in 14 models; after removing shared residual covariance, the gap has the predicted direction in every model. Base and instruction-tuned variants preserve induction-head score structure (

r{\geq}0.98

) without a consistent shift of operator signatures towards later layers. These results support CPC as a shared vocabulary for comparing transformer mechanisms while showing that their depth and geometric expression remain architecture-specific.

Abstract

Transformer Interpretability

Explore similar work

Align, Unify, Suppress, Route: A Coherentist View of Transformer Computation

Reader--Writer Coupling in Transformer MLPs

The Geometry of Inference in Transformer Residual Streams