Understanding relationships between attention heads is essential for interpreting the internal structure of Transformers, yet existing metrics do not capture this structure well. We focus on the subspaces spanned by attention-head weight matrices and quantify head-to-head relationships using the Projection Kernel (PK), a principal-angle-based measure of subspace similarity. Experiments on GPT2-small, BLOOM-560M, and Qwen2-0.5B-Instruct show that PK reproduces known head-to-head interactions on the IOI task more clearly than prior metrics such as the Composition Score. We further introduce a framework to quantify the informativeness of PK distributions by comparing them with a reference distribution derived from random orthogonal subspaces. As an application, we analyze a directed graph constructed from PK and show that, in GPT2-small, L4H7 acts as a hub by functioning as an Identity Head.
Figures & tables
Figure 1: For the attention-head weights of GPT2-small, we compute similarities between heads using (top) the Projection Kernel (PK) and (bottom) the Composition Score (CS), and visualize the top 20 scores as edges for each weight pairing (OQ, OK, and OV). Edge opacity corresponds to the score normalized by the maximum score. The annotations for the seven head classes other than Identity Heads follow Wang et al. (2023) . With PK, many heads from IOI-relevant head classes appear, and edges concentrate among these heads. In contrast, this tendency is weaker with CS; instead, many edges connect heads that do not belong to any head class.
Figure 2: Illustration of measuring similarity using the Projection Kernel (PK) in ( 9 ) between the dhead -dimensional subspace S(h,O)⊂Rd spanned by the output weights WOh∈Rd×dhead of head h and the dhead -dimensional subspace S(h′,Q)⊂Rd spanned by the query weights WQh′⊤∈Rd×dhead of head h′ . The layer indices of heads h and h′ are denoted by ℓh and ℓh′ with ℓh<ℓh′ . A larger PK(S(h,O),S(h′,Q)) indicates that the subspaces share more directions with small principal angles in ( 7 ), and thus that the output of head h is more strongly incorporated into the query of head h′ .
Figure 3: Principal angles θ1=0 and θ2 between two planes S,S′⊂R3 with dimS=dimS′=2 , and the corresponding principal vectors s1,s2∈S and s1′,s2′∈S′ . Inspired by Mandolesi (2021, Fig. 1) .
Figure 4: For a two-layer attention-only Transformer, we visualize head-to-head similarity strengths using (a) PK and (b) CS. We show the top two Previous Token Heads in the first layer and the top two Induction Heads in the second layer, coloring them by their head scores (see Appendix F for details).
Figure 5: We compute PK and CS scores for OQ, OK, and OV and plot them as scatter plots. Spearman’s ρ values over all pairs are 0.885 , 0.844 , and 0.866 , indicating strong correlations. In contrast, when we restrict to pairs that belong to the top 3% in either metric (the ⧫ , ■ , and ▲ symbols), the coefficients drop to −0.251 , −0.110 , and 0.186 , and the correlations become weak. In particular, for pairs that belong only to the top 3% in CS (the ▲ symbol), many have small PK scores. For visibility, we plot values up to the 99.99 th percentile.
Figure 6: Visualization of the top 20 scores of PKOQ , PKOK , and PKOV . For each head h , we show the top 10 tokens that represent WOh using the projection method in Section 4 . For readability, we replace Ġ in tokens with _. The L4H7 head is an Identity Head (see Section 8 ). Appendix J.1 shows results for the top 10 and top 30 scores.
Method
OQ
OK
OV
Avg.
PR-AUC
PK
0.446
0.451
0.289
0.395
CS
0.227
0.352
0.215
0.265
Simple-CS
0.245
0.359
0.305
0.303
Linear CKA
0.337
0.323
0.409
0.356
Proc. Sim.
0.328
0.269
0.428
0.342
Table 1: PR-AUC scores for detecting annotated heads in GPT2-small. We treat heads in head classes for the IOI task (36 out of 144) as positives, rank head pairs by similarity, and compute PR-AUC from head-level precision and recall. The best score is shown in bold and the second best score is underlined.
Method
PR-AUC
ROC-AUC
PK
0.047
0.809
CS
0.013
0.785
Simple-CS
0.016
0.792
Linear CKA
0.021
0.715
Proc. Sim.
0.038
0.787
Table 2: Mean PR-AUC and ROC-AUC, averaged over the same-type pairings (QQ, KK, VV, and OO) and head classes for the IOI task in GPT2-small. For each pairing type and head class, we treat within-class head pairs as positives and compute PR-AUC and ROC-AUC based on similarities, then average the scores. The best score is shown in bold and the second best is underlined. See Appendix K.2 for details.
Figure 7: For GPT2-small, (left) Identity Head scores for each head and (right) an example attention pattern of the L0H3 head, which has the largest score. In particular, several heads in early layers and L4H7 have large scores. See Appendix H for details.
Top 1
Top 2
Top 3
Top 4
Top 5
Inlet
OQ
L10H7 N-NM
L5H6 Prev
L7H1
L8H10 S-Inh
L10H11
OK
L5H1 Ind
L7H7
L5H5 Ind
L11H3 B-NM
L11H8
OV
L4H7 Id
L11H3 B-NM
L9H3
L5H2
L11H8
Outlet
OQ
L7H6
L1H1
L4H7 Id
L8H10 S-Inh
L5H2
OK
L4H7 Id
L4H11 Prev
L0H9
L1H11 Id
L5H10
OV
L0H1 Dup, Id
L3H0 Dup, Id
L4H7 Id
L5H10
L5H11
Table 3: Top five heads in GPT2-small by inlet and outlet scores for OQ, OK, and OV, defined in ( 83 ) and ( 85 ). We treat the top 10 heads by Identity Head score in Fig. 7 as Identity Heads and highlight them in bold. For heads that belong to head classes for the IOI task (including Identity Heads), we also show the class labels. See Appendix L for details.
Figure 8: (Left) KL divergence between the PK distribution for random orthogonal matrices in ( 14 ) and the PK distributions measured for each weight pairing type in GPT2-small. (Right) The PK distributions for QQ, which has the largest KL divergence, and VO, which has the smallest KL divergence. In the left panel, the KL scores are large for OQ, OK, and OV, for same-type pairings, and for query and key pairings, and are small otherwise. In the right panel, the QQ distribution differs substantially from the random distribution, whereas the VO distribution is close to the random one.
Table 4: Detection performance for annotated heads on the IOI task in BLOOM-560M and Qwen2-0.5B-Instruct, evaluated by PR-AUC as in Table 1 .
BLOOM-560M
Qwen2-0.5B-Instruct
Method
PR-AUC
ROC-AUC
PR-AUC
ROC-AUC
PK
0.027
0.866
0.032
0.781
CS
0.003
0.844
0.005
0.804
Simple-CS
0.015
0.869
0.004
0.794
Linear CKA
0.001
0.533
0.024
0.509
Proc. Sim.
0.001
0.568
0.024
0.557
Table 5: Classification performance for annotated heads on the IOI task in BLOOM-560M and Qwen2-0.5B-Instruct, evaluated by PR- and ROC-AUC as in Table 2 .
Appendix figures & tables56 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: For the QK matrix in ( 1 ) and the OV matrix in ( 2 ), we plot the Frobenius norms across layers in GPT2-small. For each layer, we also show the 1σ range.
Figure 10: For each pair consisting of a head in an earlier layer and a head in a later layer in GPT2-small, totaling 9,504 pairs, we plot the OQ, OK, and OV CS scores against the corresponding denominator values. Each panel also reports Spearman’s ρ between these quantities.
Figure 11: Singular values of GPT2-small QK matrices WQKh .
Figure 12: Singular values of GPT2-small OV matrices WOVh .
Figure 13: Summands of PK for the top 5 PK head pairs for each of OQ, OK, and OV (rows) in GPT2-small; the columns correspond to the top five head pairs. Each panel shows the dhead2=642=4096 summands (vih′⊤ujh)2 , which add up to the PK value. Here vih′ and ujh are the leading dhead right and left singular vectors of the matrices used to compute the corresponding CS. The summands are sorted in descending order (solid line, left axis), with their cumulative fraction of the total shown by the dashed line (right axis).
Figure 14: Summands of squared CS for the top 5 CS head pairs for each of OQ, OK, and OV (rows) in GPT2-small; the columns correspond to the top five head pairs. Each panel shows the dhead2=642=4096 summands {(σih′vih′/∥Σh′∥F)⊤(σjhujh/∥Σh∥F)}2 . Here σih′,vih′ and σjh,ujh are the singular values and the right and left singular vectors of the source and target matrices of CS, respectively. The summands are sorted in descending order (solid line, left axis), with their cumulative fraction of the total shown by the dashed line (right axis).
Figure 15: Scatter plots of the norm of unembedding vectors in GPT2-small and the inner product with the bias term βfinal of the final layer normalization in ( 70 ), (left) without centering and (right) with centering. Since GPT2 uses byte pair encoding (BPE), the token ID can be viewed as a proxy for frequency, where larger token IDs tend to be less frequent Hayase et al. (2024) (see Fig. 16 ). Without centering (left), the norm and the inner product with βfinal show a strong negative correlation. With centering (right), the correlation becomes slightly weaker but the same trend remains.
Figure 16: Scatter plot of GPT2 token IDs versus token frequencies in the OpenWebText Corpus Gokaslan et al. (2019) (9B tokens). Despite differences between OpenWebText and the undisclosed GPT2 tokenizer training corpus, larger token IDs tend to be less frequent, consistent with Hayase et al. (2024) (Spearman’s ρ=−0.725 ). Since token frequencies vary across datasets, we use token ID as a dataset-agnostic proxy for frequency. Many of the 141 glitch tokens identified by Rumbelow and Watkins (2023) in Table 6 have zero frequency in OpenWebText.
ID
Token
ID
Token
ID
Token
188
\x00
29372
_guiActiveUn
39177
ItemThumbnailImage
189
\x01
30202
_guiName
39253
_UCHIJ
190
\x02
30208
_externalTo
39446
_SetFontSize
191
\x03
30209
_unfocusedRange
39749
DeliveryDate
192
\x04
30210
_guiActiveUnfocused
39752
quickShip
193
\x05
30211
_guiIcon
39753
quickShipAvailable
Appendix
Table 6: The 141 glitch tokens in GPT2 identified by Rumbelow and Watkins (2023) . Here, _ denotes a leading space, and \xNN a non-printable byte with hexadecimal code NN (e.g., \x00 for ID 188 and \x7F for ID 221).
f(et)
Unembed. Norm
Query
Key
Value
Output
All
et
∥et∥
0.750±0.049
0.757±0.045
0.745±0.063
0.670±0.091
0.731±0.074
et−eˉ
∥et−eˉ∥
0.605±0.084
0.625±0.078
0.670±0.108
0.547±0.147
0.612±0.116
et/∥et∥
∥et∥
−0.392±0.092
−0.363±0.086
−0.382±0.127
−0.402±0.145
−0.385±0.116
(et−eˉ)/∥et−eˉ∥
∥et−eˉ∥
−0.198±0.138
−0.168±0.139
−0.054±0.226
−0.163±0.185
−0.146±0.184
Appendix
Table 7: For the 144 heads in GPT2-small, we report the mean and standard deviation of Spearman’s ρ between the unembedding norm and the norm of the vector projected onto Sr after preprocessing f , separately for each weight type. We also report the results when ignoring weight types (All). Without normalization ( f(et)=et or f(et)=et−eˉ ), the pre- and post-projection norms show a strong positive correlation. With normalization ( f(et)=et/∥et∥ or f(et)=(et−eˉ)/∥et−eˉ∥ ), we instead observe a weak negative correlation.
Figure 17: Scatter plots of the inner product with βfinal before and after normalization, (left) without centering and (right) with centering. As in Fig. 15 , we indicate token IDs as a proxy for frequency, where larger token IDs tend to be less frequent, and we label glitch tokens. Without centering (left), the inner products become nearly uncorrelated after normalization. With centering (right), the inner products remain strongly correlated before and after normalization.
Table 8: For Identity, Duplicate Token, Previous Token, and Induction Heads in GPT2-small, we list the top 10 heads and their head scores. See Appendix H.1 for how we compute these scores.
Table 9: Head classes and their corresponding heads in our experimental setup for GPT2-small. Heads that are not included in Wang et al. (2023, Fig. 17) are marked with “ ∗ ”. The total number of annotated heads is 36.
Figure 18: For Duplicate Token, Previous Token, and Induction Heads, we show (left) head scores for all heads in GPT2-small and (right) an example attention pattern for the head with the largest score. The head-score patterns broadly reproduce the results of Wang et al. (2023, Fig. 12) . For each head class, the attention pattern of the top-scoring head captures the characteristic behavior of that class.
Heads
Edges
Method
WP
#Total
#Annot.
#Unannot.
#Total
#Both Annot.
#Either Annot.
Figure
Top10
PK
30
15 (50.0%)
15 (50.0%)
30
10 (33.3%)
22 (73.3%)
Fig. 19
CS
30
9 (30.0%)
21 (70.0%)
30
3 (10.0%)
9 (30.0%)
Fig. 21
Top20
PK
52
22 (42.3%)
30 (57.7%)
60
18 (30.0%)
42 (70.0%)
Fig. 6
✓
51
21 (41.2%)
30 (58.8%)
60
17 (28.3%)
41 (68.3%)
Fig. 37
CS
46
14 (30.4%)
32 (69.6%)
60
3 (5.0%)
20 (33.3%)
Fig. 22
Appendix
Table 10: Number of heads and edges appearing in the wiring diagrams of GPT2-small. For each of OQ, OK, and OV, a score is computed for each pair of a head in an earlier layer and a head in a later layer ( 9,504 pairs), and the top- k pairs are drawn as edges. A head is counted if it is an endpoint of at least one edge. Annotated heads are those registered in the head classes, and an edge is counted in the “Both Annot.” column when both of its endpoints are annotated and in the “Either Annot.” column when at least one of its endpoints is annotated. The symbol ✓in the “WP” column indicates that the method is applied to the preprocessed weights described in Appendix O .
Figure 19: For GPT2-small, we visualize the top 10 scores of PKOQ , PKOK , and PKOV . For each head h , we show the top 10 tokens that represent WOh using the projection method in Section 4 . For readability, we replace Ġ in tokens with _.
Figure 20: For GPT2-small, we visualize the top 30 scores of PKOQ , PKOK , and PKOV . For each head h , we show the top 10 tokens that represent WOh using the projection method in Section 4 . For readability, we replace Ġ in tokens with _.
Figure 21: For GPT2-small, we visualize the top 10 scores of CSOQ , CSOK , and CSOV . For each head h , we show the top 10 tokens that represent WOh using the projection method in Section 4 . For readability, we replace Ġ in tokens with _.
Figure 22: For GPT2-small, we visualize the top 20 scores of CSOQ , CSOK , and CSOV . For each head h , we show the top 10 tokens that represent WOh using the projection method in Section 4 . For readability, we replace Ġ in tokens with _.
Figure 23: For GPT2-small, we visualize the top 30 scores of CSOQ , CSOK , and CSOV . For each head h , we show the top 10 tokens that represent WOh using the projection method in Section 4 . For readability, we replace Ġ in tokens with _.
Figure 24: For GPT2-small, we visualize the top 20 scores of Simple-CS for the pairing types OQ, OK, and OV. For each head h , we show the top 10 tokens that represent WOh using the projection method in Section 4 . For readability, we replace Ġ in tokens with _.
Figure 25: For GPT2-small, we visualize the top 20 scores of Linear CKA for the pairing types OQ, OK, and OV. For each head h , we show the top 10 tokens that represent WOh using the projection method in Section 4 . For readability, we replace Ġ in tokens with _.
Figure 26: For GPT2-small, we visualize the top 20 scores of Procrustes Similarity for the pairing types OQ, OK, and OV. For each head h , we show the top 10 tokens that represent WOh using the projection method in Section 4 . For readability, we replace Ġ in tokens with _.
Figure 27: Projection Kernel (PK) between all head pairs of GPT2-small for each weight pairing type (source, target) of Q, K, V, and O, where the tick label L ℓ denotes layer ℓ .
Figure 28: Composition Score (CS) between all head pairs of GPT2-small for each weight pairing type (source, target) of Q, K, V, and O, where the tick label L ℓ denotes layer ℓ .
Figure 29: Simple-CS between all head pairs of GPT2-small for each weight pairing type (source, target) of Q, K, V, and O, where the tick label L ℓ denotes layer ℓ .
Figure 30: Linear CKA between all head pairs of GPT2-small for each weight pairing type (source, target) of Q, K, V, and O, where the tick label L ℓ denotes layer ℓ .
Figure 31: Procrustes Similarity between all head pairs of GPT2-small for each weight pairing type (source, target) of Q, K, V, and O, where the tick label L ℓ denotes layer ℓ .
Figure 32: Layerwise PK, CS, Simple-CS, Linear CKA, and Procrustes Similarity of GPT2-small, where each panel shows the mean over all 66 head pairs within each layer (all pairwise combinations of the 12 heads) for the weight pairing types QQ, KK, VV, and OO. The dashed line marks the mean over all (2144)=10,296 head pairs for each of the four weight pairing types, yielding a total of 4×(2144)=41,184 comparisons: PK=7.65 , CS=0.05 , Simple-CS=0.05 , Linear CKA=0.81 , and Proc. Sim.=0.88 .
Figure 33: PR curves of head detection for OQ, OK, and OV in GPT2-small, shown for each method.
Table 11: PR-AUC and ROC-AUC for each head class in GPT2-small, computed separately for the weight pairing types QQ, KK, VV, and OO, together with the mean over the four weight pairing types for each head class. These values are used to compute the summary reported in Table 2 . The maximum is shown in bold and the second largest is underlined. We consider the (2144)=10,296 directed head pairs whose source is in an earlier layer than the target; for heads in the same layer, the head with the smaller index is treated as the source. Dup, Prev, Ind, NM, N-NM, B-NM, and S-Inh denote Duplicate Token, Previous Token, Induction, Name Mover, Negative Name Mover, Backup Name Mover, and S-Inhibition heads, respectively.
Table 12: For OQ, OK, and OV in GPT2-small, we report (a) the top 10 heads and their inlet scores in ( 83 ) and (b) the top 10 heads and their outlet scores in ( 85 ). When a head belongs to the head classes in Table 9 or to the top 10 heads by Identity Head score in Table 8(a) , we also report this information. Dup, Prev, Ind, NM, N-NM, B-NM, and S-Inh denote Duplicate Token, Previous Token, Induction, Name Mover, Negative Name Mover, Backup Name Mover, and S-Inhibition heads, respectively.
Figure 34: For random orthogonal matrices in Vm(Rd) , we show (a) a histogram of entry values, (b) squared entry values, and (c) PK values between pairs of random orthogonal matrices. We use d=768 and m=64 , following GPT2-small; (a) and (b) are based on 100,000 randomly sampled entries, and (c) on 100,000 randomly sampled pairs. For (a) and (b), we also plot the corresponding density and its asymptotic approximation as d→∞ . For (c), we plot two approximations to the PK distribution: the tight approximation in Appendix M.3 and the loose approximation in Appendix M.2 . The two approximations are nearly indistinguishable for (a) and (b), while for (c) the tight approximation matches the histogram better, especially near the mean.
GPT2-small
BLOOM-560M
Qwen2-0.5B-Instruct
# parameters
124M
560M
494M
# layers
12
24
24
# query heads
12
16
14
# KV heads
12
16
2
Attention architecture
MHA
MHA
GQA
d
768
1024
896
Appendix
Table 13: Architectural configurations of the models used in our experiments. In particular, GPT2-small and BLOOM-560M use multi-head attention (MHA) Vaswani et al. (2017) , whereas Qwen2-0.5B-Instruct uses grouped-query attention (GQA) Ainslie et al. (2023) and thus has fewer KV heads than query heads.
Table 14: For Identity, Duplicate Token, Previous Token, and Induction Heads in BLOOM-560M, we list the top 10 heads and their head scores. See Appendix H for how we compute these scores.
Table 15: For Identity, Duplicate Token, Previous Token, and Induction Heads in Qwen2-0.5B-Instruct, we list the top 10 heads and their head scores. See Appendix H for how we compute these scores.
Table 16: Head classes and their corresponding heads in our experimental setup for BLOOM-560M. Heads that are not included in Zhang et al. (2025, Fig. 2) are marked with “ ∗ ”. The total number of annotated heads is 44.
Table 17: Head classes and their corresponding heads in our experimental setup for Qwen2-0.5B-Instruct. Heads that are not included in Zhang et al. (2025, Fig. 9) are marked with “ ∗ ”. The total number of annotated heads is 41.
Figure 35: PR curves of head detection for OQ, OK, and OV in BLOOM-560M, shown for each method.
Figure 36: PR curves of head detection for OQ, OK, and OV in Qwen2-0.5B-Instruct, shown for each method.
Table 18: For OQ, OK, and OV in BLOOM-560M, we report (a) the top 10 heads and their inlet scores in ( 83 ) and (b) the top 10 heads and their outlet scores in ( 85 ). When a head belongs to the head classes in Table 16 or to the top 10 heads by Identity Head score in Table 14(a) , we also report this information. Dup, Prev, Ind, NM, N-NM, and S-Inh denote Duplicate Token, Previous Token, Induction, Name Mover, Negative Name Mover, and S-Inhibition heads, respectively.
Table 19: PR-AUC and ROC-AUC for each head class in BLOOM-560M, computed separately for the weight pairing types QQ, KK, VV, and OO, together with the mean over the four weight pairing types for each head class. These values are used to compute the summary reported in Table 5 . The maximum is shown in bold and the second largest is underlined. We consider the (224×16)=(2384)=73,536 directed head pairs whose source is in an earlier layer than the target; for heads in the same layer, the head with the smaller index is treated as the source. Dup, Prev, Ind, NM, N-NM, and S-Inh denote Duplicate Token, Previous Token, Induction, Name Mover, Negative Name Mover, and S-Inhibition heads, respectively.
Table 20: PR-AUC and ROC-AUC for each head class in Qwen2-0.5B-Instruct, computed separately for the weight pairing types QQ, KK, VV, and OO, together with the mean over the four weight pairing types for each head class. These values are used to compute the summary reported in Table 5 . The maximum is shown in bold and the second largest is underlined. We consider the (224×14)=(2336)=56,280 directed head pairs whose source is in an earlier layer than the target; for heads in the same layer, the head with the smaller index is treated as the source. Dup, Prev, Ind, NM, N-NM, and S-Inh denote Duplicate Token, Previous Token, Induction, Name Mover, Negative Name Mover, and S-Inhibition heads, respectively.
Weight preprocessing
Module
Original
Preprocessed
Shape
Layer normalization folding
LN(x)
γ⊙Std(x)x−Mean(x)1d+β
Std(x)x−Mean(x)1d
Rd→Rd
Weight after LN
Win
WinDγCd
Rdmid×d
Bias after LN
bin
Winβ+bin
Rdmid
Centering writing weights
Weight before LN
Wout
CdWout
Rd×dmid
Bias before LN
bout
Cdbout
Rd
Centering the unembedding matrix
Unembedding matrix
Eout
EoutCT
Rd×T
Appendix
Table 21: Summary of weight preprocessing Elhage et al. (2021) ; Nanda and Bloom (2022) . Let d be the model dimension and dmid the intermediate dimension. Here, γ,β∈Rd are the LN scaling weight and bias, and Dγ∈Rd×d is the diagonal matrix with diagonal entries γ . The matrices Cd∈Rd×d and CT∈RT×T are the centering matrices in ( 126 ) for dimensions d and T , where T is the vocabulary size. Note that only LN(x) is a function, whereas the other entries are parameters. See Appendix O.1 for more details.
Figure 37: For GPT2-small, to compare with the wiring diagram constructed from PK on the original weights in Fig. 6 , we visualize the top 20 scores of PKOQ , PKOK , and PKOV computed on the preprocessed weights.
Figure 38: For GPT2-small, to compare with the wiring diagram constructed from CS on the original weights in Fig. 22 , we visualize the top 20 scores of CSOQ , CSOK , and CSOV computed on the preprocessed weights.
Figure 39: For GPT2-small, we report the MSE between scores from the original and preprocessed weights for PK and CS, by weight pairing type. Normalized PK denotes PK normalized by dhead , which maps it to the same [0,1] scale as CS.
Figure 40: For GPT2-small, we report Pearson’s r between scores from the original and preprocessed weights for PK and CS, by weight pairing type.
Figure 41: Layerwise PK and CS with weight preprocessing of GPT2-small, where each panel shows the mean over all 66 head pairs within each layer (all pairwise combinations of the 12 heads) for the weight pairing types QQ, KK, VV, and OO. The dashed line marks the mean over all (2144)=10,296 head pairs for each of the four weight pairing types, yielding a total of 4×(2144)=41,184 comparisons: PK=7.71 and CS=0.05 .
Figure 42: Projection Kernel (PK) with weight preprocessing between all head pairs of GPT2-small for each weight pairing type (source, target) of Q, K, V, and O, where the tick label L ℓ denotes layer ℓ .
Figure 43: Composition Score (CS) with weight preprocessing between all head pairs of GPT2-small for each weight pairing type (source, target) of Q, K, V, and O, where the tick label L ℓ denotes layer ℓ .
Figure 44: Effective ranks of WQh , WKh , WVh , and WOh across all 144 heads of GPT2-small. The maximum possible effective rank is dhead=64 . The black dashed line marks the mean of each panel.
Figure 45: Singular values of GPT2-small attention-head weights WQh .
Figure 46: Singular values of GPT2-small attention-head weights WKh .
Figure 47: Singular values of GPT2-small attention-head weights WVh .
Figure 48: Singular values of GPT2-small attention-head weights WOh .