Table 2: Encoding maps of layers 1 and 2 , positions
1 –
16 (positions
17 –
31 repeat the pattern). Cells: percentage of the activation variance occupied by the code of each node at position
t (
P=218 ), blank below
0.05% , bold at or above
20% . The row closes depth gives the depth of the deepest variable whose span ends at
t . Tokens and depth-
1 and depth-
2 variables are listed by their role relative to
t , with
p=⌊t/2⌋ and
q=⌊t/4⌋ the numbers of completed pairs and 4-spans:
Xt is the current token and
Xt+1 the next;
Xp(4) is the last completed pair,
Xp−1(4) the one before it and
Xp+1(4) the current open pair for odd
t , or the following one for even
t . Likewise for the 4-spans
Xq(3) ,
Xq−1(3) ,
Xq+1(3) . The other rows sum the remaining nodes of that depth. Deeper variables are listed by name, with their span in brackets. Sum of codes adds the strengths of all
63 nodes, explained is the variance of the sum of the codes, and residual its complement, i.e. the part of the activation that is not a sum of single-node functions. Layer
1 holds essentially one code per position, with odd positions encoding the current token (
100% at positions
t≡1(mod4) and
87 –
94% at positions
t≡3(mod4) , where the variable of the pair just completed is retained at
3.5 –
7.4% ), and even positions the depth-
1 parent of the completed pair (
97.5 –
99.1% at positions
t≡2(mod4) , and
89 –
95% at the multiples of
4 , where the current token takes
2 –
7% ). Layer
2 adds depth-
2 variables at the multiples of
4 : at
98 –
99% at positions
4 ,
12 ,
20 and
28 , and at
70 –
79% at positions
8 ,
16 and
24 , where all latent variables of the completed
8 -span linger (c.f.
t=24 tree of Fig. 12 ). Note positions
t≡3(mod4) , where the depth-
1 variable of the last completed pair is kept at
37 –
49% next to the current token at
29 –
43% . This is persistence : the code of a pair survives the position after the pair closes, and is dropped at the next one, when the span of its parent closes.