Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction

Kingsuk Maitra; Shagun Sood Morteza Hosseini; Suman Gunnala; Vikram Gupta

Figure 12: RoPE rotations across frequency blocks. Effect on four representative frequency blocks of a projected query stream, for a 20-token sequence with

d_{\mathrm{model}}{=}64

,

d_{k}{=}32

. Blocks with large

\theta_{m}

(e.g. block 0,

\theta{=}1.0000

) rotate rapidly with position; blocks with small

\theta_{m}

(block 15,

\theta{=}0.0002

) barely rotate over the full sequence and therefore carry long-range positional information. Original query dots (blue) and their rotated counterparts (orange) lie on coincident norm shells, confirming norm preservation; the verification np.allclose(

\|q\|

,

\|\tilde{q}\|

) returned True to double precision.

Figure 13: EMA momentum magnitude as a function of

\beta

. Momentum magnitude

\|p_{q,n}\|

versus position for four values of

\beta

, with a 50-token sequence generated from structured (oscillating) embeddings. The effective memory window is approximately

(1-\beta)^{-1}

:

\beta{=}0.5

yields a 2-step window and tracks the instantaneous kinematic difference closely;

\beta{=}0.95

yields a 20-step window and produces a heavily smoothed trajectory. All four traces start at

\|p_{q,n}\|{=}0

by construction.

Figure 14: Causal attention patterns across four momentum strengths at

\beta{=}0.9

. The upper triangle is masked. Visually the patterns are closely similar across the four panels: the entropy and per-row spread are identical to three decimals in all four cases (average entropy

2.549

, mean standard deviation

0.0410

). This is the expected behavior at

\beta{=}0.9

: the momentum signal is heavily damped, and the augmentation contributes only a small perturbation to scores already in the high-temperature regime. The next subsection quantifies this through the four-term decomposition.

Figure 15: Attention score decomposition at

i{=}20

,

\gamma{=}0.15

,

\beta{=}0.9

. The four terms of Equation ( 70 ) are plotted against key index

j

. The position–position term dominates in magnitude; the two cross terms are of comparable, small size; the momentum–momentum term is three orders of magnitude below the leader, consistent with the

\gamma^{2}

scaling and with the Section A3 finding that the EMA at

\beta=0.9

damps the momentum signal severely.

Component	Mean $\|\cdot\|$	Ratio to $T_{1}$
$T_{1}$ : $\tilde{q}_{i}^{\!\top}\tilde{k}_{j}$ (position–position)	$0.0523$	$1.000$
$T_{2}$ : $\gamma_{Q}p_{q,i}^{\!\top}\tilde{k}_{j}$ (Q-mom $\cdot$ K-pos)	$0.0008$	$0.015$
$T_{3}$ : $\gamma_{K}\tilde{q}_{i}^{\!\top}p_{k,j}$ (Q-pos $\cdot$ K-mom)	$0.0009$	$0.017$
$T_{4}$ : $\gamma_{Q}\gamma_{K}p_{q,i}^{\!\top}p_{k,j}$ (momentum–momentum)	$1.3\!\times\!10^{-5}$	$2.5\!\times\!10^{-4}$

Appendix

Table 14: Mean absolute magnitude of the four score components at query position

i{=}20

, with

\gamma_{Q}=\gamma_{K}=0.15

and

\beta=0.9

. The position–position term dominates by approximately

65\times

over either cross term; the two cross terms are nearly identical, a sanity check on Q/K symmetry of the EMA.

Figure 16: Attention distribution at

i=20

with and without momentum. Left: weight over the 21 accessible keys; right: pointwise difference

w^{\mathrm{mom}}-w^{\mathrm{no\text{-}mom}}

. The maximum attention weight relocates from position 20 (self) to position 10 when momentum is enabled, and the total absolute change in the distribution over the 21 keys is

0.0109

. The effect is small in magnitude but structural: the momentum reweights attention towards a specific historical position rather than smearing the distribution.

Figure 17: Causal attention as a function of

\gamma

at

\beta=0.9

. Differences between panels are visible only on fine inspection. They become dramatic once

\beta

is dropped, as Section A3 demonstrates.

Parameter	Value
Architecture	4-layer transformer decoder, causal mask
$d_{\mathrm{model}}/n_{\mathrm{heads}}/$ head dim	$128/4/32$
FFN dimension	512
Dropout	0.1
Vocabulary size	202
Momentum coupling	$\gamma=0.5$ (fixed)

Appendix

Table 15: Model and training configuration for the

\beta

-sweep.

$\beta$	$L=4$	$L=8$	$L=12$	$L=16$	$L=20$
0.0	94.0	86.5	49.3	12.1	5.1
0.1	87.9	71.3	20.0	6.7	3.9
0.2	74.7	40.7	11.2	5.5	3.0
0.3	56.7	21.4	8.1	3.9	3.5
0.4	43.6	16.0	6.5	3.9	2.9
0.5	34.3	12.0	6.1	3.3	2.9

Appendix

Table 16: Mean test accuracy (%) on associative recall, by

\beta

and chain length

L

. Three seeds per cell. Momentum coupling

\gamma=0.5

. The

\beta=0.0

row dominates across all chain lengths; accuracy collapses monotonically as

\beta

increases.

Quantity	Value
Mean accuracy, $\beta=0.0$ (pure high-pass, momentum)	$49.4\%$
Mean accuracy, $\beta=0.9$ (heavy low-pass, momentum)	$9.5\%$
Mean accuracy, vanilla attention (no momentum)	$10.0\%$
Pearson $\rho(\|H(\pi)\|,\text{accuracy})$	$0.507$
Correlation $p$ -value	$3.47\!\times\!10^{-11}$
$R^{2}$	$0.257$

Appendix

Table 17: Headline numbers averaged across all chain lengths and seeds.

Figure 18: Main results of the

\beta

-sweep. (A) Accuracy vs.

\beta

per chain length, with vanilla-baseline horizontal lines (dashed) per length; in every case the

\beta{=}0

curve lands above the vanilla line and the

\beta\!\to\!0.9

limit coincides with it. (B) Accuracy vs. Nyquist gain

|H(\pi)|=(1-\beta)/(1+\beta)

: monotone increasing trend with fitted slope well separated from zero (

\rho{=}0.51,\ p{=}3.5\!\times\!10^{-11}

). (C) Mean gain over vanilla per

\beta

: positive at

\beta\leq 0.2

, crosses zero around

\beta=0.5

, slightly negative at

\beta\geq 0.6

. (D) Accuracy heatmap over

(\beta,L)

(numerically Table 16 ). (E) Theory vs. experiment: the blue curve is the analytical Nyquist gain scaled to

[0,100]

; the red curve is empirical accuracy averaged over chain lengths. The two share a shape but the empirical curve is substantially attenuated.

Figure 19: Detailed per-chain-length decomposition. Top-left:

\beta{=}0

vs.

\beta{=}0.9

bar comparison per chain length — the

\beta{=}0

bars are uniformly taller at

L=4,8,12

and the gap closes only at

L\geq 16

because both configurations approach the random baseline. Top-center: mean gain over vanilla per

\beta

. Top-right: Nyquist gain vs.

\beta

with the ten sampled points marked. Bottom-left: Cohen’s

d

(

\beta{=}0

vs.

\beta{=}0.9

) per chain length, with the conventional

d=0.8

“large effect” threshold marked; at

L=4

and

L=8

the observed

d

exceeds the threshold comfortably. Bottom-center: three-way comparison of

\beta{=}0

,

\beta{=}0.9

, and vanilla. Bottom-right: narrative summary panel.

Parameter	Value
Architecture	4-layer transformer decoder, 4 heads, head dim 32
$d_{\mathrm{model}}/d_{k}/d_{\mathrm{ff}}$	$128/32/512$
Dropout	0.1
Vocabulary size	200
Chain length $L$	12 KV pairs (sequence length 25)
Momentum form	Pure high-pass ( $\beta=0$ )

Appendix

Table 18: Model and training configuration for the

\gamma

-sweep.

$\gamma$	Sinusoidal PE (%)		RoPE (%)
0.00	4.9 $\pm$	1.2	5.5 $\pm$	0.5
0.01	4.4 $\pm$	0.3	6.5 $\pm$	1.2
0.02	5.1 $\pm$	1.0	5.1 $\pm$	0.7
0.03	4.3 $\pm$	1.1	5.4 $\pm$	0.3
0.04	5.0 $\pm$	0.2	5.9 $\pm$	0.4
0.05	4.1 $\pm$	0.8	7.1 $\pm$	0.2

Appendix

Table 19: Test accuracy (%) on associative recall (

L=12

, vocab 200), mean

\pm

standard deviation over three seeds, for both sinusoidal PE and RoPE . The phase transition is visible in both columns as a sigmoidal jump from near-random to near-perfect accuracy between

\gamma\!\in\![0.20,0.40]

.

Encoding	$\gamma_{c}$	Peak acc.	at $\gamma$
RoPE	$0.225$	$99.4\%$	$3.00$
Sinusoidal PE	$0.275$	$99.5\%$	$1.50$
Ratio $\gamma_{c}^{\mathrm{sin}}/\gamma_{c}^{\mathrm{RoPE}}$	$1.22\times$
Implied $r$	$0.818$

Appendix

Table 20: Detected phase-transition parameters under post-encoding placement.

Figure 20: Phase transition: RoPE vs. sinusoidal PE. (A) Full

\gamma

-range, log scale: both curves show near-random accuracy for

\gamma<0.1

, sharp transition between

\gamma=0.2

and

\gamma=0.4

, saturation near

99\%

at

\gamma\geq 0.5

. Vertical dashed lines mark detected

\gamma_{c}

values (

0.225

for RoPE ,

0.275

for sinusoidal). (B) Critical region

[0,0.5]

, linear scale, with the critical-region band shaded. (C) Fine-grained zoom of

[0,0.25]

showing the leading edge: RoPE begins accelerating earlier (by

\gamma\sim 0.12

–

0.14

), both converge by

\gamma=0.30

. (D) Susceptibility

\mathrm{d\,acc}/\mathrm{d}\gamma

peaks sharply at the respective

\gamma_{c}

and is an order of magnitude larger for sinusoidal PE (

\sim 1200\%

/unit) than for RoPE (

\sim 600\%

/unit), reflecting a narrower transition width for sinusoidal. (E) Bar comparison of the two

\gamma_{c}

estimates, annotated with the

1.22\times

ratio. (F) Headline summary panel.

Figure 21: Detailed transition analysis. (A) S-curves from cubic interpolation between sampled

\gamma

-values, with

\gamma_{c}

vertical lines; both qualitatively sigmoidal. (B) Susceptibility (piecewise derivative used to detect

\gamma_{c}

); the sinusoidal curve shows a taller, narrower peak at

\gamma\sim 0.275

, RoPE a broader peak near

\gamma=0.225

. (C) Bar summary of the two

\gamma_{c}

values and their

1.22\times

ratio.

Parameter	Value
Vocabulary size $V_{\text{vocab}}$	$64$
$d_{\mathrm{model}}$	$64$
$n_{\text{heads}}\times d_{\text{head}}$	$4\times 16$
$d_{\text{ff}}$	$256$
$N_{\text{layers}}$	$1$ (strictly)
RoPE base	$10\,000$

Appendix

Table 21: Experiment 16 configuration.

$\gamma$	Acc (%)	$\gamma$	Acc (%)	$\gamma$	Acc (%)
0.00	1.2	1.00	70.2	2.40	75.0
0.05	4.4	1.10	69.4	2.50	77.6
0.10	4.6	1.20	62.6	3.00	76.8
0.15	6.4	1.30	65.8	3.20	75.6
0.20	7.8	1.40	70.4	3.50	79.0
0.30	18.2	1.50	74.8	4.00	83.4

Appendix

Table 22: Experiment 16: accuracy as a function of momentum coupling

\gamma

at

N{=}1

, single seed. Random chance is

1.56\%

; the

\gamma{=}0

baseline lands at

1.2\%

(essentially chance, confirming that single-layer standard attention cannot perform induction); the peak is

\mathbf{83.4\%}

at

\gamma=4.0

, a

\mathbf{+82.2}

percentage-point gain over baseline and

\mathbf{53\times}

the random-chance accuracy. The dataset is deliberately harder than the disjoint-alphabet phase-transition configuration of Section A4 : a single shared

V{=}64

alphabet (no role-flag information from token IDs),

14

K-V pairs (vs.

8

there), and explicit padding tokens in the sequence; this design tests robustness rather than the absolute performance ceiling.

$N$	Predicted $\gamma^{*}$	$\gamma$ sweep
1	$4.00$	$\{0,1,2,3,4,5,6\}$
2	$2.00$	$\{0,0.5,1,1.5,2,2.5,3\}$
4	$1.00$	$\{0,0.25,0.5,0.75,1,1.25,1.5\}$
8	$0.50$	$\{0,0.1,0.25,0.4,0.5,0.6,0.75\}$

Appendix

Table 23: Experiment 17

\gamma

grid by depth (centered on predicted

\gamma^{*}=4/N

).

$N$	accuracy (%) at each $\gamma$
$1$	$0{:}2.2$	$1.0{:}65.4$	$2.0{:}76.4$	$3.0{:}81.0$	$\mathbf{4.0{:}83.0}$	$5.0{:}79.4$	$6.0{:}77.8$
$2$	$0{:}2.8$	$0.5{:}36.8$	$1.0{:}66.6$	$1.5{:}73.6$	$2.0{:}77.6$	$2.5{:}80.2$	$\mathbf{3.0{:}86.0}$
$4$	$0{:}3.4$	$0.25{:}8.6$	$0.5{:}31.2$	$0.75{:}50.4$	$1.0{:}65.4$	$\mathbf{1.25{:}71.2}$	$1.5{:}64.2$
$8$	$0{:}2.4$	$0.1{:}3.8$	$0.25{:}6.6$	$0.4{:}13.4$	$0.5{:}20.2$	$0.6{:}30.4$	$\mathbf{0.75{:}37.6}$

Appendix

Table 24: Experiment 17: accuracy (%) for every

(N,\gamma)

configuration, single seed. Best per-depth entries are bold .

Figure 22: Experiment 17: the four-panel scaling-law figure. (A) Accuracy vs.

\gamma

by depth

N

; the curves shift leftward as

N

grows, exactly as the scaling law predicts. (B) Optimal coupling

\gamma^{*}

vs.

N

, with the fitted

4.17/N^{0.73}

(

R^{2}=0.947

) and the theoretical

4/N

overlay (

\alpha=1

). (C) Log–log plot showing the power-law slope of

-\alpha\approx-0.73

. (D) The product

\gamma^{*}\!\cdot N

per depth: values

4.0,6.0,5.0,6.0

are not constant, consistent with

\alpha<1

.

Figure 23: Experiment 17 summary figure. The per-depth accuracy-vs-

\gamma

families and the fitted scaling curve are reproduced alongside the peak-accuracy bar chart.

Parameter	Value
Architecture	nano (identical to E16, E17)
Chain length $L$	$30$ KV pairs
Sequence length	$64$ tokens
Depth grid $N$	$\{1,2,3,4,6,8\}$ — six values
Seeds per config	$3$
$\gamma$ grid size	15 values per depth, centered on predicted $\gamma^{*}$

Appendix

Table 25: Experiment 18 configuration.

$N$	Predicted $\gamma^{*}$	Observed $\gamma^{*}$	Best acc. (%)	SEM (%)	Baseline (%)
$1$	$4.17$	$2.00$	$\mathbf{57.4}$	$3.4$	$1.7$
$2$	$2.51$	$2.75$	$\mathbf{61.9}$	$4.1$	$2.0$
$3$	$1.87$	$2.00$	$\mathbf{29.0}$	$15.6$	$1.3$
$4$	$1.52$	$3.00$	$\mathbf{51.2}$	$11.1$	$1.6$
$6$	$1.13$	$2.50$	$\mathbf{40.3}$	$16.2$	$1.4$
$8$	$0.91$	$2.50$	$\mathbf{24.3}$	$12.6$	$1.7$

Appendix

Table 26: Experiment 18: mean test accuracy (%) across three seeds at chain length

L=30

, for each

(N,\gamma)

configuration reported by the notebook. Predicted

\gamma^{*}

from E17 in the second column. Per-row best is bold .

Parameter	E17 ( $L=14$ )	E18 ( $L=30$ )
$\gamma_{0}$	$4.17\pm 0.41$	$2.73\pm 0.47$
$\alpha$	$0.73\pm 0.15$	$0.10\pm 0.14$
$R^{2}$	$0.947$	$-0.607$
Seeds/config	1	3
Depths fitted	4	6

Appendix

Table 27: E17 vs. E18 scaling-law fit comparison.

Figure 24: Experiment 18 scaling-law validation at

L=30

. Accuracy-vs-

\gamma

with error bars (three-seed SEM) by depth; the per-depth optima are scattered across

\gamma\!\in\![2.0,3.0]

with overlapping error bars, which is the quantitative statement of why the fit fails.

Figure 25: Experiment 18 heatmap. Accuracy heatmap across all

(N,\gamma)

configurations at

L=30

. White stars mark the per-depth argmax

\gamma^{*}

. The heatmap confirms the flat, noisy accuracy plateau at each depth

\geq 3

— the primary reason a clean scaling law does not emerge in this regime.

Parameter	Value
Architecture	nano, post- RoPE PSA
$L$ (KV pairs)	$30$
Sequence length	$64$
Query position	$60=2L$
Training steps	$3\,000$
RoPE base	$10\,000$

Appendix

Table 28: Validation Suite v7 configuration.

$\gamma$	Acc (%)	$\gamma$	Acc (%)	$\gamma$	Acc (%)
0.00	1.3	1.20	66.9	2.40	60.3
0.20	2.0	1.40	70.4	2.60	59.3
0.40	9.3	1.60	71.5	2.80	49.0
0.60	29.8	1.80	72.6	3.00	29.4
0.80	45.5	2.00	74.2
1.00	57.6	2.20	73.2

Appendix

Table 29: v7 S1: single-layer

\gamma

-sweep, three-seed means.

Figure 26: v7 S1: single-layer induction phase transition at

L=30

with three-seed means. (A) Accuracy bars with the

\gamma=0

“FAIL” bar highlighted in red and the

75\%

benchmark dashed. (B) Phase-transition visual: static regime (

\gamma<0.4

, red shaded) and kinematic regime (

\gamma\in[0.6,3.0]

, green shaded), transition edge at

\gamma\approx 0.5

.

$N$	$\gamma^{*}_{\text{obs}}$	Accuracy (%)	Baseline (%)
1	1.60	73.2	1.5
2	1.60	71.3	1.5
3	2.40	71.1	2.3
4	1.80	60.1	1.7
6	1.50	59.8	1.6
8	1.00	14.1	1.4

Appendix

Table 30: v7 S2: per-depth scaling law at

L=30

.

Figure 27: v7 S2: scaling-law figure at

L=30

. Accuracy-vs-

\gamma

family across six depths; fitted

1.88/N^{0.11}

with

R^{2}=0.112

; product

\gamma^{*}\!\cdot N

bar chart clearly not constant. The dotted theoretical

4/N

reference curve is shown for comparison.

$\gamma$	Mean	Std	SS regime?
0.0	$1.4257$	$0.0217$	NO
0.6	$1.4529$	$0.0200$	NO
1.2	$1.5878$	$0.0254$	NO
1.8	$1.7043$	$0.0341$	NO
2.4	$1.7176$	$0.0415$	NO
3.0	$1.4991$	$0.0235$	NO

Appendix

Table 31: v7 S3: SS ratio

\|p_{q}\|/\|q^{\mathrm{rot}}\|

(mean

\pm

std, 300 probes).

$\gamma$	$T_{1}$ gap	$T_{2}$ gap	$T_{3}$ gap	$T_{4}$ gap	$\|T_{4}\|>\|T_{1}\|$ ?
0.0	$T_{2}=T_{3}=T_{4}=0$ by construction
0.6	$0.0142$	$0.0053$	$0.7608$	$0.4677$	YES
1.2	$0.0123$	$0.0048$	$0.8561$	$1.0382$	YES
1.8	$0.0128$	$0.0219$	$0.6866$	$1.2635$	YES
2.4	$0.0082$	$0.0134$	$0.5038$	$1.2378$	YES
3.0	$-0.0022$	$-0.0183$	$0.2375$	$0.6514$	YES

Appendix

Table 32: v7 S4–5: four-term gaps at

j^{*}

versus other positions, averaged over 100 probes.

$\gamma$	$\delta_{\mathrm{emp}}$	$\Delta_{\mathrm{symp,th}}$	Ratio	$H_{0}$
0.0	$0$	$0$	–	$6.7666$
0.6	$3.22\!\times\!10^{-4}$	$5.77\!\times\!10^{-2}$	$0.0056$	$6.3565$
1.2	$8.85\!\times\!10^{-4}$	$1.13\!\times\!10^{-1}$	$0.0078$	$3.4900$
1.8	$1.00\!\times\!10^{-3}$	$1.42\!\times\!10^{-1}$	$0.0071$	$2.0104$
2.4	$9.26\!\times\!10^{-4}$	$1.65\!\times\!10^{-1}$	$0.0056$	$1.3252$
3.0	$1.45\!\times\!10^{-3}$	$2.44\!\times\!10^{-1}$	$0.0060$	$1.2549$

Appendix

Table 33: v7 S6–7: AC energy, over-symplectic energy bound, and the derived ratio.

Figure 28: v7 S6–7: AC-energy diagnostics. (A)

\delta_{\mathrm{emp}}

versus

\gamma

on a log scale. (B)

\delta_{\mathrm{emp}}(t)

versus position

t

, showing a slow fall-off with position rather than a flat trace — the causal-task confound addressed by v9 C9. (C) Empirical

\delta_{\mathrm{emp}}

alongside theoretical

\Delta_{\mathrm{symp,th}}

bars; the empirical values are orders of magnitude below the symplectic-theory bound, confirming the bound.

$\gamma$	Mean $H_{0}$	Rel. range	Flat ( $<15\%$ )?
0.6	$6.5560$	$1.0322$	DRIFT
1.2	$3.5985$	$1.0438$	DRIFT
1.8	$2.0713$	$1.0500$	DRIFT
2.4	$1.3632$	$1.0350$	DRIFT
3.0	$1.2905$	$0.9886$	DRIFT

Appendix

Table 34: v7 S8:

H_{0}

relative range across positions

t\geq 4

.

$\gamma$	$\delta_{\mathrm{s,post}}$	$\delta_{\mathrm{ns,post}}$	HSF post	$\delta_{\mathrm{s,pre}}$	$\delta_{\mathrm{ns,pre}}$	HSF pre
0.0	0	0	–	0	0	–
0.6	$3.22\!\times\!10^{-4}$	$1.38\!\times\!10^{-3}$	$4.2896$	$0.3528$	$0.5206$	$1.4758$
1.2	$8.85\!\times\!10^{-4}$	$2.91\!\times\!10^{-3}$	$3.2906$	$0.6942$	$4.808$	$6.9255$
1.8	$1.00\!\times\!10^{-3}$	$2.29\!\times\!10^{-3}$	$2.2821$	$0.7582$	$2.674$	$3.5262$
2.4	$9.26\!\times\!10^{-4}$	$1.59\!\times\!10^{-3}$	$1.7184$	$0.8714$	$1.593$	$1.8284$
3.0	$1.45\!\times\!10^{-3}$	$1.73\!\times\!10^{-3}$	$1.1930$	$1.634$	$1.629$	$0.9967$

Appendix

Table 35: v7 S8b: HSF values and AC energies (v7 causal-mask fix applied).

Figure 29: v7 S8/S8b:

H_{0}

conservation and HSF. (A)

H_{0}(t)

flatness at five tested

\gamma

: traces essentially constant across

4\!\leq\!t\!\leq\!60

, with a sharp drop at the query position

t=60

(an artefact of the final-token boundary). (B) HSF post- and pre-softmax across

\gamma

; HSF post monotone decreasing, HSF pre peaks at

\gamma=1.2

(v9 C8 explains). (C) Pre-softmax

\delta

for symmetric PSA versus asymmetric NS, confirming that NS produces substantially larger AC energy.

	Post-softmax			Pre-softmax (v7 fix)
$\gamma$	mean $r$	mean Nyq	theory Nyq	mean $r$	mean Nyq	theory Nyq
0.0	$0.0000$	$0.1273$	$1.0000$	$0.0000$	$0.0000$	$1.0000$
0.6	$0.9558$	$2.1589$	$4.8400$	$0.9982$	$75.97$	$4.8400$
1.2	$0.9278$	$2.5142$	$11.5600$	$0.9987$	$85.90$	$11.5600$
1.8	$0.9262$	$2.5378$	$21.1600$	$0.9991$	$100.73$	$21.1600$
2.4	$0.9331$	$2.5840$	$33.6400$	$0.9991$	$151.99$	$33.6400$

Appendix

Table 36: v7 S9: Bode forensics, post- and pre-softmax.

Figure 30: v7 S9: Bode forensics. Top row: post-softmax observed AC-power spectrum (solid) versus theory (dashed) at the six tested

\gamma

; Nyquist gain compressed relative to theory by

\sim 15

–

18\times

at

\gamma=3.0

. Bottom row (v7 fix): pre-softmax AC power; the empirical curves (solid) track the theoretical

H^{2}(\omega)

with

r

up to

0.9994

. The

\gamma=0

baseline is zero at all frequencies, as expected.

$\gamma$	$E_{\mathrm{DC}}$	$E_{\mathrm{AC}}$	frac AC
0.0	$3.5955$	$8.8273$	$0.7194$
0.6	$8.9203$	$46.8653$	$0.8417$
1.2	$8.3484$	$60.8577$	$0.8827$
1.8	$6.1622$	$58.4498$	$0.9086$
2.4	$4.3498$	$64.2329$	$0.9405$
3.0	$4.3834$	$116.1734$	$0.9637$

Appendix

Table 37: v7 S10: spectral complementarity.

Figure 31: v7 S10/S11: spectral complementarity and Parseval. Panel covering frac AC versus

\gamma

, energies

E_{\mathrm{DC}}

and

E_{\mathrm{AC}}

, and the Parseval identity test across the spectral

\gamma

grid.

$\gamma$	$\mathbb{E}\\|\hat{q}\\|^{2}$ (LHS)	RHS	discrepancy	zero-sum mean viol.
0.0	$7.0491$	$7.0491$	$0$	$0$
0.6	$19.1152$	$19.1152$	$3.52\!\times\!10^{-8}$	$4.63\!\times\!10^{-8}$
1.2	$22.3856$	$22.3856$	$5.47\!\times\!10^{-8}$	$4.90\!\times\!10^{-8}$
1.8	$22.6853$	$22.6853$	$4.05\!\times\!10^{-8}$	$4.94\!\times\!10^{-8}$
2.4	$23.3368$	$23.3368$	$6.38\!\times\!10^{-8}$	$4.89\!\times\!10^{-8}$
3.0	$31.8507$	$31.8507$	$2.73\!\times\!10^{-9}$	$5.14\!\times\!10^{-8}$

Appendix

Table 38: v7 S11: Parseval identity check and zero-sum violation magnitudes.

band	$\theta$ (rad)	theory $\|\mathrm{err}\|$	observed rel. err
0	$1.000000$	$0.958851$	$0.407971$
1	$0.316228$	$0.314912$	$0.041205$
2	$0.100000$	$0.099958$	$0.004453$
3	$0.031623$	$0.031621$	$0.000411$
4	$0.010000$	$0.010000$	$0.412654$
5	$0.003162$	$0.003162$	$0.042761$

Appendix

Table 39: v7 S12: Coriolis per RoPE band at

\gamma=3.0

, sorted by

\theta

descending.

Figure 32: v7 S12: Coriolis error per RoPE band, sorted by

\theta

descending. (A) Observed relative-error bars alongside the theoretical

2|\sin(\theta/2)|

curve; band-4 (fifth bar) is a

41\times

anomaly. (B) Scatter of observed vs. theoretical, with the

+0.307

positive slope confirming Corollary 41 directionally despite the band-4 outlier.

$\gamma$	mean	std	status
0.0	$1.4257$	$0.0217$	non-SS
0.6	$1.4529$	$0.0200$	non-SS
1.2	$1.5878$	$0.0254$	non-SS
1.8	$1.7043$	$0.0341$	non-SS
2.0	$\mathbf{1.7107}$	$\mathbf{0.0344}$	PASS — FP-18b CONFIRMED ( $\gamma^{*}$ )
2.4	$1.7176$	$0.0415$	non-SS

Appendix

Table 40: v8 C6 (FP-18b): SS ratio

\|p_{q}\|/\|q^{\mathrm{rot}}\|

across the v8 grid.

Figure 33: v8 C6: FP-18b direct measurement. Direct-measurement SS ratio at

\gamma^{*}=2.0

(red marker). The threshold line at

1.0

is crossed everywhere; the confirmation at

\gamma^{*}

turns the v7 CHECK into a v8 PASS.

$\gamma$	$r_{\mathrm{buggy}}$	$r_{\mathrm{fixed}}$	$\Delta r$	verdict
0.0	$0.0000$	$0.0000$	$0$	baseline
0.6	$0.0000$	$0.9982$	$+0.9982$	FIX DECISIVE
1.2	$0.0000$	$0.9987$	$+0.9987$	FIX DECISIVE
1.8	$0.0000$	$0.9991$	$+0.9991$	FIX DECISIVE
2.0	$0.0000$	$0.9990$	$+0.9990$	FIX DECISIVE
2.4	$0.0000$	$0.9991$	$+0.9991$	FIX DECISIVE

Appendix

Table 41: v8 C7: Bode A/B, pre-softmax, same weights.

Figure 34: v8 C7: pre-softmax Bode A/B on the same trained weights. Top row: the fixed v7/v8 analysis produces the expected

H^{2}(\omega;\gamma)

-shaped spectrum with

r

up to

0.9994

. Middle row: the buggy pre-v7 analysis produces all-NaN outputs and therefore

r=0

. Bottom row: the bar summary across the entire

\gamma

grid.

	Post-softmax		Pre-softmax (v7/v8 fix)
$\gamma$	mean $r$	mean Nyq	mean $r$	mean Nyq
0.0	$0.0000$	$0.1273$	$0.0000$	$0.0000$
0.6	$0.9558$	$2.1589$	$0.9982$	$75.97$
1.2	$0.9278$	$2.5142$	$0.9987$	$85.90$
1.8	$0.9262$	$2.5378$	$0.9991$	$100.73$
2.0	$\mathbf{0.9261}$	$\mathbf{2.5209}$ ( $\gamma^{*}$ )	$\mathbf{0.9990}$	$\mathbf{113.56}$ ( $\gamma^{*}$ )

Appendix

Table 42: v8 C8: post- and pre-softmax Bode (full v8 grid).

Figure 35: v8 C8: full Bode forensics with

\gamma^{*}=2.0

included. Post-softmax

r

near

0.93

across the grid; pre-softmax

r

up to

0.9994

throughout.

Figure 36: v8 summary figure. FP-18b direct measurement, Bode A/B, and the updated scorecard in a single panel.

Parameter	Value
Architecture	nano (identical to v7/v8)
$\gamma$ grid	$\{0.0,0.4,0.6,0.8,\ldots,3.2\}$ — 16 pts
Fine T3/T4 grid	$\{0.60,0.65,0.70,\ldots,1.40\}$ — 17 pts
$d_{k}$ sweep (FP-21/22)	$\{8,16,32\}$ with $d_{\mathrm{model}}=4d_{k}$
Seeds (FP-24)	$\{42,123,456,789,1011\}$
Training steps	$3\,000$

Appendix

Table 43: v9 configuration.

$\gamma$	mean $r$	Nyq obs	Nyq theory	compression
0.0	$0.0000$	$0.1273$	$1.0000$	–
0.4	$0.9683$	$1.8704$	$3.2400$	$0.5773$
0.6	$0.9511$	$2.2682$	$4.8400$	$0.4686$
0.8	$0.9346$	$2.5167$	$6.7600$	$0.3723$
1.0	$0.9260$	$2.6563$	$9.0000$	$0.2951$
1.2	$0.9195$	$2.7735$	$11.5600$	$0.2399$

Appendix

Table 44: v9 C6: post-softmax Bode Nyquist versus theory, with compression ratio.

Figure 37: v9 C6: post-softmax Bode on the v9 16-point grid. Observed Nyquist versus theory and the compression ratio monotonically decreasing from

0.577

to

0.065

.

$\gamma$	$T_{1}$ gap	$T_{2}$ gap	$T_{3}$ gap	$T_{4}$ gap	dominant
0.60	$-1.2583$	$-1.2967$	$-1.2491$	$-1.2705$	$T_{3}$
0.65	$-1.2216$	$-1.2603$	$-1.2156$	$-1.2382$	$T_{3}$
0.70	$-1.1683$	$-1.2063$	$-1.1630$	$-1.1857$	$T_{3}$
0.75	$-1.1085$	$-1.1459$	$-1.1028$	$-1.1237$	$T_{3}$
0.80	$-1.0485$	$-1.0853$	$-1.0421$	$-1.0627$	$T_{3}$
0.85	$-0.9963$	$-1.0337$	$-0.9881$	$-1.0119$	$T_{3}$

Appendix

Table 45: v9 C7:

T_{3}

vs

T_{4}

gaps at the induction target on fine

\gamma

grid.

Figure 38: v9 C7:

T_{3}\to T_{4}

crossover.

\gamma T_{3}

and

\gamma^{2}T_{4}

curves cross at

\gamma\approx 0.97

. Below crossover,

T_{3}

(linear in

\gamma

) dominates; above, the

\gamma^{2}T_{4}

term overtakes through its quadratic growth.

$\gamma$	HSF post	HSF pre	pre $\geq$ post?	regime
0.4	$1.9538$	$0.8210$	FAIL	low- $\gamma$ Fréchet
0.6	$4.2608$	$1.4674$	FAIL	low- $\gamma$ Fréchet
0.8	$2.3820$	$2.3097$	FAIL	low- $\gamma$ Fréchet
1.0	$3.5909$	$6.8612$	OK	interior
1.2	$3.2976$	$6.9316$	OK	interior
1.4	$3.2228$	$7.4056$	OK	interior

Appendix

Table 46: v9 C8: HSF pre/post ordering across the v9 grid.

Figure 39: v9 C8: HSF pre vs post. HSF pre/post analysis across the v9 grid, with the Fréchet boundary regions highlighted. Interior regime

\gamma\!\in\![1.0,2.4]

shows the predicted ordering; the two extremes are explained by softmax non-linearity at low

\gamma

and PSA over-rotation at high

\gamma

.

$\gamma$	causal rel. range	stationary rel. range	verdict
0.6	$0.0370$	$0.0550$	CHECK (stat)
1.2	$0.0366$	$0.0362$	PASS (stat flat)
1.8	$0.0262$	$0.0496$	PASS (stat flat)
2.0	$0.0315$	$0.0576$	CHECK (stat)
2.4	$0.0337$	$0.0727$	CHECK (stat)
3.0	$0.0179$	$0.0839$	CHECK (stat)

Appendix

Table 47: v9 C9:

H_{0}

relative range, causal vs stationary sequences.

Figure 40: v9 C9:

H_{0}

flatness causal vs stationary. Stationary traces are visibly flatter, confirming the scope qualification.

$\gamma$	PSA slope/pos	NS slope/pos	NS slope $>$ PSA?
0.6	$-0.001050$	$+0.000231$	CONFIRMED
1.2	$-0.000722$	$-0.003178$	CHECK (NS more negative)
2.0	$-0.000120$	$-0.000905$	CHECK (NS more negative)
3.0	$-0.000183$	$+0.000066$	CONFIRMED

Appendix

Table 48: v9 C10:

H_{0}

slope vs position, PSA vs NS, stationary sequences.

Figure 41: v9 C10: FP-5 NS energy drift. Slopes of

H_{0}(t)

versus position for PSA and NS, across four

\gamma

values. Bar chart shows NS systematically exhibits larger-magnitude slopes, consistent with non-symplectic drift.

$\gamma$	PSA acc	ablated acc	std attn acc	FP-16
1.0	$56.8\%$	$19.7\%$	$1.7\%$	CONFIRMED
1.4	$68.8\%$	$29.5\%$	$1.7\%$	CONFIRMED
2.0	$75.6\%$	$40.8\%$	$1.7\%$	partial
2.4	$59.1\%$	$46.3\%$	$1.7\%$	partial
3.0	$27.2\%$	$49.9\%$	$1.7\%$	partial (reversed)

Appendix

Table 49: v9 C11:

T_{4}

ablation across

\gamma

values.

Figure 42: v9 C11:

T_{4}

ablation. PSA, ablated (

T_{4}

zeroed), and standard-attention accuracies across

\gamma

. The collapse of ablated accuracy toward standard attention at

\gamma\!\in\![1.0,1.4]

measures the size of the

T_{4}

contribution; the residual gap above the standard-attention baseline is the contribution of the channels that survive the ablation.

$d_{k}$	$8L/d_{k}$	observed $\gamma_{c}$ (50% thresh.)	$\Delta\gamma$	peak acc
$8$	$4.14$	(never reaches 50%)	$\sim 2.8$	$9.0\%$ at $\gamma=2.8$
$16$	$2.07$	$1.0$	$0.80$	$74.6\%$ at $\gamma=2.0$
$32$	$1.03$	$0.4$	$0.40$	$93.4\%$ at $\gamma=2.0$

Appendix

Table 50: v9 C12:

\gamma_{c}

scaling results across

d_{k}

.

Figure 43: v9 C12:

\gamma_{c}

and

\Delta\gamma

vs head dimension

d_{k}

. Three accuracy curves for

d_{k}\!\in\!\{8,16,32\}

show the dramatic quality improvement with larger heads.

seed	PSA-1L ( $\gamma=2.0$ )	Std-1L	Std-2L
42	$74.4\%$	$1.7\%$	$1.1\%$
123	$72.7\%$	$1.8\%$	$1.4\%$
456	$58.7\%$	$2.1\%$	$2.0\%$
789	$79.4\%$	$1.0\%$	$1.8\%$
1011	$60.2\%$	$1.6\%$	$1.7\%$
mean $\pm$ std	$\mathbf{69.1\pm 8.2\%}$	$\mathbf{1.6\pm 0.4\%}$	$\mathbf{1.6\pm 0.3\%}$

Appendix

Table 51: v9 C13: single-layer PSA vs single-layer standard attention, five seeds.

Figure 44: v9 C13: FP-24 single-layer comparison. Five-seed bars for PSA

N=1

, standard attention

N=1

, standard attention

N=2

. The PSA bar stands at

69\%

with five-seed error bars; the two standard-attention bars sit on top of the chance line.

band	$\theta$	theory	observed	obs/th	joint align
0	$1.000000$	$0.958851$	$0.407971$	$0.43$	$0.9806$
1	$0.316228$	$0.314912$	$0.041205$	$0.13$	$0.9893$
2	$0.100000$	$0.099958$	$0.004453$	$0.04$	$1.0174$
3	$0.031623$	$0.031621$	$0.000411$	$0.01$	$1.0118$
4	$\mathbf{0.010000}$	$\mathbf{0.010000}$	$\mathbf{0.412654}$	$\mathbf{41.27}$	$\mathbf{0.9541}$
5	$0.003162$	$0.003162$	$0.042761$	$13.52$	$1.0593$

Appendix

Table 52: v9 C14: per-band Coriolis and

W_{Q}/W_{K}

joint alignment,

\gamma=3.0

.

Figure 45: v9 C14: band-4 Coriolis anomaly. Per-band observed/theory ratio (top) and joint

W_{Q}/W_{K}

alignment (bottom) across the three analyzed

\gamma

values. Band-4’s anomaly persists; its alignment varies with

\gamma

but remains finite.

$\gamma$	$r_{\mathrm{PSA}}$	$r_{\mathrm{NS}}$	$\Delta r$	FP-29
0.6	$0.9528$	$0.0305$	$+0.9223$	CONFIRMED
1.2	$0.9225$	$0.8715$	$+0.0510$	CONFIRMED
1.8	$0.9206$	$0.9041$	$+0.0166$	CONFIRMED
2.0	$0.9206$	$0.9113$	$+0.0093$	CONFIRMED
2.4	$0.9282$	$0.9175$	$+0.0107$	CONFIRMED
3.0	$0.9166$	$0.9118$	$+0.0047$	CONFIRMED

Appendix

Table 53: v9 C15: PSA vs NS post-softmax Bode

r

.

Figure 46: v9 C15: FP-29 symmetric advantage. PSA (symmetric) versus NS (asymmetric) post-softmax Bode

r

across the six tested

\gamma

. PSA is above NS at every

\gamma

; the shaded region shows the

r

advantage.

Figure 47: v9 summary dashboard. (A) Cumulative PASS/CHECK/open FP counts across v6, v7, v8, v9 — the green bar grows from

12

in v6 to

21

in v9; the open-FP column shrinks from

30

to

13

. (B) T3

\to

T4 crossover at

\gamma\approx 0.97

. (C) FP-24 bar chart. (D) FP-5 drift slopes. (E) FP-29

r

-advantage. (F) FP-16

T_{4}

-ablation collapse.

Parameter	Value
Vocabulary size	$V=128$
$d_{\mathrm{model}}$ , $H$ , $d_{\mathrm{head}}$	$64$ , $4$ , $16$
Chain length $L$	$8$ KV pairs (sequence length $17$ )
Training	$2000$ samples, batch $128$ , $8$ epochs
Optimizer	AdamW, lr $10^{-3}$ , wd $10^{-2}$
RoPE family	Bandpass, bandwidth $b=0.2$

Appendix

Table 54: F1 configuration. A single-seed exploratory scan over six center frequencies and eight momentum couplings.

$\theta\backslash\gamma$	$0.00$	$0.25$	$0.50$	$0.75$	$1.00$	$1.50$	$2.00$	$3.00$
$\theta=2.00$	$0.75$	$0.80$	$0.80$	$\mathbf{0.82}$	$\mathbf{0.82}$	$0.75$	$0.72$	$0.56$
$\theta=1.00$	$0.71$	$0.75$	$\mathbf{0.80}$	$\mathbf{0.80}$	$0.79$	$0.76$	$0.68$	$0.54$
$\theta=0.50$	$0.69$	$0.79$	$\mathbf{0.79}$	$0.78$	$\mathbf{0.79}$	$0.76$	$0.70$	$0.55$
$\theta=0.25$	$0.70$	$0.76$	$\mathbf{0.80}$	$\mathbf{0.80}$	$0.79$	$0.75$	$0.68$	$0.55$
$\theta=0.10$	$0.67$	$0.80$	$0.87$	$\mathbf{0.89}$	$0.88$	$0.77$	$0.73$	$0.57$
$\theta=0.05$	$0.68$	$0.82$	$0.91$	$\mathbf{0.95}$	$0.93$	$0.85$	$0.74$	$0.57$

Appendix

Table 55: F1: full 48-point accuracy surface from the exploratory scan. Row =

\theta

(descending), Column =

\gamma

. Peak entries per row are in bold . Global peak:

(\theta,\gamma)=(0.05,0.75)

at

0.95

.

Figure 48: F1: bandpass RoPE exploratory phase diagram. Rows are center frequencies

\theta\!\in\!\{0.05,0.10,0.25,0.50,1.00,2.00\}

(top to bottom in this ordering); columns are momentum couplings

\gamma\!\in\!\{0.0,0.25,0.5,0.75,1.0,1.5,2.0,3.0\}

. The brightest cell is

(0.05,0.75)=0.95

. The bottom-right corner (

\gamma=3.0

) collapses uniformly to

\sim 0.55

regardless of

\theta

, indicating that very large momentum coupling dominates the signal globally.

Parameter	Value
Architecture	nano single-block (identical to F1)
$V$ , $d_{\mathrm{model}}$ , $H$ , $d_{\mathrm{head}}$	$128$ , $64$ , $4$ , $16$
Chain length $L$	$8$ KV pairs
Training	$2000$ samples, batch $128$ , $8$ epochs, AdamW lr $10^{-3}$
RoPE family	Bandpass, $b=0.2$
$\theta$ grid (15 log-spaced)	from $0.03$ to $3.0$ rad

Appendix

Table 56: F2 configuration. Architecture identical to F1; sweep grid substantially denser, three seeds per cell.

$\gamma$	mean acc	std	$\gamma$	mean acc	std
$0.00$	$0.356$	$0.002$	$0.90$	$0.593$	$0.041$
$0.10$	$0.393$	$0.005$	$1.00$	$0.581$	$0.046$
$0.20$	$0.428$	$0.004$	$1.20$	$0.545$	$0.053$
$0.30$	$0.472$	$0.008$	$1.65$	$0.484$	$0.050$
$0.40$	$0.511$	$0.015$	$2.10$	$0.436$	$0.033$
$0.50$	$0.546$	$0.021$	$2.55$	$0.407$	$0.034$

Appendix

Table 57: F2

\gamma

-sweep at the peak frequency

\theta=0.0300

, three-seed mean and standard deviation. Inverted-U with peak at

\gamma=0.80

.

$\theta$	peak accuracy	at $\gamma$
$0.030$	$0.599$	$0.80$
$0.080$	$0.57$	$0.70$
$0.300$	$0.43$	$0.4$ – $1.0$
$3.000$	$0.49$ (wrap-around)	$0.80$

Appendix

Table 58: F2 cross-section by frequency: peak accuracy at four representative

\theta

values. The peak is monotone decreasing with

\theta

for

\theta\!\leq\!0.3

.

Figure 49: F2: low-pass induction heatmap with iso-accuracy contours.

\theta

is on a log scale,

\gamma

is linear. The

0.5

-contour (yellow) encloses a region around

(\theta,\gamma)\!\in\!([0.03,0.2],[0.4,1.5])

— the high-fidelity induction zone. The upper band at

\theta\!\in\![1,3]

is the noise-corrupted regime.

Figure 50: F2: accuracy-versus-

\gamma

slices at four

\theta

. Peak height monotone decreasing with

\theta

for

\theta\!\leq\!0.3

;

\theta=3.0

shows the wrap-around recovery.

Figure 51: F2: accuracy-versus-

\theta

slices at five

\gamma

. For every

\gamma>0

, accuracy falls sharply around

\theta\!\approx\!0.2

— the empirical cut-off frequency

\theta_{c}

of the low-pass filter.

Parameter	Value
Architecture	nano single-block (identical to F1, F2)
$d_{\mathrm{model}}$ , $H$ , $d_{\mathrm{head}}$	$64$ , $4$ , $16$
Vocabulary, chain length	$V=128$ , $L=8$
RoPE family	Monochromatic — single scalar $\theta$ for all $d/2$ dimensions
Wavelength grid (6)	$\{2,4,8,16,32,64\}$ positions
Corresponding $\theta$ values	$\{3.14,1.57,0.78,0.39,0.19,0.09\}$ rad

Appendix

Table 59: F3 configuration: monochromatic RoPE control.

wavelength	$\theta$ (rad)	$\gamma{=}0.1$	$\gamma{=}0.5$	$\gamma{=}1.0$	$\gamma{=}2.0$	$\gamma{=}4.0$
$2$	$3.14$	$0.46$	$\mathbf{0.62}$	$0.60$	$0.42$	$0.33$
$4$	$1.57$	$0.46$	$0.49$	$\mathbf{0.51}$	$0.40$	$0.27$
$8$	$0.78$	$0.48$	$\mathbf{0.53}$	$0.50$	$0.41$	$0.30$
$16$	$0.39$	$\mathbf{0.51}$	$\mathbf{0.51}$	$\mathbf{0.51}$	$0.42$	$0.29$
$32$	$0.19$	$0.48$	$\mathbf{0.55}$	$0.52$	$0.40$	$0.31$
$64$	$0.09$	$0.46$	$0.59$	$\mathbf{0.60}$	$0.41$	$0.28$

Appendix

Table 60: F3: monochromatic RoPE accuracy by wavelength (positions),

\theta

(radians), and

\gamma

. Peak per row in bold .

Figure 52: F3: monochromatic phase diagram. Two peaks visible at

(\lambda=2,\gamma=0.5)=0.62

and

(\lambda=64,\gamma=1.0)=0.60

. The

\gamma=4.0

column collapses to

0.27

–

0.33

, consistent with F1 and F2’s universal high-

\gamma

collapse.

Parameter	Value
Architecture	nano single-block
$V$ , $d_{\mathrm{model}}$ , $H$ , $d_{\mathrm{head}}$	$128$ , $64$ , $4$ , $16$
Chain length $L$	$8$ KV pairs (sequence length $17$ )
Training	$2000$ samples, batch $128$ , $10$ epochs, AdamW lr $10^{-3}$ , wd $10^{-2}$
RoPE family	Bandpass, $b=0.2$
$\theta$ grid (20 log-spaced)	from $0.02$ to $\pi\approx 3.14$ rad

Appendix

Table 61: G configuration.

$\theta$ (rad)	$\theta$ (deg)	$\gamma^{*}$	peak acc	baseline	gain	noise
$0.0200$	$1.1$	$0.80$	$0.778$	$0.447$	$+0.331$	$0.0200$
$0.0261$	$1.5$	$0.80$	$0.776$	$0.446$	$+0.330$	$0.0261$
$0.0341$	$2.0$	$0.80$	$0.771$	$0.445$	$+0.326$	$0.0341$
$0.0444$	$2.5$	$0.80$	$0.769$	$0.447$	$+0.321$	$0.0444$
$0.0580$	$3.3$	$0.80$	$0.761$	$0.451$	$+0.310$	$0.0580$
$0.0757$	$4.3$	$0.80$	$0.747$	$0.455$	$+0.292$	$0.0757$

Appendix

Table 62: G: optimal

\gamma^{*}

, peak accuracy, baseline accuracy, momentum gain, and noise level for all 20 tested

\theta

values. This is the canonical empirical table of the Low-Pass Induction Filter.

Figure 53: G main figure. (A) Full heatmap (log

\theta

, linear

\gamma

) with the clean induction zone at low

\theta

. (B) Inverted-U slice plots at

5

representative

\theta

. (C) Scatter of momentum gain versus rotational noise with the fitted line and the

r=-0.679

,

p=9.9\!\times\!10^{-4}

annotation. (D) Empirical

\gamma^{*}(\theta)

(scatter, colored by peak accuracy) against the

\gamma^{*}\propto 1/\sin(\theta/2)

theory curve. Theory overestimates

\gamma^{*}

at low

\theta

; empirical

\gamma^{*}

sits uniformly at

\sim 0.80

for

\theta<0.2

.

Figure 54: G theory validation panel. (A) Hamiltonian decomposition: the

2\sin(\theta/2)

noise curve (red, shaded) versus the unit signal curve (green), with the low-pass-regime boundary annotated. (B) SNR curves for five

\gamma

: at high

\gamma

, SNR falls below unity at modest

\theta

. (C) Experimental noise–gain scatter with fit

y=-0.110x+0.245

. (D) Textual summary.

Figure 55: G supplementary analysis. (A) Phase diagram with iso-accuracy contours. (B) Box plots of accuracy by frequency band (Low/Mid/High); the low band is strongly right-shifted. (C) Variance analysis: std versus mean accuracy colored by

\log_{10}\theta

. (D) Peak accuracy versus

\theta

(sharp fall-off around

\theta=0.2

). (E) Momentum benefit per frequency band (bars). (F) Top-

10

frequencies summary — all at

\gamma^{*}=0.8

with peak accuracies from

0.778

down to

0.612

.

Parameter	Value
Architecture	3-layer transformer with pre-LN, GELU MLP
$d_{\mathrm{model}}$ , $H$ , $d_{\mathrm{head}}$ , $d_{\mathrm{ff}}$	$128$ , $4$ , $32$ , $256$
Vocabulary	$V=200$
Chain length $L$	$\in[8,12]$ KV pairs (random per example)
Max sequence length	$100$ tokens
Dataset	$5000$ train / $1000$ test (fresh samples)

Appendix

Table 63: H configuration. Three-layer transformer, larger

d_{\mathrm{model}}

, longer chain length,

60

training epochs.

RoPE type	$\theta_{\min}$	$\theta_{\max}$	range ratio
Single-frequency	$0.100000$	$0.100000$	$1.0\!\times$
Bandpass	$0.080000$	$0.120000$	$1.5\!\times$
Multi-frequency	$0.000178$	$1.000000$	$5{,}623.4\!\times$

Appendix

Table 64: H frequency spectrum properties at

d_{k}=32

(so

16

frequency slots).

	Single-frequency			Bandpass			Multi-frequency
$\gamma$	acc (%)	SEM	$\Delta$	acc (%)	SEM	$\Delta$	acc (%)	SEM	$\Delta$
$0.0$	$11.1$	$0.3$	base	$10.8$	$0.3$	base	$11.6$	$0.2$	base
$0.3$	$44.5$	$0.6$	$+33.4$	$43.8$	$1.0$	$+33.0$	$66.6$	$0.8$	$+55.1$
$0.5$	$72.1$	$0.4$	$+61.0$	$72.2$	$0.8$	$+61.4$	$89.7$	$0.6$	$+78.2$
$0.7$	$82.7$	$0.5$	$+71.6$	$82.3$	$0.2$	$+71.5$	$94.7$	$0.4$	$+83.2$
$1.0$	$\mathbf{86.8}$	$\mathbf{0.2}$	$\mathbf{+75.7}$	$\mathbf{87.5}$	$\mathbf{0.3}$	$\mathbf{+76.7}$	$\mathbf{96.2}$	$\mathbf{0.2}$	$\mathbf{+84.7}$

Appendix

Table 65: H test accuracy and momentum gain (over

\gamma=0

baseline) for each ( RoPE type,

\gamma

) cell, three-seed mean

\pm

SEM. Per-family baseline and peak highlighted.

Figure 56: H: RoPE frequency design space. (A/B/C) Per-dimension frequency spectra for the three families. Single-frequency (A) shows

16

identical bars at

\theta=0.1

(no escape routes); Bandpass (B) shows a linear ramp from

0.08

to

0.12

(limited escape); Multi-frequency (C, log scale) shows the exponential spread from

\sim 10^{-4}

to

10^{0}

(many escape routes). (D) Accuracy versus

\gamma

for all three families, all inverted-U, all peaking at

\gamma=1.0

, with multi-frequency cleanly above the others across the whole

\gamma

range. (E) Theoretical-framework text panel.

Figure 57: H curve shapes. Per-family curves with SEM shading and peak annotations. All three are inverted-U with peak at

\gamma=1.0

, but the multi-frequency curve is shifted upward by

\sim 9\%

at every

\gamma>0

— the Escape Routes advantage manifests as a level shift, not a shape change.

Task	$\gamma=0$ baseline	$\gamma=0.5$ momentum	$\Delta$
Associative Recall ( $\nabla$ )	$14.0\%$	$50.8\%$	$+36.8$ pp
Variable Tracking ( $\nabla$ )	$59.0\%$	$54.4\%$	$-4.6$ pp
Global Counting ( $\int$ )	$60.8\%$	$28.6\%$	$-32.2$ pp

Appendix

Table 66: Mechanistic-visualization paired training,

\theta=0.03

,

\gamma\!\in\!\{0,0.5\}

, single seed.

$\theta\backslash\gamma$	$0.0$	$0.2$	$0.4$	$0.6$	$0.8$	$1.0$
$0.01$	$13.2$	$24.0$	$56.8$	$72.6$	$\mathbf{76.0}$	$72.4$
$0.03$	$14.0$	$21.4$	$42.6$	$56.4$	$\mathbf{58.0}$	$54.0$
$0.05$	$14.8$	$19.8$	$33.8$	$\mathbf{42.4}$	$40.8$	$39.6$
$0.10$	$14.0$	$16.2$	$22.0$	$\mathbf{26.6}$	$24.0$	$19.8$
$0.20$	$12.6$	$13.0$	$13.2$	$\mathbf{13.4}$	$12.4$	$13.0$
$0.30$	$13.4$	$12.8$	$12.6$	$12.4$	$12.4$	$9.4$

Appendix

Table 67: Heatmap: Associative Recall accuracy (%) by

(\theta,\gamma)

, single seed,

36

training runs. Row maxima in bold .

Task	$\theta$	$0.0$	$0.2$	$0.4$	$0.6$	$0.8$	$1.0$	$1.2$
Associative Recall	$0.03$	$14.0$	$21.4$	$42.6$	$56.4$	$\mathbf{58.0}$	$54.0$	$45.0$
Associative Recall	$0.30$	$\mathbf{13.4}$	$12.8$	$12.6$	$12.4$	$12.4$	$9.4$	$9.2$
Variable Tracking	$0.03$	$\mathbf{59.0}$	$57.4$	$54.6$	$50.8$	$49.6$	$48.4$	$49.6$
Variable Tracking	$0.30$	$\mathbf{56.4}$	$53.2$	$52.8$	$51.6$	$47.8$	$48.6$	$47.6$
Global Counting	$0.03$	$\mathbf{60.8}$	$38.8$	$31.2$	$26.6$	$23.6$	$23.0$	$23.6$
Global Counting	$0.30$	$\mathbf{25.6}$	$24.4$	$23.6$	$22.4$	$21.6$	$21.2$	$20.8$

Appendix

Table 68: Accuracy versus

\gamma

at two frequencies for all three tasks. Row peaks in bold .

Figure 58: Mechanistic analysis. Panels (A)–(C): theory. (A) RoPE position encoding

\cos(\theta t)

at four frequencies showing slow vs. fast rotation. (B) Discrete momentum

p_{t}=q_{t}-q_{t-1}

versus the analytic derivative

-\theta\sin(\theta t)

; numerical and theoretical curves coincide modulo the discrete-step correction. (C) Transfer-function magnitude

|H(\omega)|=|1+\gamma(e^{-j\omega}-1)|

for five

\gamma

. Panels (D)–(F): attention maps,

\gamma=0

versus

\gamma=0.5

. (D) Associative Recall: sharper attention to the target position at

\gamma=0.5

. (E) Variable Tracking: stronger diagonal structure at

\gamma=0.5

. (F) Global Counting (negative control): attention becomes less uniform and therefore worse at counting. Panel (H): accuracy-versus-

\gamma

curves. Panel (I): the

36

-cell heatmap with white stars marking per-row optima.

Figure 59:

\gamma

-progression on Associative Recall. Evolution of the attention map on a single held-out example as

\gamma

is swept from

0

to

1.0

. Each panel header reports the accuracy of an independently-trained model at that

\gamma

under

\theta=0.03

. Accuracy rises from

14.0\%

at

\gamma=0

to

57.2\%

at

\gamma=0.7

before descending at

\gamma=1.0

. The pattern visibly sharpens through

\gamma=0.7

and begins to fragment at

\gamma=1.0

.

Variant	Hops	Negation rate	Distractors	Baseline target
Original (pre-easing)	$6$ – $16$	$30$ – $50\%$	$2$ – $6$	$\sim 52\%$
v2 easing	$4$ – $9$	$30$ – $40\%$	$1$ – $3$	$\sim 59\%$
v3 easing	$2$ – $5$	$20$ – $30\%$	$0$ – $1$	$100\%$ ceiling
v4 (this section)	$3$ – $6$	$25$ – $30\%$	$1$ – $2$	target $70$ – $85\%$

Appendix

Table 69: Eased multi-hop configurations relative to the original (harder) and to prior easing attempts.

Difficulty	$\theta$	$\gamma=0$	$\gamma=0.3$	$\gamma=0.5$	$\gamma=0.7$	$\gamma=1.0$
Easy	$0.03$	$65.0$	$63.6$	$63.8$	$62.3$	$62.1$
Easy	$0.30$	$63.0$	$62.6$	$62.6$	$62.0$	$62.0$
Medium	$0.03$	$59.1$	$58.3$	$59.0$	$59.8$	$58.6$
Medium	$0.30$	$56.1$	$54.4$	$54.4$	$55.0$	$53.9$
Hard	$0.03$	$52.6$	$52.1$	$52.2$	$52.3$	$52.7$
Hard	$0.30$	$53.1$	$52.6$	$52.4$	$51.9$	$52.7$

Appendix

Table 70: All 90 runs of the multi-hop eased sweep. Each cell is mean accuracy (%) over three seeds.

Figure 60: Eased multi-hop results. Three panels, one per difficulty level, show accuracy against

\gamma

with seed-variance error bars and the two theta values superimposed. The “Random” line at

50\%

marks binary-task chance level. No configuration exceeds its

\gamma=0

baseline by more than

+0.7

pp.

Difficulty	$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$1.0$
Easy	$0.03$	$94.4$	$96.3$	$97.9$	$96.5$	$\mathbf{98.0}$
Easy	$0.30$	$93.9$	$94.4$	$\mathbf{95.0}$	$94.6$	$94.7$
Medium	$0.03$	$82.4$	$88.7$	$88.8$	$88.8$	$\mathbf{89.8}$
Medium	$0.30$	$81.1$	$81.2$	$\mathbf{81.5}$	$79.8$	$80.7$
Hard	$0.03$	$73.7$	$75.2$	$\mathbf{80.9}$	$74.0$	$80.5$
Hard	$0.30$	$65.2$	$70.1$	$\mathbf{71.4}$	$68.8$	$67.7$

Appendix

Table 71: Variable Tracking sweep, 3-seed means in %. Row maxima in bold .

Difficulty	$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$1.0$
Easy	$0.03$	$52.4$	$52.2$	$52.3$	$52.0$	$52.5$
Easy	$0.30$	$52.7$	$52.1$	$52.1$	$52.2$	$52.9$
Medium	$0.03$	$53.1$	$52.8$	$53.0$	$53.0$	$53.2$
Medium	$0.30$	$52.9$	$54.3$	$52.2$	$52.6$	$53.0$
Hard	$0.03$	$53.0$	$52.7$	$53.1$	$52.5$	—
Hard	$0.30$	—	—	—	—	—

Appendix

Table 72: Multi-Hop with Negations sweep, 3-seed means in %. All gains within seed noise.

Difficulty	$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$1.0$
Easy	$0.03$	$100.0$	$100.0$	$100.0$	$\mathbf{100.0}$	$99.7$
Easy	$0.30$	$90.6$	$78.4$	$70.4$	$54.7$	$62.3$
Medium	$0.03$	$85.4$	$86.1$	$85.2$	$\mathbf{97.4}$	$80.1$
Medium	$0.30$	$70.6$	$69.5$	$\mathbf{81.7}$	$58.0$	$22.0$
Hard	$0.03$	$12.8$	$\mathbf{73.4}$	$70.2$	$28.0$	$38.7$
Hard	$0.30$	$56.8$	$49.4$	$30.6$	$46.6$	$25.0$

Appendix

Table 73: Arithmetic CoT sweep, 3-seed means in %. Predict the final digit of the sum of two

n

-digit numbers. Row maxima in bold .

Difficulty	$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$1.0$
Easy	$0.03$	$31.7$	$31.6$	$31.6$	$31.8$	$31.8$
Easy	$0.30$	$31.7$	$31.4$	$31.6$	$32.0$	$31.6$
Medium	$0.03$	$25.3$	$24.8$	$24.8$	$25.1$	$25.2$
Medium	$0.30$	$25.4$	$24.9$	$24.7$	$25.4$	$25.4$
Hard	$0.03$	$20.4$	$20.8$	$20.7$	$20.5$	$20.6$
Hard	$0.30$	$20.7$	$21.0$	$20.7$	$20.8$	$20.5$

Appendix

Table 74: Global Counting sweep, 3-seed means in %. The

\int

-task’s expected behavior: no gain.

Task	Mechanism	Baseline	Peak	Relative gain
Variable Tracking	$\nabla$	$82.4\%$	$98.0\%$	$+19\%$
Multi-Hop w/ Negations	$\nabla$	$53.1\%$	$53.2\%$	$+0\%$
Arithmetic CoT	$\nabla$	$85.4\%$	$100.0\%$	$+17\%$
Global Counting	$\int$	$25.3\%$	$31.8\%$	$+1\%$

Appendix

Table 75: CoT cross-task summary at medium difficulty,

\theta=0.03

.

Task	$\gamma=0$	$0.5$	$0.7$	$0.9$	$1.2$
Arithmetic Carry	$0.888$	$0.917$	$0.957$	$0.966$	$\mathbf{0.970}$
List Reversal	$1.000$	$1.000$	$1.000$	$1.000$	$\mathbf{1.000}$
Parity	$0.486$	$\mathbf{0.515}$	$0.500$	$0.497$	$0.497$
Sorting (Min)	$\mathbf{0.997}$	$0.995$	$0.993$	$0.991$	$0.987$
Natural Induction	$0.127$	$0.578$	$0.796$	$0.870$	$\mathbf{0.906}$

Appendix

Table 76: Real-world reasoning main sweep at

\theta=0.03

. Accuracy values are fractions; each cell is the mean over five seeds. Row maxima in bold .

Figure 61: Real-world reasoning overview. (A) Inverted-U accuracy curves for all five tasks at

\theta=0.03

. Natural Induction (orange) is the dramatic mover. (B) Peak momentum gains bar chart; Natural Induction’s

+0.74

dwarfs everything else. (C) Heatmap of accuracy at

\theta=0.03

across all

(\mathrm{task},\gamma)

combinations. (D) Low-pass filter effect: gain against

\theta

at optimal

\gamma\!\in\![0.5,1.0]

, color per task. Natural Induction’s gain decays as

\theta

increases, exactly as predicted. (E) Histogram of per-task optimal

\gamma

across the five tasks — concentrating around

0.5

–

0.7

with Natural Induction extending to

1.2

. (F) Summary text panel.

$\theta$	$\gamma=0.0$	$0.3$	$0.5$	$0.7$	$0.9$	$1.2$
$0.03$	$0.127$	$0.243$	$0.578$	$0.796$	$0.870$	$\mathbf{0.906}$
$0.10$	$0.179$	$0.249$	$0.533$	$0.769$	$0.858$	$\mathbf{0.923}$
$0.30$	$0.281$	$0.348$	$0.498$	$0.651$	$0.760$	$\mathbf{0.851}$
$1.00$	$0.178$	$0.163$	$0.160$	$0.152$	$0.161$	$\mathbf{0.158}$

Appendix

Table 77: Natural Induction: full

(\theta,\gamma)

grid, five-seed means. Row maxima in bold .

$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$0.9$	$1.2$
$0.03$	$0.888$	$0.896$	$0.917$	$0.957$	$0.966$	$\mathbf{0.970}$
$0.10$	$0.889$	$0.910$	$0.921$	$0.938$	$\mathbf{0.946}$	$0.930$
$0.30$	$0.898$	$0.927$	$0.934$	$0.944$	$\mathbf{0.946}$	$0.932$
$1.00$	$\mathbf{0.974}$	$0.973$	$0.963$	$0.953$	$0.947$	$0.930$

Appendix

Table 78: Arithmetic Carry: full

(\theta,\gamma)

grid, 5-seed means. Row maxima in bold .

Figure 62: Per-task inverted-U curves. Each panel sweeps

\gamma

at four

\theta

values (color scale dark purple at

\theta=0.03

to yellow at

\theta=1.00

). The Natural Induction panel is the most striking: at

\theta=0.03

the curve rises steeply from

12.7\%

at

\gamma=0

to

90.6\%

at

\gamma=1.2

, while at

\theta=1.00

the curve is essentially flat around

16\%

.

Task	Baseline	Peak	Peak $\gamma$	Peak $\theta$	Peak gain
Majority	$1.000$	$1.000$	$0.0$	$0.03$	$+0.000$
Induction	$0.275$	$0.846$	$1.2$	$0.10$	$+0.591$
Trajectory	$0.691$	$0.744$	$1.2$	$1.00$	$+0.040$
Dyck	$0.898$	$0.946$	$0.7$	$1.00$	$+0.016$

Appendix

Table 79: Multi-task validation main summary. Note: Majority baseline is

1.000

— the single-layer transformer with this small capacity solves the bag-of-tokens task cleanly.

$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$0.9$	$1.2$	$1.8$
$0.03$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$
$0.10$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$
$0.30$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$
$1.00$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$

Appendix

Table 80: Majority task: every one of the

28

cells (5 seeds each,

=140

runs) returns

1.000

accuracy. The negative control behaving exactly as theory predicts.

$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$0.9$	$1.2$	$1.8$
$0.03$	$0.152$	$0.243$	$0.397$	$0.577$	$0.701$	$\mathbf{0.787}$	$0.717$
$0.10$	$0.255$	$0.328$	$0.479$	$0.653$	$0.770$	$\mathbf{0.846}$	$0.801$
$0.30$	$0.354$	$0.438$	$0.531$	$0.611$	$\mathbf{0.648}$	$0.640$	$0.522$
$1.00$	$0.340$	$0.368$	$0.409$	$0.442$	$\mathbf{0.454}$	$0.453$	$0.444$

Appendix

Table 81: Induction task across the full

28

-cell grid, 5-seed means. Row maxima in bold .

$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$0.9$	$1.2$	$1.8$
$0.03$	$0.684$	$0.697$	$0.701$	$0.703$	$0.705$	$\mathbf{0.708}$	$0.697$
$0.10$	$0.688$	$0.697$	$0.704$	$0.708$	$0.709$	$\mathbf{0.709}$	$0.703$
$0.30$	$0.689$	$0.701$	$0.712$	$0.717$	$0.718$	$\mathbf{0.718}$	$0.704$
$1.00$	$0.704$	$0.725$	$0.734$	$0.740$	$0.743$	$\mathbf{0.744}$	$0.735$

Appendix

Table 82: Trajectory extrapolation across the full grid. The peak

\theta

is

1.00

, not in the low-pass region.

$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$0.9$	$1.2$	$1.8$
$0.03$	$0.871$	$\mathbf{0.912}$	$0.905$	$0.900$	$0.896$	$0.885$	$0.826$
$0.10$	$0.886$	$\mathbf{0.917}$	$0.910$	$0.898$	$0.904$	$0.890$	$0.839$
$0.30$	$0.905$	$0.913$	$\mathbf{0.920}$	$0.915$	$0.904$	$0.915$	$0.843$
$1.00$	$0.930$	$0.942$	$\mathbf{0.946}$	$0.946$	$0.932$	$0.917$	$0.877$

Appendix

Table 83: Dyck language across the grid. Gentler U with peak near

\gamma\!\in\![0.3,0.5]

, clear over-rotation collapse at

\gamma=1.8

.

$t$ -statistic	$-2.910$
$p$ -value	$4.51\!\times\!10^{-3}$
Majority mean gain (at $\gamma>0$ )	$+0.000$
Non-majority mean gain (at $\gamma>0$ )	$+0.107$

Appendix

Table 84: Hypothesis test: is the Majority gain distribution significantly different from the pooled other-tasks distribution?

Figure 63: Multi-task validation main figure. (A) U-curves for all four tasks at

\theta=0.03

: induction (green) rises sharply, trajectory (blue) and Dyck (red) rise modestly, majority (grey) is flat at

1.0

. (B) Peak-gain bar chart with the majority bar at exactly zero (negative-control annotation). (C) Heatmap at

\theta=0.03

. (D) Low-pass effect: gain-versus-

\theta

at each task’s optimal

\gamma

; induction’s curve decays strongly with

\theta

.

Figure 64: Per-task detailed analysis. Four subplots with U-curves at each of the four

\theta

values. Majority’s subplot is rigid at

1.0

; induction’s is the cleanest U-family; trajectory’s U is shallow but consistent; Dyck’s is narrow with over-rotation damage at high

\gamma

.

Figure 65: Task classification by momentum benefit. Scatter plots (peak gain, baseline) for each task; the green shaded region is the “momentum benefit zone” (peak gain

>0.05

); red is the “no-benefit zone.” Majority lands squarely in red; induction, trajectory and Dyck in green.

Chain $\backslash$ Vocab	$64$	$128$	$256$	$512$
$4$	$0.577$	$0.624$	$0.667$	$0.707$
$8$	$0.675$	$0.729$	$0.779$	$0.826$
$12$	$0.729$	$0.787$	$0.842$	$0.893$
$16$	$0.766$	$0.827$	$0.885$	$0.938$
$20$	$0.794$	$0.858$	$0.917$	$0.973$
$24$	$0.816$	$0.882$	$0.943$	$1.000$

Appendix

Table 85: Task difficulty

d(\ell,V)

for each

(\mathrm{chain\,length},\mathrm{vocab\,size})

combination. Higher is harder.

$\ell$	$V$	diff	baseline	peak	gain	best $\theta$	best $\gamma$
$4$	$64$	$0.577$	$0.651$	$1.000$	$+0.349$	$0.03$	$0.7$
$4$	$128$	$0.624$	$0.703$	$0.994$	$+0.291$	$0.03$	$0.9$
$4$	$256$	$0.667$	$0.653$	$0.911$	$+0.258$	$0.03$	$0.9$
$4$	$512$	$0.707$	$0.590$	$0.748$	$+0.158$	$0.03$	$0.7$
$8$	$64$	$0.675$	$0.482$	$0.999$	$+0.517$	$0.03$	$0.9$
$8$	$128$	$0.729$	$0.527$	$0.875$	$+0.348$	$0.03$	$0.9$

Appendix

Table 86: Per-

(\ell,V)

best case: best

(\theta,\gamma)

cell in each of

24

difficulty cells. Observe: best

\theta=0.03

for every low-difficulty cell but shifts to higher values at high difficulty; best

\gamma

drifts from

0.7

toward

1.2

–

1.8

as difficulty grows.

Bin	Difficulty range	Mean baseline acc	Mean momentum gain
Hard	$0.5$ – $0.7$	$0.652$	$+0.243\pm 0.119$
Very Hard	$0.7$ – $1.0$	$0.450$	$+0.167\pm 0.081$

Appendix

Table 87: Difficulty-binned momentum gain. Mean over all

(\theta,\gamma)

combinations within each chain-length/vocab bin.

Pair	Pearson $r$	$p$
Rotational noise $2\sin(\theta/2)$ vs gain	$-0.3719$	$2.88\!\times\!10^{-5}$
Task difficulty $d(\ell,V)$ vs gain	$-0.2470$	$6.54\!\times\!10^{-3}$
Baseline accuracy vs gain	$-0.0164$	$0.859$

Appendix

Table 88: Correlations between task-level quantities and momentum gain. Rows sorted by

|r|

.

Prediction	Result	Verdict
Low-pass filter (noise–gain $\rho<0$ )	$r=-0.372$ , $p=2.9\!\times\!10^{-5}$	PASS
Difficulty sweet spot (Hard $>$ Very Hard)	$F=12.04$ , $p=7.4\!\times\!10^{-4}$	PASS
Universal optimal $\gamma$ (clusters in $[0.7,1.2]$ )	median $\sim 0.9$	PASS

Appendix

Table 89: Three pre-registered predictions, each with its independent statistical test.

Figure 66: Multi-difficulty main validation figure. (A) Sweet-spot scatter: momentum gain versus task difficulty, colored by

\theta

. Quadratic fit overlay (dashed red) is the inverted-U shape the sweet-spot hypothesis predicts. (B) Average-over-

\theta

gain heatmap on the

(\mathrm{chain},\mathrm{vocab})

grid. (C) Difficulty-regime U-curves. (D) Low-pass filter scatter: gain versus rotational noise, colored by difficulty; the

r=-0.372

trend line is shown.

Figure 67: Phase diagrams by chain length (at

\theta=0.03

). Six subplots for chain lengths

\{4,8,12,16,20,24\}

, each showing an

(\mathrm{vocab},\gamma)

accuracy heatmap with the red star marking the optimum. As chain length grows, the optimum shifts diagonally — lower vocab, lower

\gamma

— and peak accuracy falls (from

\approx 0.98

at chain

4

to

\approx 0.62

at chain

24

).

Figure 68: Theoretical-validation figure. (A) Predicted benefit versus observed gain (normalized), color by

\theta

. (B) Optimal

\gamma

versus difficulty, colored by peak accuracy; horizontal dashed red line at

\gamma=0.8

marks the theory-predicted optimum. Optimal

\gamma

tightly clustered in

[0.7,1.2]

across difficulties. (C) Low-pass effect stratified by difficulty bin. (D) Summary panel.

Section	Task	Class	Peak gain	Low- $\theta$ best?	PSA-consistent?
A18	Associative Recall	$\nabla$	$+0.368$	yes	yes
A18	Variable Tracking	$\nabla$	$-0.046$	—	neutral (narrow sweep)
A18	Global Counting	$\int$	$-0.322$	—	yes (harm as predicted)
A19	Multi-Hop Eased	$\nabla$	$+0.007$	—	null (task out of band)
A20	Variable Tracking	$\nabla$	$+0.156$	yes	yes
A20	Multi-Hop w/ Negations	$\nabla$	$+0.001$	—	null (task out of band)

Appendix

Table 90: Task-by-task verdict matrix across the reasoning arc. Each row is a distinct task; columns are class (

\nabla

vs

\int

), peak gain in the best-case cell, whether low-

\theta

produced the best gain (low-pass-filter signature), and whether the outcome matches the PSA prediction.

Metric	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.7$ )	$\Delta$
$L_{\mathrm{new}}$	$8.0861$	$8.0713$	$-0.0147$
$L_{\mathrm{second}}$	$3.4099$	$3.4966$	$+0.0867$
$L_{\mathrm{rep}}$	$2.1103$	$2.2220$	$+0.1117$
$\Delta_{1\!-\!2}$	$4.6761$	$4.5747$	$-0.1014$
Train loss	$3.9424$	$4.0158$	$+0.0734$

Appendix

Table 91: Burstiness final metrics at step

10{,}000

.

\gamma=0.7

,

\beta=0

.

Hyp.	Statement	Measured	Verdict
H1	$\|\Delta L_{\mathrm{new}}\|<10\%$	$0.2\%$	PASS
H2	$L_{\mathrm{rep}}$ decreases ( $\Delta<0$ )	$+5.3\%$	FAIL
H3	$\Delta_{1\!-\!2}$ increases ( $\Delta>0$ )	$-0.1014$	FAIL

Appendix

Table 92: Hypothesis tests on the burstiness task.

Figure 69: Burstiness training trajectory. Training loss (top-left),

L_{\mathrm{new}}

and

L_{\mathrm{rep}}

(top-middle, right), first-second gap

\Delta_{1\!-\!2}

(bottom-left), per-occurrence learning curve (bottom-middle), final bar-chart comparison (bottom-right). Baseline and momentum curves visibly interleaved throughout training on this

\int

-task.

Figure 70: Learning curve by occurrence. The canonical visualization for this experiment. Baseline and momentum curves indistinguishable beyond the first occurrence. No “L-shape” for the momentum model, which would have been the signature of one-shot learning.

Metric	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.7$ )	$\Delta$
$L_{\mathrm{new}}$	$7.3450$	$7.3477$	$+0.0026$
$L_{\mathrm{second}}$	$2.1191$	$2.2813$	$+0.1622$
$L_{\mathrm{rep}}$	$1.6274$	$1.7687$	$+0.1413$
$\Delta_{1\!-\!2}$	$5.2259$	$5.0664$	$-0.1595$

Appendix

Table 93: Structural ICL without anchoring: final metrics.

\gamma=0.7

.

Hyp.	Statement	Measured	Verdict
H1	$\|\Delta L_{\mathrm{new}}\|<15\%$	$0.0\%$	PASS
H2	$L_{\mathrm{rep}}$ decreases	$+8.7\%$	FAIL
H3	$\Delta_{1\!-\!2}$ increases	$-0.1595$	FAIL

Appendix

Table 94: Hypothesis verdicts.

Figure 71: Structural ICL without anchoring: training curves. The baseline edges ahead on

L_{\mathrm{rep}}

and the momentum model loses the first-second gap. The opposite of the predicted behavior — which motivates the context-mismatch diagnosis in Section A27 .

Figure 72: Per-occurrence learning curve. Baseline (blue) below momentum (red) for all

k\!\geq\!1

, consistently.

Metric	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.2$ )	$\Delta$
$L_{\mathrm{new}}$	$7.1529$	$7.1458$	$-0.0071$
$L_{\mathrm{second}}$	$3.4463$	$3.9337$	$+0.4874$
$L_{\mathrm{rep}}$	$1.6496$	$2.1261$	$+0.4765$
$\Delta_{1\!-\!2}$	$3.7066$	$3.2121$	$-0.4946$

Appendix

Table 95: Chained ICL without anchoring: final metrics (

L=10

,

\gamma=0.2

).

Hyp.	Statement	Measured	Verdict
H1	$\|\Delta L_{\mathrm{new}}\|<15\%$	$0.1\%$	PASS
H2	$L_{\mathrm{rep}}$ decreases	$+28.9\%$	FAIL
H3	$\Delta_{1\!-\!2}$ increases	$-0.4946$	FAIL
H4	Deep-link improvement $k>7$ , $\Delta>0$	$-0.3492$	FAIL

Appendix

Table 96: Hypothesis verdict.

Depth $k$	Baseline	Momentum	$\Delta$ (B $-$ M)	Winner
$0$	$7.1529$	$7.1458$	$+0.0071$	Momentum
$1$	$3.4463$	$3.9337$	$-0.4874$	Baseline
$2$	$1.7209$	$2.3737$	$-0.6528$	Baseline
$3$	$1.4872$	$2.0387$	$-0.5515$	Baseline
$4$	$1.3765$	$1.8821$	$-0.5056$	Baseline
$5$	$1.3035$	$1.7631$	$-0.4596$	Baseline

Appendix

Table 97: Chain-depth breakdown: momentum loses at every depth

1

through

14

. The only “win” is at depth

0

(first-occurrence position).

Figure 73: Chained ICL training curves (no anchor). Baseline (blue) consistently below momentum (red).

Figure 74: Per-depth loss (no anchor). Baseline wins at every depth

\geq 1

; the gap is largest at depths

2

–

5

(the early chain positions where induction-head formation is easiest).

Metric	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.2$ )	$\Delta$
$L_{\mathrm{new}}$	$7.0849$	$7.0803$	$-0.0046$
$L_{\mathrm{second}}$	$3.0544$	$2.8714$	$-0.1830$
$L_{\mathrm{rep}}$	$1.2785$	$1.2262$	$-0.0523$
$\Delta_{1\!-\!2}$	$4.0305$	$4.2089$	$+0.1784$

Appendix

Table 98: Anchored ICL final metrics (

L=10

,

\gamma=0.2

). Compare against Section A27 ’s Table 95 :

L_{\mathrm{rep}}

flips from

+28.9\%

to

-4.1\%

on a one-token intervention.

Hyp.	Statement	Measured	Verdict
H1	$\|\Delta L_{\mathrm{new}}\|<15\%$	$0.1\%$	PASS
H2	$L_{\mathrm{rep}}$ decreases	$-4.1\%$	PASS
H3	$\Delta_{1\!-\!2}$ increases	$+0.1784$	PASS

Appendix

Table 99: Hypothesis verdict (anchored).

Configuration	Baseline $L_{\mathrm{rep}}$	Momentum $L_{\mathrm{rep}}$	Verdict
No anchor (Section A27 )	$1.6496$	$2.1261$	$+29\%$ FAIL
Anchored (this section)	$1.2785$	$1.2262$	$-4.1\%$ PASS

Appendix

Table 100: The no-anchor / anchored transition. One-token intervention flips the sign of the momentum effect on the same underlying task.

Figure 75: Anchored ICL training curves. Momentum model consistently at or below baseline throughout training, particularly on

L_{\mathrm{rep}}

.

\Delta_{1\!-\!2}

widens in the expected direction.

Figure 76: No-anchor vs anchored side-by-side. Left: per-depth loss; right: aggregate

L_{\mathrm{rep}}

. Anchoring flips the sign of the momentum effect at every depth.

Metric	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.2$ )	$\Delta$
$L_{\mathrm{new}}$	$6.9860$	$7.0202$	$+0.0342$
$L_{\mathrm{second}}$	$2.3598$	$1.2309$	$-1.1289$
$L_{\mathrm{rep}}$	$1.7451$	$0.8288$	$-0.9163$
$\Delta_{1\!-\!2}$	$4.6262$	$5.7893$	$+1.1632$

Appendix

Table 101:

L=30

anchored stress test final metrics.

L_{\mathrm{rep}}

drops by more than half — from

1.7451

to

0.8288

.

Hyp.	Statement	Measured	Verdict
H1	$\|\Delta L_{\mathrm{new}}\|<15\%$	$0.5\%$	PASS
H2	$L_{\mathrm{rep}}$ decreases	$-52.5\%$	PASS
H3	$\Delta_{1\!-\!2}$ increases	$+1.1632$	PASS
H4	Gap larger than at $L=10$ ( $\Delta>0.0523$ )	$+0.9163$	PASS

Appendix

Table 102: Hypothesis verdict,

L=30

. All four pass.

Depth $k$	Baseline	Momentum	$\Delta$ (B $-$ M)	Winner
$0$	$6.9860$	$7.0202$	$-0.0342$	Baseline
$1$	$2.3598$	$1.2309$	$+1.1289$	Momentum
$2$	$1.5645$	$0.6515$	$+0.9131$	Momentum
$3$	$1.4266$	$0.5334$	$+0.8931$	Momentum
$4$	$1.4771$	$0.5769$	$+0.9002$	Momentum
$5$	$1.2770$	$0.5402$	$+0.7368$	Momentum

Appendix

Table 103: Per-depth loss on

L=30

anchored chains. Momentum wins at every depth

1

through

19

(only depth

0

goes to baseline, by

0.03

nats). Deep-chain improvement alone

+0.133

; average

+0.406

. Mirror image of Section A27 ’s negative-per-depth story.

Figure 77:

L=30

stress-test training trajectory.

L_{\mathrm{rep}}

separation is large and widens throughout training, the momentum model finishing at less than half the baseline’s loss.

Figure 78: Three-panel stress-test analysis. Left: per-depth loss, “momentum wins everywhere” signature. Middle: smoothed-delta plot. Right: the exponential-vs-linear decay interpretation — the baseline exhibits the predicted

p^{L}

curve shape while the momentum model’s loss grows more slowly (closer to linear in depth).

Metric	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.2$ )	$\Delta$ (M $-$ B)
$L_{\mathrm{new}}$	$7.1026$	$7.2029$	$+0.1003$
$L_{\mathrm{rep}}$	$1.1446$	$1.1910$	$+0.0464$
$\Delta_{1\!-\!2}$	$5.9580$	$6.0119$	$+0.0539$
Improvement $(L_{\mathrm{rep}}^{B}-L_{\mathrm{rep}}^{M})/L_{\mathrm{rep}}^{B}$ :			$-4.1\%$

Appendix

Table 104: ICL metrics on anchored

L=30

chains with embedding-level momentum (notebook A). Despite using the correct transfer function in mathematical form, placing the shear at the embedding layer regresses the result.

Figure 79: Notebook A: training curves. Training loss,

L_{\mathrm{rep}}

, and

\Delta_{1\!-\!2}

are near-identical for baseline and momentum; the momentum model is slightly worse at the end.

Quantity	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.2$ )
Sensitivity range	$[0.08,37.8]$	$[0.08,38.5]$
DC component	$98.98$	$95.93$
Nyquist component	$38.07$	$31.81$
DC / Nyquist ratio	$2.60$	$3.015$

Appendix

Table 105: Notebook A: spectral biopsy at Layer

0

. The DC/Nyquist ratio increases from

2.60

(baseline) to

3.015

(momentum) — the opposite of what a high-pass filter should produce.

Layer	DC (baseline)	Nyq (baseline)	DC (momentum)	Nyq (momentum)
$0$	$96.46$	$37.24$	$98.99$	$31.99$
$1$	$200.40$	$43.83$	$168.71$	$36.72$
$2$	$457.82$	$62.70$	$221.18$	$33.87$
$3$	$844.17$	$98.21$	$600.45$	$88.68$

Appendix

Table 106: Notebook A: layer-wise DC and Nyquist evolution. In the baseline, DC grows rapidly with depth (96

\to

844); the momentum model also grows, less aggressively but in the same direction — the opposite of what a working high-pass filter should produce at the shear’s intended location.

Hyp.	Statement	Measured	Verdict
H(a)	baseline is low-pass	DC/Nyq $=2.60$	PASS
H(b)	momentum is high-pass	DC/Nyq $=3.015$	FAIL
H(c)	momentum improves ICL	$-4.1\%$ regression	FAIL

Appendix

Table 107: Notebook A: hypothesis scorecard.

Figure 80: Notebook A: Bode-plot diagnostics. Top row: sensitivity vectors and their FFTs for baseline and momentum. Bottom row: FFT magnitudes and the ratio. The predicted high-pass response does not emerge in this configuration; the DC-to-Nyquist ratio rises in the wrong direction.

Metric	§ A29 Base	§ A29 Mom	B Base	B Mom
$L_{\mathrm{new}}$	$6.9860$	$7.0202$	$6.9860$	$7.0202$
$L_{\mathrm{rep}}$	$1.7451$	$0.8288$	$1.7451$	$0.8288$
$\Delta_{1\!-\!2}$	$4.6262$	$5.7893$	$4.6262$	$5.7893$
$L_{\mathrm{rep}}$ improvement	$-52.5\%$		$-52.5\%$

Appendix

Table 108: Notebook B: reproduction of the Section A29 main result. Baseline and momentum

L_{\mathrm{new}}

,

L_{\mathrm{rep}}

, and

\Delta_{1\!-\!2}

are reproduced to machine precision because notebook B shares that section’s configuration; see § A30.4 on why notebook A does not.

Quantity	Baseline	Momentum
Spectral entropy $H$	$6.2159$	$6.2354$
$\Delta$ -entropy (M $-$ B)	$+0.0196$
Theory–experiment correlation	$\mathbf{r=0.9865}$

Appendix

Table 109: Notebook B: spectral metrics on the attention-score spectrum. Theory–experiment correlation is the headline number: at

r=0.9865

, the measured AC-power spectrum at each frequency

\omega

is

99\%

correlated with

|H(\omega)|^{2}=|1+\gamma(1-e^{-j\omega})|^{2}

.

Figure 81: Notebook B: the Bode plot of emergence. Six panels: (top-left) baseline attention spectrum showing the diffuse low-pass structure; (top-middle) PSA spectrum with clear peaks; (top-right) the spectral ratio

\log(S_{\mathrm{mom}}/S_{\mathrm{base}})

overlaid on the theoretical

|H|^{2}

curve — they match; bottom row: further diagnostics including the theory-experiment correlation curve.

Figure 82: Notebook B: training dynamics. Both baseline and momentum follow similar training-loss trajectories, but the decomposed metrics (

L_{\mathrm{rep}}

,

\Delta_{1\!-\!2}

) show the expected separation: baseline consistently above momentum on

L_{\mathrm{rep}}

and below on

\Delta_{1\!-\!2}

.

Parameter	Value
Architecture	12-layer, 12-head GPT-style decoder
$d_{\mathrm{model}}$	$768$
Vocabulary	$8{,}192$
Block size	$512$
Total parameters	$91.7$ M
RoPE base	$10{,}000$

Appendix

Table 110: Section A31 configuration.

$\gamma$	Fluency loss	Logic loss	$\|\det J-1\|^{\dagger}$	Energy ratio $R$	$\kappa(J)$
† saturated: $1.0000$ here means $\det J\approx 0$ , not $\det J\approx 1$ . See § A31.4 .
$0$	$7.9926$	$0.6934$	$1.0000$	$0.552$	$5.3\!\cdot\!10^{2}$
$10^{-4}$	$7.9838$	$0.6934$	$1.0000$	$0.544$	$4.0\!\cdot\!10^{2}$
$2{\cdot}10^{-4}$	$7.9910$	$0.6934$	$1.0000$	$0.450$	$2.0\!\cdot\!10^{7}$ (spike)
$5{\cdot}10^{-4}$	$7.9795$	$0.6934$	$1.0000$	$0.461$	$1.6\!\cdot\!10^{2}$
$10^{-3}$	$7.9672$	$0.6934$	$1.0000$	$0.543$	$2.0\!\cdot\!10^{3}$

Appendix

Table 111: Per-

\gamma

final values after

10{,}000

training steps. The

|\det J-1|

column is flat at

1.0000

across all

13

runs, which means

\det J\approx 0

: the projected Jacobian is numerically singular and the statistic is saturated (§ A31.4 ). It is retained for completeness and carries no inference. The condition number

\kappa

exhibits intermittent spikes, up to

10^{8}

, and is recorded for completeness; since the probe from which it is computed is degenerate, no mechanistic reading of those spikes — as phase transitions in learning or as anything else — is offered here.

Figure 83: The 13-point

\gamma

-sweep at 91.7M parameters. Learning curves (top row), Jacobian and energy diagnostics (middle rows), and the Jacobian condition number over training (bottom). The leakage-free energy ratio is the panel that carries inference; the Jacobian panels are saturated (§ A31.4 ) and the condition-number spikes are displayed without interpretation.

$\gamma$	emerged	steps to $50\%$ , by seed	mean	$\bar{E}$
$0.00$	$2/3$	$2900,\ 2500,\ \mathrm{NE}$	$2700$	$13.1$
$0.05$	$1/2$	$\mathrm{NE},\ 1750$	$1750$	$9.7$
$0.10$	$1/2$	$\mathrm{NE},\ 1550$	$1550$	$21.2$
$0.20$	$0/2$	$\mathrm{NE},\ \mathrm{NE}$	—	$64.7$
$0.40$	$3/3$	$1000,\ 2200,\ 2150$	$1783$	$12.0$
$0.50$	$3/3$	$1000,\ 1000,\ 2400$	$1467$	$6.9$

Appendix

Table 112: Single-layer induction emergence at

d_{k}=128

,

N=12

,

91.3

M parameters. Rows at

\gamma\leq 0.40

come from the width sweep, rows at

\gamma\geq 0.50

from the supercritical sweep; the

\gamma=0.00

and

0.40

rows carry a third seed contributed by the latter. “NE” = did not reach

50\%

within

3000

steps.

\bar{E}

is the mean final leakage-free energy ratio of § A31 . The two-seed rows are two-seed because sweep 1 was stopped at

49

of

80

runs by design, not because runs were discarded.

Figure 84: The emergence band at

d_{k}=128

,

N=12

,

91.3

M parameters. Top: steps to

50\%

induction accuracy, one marker per seed, bar at the mean; crosses at the top mark runs that never emerged within

3000

steps. Middle: emerged fraction. Bottom: final leakage-free energy ratio

E

, log scale, green for emerged and red for failed. Shading marks the band observed at the constant learning rate

3\!\times\!10^{-4}

used throughout ; its upper edge is a property of that step size and not of the operator (§ A31.11 ). Vertical lines:

\gamma_{c}^{T_{3}T_{4}}=0.413

(dotted), the depth-law prediction

\gamma^{*}(12)=0.680

(dash-dot), and

\gamma_{c}^{T_{4}}=0.787

(dashed). The

T_{4}

law’s second root at

2.245

(Remark 65 ) lies off the displayed range.

arm	outcome	$E_{\max}$ range	steps to $50\%$	verdict
A	$0/1$	$151.8$	—	pinned
B	$0/3$	$36.0$ – $106.4$	—	pinned
C	$\mathbf{2/3}$	$\mathbf{0.94}$ – $\mathbf{1.26}$	$\mathbf{1350,\ 1750}$	third still descending
D	$1/2$	$15.6$ – $27.7$	$1650$	second at $0.449$ and rising
E	$1/2$	$0.54$ – $0.74$	$1300$	second still descending

Appendix

Table 113: The decisive experiment. “descending” marks a run that had not reached

50\%

at its budget but whose induction loss was still falling; “pinned” marks one flat at chance.

E_{\max}

is the peak leakage-free energy ratio over the run.

Figure 85: The upper cutoff is optimiser conditioning. Left: peak energy ratio

E_{\max}

by arm, log scale. Every run above

E=30

is pinned at chance; every run below it emerges or is still descending. Right: steps to

50\%

induction accuracy; markers at the top mark runs that did not emerge within budget. Open grey squares are the constant-LR runs of Table 112 at the same couplings, shown for reference.

Model	Params	Layers	$d_{\mathrm{model}}$	Heads	Momentum
Goliath	406M	24	$1024$	$16$	none
David	163M	12	$768$	$12$	$\gamma=0.8$
Goliath Jr	163M	12	$768$	$12$	none

Appendix

Table 114: R 1 : model configurations. All three models use RoPE with base

10{,}000

and block size

512

. “David” adds momentum at

\gamma=0.8

.

Model	Params	Momentum	Fluency loss	Logic loss	Wall-clock
Goliath	$406$ M	none	$10.7745$	$0.6935$	$245.4$ min
David	$163$ M	$\gamma=0.8$	$10.8214$	$0.6933$	$93.3$ min
Goliath Jr	$163$ M	none	$10.7863$	$0.6934$	$88.2$ min

Appendix

Table 115: R 1 : final losses after

10{,}000

steps.

Hyp.	Statement	Margin	Verdict
H1	David ( $163$ M+mom) beats Goliath ( $406$ M) on fluency	$-0.4\%$	FAIL
H2	Goliath beats David on logic	$-0.0\%$	FAIL (tied at chance)
H3	David beats Goliath Jr at equal size	$-0.3\%$	FAIL

Appendix

Table 116: R 1 : hypothesis verdicts. All three hypotheses fail; all margins are

<0.5\%

.

Figure 86: R 1 : training curves. Fluency loss (left), logic loss (middle), and combined training loss (right) over

10{,}000

training steps. All three fluency curves stay near the initial

\ln(50{,}257)\approx 10.82

plateau throughout training. Logic loss is stuck at

\ln(2)=0.6931

from step

\sim 500

onward.

Patch	Before (R 1 )	After (R 2 )	Rationale
Batch size	$8$	$64$	Reduce gradient variance $\sim 2.83\times$
Vocabulary	$50{,}257$	$8{,}192$	Increase $P(\text{token recurrence})\sim 6\times$
Learning rate	$3\!\times\!10^{-4}$	$6\!\times\!10^{-4}$	$\sqrt{B_{\mathrm{new}}/B_{\mathrm{old}}}$ scaling

Appendix

Table 117: R 2 : the three patches relative to R 1 .

Model	Params	Layers	$d_{\mathrm{model}}$	Heads	Momentum
Goliath	$319$ M	$24$	$1024$	$16$	none
David	$98$ M	$12$	$768$	$12$	$\gamma=0.8$
Goliath Jr	$98$ M	$12$	$768$	$12$	none

Appendix

Table 118: R 2 : revised model configurations.

Model	Params	Momentum	Fluency loss	Logic loss	Wall-clock
Goliath	$319$ M	none	$7.7732$	$0.6932$	$1565.1$ min
David	$98$ M	$\gamma=0.8$	$8.1538$	$0.6933$	$592.2$ min
Goliath Jr	$98$ M	none	$7.9205$	$0.6934$	$542.1$ min

Appendix

Table 119: R 2 : final losses after

10{,}000

steps with patches. Fluency does escape the plateau (drops from

9.03

to

7.77

–

8.15

), confirming that the patches are working at the SNR level.

Hyp.	Statement	Margin	Verdict
H1	David ( $98$ M+mom) beats Goliath ( $319$ M) on fluency	$-4.9\%$	FAIL
H2	Goliath ties or beats David on logic	$+0.0\%$	PASS
H3	David beats Goliath Jr at equal size	$-2.9\%$	FAIL

Appendix

Table 120: R 2 : hypothesis verdicts (patched). H2 passes narrowly because all three models tie at chance on logic; H1 and H3 fail with margins between

3

and

5\%

.

Source	Task	Class	Peak gain	Low- $\theta$ best?	PSA-consistent?
§ A18	Associative Recall	$\nabla$	$+0.368$	yes	yes
§ A18	Variable Tracking	$\nabla$	$-0.046$	—	neutral (narrow sweep)
§ A18	Global Counting	$\int$	$-0.322$	—	yes (harm as predicted)
§ A19	Multi-Hop eased	$\nabla$	$+0.007$	—	null (out of band)
§ A20	Variable Tracking	$\nabla$	$+0.156$	yes	yes
§ A20	Multi-Hop w/ neg.	$\nabla$	$+0.001$	—	null (out of band)

Appendix

Table 121: Task-by-task verdict matrix for the Tranche 2 reasoning and ICL programs. Class:

\nabla

(derivative-class) or

\int

(integral-class, order-invariant). “Low-

\theta

best?” indicates whether low-frequency RoPE produced the largest gain, the empirical signature of the Low-Pass Induction Filter.

the claim of the body	Anchored in	Verdict	Evidence
Theorem 11 (four-term decomposition)	§ A2 , § A8 S4–5	PASS	NumPy verification $+$ $T_{4}$ -dominance at all $\gamma>0$
Theorem 27 (uniqueness in $\mathcal{C}_{\mathrm{PSA}}$ )	§ A3 , § A10 C15	PASS	C2 high-pass, FP-29 symmetric advantage
Cor. 41 (post- RoPE placement)	§ A2 , § A30	PASS	directional: $+4.1\%$ vs $-52.5\%$ ; runs unmatched
Cor. 41 , eq. ( 38 ) (Coriolis error vs. $\theta$ )	§ A8 S12	PASS ∗	directional slope $+0.307$ ; band-4 anomaly noted
Lemma 4 ( $M_{\gamma}$ is exact $\gamma$ -flow)	C6	PASS	exact operator residual, scale-independent; no training-scale evidence is claimed (§ A31.4 )
Thm. 38 (ii) (invariant conserved)	§ A10 C9	PASS ∗	flat on stationary inputs; causal-task confound

Appendix

Table 122: Audit of the claims of the body tested in these appendices. “Anchored in” gives the section number where the empirical content lives; “Verdict” summarizes the outcome, with PASS/CHECK/null tags as defined in the validation-suite framework of Section A8 .

Operator	Order	Accuracy	What it carries
Query-only $(\gamma,0)$	$(1,0)$	$0.033$	Nothing: wrong axis; capped at $1/(T{-}2)=0.077$ , and below chance because the previous-token position takes the argmax (Cor. 68 , Rem. 2 )
Key-only $(0,\gamma)$	$(0,1)$	$\mathbf{0.949}$	Representability, and the best accuracy in the battery. Its own transition law (Cor. 69 )
Asymmetric $(\gamma_{q},\gamma_{k})$	$(1,1)^{*}$	$0.875$	Symplectic and energy-preserving, but an exact score bias and no transport identity (Prop. 36 )
Symmetric $(\gamma,\gamma)$	$(1,1)$	$0.811$	Uniqueness (Thm. 27 ); transport identity (Prop. 34 ); the $\gamma^{2}$ channel and the two-branch law of the reduced model (Thm. 64 ); self-adjoint higher-order completion; Differential Transformer at first order (Prop. 81 )

Family	(C1)	(C2)	Operational	Verdict
Upper shear $M_{\gamma}$	yes	yes (Prop. 13 )	yes	admissible
Lower shear $M_{\gamma}^{l}$	yes	no, all-pass (Cor. 21 )	no (Prop. 20 )	dismissed
Skew (F3)	yes	—	moves the momentum block (Prop. 24 )	outside the class
Rotation ( RoPE reuse)	yes	yes	breaks factorisation (Prop. 26 )	dismissed

Classical statement (integrator)	Requires	Attention-side analogue
$M_{\gamma}$ is the exact flow of a quadratic Hamiltonian; symplectic at each application	one application	Asserted. Theorem 38 (i)–(ii); certificate C6
Order of the stencil is raised from $1$ to $2$ by passing to phase space	algebra only	Asserted in the filter-order image: joint order $(0,0)\to(1,1)$ (§ 2 , Thm. 27 )
Self-adjointness of the symmetric composition	algebra only	Asserted as the transport identity, Prop. 34
Existence of a modified Hamiltonian $\widetilde{H}$ conserved to all orders	iterated map	Not asserted. No iterate exists; see ( 36 )
Energy error bounded over times $e^{c/\Delta t}$ ; secular drift absent	iterated map, small step, analytic field	Not asserted. No long-time or depth-wise stability claim is made
Secular energy growth $E_{n}/E_{0}=(1+\Delta t^{2})^{n}$ for the non-symplectic alternative	iterated map	Not asserted. The perturbation statement we make is per application, Thm. 38 (iii)

Notebook	Protocol	Hardware	Wall time
Certificates C1–C10	SymPy ring normal forms and float64 numerics; ten independent algebraic and numerical checks	CPU	$<1$ min
Score-model $d_{k}$ sweep	$d_{k}\in\{4,8,16,32,64,128\}$ $\times$ $T\in\{10,15,25\}$ ; 5 seeds, $60{,}000$ trials per point, 61-point $\gamma$ grid on $[0,8]$	A100	198 s
$T$ -axis validation	$d_{k}=16$ with $T\in\{6,8,9,10,12,15,20,25,30,40\}$ ; same extraction	CPU	$<2$ min
Trained $d_{k}$ ladder	$d_{k}\in\{4,\dots,64\}$ , $d_{\mathrm{model}}=4d_{k}$ , $d_{\mathrm{ff}}=16d_{k}$ ; 3 seeds $\times$ 3,000 steps on an eleven-point $\gamma$ grid, $T{-}1=14$ pairs	A100	47.0 min
Matched control battery	Seven mechanisms, one protocol: $T{-}1=14$ , $d_{k}=16$ , $d_{\mathrm{model}}=64$ , 3 seeds $\times$ 3,000 steps, taps post- RoPE , $V$ untouched	A100	24.2 min
Spectral rerun	Retraining from scratch at each of six couplings; $d_{k}=16$ , $T=64$ with 30 pairs, 3,000 steps; 200 evaluation sequences	A100	1.9 min

Theorem	Prediction	Metric	Result	Failure mode
Prop. 13	Closed-form $\|H(e^{j\omega})\|^{2}$	Pearson $r$ vs. closed-form	$r{=}0.895$ – $0.9966$ pre-softmax across $\gamma{>}0$	NS baseline: $r{=}0.03$ at $\gamma{=}0.6$ (Appendix A10, C15)
Thm. 38	$\\|M_{\gamma}^{\top}JM_{\gamma}-J\\|{=}0$ at operator level	Symplectic residual; HSF/ $\tau$ scaling	$0.0$ exactly (C6); HSF $\pm 4\%$ (Appendix A10)	$\tau{=}0.1$ ablation: Bode $r{\to}0.41$ (Appendix A10)
Prop. 34	Symmetric score is the transported pairing	Symbolic identity	Exact; asymmetric current is no total difference (C9)	—
Prop. 36	Unequal shears symplectic; bias linear in $\varepsilon$	Residual; exact score difference	$0.0$ exactly (C4); bias exact (C5); trains to $0.875$	—
Cor. 41	Coriolis penalty pre- RoPE	Controlled placement comparison	Post- RoPE better under the tested protocols, $\sim$ 2k runs (Appendix A30)	—
Thm. 64	$\gamma_{c}^{2}{=}\tfrac{2L}{\sqrt{d_{k}}+\sqrt{d_{k}-8L}}$ ( $d_{k}{\geq}8L$ ), $\gamma_{c}^{2}{=}\tfrac{1}{2}c^{2}\sqrt{2L}$ ( $d_{k}{<}8L$ )	Held-out MC grid	$r{=}0.9876$ (9 pts, 0 params); $69.1{\pm}8.2\%$ at $\gamma^{\star}{=}2.0$ vs. $1.6\%$	Depth law fails at $L{=}30$ (Appendix A7)

Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction

Abstract

Figures & tables

Appendix

Transformer Attention

HySPE: Positional Encoding via Symplectic Dual Shears

RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies

Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

Notebook	Where	What it establishes
Battery (§ 7.1 ): seven notebooks, one matched protocol.
PSA_Certificates	§ 7.15	Certificates C1–C10: the annealed crossing, the crossover $d_{k}^{*}{=}8L$ , the quartic root and its asymptote, symplecticity and energy preservation of unequal shears, the exact score bias, the operator residual, the plateau quadrature, the quartic against fresh-seed Monte Carlo, the transport identity, and Pearson scale invariance
PSA_MC_dk_sweep_with_results	§ 7.9	Score-model Monte-Carlo grid, $d_{k}{\times}T$ , five seeds and $60{,}000$ trials per point; the held-out mean-field test, $r{=}0.9876$ over nine points with no fitted parameter
PSA_T_axis_condensed_validation	§ 7.10	The condensed branch along the $T$ axis at fixed $d_{k}{=}16$ ; $r{=}0.9995$ on branch, and the condensation constant on a second axis
PSA_trained_dk_overlay_with_results	§ 7.11	The trained $d_{k}$ ladder: $\gamma_{c}$ falling with head width, plateaus against the zero-parameter score-model argmax limit, and the measured operating-point ratio
PSA_control_battery_results	§ 7.5	Seven mechanisms under one protocol: query-only never above chance, key-only strongest, low-pass lagging, and the learned two-tap converging to a pure key-side difference

Variant	Order	Best accuracy	Attained at	Reading
One-layer baseline	$(0,0)$	$0.033\pm 0.006$	—	below the $0.077$ ceiling; cf. Prop. 1 , Rem. 2
Query-only shift	$(1,0)$	$0.033\pm 0.007$	$\gamma{=}0.5$	same class, same value at every $\gamma$ ; cf. Cor. 68
Key-only shift	$(0,1)$	$0.949\pm 0.006$	$\gamma{=}3.0$	minimal member; strongest fixed variant
PSA symmetric	$(1,1)$	$0.811\pm 0.006$	$\gamma{=}3.0$	rises monotonically from $0.505$
Asymmetric $(\gamma_{q},\gamma_{k})$	$(1,1)^{*}$	$0.875\pm 0.008$	$(1.0,3.0)$	bias absorbed, as Prop. 36 predicts
Low-pass two-tap	non-DC-neutral	$0.428\pm 0.018$	$\gamma{=}3.0$	differencing, not generic context

Attention side	Filter order	Integrator analogue	Measured
Standard bilinear	$(0,0)$	no lift: the obstruction	$0.033$
Query-only shear	$(1,0)$	first-order rule, wrong axis	$0.033$
Key-only shear	$(0,1)$	exact flow of $\tfrac{1}{2}\\|p_{k}\\|^{2}$ ; no swap symmetry	$0.949$
Asymmetric $\gamma_{q}\neq\gamma_{k}$	$(1,1)^{*}$	exact flow of a two-coefficient quadratic; no transport identity (C9)	$0.875$
Low-pass two-tap	non-DC-neutral	dissipative smoother	$0.428$
Learned two-tap	learned	discovers $(0,1)$ from data	$1.000$

accuracy drop	$N{=}2$	$4$	$5$	$6$	$7$	$8$	$12$	$16$
global $\gamma\!\to\!0$	$0.861$	$0.880$	$0.885$	$0.888$	$0.888$	$0.889$	$0.891$	$0.895$
first block only	$0.856$	$0.874$	$0.878$	$0.879$	$0.878$	$0.878$	$0.849$	$0.805$
last block only	$0.021$	$0.008$	$0.003$	$0.002$	$0.001$	$0.000$	$0.000$	$0.000$
first $-$ global	$-0.005$	$-0.006$	$-0.007$	$-0.009$	$-0.010$	$-0.011$	$-0.042$	$-0.090$

	Supports	Verifies	Outcome
C1	Thm. 64 (2)	Gaussian MGF by direct integration; annealed exponent $\Lambda=L+2\gamma^{4}$	exact, symbolic
C2	Thm. 64 (4)	$\max_{\gamma}(a-\Lambda)=d_{k}/8-L$ at $\gamma^{2}=\sqrt{d_{k}}/4$ ; crossover $d_{k}^{*}=8L$	exact, symbolic
C3	Thm. 64 (3)	quartic root closed form, $(L^{2}/d_{k})^{1/4}$ asymptote, monotonicity in both arguments	exact, symbolic
C4	Prop. 36 (i)–(ii)	asymmetric shear is symplectic and preserves $H_{\mathrm{joint}}$	residual exactly $0.0$ at $d_{k}{=}64$
C5	Prop. 36 (iii)	$s^{\mathrm{asym}}-s^{\mathrm{sym}}=\varepsilon(q^{\top}p_{k}+\gamma_{q}p_{q}^{\top}p_{k})$	exact, symbolic
C6	Thm. 38 (i)	$M_{\gamma}^{\top}JM_{\gamma}=J$ symbolically and numerically	residual $0.0$ exactly at $d_{k}{=}64$

configuration	$d$	$T$	overhead
small on-device	1024	2048	$1.8\times 10^{-4}$
mid on-device	2048	4096	$9.2\times 10^{-5}$
server class	4096	4096	$5.2\times 10^{-5}$
long context	4096	32768	$2.6\times 10^{-5}$

$\gamma$	range factor $1+2\gamma$	$\Delta b$ (bits)	INT8 levels retained
0.05	1.10	0.14	233/256
0.10	1.20	0.26	213/256
0.20	1.40	0.49	183/256
0.40	1.80	0.85	142/256

quantity	PSA relative to standard attention
parameters	identical: the operator adds none
KV-cache size	identical; cache the sheared keys (Prop. 73 )
extra persistent state	$2dL$ scalars, independent of $T$ ( $1/T$ of the cache)
FLOPs per token, per layer	$+6d$ , i.e. $6/(24d+4T)$ ; $<10^{-4}$ in practice
prefill	same fraction; fully vectorised, no recurrence
prefix caching	unaffected (causal two-tap FIR, Prop. 76 )

Abstract

Transformer Attention

Explore similar work

HySPE: Positional Encoding via Symplectic Dual Shears

RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies

Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention