Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction

Kingsuk Maitra; Shagun Sood Morteza Hosseini; Suman Gunnala; Vikram Gupta

cs.LGSep 26, 2026

Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction

Authors: Kingsuk Maitra, Shagun Sood Morteza Hosseini, Suman Gunnala, Vikram Gupta

Organizations: Qualcomm Cloud AI Division, Santa Clara, CA

Abstract

We circumvent the Sanford-Hsu-Telgarsky (SHT) single-layer induction obstruction within a linear, one-step, causal, bilinear, symplectically consistent design class on the post-RoPE substrate, by lifting attention onto a symplectic phase space, mirroring Hairer's lift of Stormer-Verlet. The lift exits the premise of the SHT counting argument rather than the bound itself. The reframing exhibits the obstruction as a filter-order gap: a one-layer bilinear score realises a $z$ -transform of joint order $(0,0)$ , whereas the induction discriminator requires key-side order $\geq 1$ . Applying the symplectic upper shear $M_γ:(q,p)\mapsto(q+γp,p)$ to the post-RoPE query and key streams closes it. We prove this lift is unique within the factorised subclass, exactly symplectic at operator level, and requires post-RoPE placement; and in an explicit $T_4$ -only Gaussian reduction we derive a closed-form two-branch induction phase transition, held out at $r=0.9876$ with zero fitted parameters. That law is an analytically solvable limit, not a robust prediction: restoring the $T_3$ channel moves $γ_c$ at $d_k=64$ from 1.030 to 0.569 and removes the crossover. Deployability follows by exact derivation: the KV cache is unchanged, prefix reuse and speculative decoding are preserved, overhead is $6d$ FLOPs per token per layer, INT8 headroom grows by at most $\log_2(1+2γ)$ bits, fused kernels are unmodified, and no parameters are added. At 91.3M parameters a supercritical sweep locates an emergence band: induction forms 3/3 seeds at $γ=0.80$ in a mean of 717 steps, against 2/3 seeds and 2700 steps at $γ=0$ . Adverse results are reported as directly: a key-only half-lift reaches 0.949 against 0.811 for the symmetric operator, so if induction accuracy is the objective, the half-lift is the better construction. Forty-one notebooks and result files ship as ancillary material.

Figures & tables

Operator	Order	Accuracy	What it carries
Query-only $(\gamma,0)$	$(1,0)$	$0.033$	Nothing: wrong axis; capped at $1/(T{-}2)=0.077$ , and below chance because the previous-token position takes the argmax (Cor. 68 , Rem. 2 )
Key-only $(0,\gamma)$	$(0,1)$	$\mathbf{0.949}$	Representability, and the best accuracy in the battery. Its own transition law (Cor. 69 )
Asymmetric $(\gamma_{q},\gamma_{k})$	$(1,1)^{*}$	$0.875$	Symplectic and energy-preserving, but an exact score bias and no transport identity (Prop. 36 )
Symmetric $(\gamma,\gamma)$	$(1,1)$	$0.811$	Uniqueness (Thm. 27 ); transport identity (Prop. 34 ); the $\gamma^{2}$ channel and the two-branch law of the reduced model (Thm. 64 ); self-adjoint higher-order completion; Differential Transformer at first order (Prop. 81 )

Table 1: The trade-off, stated once. Accuracies are the maximum over the coupling grid, so each is a best-of-

k

statistic and is upward biased relative to a value selected on held-out data; the comparison between variants is fair because every variant is scored by the same rule on the same grid. Protocol as in § 7.5 (three seeds, chance

0.071

; Fig. 4 and Table 7 ). Every entry in the left column is exactly symplectic; symplecticity does not separate them (Prop. 36 ).

Figure 1: Roadmap from the SHT obstruction to PSA via Hairer’s symplectic lift. (1) Standard attention is order

(0,0)

; the induction discriminator needs key-side order

\geq 1

, whose minimal member is

(0,1)

. (2) The same filter-order gap obstructs forward Euler; Störmer–Verlet attains order 2. (3) The lift extends the state to

(q,p)

and closes the gap with the shear

M_{\gamma}

, the exact

\gamma

-flow of

H(q,p)=\tfrac{1}{2}\|p\|^{2}

(Lem. 4 ); symplecticity is shared by every member of the order lattice and is therefore not what distinguishes it (Prop. 36 ). (4) Transporting

M_{\gamma}

to post- RoPE

Q,K

adds the two single-ghost cross terms and the

\gamma^{2}

coherence channel, raising the kernel order to

(1,1)

; the placement corollary requires post- RoPE application. (5) Within the factorised subclass

M_{\gamma}

is the unique causal, one-step, linear augmentation that is per-stream high-pass, DC-neutral and swap-symmetric (Thm. 27 ), unique up to the scalar

\gamma

. What symmetry buys is structure, not accuracy: on the induction task the

(0,1)

half-lift scores higher (§ 7.5 ). Derivations: § 2 (panel 1), § 3 (panel 2), § 3.2 (panel 3), § 4.1 – 4.2 and § 5.3 (panel 4), § 4.5 and § 4.8 (panel 5).

Family	(C1)	(C2)	Operational	Verdict
Upper shear $M_{\gamma}$	yes	yes (Prop. 13 )	yes	admissible
Lower shear $M_{\gamma}^{l}$	yes	no, all-pass (Cor. 21 )	no (Prop. 20 )	dismissed
Skew (F3)	yes	—	moves the momentum block (Prop. 24 )	outside the class
Rotation ( RoPE reuse)	yes	yes	breaks factorisation (Prop. 26 )	dismissed

Table 2: The local design space. Each family is eliminated by a separate proved statement, not by a preference. “Operational” asks whether the map changes the score at all.

Classical statement (integrator)	Requires	Attention-side analogue
$M_{\gamma}$ is the exact flow of a quadratic Hamiltonian; symplectic at each application	one application	Asserted. Theorem 38 (i)–(ii); certificate C6
Order of the stencil is raised from $1$ to $2$ by passing to phase space	algebra only	Asserted in the filter-order image: joint order $(0,0)\to(1,1)$ (§ 2 , Thm. 27 )
Self-adjointness of the symmetric composition	algebra only	Asserted as the transport identity, Prop. 34
Existence of a modified Hamiltonian $\widetilde{H}$ conserved to all orders	iterated map	Not asserted. No iterate exists; see ( 36 )
Energy error bounded over times $e^{c/\Delta t}$ ; secular drift absent	iterated map, small step, analytic field	Not asserted. No long-time or depth-wise stability claim is made
Secular energy growth $E_{n}/E_{0}=(1+\Delta t^{2})^{n}$ for the non-symplectic alternative	iterated map	Not asserted. The perturbation statement we make is per application, Thm. 38 (iii)

Table 3: The boundary of the correspondence. The left column lists the classical statements of § 3 ; the right column records whether an analogue is asserted for the attention operator. The distinguishing hypothesis in every declined row is iteration of the map on its own output, which ( 36 ) shows does not occur.

Figure 2: The freezing transition, from disordered systems to attention. (a) The annealed entropy of the random energy model,

s(b)=L-b^{2}/2

, at

L=\log 511

. It vanishes at

b_{c}=\sqrt{2L}=3.53

: to the left the partition sum is shared among

e^{s(b)}

states; to the right the sum is extreme-value dominated, carried by the leading extremal states, and the annealed evaluation is no longer valid. (b) The same two phases as the two branches of ( 52 ), in the same two colours. The panel concerns the

T_{4}

-only reduction

\mathcal{G}

and nothing wider: § 10 finds the branch structure to be the most reduction-sensitive feature of the theory, not surviving the restoration of

T_{3}

. The condensed branch is

\gamma_{c}=c(2L)^{1/4}/\sqrt{2}

, flat in

d_{k}

and carrying the one calibrated constant; the mean-field branch is

\gamma_{c}^{2}=2L/(\sqrt{d_{k}}+\sqrt{d_{k}-8L})

, with no free parameter. The crossover is

d_{k}^{*}=8L=49.9

, where the annealed bound saturates, and the step across it is exactly the factor

c

. (c) The coupling transition and the training transition are orthogonal: the first is a property of the operator, the second of the optimiser, and neither is an estimator of the other. The horizontal line is

\gamma_{c}

, where the reduced model concentrates target mass; it is not where the shear switches on or the channel becomes representable, both of which hold at every

\gamma>0

(Thm. 27 , Cor. 14 ). § 7.14 measures trained models taking the single-layer route between

0.17\gamma_{c}

and

0.45\gamma_{c}

— inside the lower half of the panel.

Notebook	Protocol	Hardware	Wall time
Certificates C1–C10	SymPy ring normal forms and float64 numerics; ten independent algebraic and numerical checks	CPU	$<1$ min
Score-model $d_{k}$ sweep	$d_{k}\in\{4,8,16,32,64,128\}$ $\times$ $T\in\{10,15,25\}$ ; 5 seeds, $60{,}000$ trials per point, 61-point $\gamma$ grid on $[0,8]$	A100	198 s
$T$ -axis validation	$d_{k}=16$ with $T\in\{6,8,9,10,12,15,20,25,30,40\}$ ; same extraction	CPU	$<2$ min
Trained $d_{k}$ ladder	$d_{k}\in\{4,\dots,64\}$ , $d_{\mathrm{model}}=4d_{k}$ , $d_{\mathrm{ff}}=16d_{k}$ ; 3 seeds $\times$ 3,000 steps on an eleven-point $\gamma$ grid, $T{-}1=14$ pairs	A100	47.0 min
Matched control battery	Seven mechanisms, one protocol: $T{-}1=14$ , $d_{k}=16$ , $d_{\mathrm{model}}=64$ , 3 seeds $\times$ 3,000 steps, taps post- RoPE , $V$ untouched	A100	24.2 min
Spectral rerun	Retraining from scratch at each of six couplings; $d_{k}=16$ , $T=64$ with 30 pairs, 3,000 steps; 200 evaluation sequences	A100	1.9 min

Table 4: The seven-notebook battery. Each notebook is self-contained, carries deterministic seeds, and ships with embedded outputs. Wall times are for the full (non-quick) configuration on the hardware noted; the certificate notebook requires no accelerator and no data.

Theorem	Prediction	Metric	Result	Failure mode
Prop. 13	Closed-form $\|H(e^{j\omega})\|^{2}$	Pearson $r$ vs. closed-form	$r{=}0.895$ – $0.9966$ pre-softmax across $\gamma{>}0$	NS baseline: $r{=}0.03$ at $\gamma{=}0.6$ (Appendix A10, C15)
Thm. 38	$\\|M_{\gamma}^{\top}JM_{\gamma}-J\\|{=}0$ at operator level	Symplectic residual; HSF/ $\tau$ scaling	$0.0$ exactly (C6); HSF $\pm 4\%$ (Appendix A10)	$\tau{=}0.1$ ablation: Bode $r{\to}0.41$ (Appendix A10)
Prop. 34	Symmetric score is the transported pairing	Symbolic identity	Exact; asymmetric current is no total difference (C9)	—
Prop. 36	Unequal shears symplectic; bias linear in $\varepsilon$	Residual; exact score difference	$0.0$ exactly (C4); bias exact (C5); trains to $0.875$	—
Cor. 41	Coriolis penalty pre- RoPE	Controlled placement comparison	Post- RoPE better under the tested protocols, $\sim$ 2k runs (Appendix A30)	—
Thm. 64	$\gamma_{c}^{2}{=}\tfrac{2L}{\sqrt{d_{k}}+\sqrt{d_{k}-8L}}$ ( $d_{k}{\geq}8L$ ), $\gamma_{c}^{2}{=}\tfrac{1}{2}c^{2}\sqrt{2L}$ ( $d_{k}{<}8L$ )	Held-out MC grid	$r{=}0.9876$ (9 pts, 0 params); $69.1{\pm}8.2\%$ at $\gamma^{\star}{=}2.0$ vs. $1.6\%$	Depth law fails at $L{=}30$ (Appendix A7)

Table 5: Empirical validation map. Each row links a theorem to its falsifiable prediction, the metric, the headline result, and the documented failure regime. Full protocols and seed statistics are in the cited appendices.

Notebook	Where	What it establishes
Battery (§ 7.1 ): seven notebooks, one matched protocol.
PSA_Certificates	§ 7.15	Certificates C1–C10: the annealed crossing, the crossover $d_{k}^{*}{=}8L$ , the quartic root and its asymptote, symplecticity and energy preservation of unequal shears, the exact score bias, the operator residual, the plateau quadrature, the quartic against fresh-seed Monte Carlo, the transport identity, and Pearson scale invariance
PSA_MC_dk_sweep_with_results	§ 7.9	Score-model Monte-Carlo grid, $d_{k}{\times}T$ , five seeds and $60{,}000$ trials per point; the held-out mean-field test, $r{=}0.9876$ over nine points with no fitted parameter
PSA_T_axis_condensed_validation	§ 7.10	The condensed branch along the $T$ axis at fixed $d_{k}{=}16$ ; $r{=}0.9995$ on branch, and the condensation constant on a second axis
PSA_trained_dk_overlay_with_results	§ 7.11	The trained $d_{k}$ ladder: $\gamma_{c}$ falling with head width, plateaus against the zero-parameter score-model argmax limit, and the measured operating-point ratio
PSA_control_battery_results	§ 7.5	Seven mechanisms under one protocol: query-only never above chance, key-only strongest, low-pass lagging, and the learned two-tap converging to a pure key-side difference

Table 6: The forty-one accompanying notebooks. “Where” gives the section or appendix in which the notebook’s results are reported. Outputs are embedded throughout except in the four Appendix Q notebooks marked

\dagger

, whose sweeps outlived their interactive sessions and which ship with their complete result files, ledgers and logs instead (§ 7.2 ). No notebook needs to be run to read its results.

Figure 3: Single-layer induction phase transition. The two panels measure complementary quantities on deliberately different tasks; their absolute accuracies are not directly comparable. (a) Single-layer induction under distractor noise (

N{=}1

, “nano” decoder,

\sim

54 k parameters,

d_{\mathrm{model}}{=}64

, 4 heads,

d_{k}{=}16

; Appendix A8). Vocabulary 64 with shared key/target alphabet (key-vs-target role learnable only from position), 14 KV pairs, padded sequence length 30 (padding tokens act as distractors); the plotted sweep is one trained model evaluated on three seeds, peaking at

74.2\%

, and the statistic annotated at

\gamma^{\star}{=}2.0

is an independent five-seed mean

\,\pm\,

s.d. Standard attention (

\gamma{=}0

) does not exceed chance,

1/64=1.6\%

: it measures

1.27\%

on the plotted sweep and

1.6\pm 0.4\%

across the five seeds. Single-layer PSA reaches

69.1\pm 8.2\%

over those five seeds (best seed

79.4\%

), a

+67.5

pp mean phase jump that demonstrates single-layer induction recovery under noise , in a regime where the standard bilinear head does not. The shaded band marks the low-coupling range over which the head is not measurably above the standard single-layer baseline; it is not a region excluded by the SHT bound, which constrains description size rather than

\gamma

. (b) PE-agnostic phase boundary on a sanitised multi-layer setup (

N{=}4

decoder,

\sim

842 k parameters,

d_{\mathrm{model}}{=}128

, 4 heads,

d_{k}{=}32

d_{\mathrm{ff}}{=}512

; Appendix A4). Vocabulary 200 with disjoint key/target sub-alphabets (keys

\in[1,100)

, targets

\in[100,200)

; role readable from token ID), 12 KV pairs, no padding distractors—designed to locate

\gamma_{c}

cleanly across PE families. The panel plots the transition window

\gamma\in[0,0.5]

; the sweep continues to

\gamma=5

, where both families saturate together —

98.5

–

99.6\%

across

\gamma\in[0.7,3]

, so the two curves converge as well as sharing a boundary. The one exception is

\gamma=5

, at which RoPE collapses to

68.4\pm 43.1\%

over three seeds while sinusoidal holds at

98.6\pm 0.2\%

: at that coupling one seed fails outright, and we report the instability rather than truncating the sweep before it. Peak accuracy within the plotted window is

96.6\%

( RoPE ) and

97.7\%

(sinusoidal); the boundary is PE-agnostic to within

1.22\times

(

\gamma_{c}\!\approx\!0.225

RoPE ,

\gamma_{c}\!\approx\!0.275

sinusoidal), against the parameter-free

1.000

that Theorem 64 predicts, the law carrying no positional-encoding term. Panel (a) tests single-layer recovery under noise ; panel (b) tests

\gamma_{c}

scaling on the multi-layer regime where Theorem 64 applies —hence the

\sim

16\times

parameter gap and the absolute-accuracy gap.

Variant	Order	Best accuracy	Attained at	Reading
One-layer baseline	$(0,0)$	$0.033\pm 0.006$	—	below the $0.077$ ceiling; cf. Prop. 1 , Rem. 2
Query-only shift	$(1,0)$	$0.033\pm 0.007$	$\gamma{=}0.5$	same class, same value at every $\gamma$ ; cf. Cor. 68
Key-only shift	$(0,1)$	$0.949\pm 0.006$	$\gamma{=}3.0$	minimal member; strongest fixed variant
PSA symmetric	$(1,1)$	$0.811\pm 0.006$	$\gamma{=}3.0$	rises monotonically from $0.505$
Asymmetric $(\gamma_{q},\gamma_{k})$	$(1,1)^{*}$	$0.875\pm 0.008$	$(1.0,3.0)$	bias absorbed, as Prop. 36 predicts
Low-pass two-tap	non-DC-neutral	$0.428\pm 0.018$	$\gamma{=}3.0$	differencing, not generic context

Table 7: Matched control battery. Best accuracy over

\gamma\in\{0.5,1,2,3\}

(asymmetric variants over six

(\gamma_{q},\gamma_{k})

pairs), mean

\pm

s.d. over three seeds. Protocol as in the text; chance

0.071

. The order column gives the member of the lattice ( 3 ) that the variant realises.

Attention side	Filter order	Integrator analogue	Measured
Standard bilinear	$(0,0)$	no lift: the obstruction	$0.033$
Query-only shear	$(1,0)$	first-order rule, wrong axis	$0.033$
Key-only shear	$(0,1)$	exact flow of $\tfrac{1}{2}\\|p_{k}\\|^{2}$ ; no swap symmetry	$0.949$
Asymmetric $\gamma_{q}\neq\gamma_{k}$	$(1,1)^{*}$	exact flow of a two-coefficient quadratic; no transport identity (C9)	$0.875$
Low-pass two-tap	non-DC-neutral	dissipative smoother	$0.428$
Learned two-tap	learned	discovers $(0,1)$ from data	$1.000$

Table 8: The correspondence of Fig. 1 , with the attention column measured. Every row lands where the filter-order account places it, including the two rows that are not the construction we advocate. Accuracies from Table 7 ; chance

0.071

Figure 4: Matched control battery: one protocol, seven mechanisms.

T{-}1{=}14

induction pairs,

d_{k}{=}16

d_{\mathrm{model}}{=}64

, three seeds

\times

3,000 steps, taps post- RoPE , value stream untouched, chance

1/14=0.071

. (a) Maximum accuracy attained by each variant over its coupling grid — a best-of-

k

statistic, upward biased as an estimate of held-out performance, but applied identically to every variant; error bars are the seed s.d. The key-only half-lift (

0.949

) outperforms the symmetric operator (

0.811

) by

13.8

pp: if single-layer induction accuracy is the objective, the half-lift is the better construction, and what the symmetric operator buys instead is structure (Table 1 ). (b) The coupling sweeps behind those maxima. The query-only half-lift never rises above the chance benchmark at any

\gamma

, consistent with Corollary 68 ; the low-pass member of matched parameter count rises but lags every differencing variant. (c) The learned two-tap filter’s converged coefficients against the fixed operator’s. A free filter with no constraint tying the streams selects the identity on the query stream and a pure first difference on the key stream. The shaded half-plane

b/a<0

is the high-pass differencing regime.

Figure 5: Bode forensics for the score stream. (a) The closed-form per-stream transfer function

|H(e^{j\omega})|^{2}=1+4\gamma(1+\gamma)\sin^{2}(\omega/2)

(Prop. 13 ) is high-pass for every

\gamma>0

and reduces to all-pass at

\gamma=0

(standard attention). The Nyquist gain is

(1+2\gamma)^{2}

. (b) Measured Pearson correlation between the theoretical transfer function and the measured AC power, per

\gamma

, taken before and after the softmax. Protocol: one single-layer PSA model retrained from scratch at each

\gamma

(

d_{k}{=}16

d_{\mathrm{model}}{=}64

T{=}64

with 30 induction pairs, 3,000 steps, fixed seed), evaluated on 200 held-out sequences; both spectra normalised by total AC power. Standard attention is order-

(0,0)

and produces a flat AC power spectrum, giving a degenerate correlation.

Figure 6: Embedding-axis low-pass induction filter (cascade RoPE $\to$ PSA ). (a) Recall accuracy as a function of momentum coupling $\gamma$ and RoPE centre frequency $\theta$ , on a 2,000-experiment sweep over $\gamma\in[0,3]$ and $\theta\in[0.02,\pi]$ (per-configuration protocol and seeds in Appendix A16). The high-fidelity induction zone concentrates at low $\theta$ and $\gamma\in[0.4,1.4]$ . (b) Momentum gain at each $\theta$ (peak accuracy over $\gamma\in[0,3]$ at that $\theta$ , minus the $\gamma{=}0$ baseline) against $\theta$ on the same sweep, $n=20$ centre frequencies: a negative association, Pearson $r=-0.679$ ( $p=9.9\times 10^{-4}$ ). Pearson correlation measures linear association and does not establish monotonicity; no rank statistic is reported.

Figure 7: The two-branch law of Theorem 64 against the score-model Monte-Carlo grid. Protocol: eighteen

(d_{k},T)

points, five seeds and

60{,}000

trials per point,

\gamma_{c}

extracted at the midpoint between chance and the measured plateau on a 61-point coupling grid; no parameter is fitted on the

d_{k}

axis. All three panels concern the reduced model

\mathcal{G}(d_{k},T;\gamma)

of Definition 62 and simulation of it, not trained transformers. Observable. Every curve from which a

\gamma_{c}

is extracted plots the competition observable

\mathcal{A}(\gamma)=\mathbb{E}[\mathrm{softmax}_{j}(s)_{j^{*}}]

, expected target softmax mass, and not argmax classification accuracy: that is the quantity Theorem 64 governs and the quantity that transitions. The argmax ordering within

\mathcal{G}

is invariant in

\gamma

, exactly as the theorem states, so no argmax curve could show a transition in

\gamma

and none is plotted. The one place argmax appears is certificate C7, which checks the plateau

p_{\infty}

against direct argmax Monte Carlo; that is legitimate because

p_{\infty}

is the

\gamma\to\infty

limit of

\mathcal{A}

, where softmax mass and argmax indicator probability coincide. Comparison with trained-model accuracy (§ 7.11 ) is therefore a comparison between a mass and an accuracy, and is labelled approximate transfer wherever it appears. (a)

\gamma_{c}

against

d_{k}

at three context lengths. Dashed segments are the condensed branch, solid curves the mean-field branch, and the faint vertical stubs at

d_{k}^{*}{=}8L

mark the value gap between them: the piecewise law is discontinuous at the crossover while the measurement is not, which is what places the largest errors nearby. Error bars are the seed s.d. magnified

8\times

to be visible. (b) Predicted against measured on the nine mean-field points, with

\pm 5\%

and

\pm 10\%

bands. Pearson

r=0.9876

, median relative error

2.6\%

, zero fitted parameters. (c) Relative error against proximity to the crossover. The trend is positive but is not resolved as monotone at

n=9

(Spearman

\rho=0.27

p=0.49

Figure 8: The condensed branch and the accuracy ceiling. (a)

\gamma_{c}

against context length at fixed

d_{k}{=}16

, ten lengths, same extraction procedure as Fig. 7 . Circles are the eight condensed-branch points (

d_{k}<8L

), squares the two boundary-adjacent points on the mean-field side with their percentage errors annotated. On branch:

r=0.9995

, median error

3.1\%

. Error bars are the seed s.d. magnified

10\times

. (b) Pointwise estimates of the condensation constant on two orthogonal axes. The grid estimates (circles, mean and

\pm 1

s.d. shaded) are flat to

\pm 2\%

; the

T

-axis estimates (diamonds) drift systematically with

T

, which is the residual the single-constant description leaves. (c) The zero-parameter score-model argmax limit ( 51 ) as a function of head width at three context lengths, with the four certificate C7 checkpoints against direct argmax Monte Carlo marked as rings. Largest discrepancy

9\times 10^{-4}

Figure 9: The trained-model

d_{k}

ladder. Protocol:

T{-}1=14

induction pairs,

d_{\mathrm{model}}=4d_{k}

d_{\mathrm{ff}}=16d_{k}

, three seeds and 3,000 steps per coupling on an eleven-point

\gamma

grid,

T=15

d_{k}^{*}=21.1

. (a) Trained

\gamma_{c}

against

d_{k}

on log–log axes. The fitted slope is

-1.13

; the two mean-field points alone give

-0.91

; the score model’s asymptote is

-1/4

, anchored at

d_{k}=16

for comparison of slope. At

d_{k}=4

no transition is measurable. (b) Trained plateaus against the score-model argmax limit

p_{\infty}

of ( 51 ), evaluated with

m=T-2

. The two track in shape (

r=0.945

) and not in level;

p_{\infty}

is the limit of the score model and is not a bound on trained accuracy, which exceeds it at

d_{k}=16

. (c) The measured operating-point ratio

\gamma\|p\|/\|q\|

\gamma=2

. The shaded band is where a small-signal expansion would be quantitative; the trained models are outside it, and the transition theory does not require them to be inside it.

Figure 10: Coupling, depth and length characterisation. This is measurement and not theorem: see § 5.2 . Protocol:

d_{k}{=}16

, 14 induction pairs, two seeds per configuration, chance

0.071

. (a) Pre-softmax score RMS by layer in the depth-3 stack. The pre-registered check for growth below

3\times

the layer-1 value (dashed line, drawn at

3\times

the control’s layer-1 value) is failed by the

\gamma{=}0

control, not by the sheared runs. (b) Evaluation-coupling sweep for a model trained at

\gamma{=}2

with weights fixed. Filled markers lie within

90\%

of the best accuracy; the plateau spans

\gamma\in[1.5,4]

. (c) Accuracy at extended context length against the chance level at that length, for a model trained at 14 pairs. Ratios to chance are annotated. RoPE extrapolation is a confound and the vocabulary budget caps the extension at about

2.2\times

Figure 11: Where induction becomes decodable, and where it is computed. (a) Held-out ridge-probe accuracy for the next token read from the residual stream after a single attention layer, over depth and coupling; $192$ models, three seeds. The $\gamma=0$ column sits at chance ( $0.0078$ ) throughout. (b) The coupling $\gamma_{1/2}$ at which the model adopts the single-layer route, against the operating-point curve $\gamma^{*}(N)$ of Appendix A6 — imported, and measured here to be non-transferable: the in-configuration optimum is $\gamma^{*}=0$ at every depth — and the reduced model’s $\gamma_{c}$ , drawn as a reference level rather than a prediction under test. The two curves do not cross within the grid; extrapolating the fitted $\gamma^{*}$ places the crossing near $N\approx 25$ , against a registered prediction of $N\approx 8.3$ . That extrapolation falsifies the pre-registration and carries no further mechanistic weight. (c) Accuracy lost to $\gamma$ -ablation, averaged over $\gamma\geq 0.5$ . Zeroing the shear in the first block alone costs almost exactly what zeroing it everywhere costs, and zeroing it in the last block costs nothing — except at $N\geq 12$ , where the first-block and global bars separate.

accuracy drop	$N{=}2$	$4$	$5$	$6$	$7$	$8$	$12$	$16$
global $\gamma\!\to\!0$	$0.861$	$0.880$	$0.885$	$0.888$	$0.888$	$0.889$	$0.891$	$0.895$
first block only	$0.856$	$0.874$	$0.878$	$0.879$	$0.878$	$0.878$	$0.849$	$0.805$
last block only	$0.021$	$0.008$	$0.003$	$0.002$	$0.001$	$0.000$	$0.000$	$0.000$
first $-$ global	$-0.005$	$-0.006$	$-0.007$	$-0.009$	$-0.010$	$-0.011$	$-0.042$	$-0.090$

Table 9: Accuracy lost to

\gamma

-ablation, per depth, averaged over

\gamma\geq 0.5

and three seeds. Through

N=8

the first-block ablation reproduces the global one to within

0.011

and the last-block ablation costs nothing; from

N=12

the first and global rows separate, which is the distributed corner discussed below. Plotted as Fig. 11 (c).

	Supports	Verifies	Outcome
C1	Thm. 64 (2)	Gaussian MGF by direct integration; annealed exponent $\Lambda=L+2\gamma^{4}$	exact, symbolic
C2	Thm. 64 (4)	$\max_{\gamma}(a-\Lambda)=d_{k}/8-L$ at $\gamma^{2}=\sqrt{d_{k}}/4$ ; crossover $d_{k}^{*}=8L$	exact, symbolic
C3	Thm. 64 (3)	quartic root closed form, $(L^{2}/d_{k})^{1/4}$ asymptote, monotonicity in both arguments	exact, symbolic
C4	Prop. 36 (i)–(ii)	asymmetric shear is symplectic and preserves $H_{\mathrm{joint}}$	residual exactly $0.0$ at $d_{k}{=}64$
C5	Prop. 36 (iii)	$s^{\mathrm{asym}}-s^{\mathrm{sym}}=\varepsilon(q^{\top}p_{k}+\gamma_{q}p_{q}^{\top}p_{k})$	exact, symbolic
C6	Thm. 38 (i)	$M_{\gamma}^{\top}JM_{\gamma}=J$ symbolically and numerically	residual $0.0$ exactly at $d_{k}{=}64$

Table 10: The certificate set. Column three states what is verified, column four the outcome as printed by the notebook.

configuration	$d$	$T$	overhead
small on-device	1024	2048	$1.8\times 10^{-4}$
mid on-device	2048	4096	$9.2\times 10^{-5}$
server class	4096	4096	$5.2\times 10^{-5}$
long context	4096	32768	$2.6\times 10^{-5}$

Table 11: Arithmetic overhead of the shear, from ( 64 ). The fraction falls as context grows, because the attention term dominates.

$\gamma$	range factor $1+2\gamma$	$\Delta b$ (bits)	INT8 levels retained
0.05	1.10	0.14	233/256
0.10	1.20	0.26	213/256
0.20	1.40	0.49	183/256
0.40	1.80	0.85	142/256

Table 12: Worst-case quantisation cost of the shear, from Proposition 77 . The last column is the number of INT8 levels left after widening the scale by

1+2\gamma

quantity	PSA relative to standard attention
parameters	identical: the operator adds none
KV-cache size	identical; cache the sheared keys (Prop. 73 )
extra persistent state	$2dL$ scalars, independent of $T$ ( $1/T$ of the cache)
FLOPs per token, per layer	$+6d$ , i.e. $6/(24d+4T)$ ; $<10^{-4}$ in practice
prefill	same fraction; fully vectorised, no recurrence
prefix caching	unaffected (causal two-tap FIR, Prop. 76 )

Table 13: The deployment bill, relative to standard attention. Every row is exact arithmetic, not measurement.

Appendix figures & tables184 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 12: RoPE rotations across frequency blocks. Effect on four representative frequency blocks of a projected query stream, for a 20-token sequence with $d_{\mathrm{model}}{=}64$ , $d_{k}{=}32$ . Blocks with large $\theta_{m}$ (e.g. block 0, $\theta{=}1.0000$ ) rotate rapidly with position; blocks with small $\theta_{m}$ (block 15, $\theta{=}0.0002$ ) barely rotate over the full sequence and therefore carry long-range positional information. Original query dots (blue) and their rotated counterparts (orange) lie on coincident norm shells, confirming norm preservation; the verification np.allclose( $\|q\|$ , $\|\tilde{q}\|$ ) returned True to double precision.

Figure 13: EMA momentum magnitude as a function of $\beta$ . Momentum magnitude $\|p_{q,n}\|$ versus position for four values of $\beta$ , with a 50-token sequence generated from structured (oscillating) embeddings. The effective memory window is approximately $(1-\beta)^{-1}$ : $\beta{=}0.5$ yields a 2-step window and tracks the instantaneous kinematic difference closely; $\beta{=}0.95$ yields a 20-step window and produces a heavily smoothed trajectory. All four traces start at $\|p_{q,n}\|{=}0$ by construction.

Figure 14: Causal attention patterns across four momentum strengths at $\beta{=}0.9$ . The upper triangle is masked. Visually the patterns are closely similar across the four panels: the entropy and per-row spread are identical to three decimals in all four cases (average entropy $2.549$ , mean standard deviation $0.0410$ ). This is the expected behavior at $\beta{=}0.9$ : the momentum signal is heavily damped, and the augmentation contributes only a small perturbation to scores already in the high-temperature regime. The next subsection quantifies this through the four-term decomposition.

Figure 15: Attention score decomposition at $i{=}20$ , $\gamma{=}0.15$ , $\beta{=}0.9$ . The four terms of Equation ( 70 ) are plotted against key index $j$ . The position–position term dominates in magnitude; the two cross terms are of comparable, small size; the momentum–momentum term is three orders of magnitude below the leader, consistent with the $\gamma^{2}$ scaling and with the Section A3 finding that the EMA at $\beta=0.9$ damps the momentum signal severely.

Component	Mean $\|\cdot\|$	Ratio to $T_{1}$
$T_{1}$ : $\tilde{q}_{i}^{\!\top}\tilde{k}_{j}$ (position–position)	$0.0523$	$1.000$
$T_{2}$ : $\gamma_{Q}p_{q,i}^{\!\top}\tilde{k}_{j}$ (Q-mom $\cdot$ K-pos)	$0.0008$	$0.015$
$T_{3}$ : $\gamma_{K}\tilde{q}_{i}^{\!\top}p_{k,j}$ (Q-pos $\cdot$ K-mom)	$0.0009$	$0.017$
$T_{4}$ : $\gamma_{Q}\gamma_{K}p_{q,i}^{\!\top}p_{k,j}$ (momentum–momentum)	$1.3\!\times\!10^{-5}$	$2.5\!\times\!10^{-4}$

Appendix

Table 14: Mean absolute magnitude of the four score components at query position

i{=}20

, with

\gamma_{Q}=\gamma_{K}=0.15

and

\beta=0.9

. The position–position term dominates by approximately

65\times

over either cross term; the two cross terms are nearly identical, a sanity check on Q/K symmetry of the EMA.

Figure 16: Attention distribution at $i=20$ with and without momentum. Left: weight over the 21 accessible keys; right: pointwise difference $w^{\mathrm{mom}}-w^{\mathrm{no\text{-}mom}}$ . The maximum attention weight relocates from position 20 (self) to position 10 when momentum is enabled, and the total absolute change in the distribution over the 21 keys is $0.0109$ . The effect is small in magnitude but structural: the momentum reweights attention towards a specific historical position rather than smearing the distribution.

Figure 17: Causal attention as a function of $\gamma$ at $\beta=0.9$ . Differences between panels are visible only on fine inspection. They become dramatic once $\beta$ is dropped, as Section A3 demonstrates.

Parameter	Value
Architecture	4-layer transformer decoder, causal mask
$d_{\mathrm{model}}/n_{\mathrm{heads}}/$ head dim	$128/4/32$
FFN dimension	512
Dropout	0.1
Vocabulary size	202
Momentum coupling	$\gamma=0.5$ (fixed)

Appendix

Table 15: Model and training configuration for the

\beta

-sweep.

$\beta$	$L=4$	$L=8$	$L=12$	$L=16$	$L=20$
0.0	94.0	86.5	49.3	12.1	5.1
0.1	87.9	71.3	20.0	6.7	3.9
0.2	74.7	40.7	11.2	5.5	3.0
0.3	56.7	21.4	8.1	3.9	3.5
0.4	43.6	16.0	6.5	3.9	2.9
0.5	34.3	12.0	6.1	3.3	2.9

Appendix

Table 16: Mean test accuracy (%) on associative recall, by

\beta

and chain length

L

. Three seeds per cell. Momentum coupling

\gamma=0.5

. The

\beta=0.0

row dominates across all chain lengths; accuracy collapses monotonically as

\beta

increases.

Quantity	Value
Mean accuracy, $\beta=0.0$ (pure high-pass, momentum)	$49.4\%$
Mean accuracy, $\beta=0.9$ (heavy low-pass, momentum)	$9.5\%$
Mean accuracy, vanilla attention (no momentum)	$10.0\%$
Pearson $\rho(\|H(\pi)\|,\text{accuracy})$	$0.507$
Correlation $p$ -value	$3.47\!\times\!10^{-11}$
$R^{2}$	$0.257$

Appendix

Table 17: Headline numbers averaged across all chain lengths and seeds.

Figure 18: Main results of the $\beta$ -sweep. (A) Accuracy vs. $\beta$ per chain length, with vanilla-baseline horizontal lines (dashed) per length; in every case the $\beta{=}0$ curve lands above the vanilla line and the $\beta\!\to\!0.9$ limit coincides with it. (B) Accuracy vs. Nyquist gain $|H(\pi)|=(1-\beta)/(1+\beta)$ : monotone increasing trend with fitted slope well separated from zero ( $\rho{=}0.51,\ p{=}3.5\!\times\!10^{-11}$ ). (C) Mean gain over vanilla per $\beta$ : positive at $\beta\leq 0.2$ , crosses zero around $\beta=0.5$ , slightly negative at $\beta\geq 0.6$ . (D) Accuracy heatmap over $(\beta,L)$ (numerically Table 16 ). (E) Theory vs. experiment: the blue curve is the analytical Nyquist gain scaled to $[0,100]$ ; the red curve is empirical accuracy averaged over chain lengths. The two share a shape but the empirical curve is substantially attenuated.

Figure 19: Detailed per-chain-length decomposition. Top-left: $\beta{=}0$ vs. $\beta{=}0.9$ bar comparison per chain length — the $\beta{=}0$ bars are uniformly taller at $L=4,8,12$ and the gap closes only at $L\geq 16$ because both configurations approach the random baseline. Top-center: mean gain over vanilla per $\beta$ . Top-right: Nyquist gain vs. $\beta$ with the ten sampled points marked. Bottom-left: Cohen’s $d$ ( $\beta{=}0$ vs. $\beta{=}0.9$ ) per chain length, with the conventional $d=0.8$ “large effect” threshold marked; at $L=4$ and $L=8$ the observed $d$ exceeds the threshold comfortably. Bottom-center: three-way comparison of $\beta{=}0$ , $\beta{=}0.9$ , and vanilla. Bottom-right: narrative summary panel.

Parameter	Value
Architecture	4-layer transformer decoder, 4 heads, head dim 32
$d_{\mathrm{model}}/d_{k}/d_{\mathrm{ff}}$	$128/32/512$
Dropout	0.1
Vocabulary size	200
Chain length $L$	12 KV pairs (sequence length 25)
Momentum form	Pure high-pass ( $\beta=0$ )

Appendix

Table 18: Model and training configuration for the

\gamma

-sweep.

$\gamma$	Sinusoidal PE (%)		RoPE (%)
0.00	4.9 $\pm$	1.2	5.5 $\pm$	0.5
0.01	4.4 $\pm$	0.3	6.5 $\pm$	1.2
0.02	5.1 $\pm$	1.0	5.1 $\pm$	0.7
0.03	4.3 $\pm$	1.1	5.4 $\pm$	0.3
0.04	5.0 $\pm$	0.2	5.9 $\pm$	0.4
0.05	4.1 $\pm$	0.8	7.1 $\pm$	0.2

Appendix

Table 19: Test accuracy (%) on associative recall (

L=12

, vocab 200), mean

\pm

standard deviation over three seeds, for both sinusoidal PE and RoPE . The phase transition is visible in both columns as a sigmoidal jump from near-random to near-perfect accuracy between

\gamma\!\in\![0.20,0.40]

Encoding	$\gamma_{c}$	Peak acc.	at $\gamma$
RoPE	$0.225$	$99.4\%$	$3.00$
Sinusoidal PE	$0.275$	$99.5\%$	$1.50$
Ratio $\gamma_{c}^{\mathrm{sin}}/\gamma_{c}^{\mathrm{RoPE}}$	$1.22\times$
Implied $r$	$0.818$

Appendix

Table 20: Detected phase-transition parameters under post-encoding placement.

Figure 20: Phase transition: RoPE vs. sinusoidal PE. (A) Full $\gamma$ -range, log scale: both curves show near-random accuracy for $\gamma<0.1$ , sharp transition between $\gamma=0.2$ and $\gamma=0.4$ , saturation near $99\%$ at $\gamma\geq 0.5$ . Vertical dashed lines mark detected $\gamma_{c}$ values ( $0.225$ for RoPE , $0.275$ for sinusoidal). (B) Critical region $[0,0.5]$ , linear scale, with the critical-region band shaded. (C) Fine-grained zoom of $[0,0.25]$ showing the leading edge: RoPE begins accelerating earlier (by $\gamma\sim 0.12$ – $0.14$ ), both converge by $\gamma=0.30$ . (D) Susceptibility $\mathrm{d\,acc}/\mathrm{d}\gamma$ peaks sharply at the respective $\gamma_{c}$ and is an order of magnitude larger for sinusoidal PE ( $\sim 1200\%$ /unit) than for RoPE ( $\sim 600\%$ /unit), reflecting a narrower transition width for sinusoidal. (E) Bar comparison of the two $\gamma_{c}$ estimates, annotated with the $1.22\times$ ratio. (F) Headline summary panel.

Figure 21: Detailed transition analysis. (A) S-curves from cubic interpolation between sampled $\gamma$ -values, with $\gamma_{c}$ vertical lines; both qualitatively sigmoidal. (B) Susceptibility (piecewise derivative used to detect $\gamma_{c}$ ); the sinusoidal curve shows a taller, narrower peak at $\gamma\sim 0.275$ , RoPE a broader peak near $\gamma=0.225$ . (C) Bar summary of the two $\gamma_{c}$ values and their $1.22\times$ ratio.

Parameter	Value
Vocabulary size $V_{\text{vocab}}$	$64$
$d_{\mathrm{model}}$	$64$
$n_{\text{heads}}\times d_{\text{head}}$	$4\times 16$
$d_{\text{ff}}$	$256$
$N_{\text{layers}}$	$1$ (strictly)
RoPE base	$10\,000$

Appendix

Table 21: Experiment 16 configuration.

$\gamma$	Acc (%)	$\gamma$	Acc (%)	$\gamma$	Acc (%)
0.00	1.2	1.00	70.2	2.40	75.0
0.05	4.4	1.10	69.4	2.50	77.6
0.10	4.6	1.20	62.6	3.00	76.8
0.15	6.4	1.30	65.8	3.20	75.6
0.20	7.8	1.40	70.4	3.50	79.0
0.30	18.2	1.50	74.8	4.00	83.4

Appendix

Table 22: Experiment 16: accuracy as a function of momentum coupling

\gamma

N{=}1

, single seed. Random chance is

1.56\%

; the

\gamma{=}0

baseline lands at

1.2\%

(essentially chance, confirming that single-layer standard attention cannot perform induction); the peak is

\mathbf{83.4\%}

\gamma=4.0

, a

\mathbf{+82.2}

percentage-point gain over baseline and

\mathbf{53\times}

the random-chance accuracy. The dataset is deliberately harder than the disjoint-alphabet phase-transition configuration of Section A4 : a single shared

V{=}64

alphabet (no role-flag information from token IDs),

14

K-V pairs (vs.

8

there), and explicit padding tokens in the sequence; this design tests robustness rather than the absolute performance ceiling.

$N$	Predicted $\gamma^{*}$	$\gamma$ sweep
1	$4.00$	$\{0,1,2,3,4,5,6\}$
2	$2.00$	$\{0,0.5,1,1.5,2,2.5,3\}$
4	$1.00$	$\{0,0.25,0.5,0.75,1,1.25,1.5\}$
8	$0.50$	$\{0,0.1,0.25,0.4,0.5,0.6,0.75\}$

Appendix

Table 23: Experiment 17

\gamma

grid by depth (centered on predicted

\gamma^{*}=4/N

$N$	accuracy (%) at each $\gamma$
$1$	$0{:}2.2$	$1.0{:}65.4$	$2.0{:}76.4$	$3.0{:}81.0$	$\mathbf{4.0{:}83.0}$	$5.0{:}79.4$	$6.0{:}77.8$
$2$	$0{:}2.8$	$0.5{:}36.8$	$1.0{:}66.6$	$1.5{:}73.6$	$2.0{:}77.6$	$2.5{:}80.2$	$\mathbf{3.0{:}86.0}$
$4$	$0{:}3.4$	$0.25{:}8.6$	$0.5{:}31.2$	$0.75{:}50.4$	$1.0{:}65.4$	$\mathbf{1.25{:}71.2}$	$1.5{:}64.2$
$8$	$0{:}2.4$	$0.1{:}3.8$	$0.25{:}6.6$	$0.4{:}13.4$	$0.5{:}20.2$	$0.6{:}30.4$	$\mathbf{0.75{:}37.6}$

Appendix

Table 24: Experiment 17: accuracy (%) for every

(N,\gamma)

configuration, single seed. Best per-depth entries are bold .

Figure 22: Experiment 17: the four-panel scaling-law figure. (A) Accuracy vs. $\gamma$ by depth $N$ ; the curves shift leftward as $N$ grows, exactly as the scaling law predicts. (B) Optimal coupling $\gamma^{*}$ vs. $N$ , with the fitted $4.17/N^{0.73}$ ( $R^{2}=0.947$ ) and the theoretical $4/N$ overlay ( $\alpha=1$ ). (C) Log–log plot showing the power-law slope of $-\alpha\approx-0.73$ . (D) The product $\gamma^{*}\!\cdot N$ per depth: values $4.0,6.0,5.0,6.0$ are not constant, consistent with $\alpha<1$ .

Figure 23: Experiment 17 summary figure. The per-depth accuracy-vs- $\gamma$ families and the fitted scaling curve are reproduced alongside the peak-accuracy bar chart.

Parameter	Value
Architecture	nano (identical to E16, E17)
Chain length $L$	$30$ KV pairs
Sequence length	$64$ tokens
Depth grid $N$	$\{1,2,3,4,6,8\}$ — six values
Seeds per config	$3$
$\gamma$ grid size	15 values per depth, centered on predicted $\gamma^{*}$

Appendix

Table 25: Experiment 18 configuration.

$N$	Predicted $\gamma^{*}$	Observed $\gamma^{*}$	Best acc. (%)	SEM (%)	Baseline (%)
$1$	$4.17$	$2.00$	$\mathbf{57.4}$	$3.4$	$1.7$
$2$	$2.51$	$2.75$	$\mathbf{61.9}$	$4.1$	$2.0$
$3$	$1.87$	$2.00$	$\mathbf{29.0}$	$15.6$	$1.3$
$4$	$1.52$	$3.00$	$\mathbf{51.2}$	$11.1$	$1.6$
$6$	$1.13$	$2.50$	$\mathbf{40.3}$	$16.2$	$1.4$
$8$	$0.91$	$2.50$	$\mathbf{24.3}$	$12.6$	$1.7$

Appendix

Table 26: Experiment 18: mean test accuracy (%) across three seeds at chain length

L=30

, for each

(N,\gamma)

configuration reported by the notebook. Predicted

\gamma^{*}

from E17 in the second column. Per-row best is bold .

Parameter	E17 ( $L=14$ )	E18 ( $L=30$ )
$\gamma_{0}$	$4.17\pm 0.41$	$2.73\pm 0.47$
$\alpha$	$0.73\pm 0.15$	$0.10\pm 0.14$
$R^{2}$	$0.947$	$-0.607$
Seeds/config	1	3
Depths fitted	4	6

Appendix

Table 27: E17 vs. E18 scaling-law fit comparison.

Figure 24: Experiment 18 scaling-law validation at $L=30$ . Accuracy-vs- $\gamma$ with error bars (three-seed SEM) by depth; the per-depth optima are scattered across $\gamma\!\in\![2.0,3.0]$ with overlapping error bars, which is the quantitative statement of why the fit fails.

Figure 25: Experiment 18 heatmap. Accuracy heatmap across all $(N,\gamma)$ configurations at $L=30$ . White stars mark the per-depth argmax $\gamma^{*}$ . The heatmap confirms the flat, noisy accuracy plateau at each depth $\geq 3$ — the primary reason a clean scaling law does not emerge in this regime.

Parameter	Value
Architecture	nano, post- RoPE PSA
$L$ (KV pairs)	$30$
Sequence length	$64$
Query position	$60=2L$
Training steps	$3\,000$
RoPE base	$10\,000$

Appendix

Table 28: Validation Suite v7 configuration.

$\gamma$	Acc (%)	$\gamma$	Acc (%)	$\gamma$	Acc (%)
0.00	1.3	1.20	66.9	2.40	60.3
0.20	2.0	1.40	70.4	2.60	59.3
0.40	9.3	1.60	71.5	2.80	49.0
0.60	29.8	1.80	72.6	3.00	29.4
0.80	45.5	2.00	74.2
1.00	57.6	2.20	73.2

Appendix

Table 29: v7 S1: single-layer

\gamma

-sweep, three-seed means.

Figure 26: v7 S1: single-layer induction phase transition at $L=30$ with three-seed means. (A) Accuracy bars with the $\gamma=0$ “FAIL” bar highlighted in red and the $75\%$ benchmark dashed. (B) Phase-transition visual: static regime ( $\gamma<0.4$ , red shaded) and kinematic regime ( $\gamma\in[0.6,3.0]$ , green shaded), transition edge at $\gamma\approx 0.5$ .

$N$	$\gamma^{*}_{\text{obs}}$	Accuracy (%)	Baseline (%)
1	1.60	73.2	1.5
2	1.60	71.3	1.5
3	2.40	71.1	2.3
4	1.80	60.1	1.7
6	1.50	59.8	1.6
8	1.00	14.1	1.4

Appendix

Table 30: v7 S2: per-depth scaling law at

L=30

Figure 27: v7 S2: scaling-law figure at $L=30$ . Accuracy-vs- $\gamma$ family across six depths; fitted $1.88/N^{0.11}$ with $R^{2}=0.112$ ; product $\gamma^{*}\!\cdot N$ bar chart clearly not constant. The dotted theoretical $4/N$ reference curve is shown for comparison.

$\gamma$	Mean	Std	SS regime?
0.0	$1.4257$	$0.0217$	NO
0.6	$1.4529$	$0.0200$	NO
1.2	$1.5878$	$0.0254$	NO
1.8	$1.7043$	$0.0341$	NO
2.4	$1.7176$	$0.0415$	NO
3.0	$1.4991$	$0.0235$	NO

Appendix

Table 31: v7 S3: SS ratio

\|p_{q}\|/\|q^{\mathrm{rot}}\|

(mean

\pm

std, 300 probes).

$\gamma$	$T_{1}$ gap	$T_{2}$ gap	$T_{3}$ gap	$T_{4}$ gap	$\|T_{4}\|>\|T_{1}\|$ ?
0.0	$T_{2}=T_{3}=T_{4}=0$ by construction
0.6	$0.0142$	$0.0053$	$0.7608$	$0.4677$	YES
1.2	$0.0123$	$0.0048$	$0.8561$	$1.0382$	YES
1.8	$0.0128$	$0.0219$	$0.6866$	$1.2635$	YES
2.4	$0.0082$	$0.0134$	$0.5038$	$1.2378$	YES
3.0	$-0.0022$	$-0.0183$	$0.2375$	$0.6514$	YES

Appendix

Table 32: v7 S4–5: four-term gaps at

j^{*}

versus other positions, averaged over 100 probes.

$\gamma$	$\delta_{\mathrm{emp}}$	$\Delta_{\mathrm{symp,th}}$	Ratio	$H_{0}$
0.0	$0$	$0$	–	$6.7666$
0.6	$3.22\!\times\!10^{-4}$	$5.77\!\times\!10^{-2}$	$0.0056$	$6.3565$
1.2	$8.85\!\times\!10^{-4}$	$1.13\!\times\!10^{-1}$	$0.0078$	$3.4900$
1.8	$1.00\!\times\!10^{-3}$	$1.42\!\times\!10^{-1}$	$0.0071$	$2.0104$
2.4	$9.26\!\times\!10^{-4}$	$1.65\!\times\!10^{-1}$	$0.0056$	$1.3252$
3.0	$1.45\!\times\!10^{-3}$	$2.44\!\times\!10^{-1}$	$0.0060$	$1.2549$

Appendix

Table 33: v7 S6–7: AC energy, over-symplectic energy bound, and the derived ratio.

Figure 28: v7 S6–7: AC-energy diagnostics. (A) $\delta_{\mathrm{emp}}$ versus $\gamma$ on a log scale. (B) $\delta_{\mathrm{emp}}(t)$ versus position $t$ , showing a slow fall-off with position rather than a flat trace — the causal-task confound addressed by v9 C9. (C) Empirical $\delta_{\mathrm{emp}}$ alongside theoretical $\Delta_{\mathrm{symp,th}}$ bars; the empirical values are orders of magnitude below the symplectic-theory bound, confirming the bound.

$\gamma$	Mean $H_{0}$	Rel. range	Flat ( $<15\%$ )?
0.6	$6.5560$	$1.0322$	DRIFT
1.2	$3.5985$	$1.0438$	DRIFT
1.8	$2.0713$	$1.0500$	DRIFT
2.4	$1.3632$	$1.0350$	DRIFT
3.0	$1.2905$	$0.9886$	DRIFT

Appendix

Table 34: v7 S8:

H_{0}

relative range across positions

t\geq 4

$\gamma$	$\delta_{\mathrm{s,post}}$	$\delta_{\mathrm{ns,post}}$	HSF post	$\delta_{\mathrm{s,pre}}$	$\delta_{\mathrm{ns,pre}}$	HSF pre
0.0	0	0	–	0	0	–
0.6	$3.22\!\times\!10^{-4}$	$1.38\!\times\!10^{-3}$	$4.2896$	$0.3528$	$0.5206$	$1.4758$
1.2	$8.85\!\times\!10^{-4}$	$2.91\!\times\!10^{-3}$	$3.2906$	$0.6942$	$4.808$	$6.9255$
1.8	$1.00\!\times\!10^{-3}$	$2.29\!\times\!10^{-3}$	$2.2821$	$0.7582$	$2.674$	$3.5262$
2.4	$9.26\!\times\!10^{-4}$	$1.59\!\times\!10^{-3}$	$1.7184$	$0.8714$	$1.593$	$1.8284$
3.0	$1.45\!\times\!10^{-3}$	$1.73\!\times\!10^{-3}$	$1.1930$	$1.634$	$1.629$	$0.9967$

Appendix

Table 35: v7 S8b: HSF values and AC energies (v7 causal-mask fix applied).

Figure 29: v7 S8/S8b: $H_{0}$ conservation and HSF. (A) $H_{0}(t)$ flatness at five tested $\gamma$ : traces essentially constant across $4\!\leq\!t\!\leq\!60$ , with a sharp drop at the query position $t=60$ (an artefact of the final-token boundary). (B) HSF post- and pre-softmax across $\gamma$ ; HSF post monotone decreasing, HSF pre peaks at $\gamma=1.2$ (v9 C8 explains). (C) Pre-softmax $\delta$ for symmetric PSA versus asymmetric NS, confirming that NS produces substantially larger AC energy.

	Post-softmax			Pre-softmax (v7 fix)
$\gamma$	mean $r$	mean Nyq	theory Nyq	mean $r$	mean Nyq	theory Nyq
0.0	$0.0000$	$0.1273$	$1.0000$	$0.0000$	$0.0000$	$1.0000$
0.6	$0.9558$	$2.1589$	$4.8400$	$0.9982$	$75.97$	$4.8400$
1.2	$0.9278$	$2.5142$	$11.5600$	$0.9987$	$85.90$	$11.5600$
1.8	$0.9262$	$2.5378$	$21.1600$	$0.9991$	$100.73$	$21.1600$
2.4	$0.9331$	$2.5840$	$33.6400$	$0.9991$	$151.99$	$33.6400$

Appendix

Table 36: v7 S9: Bode forensics, post- and pre-softmax.

Figure 30: v7 S9: Bode forensics. Top row: post-softmax observed AC-power spectrum (solid) versus theory (dashed) at the six tested $\gamma$ ; Nyquist gain compressed relative to theory by $\sim 15$ – $18\times$ at $\gamma=3.0$ . Bottom row (v7 fix): pre-softmax AC power; the empirical curves (solid) track the theoretical $H^{2}(\omega)$ with $r$ up to $0.9994$ . The $\gamma=0$ baseline is zero at all frequencies, as expected.

$\gamma$	$E_{\mathrm{DC}}$	$E_{\mathrm{AC}}$	frac AC
0.0	$3.5955$	$8.8273$	$0.7194$
0.6	$8.9203$	$46.8653$	$0.8417$
1.2	$8.3484$	$60.8577$	$0.8827$
1.8	$6.1622$	$58.4498$	$0.9086$
2.4	$4.3498$	$64.2329$	$0.9405$
3.0	$4.3834$	$116.1734$	$0.9637$

Appendix

Table 37: v7 S10: spectral complementarity.

Figure 31: v7 S10/S11: spectral complementarity and Parseval. Panel covering frac AC versus $\gamma$ , energies $E_{\mathrm{DC}}$ and $E_{\mathrm{AC}}$ , and the Parseval identity test across the spectral $\gamma$ grid.

$\gamma$	$\mathbb{E}\\|\hat{q}\\|^{2}$ (LHS)	RHS	discrepancy	zero-sum mean viol.
0.0	$7.0491$	$7.0491$	$0$	$0$
0.6	$19.1152$	$19.1152$	$3.52\!\times\!10^{-8}$	$4.63\!\times\!10^{-8}$
1.2	$22.3856$	$22.3856$	$5.47\!\times\!10^{-8}$	$4.90\!\times\!10^{-8}$
1.8	$22.6853$	$22.6853$	$4.05\!\times\!10^{-8}$	$4.94\!\times\!10^{-8}$
2.4	$23.3368$	$23.3368$	$6.38\!\times\!10^{-8}$	$4.89\!\times\!10^{-8}$
3.0	$31.8507$	$31.8507$	$2.73\!\times\!10^{-9}$	$5.14\!\times\!10^{-8}$

Appendix

Table 38: v7 S11: Parseval identity check and zero-sum violation magnitudes.

band	$\theta$ (rad)	theory $\|\mathrm{err}\|$	observed rel. err
0	$1.000000$	$0.958851$	$0.407971$
1	$0.316228$	$0.314912$	$0.041205$
2	$0.100000$	$0.099958$	$0.004453$
3	$0.031623$	$0.031621$	$0.000411$
4	$0.010000$	$0.010000$	$0.412654$
5	$0.003162$	$0.003162$	$0.042761$

Appendix

Table 39: v7 S12: Coriolis per RoPE band at

\gamma=3.0

, sorted by

\theta

descending.

$\gamma$	mean	std	status
0.0	$1.4257$	$0.0217$	non-SS
0.6	$1.4529$	$0.0200$	non-SS
1.2	$1.5878$	$0.0254$	non-SS
1.8	$1.7043$	$0.0341$	non-SS
2.0	$\mathbf{1.7107}$	$\mathbf{0.0344}$	PASS — FP-18b CONFIRMED ( $\gamma^{*}$ )
2.4	$1.7176$	$0.0415$	non-SS

Appendix

Table 40: v8 C6 (FP-18b): SS ratio

\|p_{q}\|/\|q^{\mathrm{rot}}\|

across the v8 grid.

Figure 33: v8 C6: FP-18b direct measurement. Direct-measurement SS ratio at $\gamma^{*}=2.0$ (red marker). The threshold line at $1.0$ is crossed everywhere; the confirmation at $\gamma^{*}$ turns the v7 CHECK into a v8 PASS.

$\gamma$	$r_{\mathrm{buggy}}$	$r_{\mathrm{fixed}}$	$\Delta r$	verdict
0.0	$0.0000$	$0.0000$	$0$	baseline
0.6	$0.0000$	$0.9982$	$+0.9982$	FIX DECISIVE
1.2	$0.0000$	$0.9987$	$+0.9987$	FIX DECISIVE
1.8	$0.0000$	$0.9991$	$+0.9991$	FIX DECISIVE
2.0	$0.0000$	$0.9990$	$+0.9990$	FIX DECISIVE
2.4	$0.0000$	$0.9991$	$+0.9991$	FIX DECISIVE

Appendix

Table 41: v8 C7: Bode A/B, pre-softmax, same weights.

Figure 34: v8 C7: pre-softmax Bode A/B on the same trained weights. Top row: the fixed v7/v8 analysis produces the expected $H^{2}(\omega;\gamma)$ -shaped spectrum with $r$ up to $0.9994$ . Middle row: the buggy pre-v7 analysis produces all-NaN outputs and therefore $r=0$ . Bottom row: the bar summary across the entire $\gamma$ grid.

	Post-softmax		Pre-softmax (v7/v8 fix)
$\gamma$	mean $r$	mean Nyq	mean $r$	mean Nyq
0.0	$0.0000$	$0.1273$	$0.0000$	$0.0000$
0.6	$0.9558$	$2.1589$	$0.9982$	$75.97$
1.2	$0.9278$	$2.5142$	$0.9987$	$85.90$
1.8	$0.9262$	$2.5378$	$0.9991$	$100.73$
2.0	$\mathbf{0.9261}$	$\mathbf{2.5209}$ ( $\gamma^{*}$ )	$\mathbf{0.9990}$	$\mathbf{113.56}$ ( $\gamma^{*}$ )

Appendix

Table 42: v8 C8: post- and pre-softmax Bode (full v8 grid).

Figure 35: v8 C8: full Bode forensics with $\gamma^{*}=2.0$ included. Post-softmax $r$ near $0.93$ across the grid; pre-softmax $r$ up to $0.9994$ throughout.

Figure 36: v8 summary figure. FP-18b direct measurement, Bode A/B, and the updated scorecard in a single panel.

Parameter	Value
Architecture	nano (identical to v7/v8)
$\gamma$ grid	$\{0.0,0.4,0.6,0.8,\ldots,3.2\}$ — 16 pts
Fine T3/T4 grid	$\{0.60,0.65,0.70,\ldots,1.40\}$ — 17 pts
$d_{k}$ sweep (FP-21/22)	$\{8,16,32\}$ with $d_{\mathrm{model}}=4d_{k}$
Seeds (FP-24)	$\{42,123,456,789,1011\}$
Training steps	$3\,000$

Appendix

Table 43: v9 configuration.

$\gamma$	mean $r$	Nyq obs	Nyq theory	compression
0.0	$0.0000$	$0.1273$	$1.0000$	–
0.4	$0.9683$	$1.8704$	$3.2400$	$0.5773$
0.6	$0.9511$	$2.2682$	$4.8400$	$0.4686$
0.8	$0.9346$	$2.5167$	$6.7600$	$0.3723$
1.0	$0.9260$	$2.6563$	$9.0000$	$0.2951$
1.2	$0.9195$	$2.7735$	$11.5600$	$0.2399$

Appendix

Table 44: v9 C6: post-softmax Bode Nyquist versus theory, with compression ratio.

Figure 37: v9 C6: post-softmax Bode on the v9 16-point grid. Observed Nyquist versus theory and the compression ratio monotonically decreasing from $0.577$ to $0.065$ .

$\gamma$	$T_{1}$ gap	$T_{2}$ gap	$T_{3}$ gap	$T_{4}$ gap	dominant
0.60	$-1.2583$	$-1.2967$	$-1.2491$	$-1.2705$	$T_{3}$
0.65	$-1.2216$	$-1.2603$	$-1.2156$	$-1.2382$	$T_{3}$
0.70	$-1.1683$	$-1.2063$	$-1.1630$	$-1.1857$	$T_{3}$
0.75	$-1.1085$	$-1.1459$	$-1.1028$	$-1.1237$	$T_{3}$
0.80	$-1.0485$	$-1.0853$	$-1.0421$	$-1.0627$	$T_{3}$
0.85	$-0.9963$	$-1.0337$	$-0.9881$	$-1.0119$	$T_{3}$

Appendix

Table 45: v9 C7:

T_{3}

T_{4}

gaps at the induction target on fine

\gamma

grid.

Figure 38: v9 C7: $T_{3}\to T_{4}$ crossover. $\gamma T_{3}$ and $\gamma^{2}T_{4}$ curves cross at $\gamma\approx 0.97$ . Below crossover, $T_{3}$ (linear in $\gamma$ ) dominates; above, the $\gamma^{2}T_{4}$ term overtakes through its quadratic growth.

$\gamma$	HSF post	HSF pre	pre $\geq$ post?	regime
0.4	$1.9538$	$0.8210$	FAIL	low- $\gamma$ Fréchet
0.6	$4.2608$	$1.4674$	FAIL	low- $\gamma$ Fréchet
0.8	$2.3820$	$2.3097$	FAIL	low- $\gamma$ Fréchet
1.0	$3.5909$	$6.8612$	OK	interior
1.2	$3.2976$	$6.9316$	OK	interior
1.4	$3.2228$	$7.4056$	OK	interior

Appendix

Table 46: v9 C8: HSF pre/post ordering across the v9 grid.

Figure 39: v9 C8: HSF pre vs post. HSF pre/post analysis across the v9 grid, with the Fréchet boundary regions highlighted. Interior regime $\gamma\!\in\![1.0,2.4]$ shows the predicted ordering; the two extremes are explained by softmax non-linearity at low $\gamma$ and PSA over-rotation at high $\gamma$ .

$\gamma$	causal rel. range	stationary rel. range	verdict
0.6	$0.0370$	$0.0550$	CHECK (stat)
1.2	$0.0366$	$0.0362$	PASS (stat flat)
1.8	$0.0262$	$0.0496$	PASS (stat flat)
2.0	$0.0315$	$0.0576$	CHECK (stat)
2.4	$0.0337$	$0.0727$	CHECK (stat)
3.0	$0.0179$	$0.0839$	CHECK (stat)

Appendix

Table 47: v9 C9:

H_{0}

relative range, causal vs stationary sequences.

Figure 40: v9 C9: $H_{0}$ flatness causal vs stationary. Stationary traces are visibly flatter, confirming the scope qualification.

$\gamma$	PSA slope/pos	NS slope/pos	NS slope $>$ PSA?
0.6	$-0.001050$	$+0.000231$	CONFIRMED
1.2	$-0.000722$	$-0.003178$	CHECK (NS more negative)
2.0	$-0.000120$	$-0.000905$	CHECK (NS more negative)
3.0	$-0.000183$	$+0.000066$	CONFIRMED

Appendix

Table 48: v9 C10:

H_{0}

slope vs position, PSA vs NS, stationary sequences.

Figure 41: v9 C10: FP-5 NS energy drift. Slopes of $H_{0}(t)$ versus position for PSA and NS, across four $\gamma$ values. Bar chart shows NS systematically exhibits larger-magnitude slopes, consistent with non-symplectic drift.

$\gamma$	PSA acc	ablated acc	std attn acc	FP-16
1.0	$56.8\%$	$19.7\%$	$1.7\%$	CONFIRMED
1.4	$68.8\%$	$29.5\%$	$1.7\%$	CONFIRMED
2.0	$75.6\%$	$40.8\%$	$1.7\%$	partial
2.4	$59.1\%$	$46.3\%$	$1.7\%$	partial
3.0	$27.2\%$	$49.9\%$	$1.7\%$	partial (reversed)

Appendix

Table 49: v9 C11:

T_{4}

ablation across

\gamma

values.

Figure 42: v9 C11: $T_{4}$ ablation. PSA, ablated ( $T_{4}$ zeroed), and standard-attention accuracies across $\gamma$ . The collapse of ablated accuracy toward standard attention at $\gamma\!\in\![1.0,1.4]$ measures the size of the $T_{4}$ contribution; the residual gap above the standard-attention baseline is the contribution of the channels that survive the ablation.

$d_{k}$	$8L/d_{k}$	observed $\gamma_{c}$ (50% thresh.)	$\Delta\gamma$	peak acc
$8$	$4.14$	(never reaches 50%)	$\sim 2.8$	$9.0\%$ at $\gamma=2.8$
$16$	$2.07$	$1.0$	$0.80$	$74.6\%$ at $\gamma=2.0$
$32$	$1.03$	$0.4$	$0.40$	$93.4\%$ at $\gamma=2.0$

Appendix

Table 50: v9 C12:

\gamma_{c}

scaling results across

d_{k}

Figure 43: v9 C12: $\gamma_{c}$ and $\Delta\gamma$ vs head dimension $d_{k}$ . Three accuracy curves for $d_{k}\!\in\!\{8,16,32\}$ show the dramatic quality improvement with larger heads.

seed	PSA-1L ( $\gamma=2.0$ )	Std-1L	Std-2L
42	$74.4\%$	$1.7\%$	$1.1\%$
123	$72.7\%$	$1.8\%$	$1.4\%$
456	$58.7\%$	$2.1\%$	$2.0\%$
789	$79.4\%$	$1.0\%$	$1.8\%$
1011	$60.2\%$	$1.6\%$	$1.7\%$
mean $\pm$ std	$\mathbf{69.1\pm 8.2\%}$	$\mathbf{1.6\pm 0.4\%}$	$\mathbf{1.6\pm 0.3\%}$

Appendix

Table 51: v9 C13: single-layer PSA vs single-layer standard attention, five seeds.

Figure 44: v9 C13: FP-24 single-layer comparison. Five-seed bars for PSA $N=1$ , standard attention $N=1$ , standard attention $N=2$ . The PSA bar stands at $69\%$ with five-seed error bars; the two standard-attention bars sit on top of the chance line.

band	$\theta$	theory	observed	obs/th	joint align
0	$1.000000$	$0.958851$	$0.407971$	$0.43$	$0.9806$
1	$0.316228$	$0.314912$	$0.041205$	$0.13$	$0.9893$
2	$0.100000$	$0.099958$	$0.004453$	$0.04$	$1.0174$
3	$0.031623$	$0.031621$	$0.000411$	$0.01$	$1.0118$
4	$\mathbf{0.010000}$	$\mathbf{0.010000}$	$\mathbf{0.412654}$	$\mathbf{41.27}$	$\mathbf{0.9541}$
5	$0.003162$	$0.003162$	$0.042761$	$13.52$	$1.0593$

Appendix

Table 52: v9 C14: per-band Coriolis and

W_{Q}/W_{K}

joint alignment,

\gamma=3.0

Figure 45: v9 C14: band-4 Coriolis anomaly. Per-band observed/theory ratio (top) and joint $W_{Q}/W_{K}$ alignment (bottom) across the three analyzed $\gamma$ values. Band-4’s anomaly persists; its alignment varies with $\gamma$ but remains finite.

$\gamma$	$r_{\mathrm{PSA}}$	$r_{\mathrm{NS}}$	$\Delta r$	FP-29
0.6	$0.9528$	$0.0305$	$+0.9223$	CONFIRMED
1.2	$0.9225$	$0.8715$	$+0.0510$	CONFIRMED
1.8	$0.9206$	$0.9041$	$+0.0166$	CONFIRMED
2.0	$0.9206$	$0.9113$	$+0.0093$	CONFIRMED
2.4	$0.9282$	$0.9175$	$+0.0107$	CONFIRMED
3.0	$0.9166$	$0.9118$	$+0.0047$	CONFIRMED

Appendix

Table 53: v9 C15: PSA vs NS post-softmax Bode

r

Figure 46: v9 C15: FP-29 symmetric advantage. PSA (symmetric) versus NS (asymmetric) post-softmax Bode $r$ across the six tested $\gamma$ . PSA is above NS at every $\gamma$ ; the shaded region shows the $r$ advantage.

Figure 47: v9 summary dashboard. (A) Cumulative PASS/CHECK/open FP counts across v6, v7, v8, v9 — the green bar grows from $12$ in v6 to $21$ in v9; the open-FP column shrinks from $30$ to $13$ . (B) T3 $\to$ T4 crossover at $\gamma\approx 0.97$ . (C) FP-24 bar chart. (D) FP-5 drift slopes. (E) FP-29 $r$ -advantage. (F) FP-16 $T_{4}$ -ablation collapse.

Parameter	Value
Vocabulary size	$V=128$
$d_{\mathrm{model}}$ , $H$ , $d_{\mathrm{head}}$	$64$ , $4$ , $16$
Chain length $L$	$8$ KV pairs (sequence length $17$ )
Training	$2000$ samples, batch $128$ , $8$ epochs
Optimizer	AdamW, lr $10^{-3}$ , wd $10^{-2}$
RoPE family	Bandpass, bandwidth $b=0.2$

Appendix

Table 54: F1 configuration. A single-seed exploratory scan over six center frequencies and eight momentum couplings.

$\theta\backslash\gamma$	$0.00$	$0.25$	$0.50$	$0.75$	$1.00$	$1.50$	$2.00$	$3.00$
$\theta=2.00$	$0.75$	$0.80$	$0.80$	$\mathbf{0.82}$	$\mathbf{0.82}$	$0.75$	$0.72$	$0.56$
$\theta=1.00$	$0.71$	$0.75$	$\mathbf{0.80}$	$\mathbf{0.80}$	$0.79$	$0.76$	$0.68$	$0.54$
$\theta=0.50$	$0.69$	$0.79$	$\mathbf{0.79}$	$0.78$	$\mathbf{0.79}$	$0.76$	$0.70$	$0.55$
$\theta=0.25$	$0.70$	$0.76$	$\mathbf{0.80}$	$\mathbf{0.80}$	$0.79$	$0.75$	$0.68$	$0.55$
$\theta=0.10$	$0.67$	$0.80$	$0.87$	$\mathbf{0.89}$	$0.88$	$0.77$	$0.73$	$0.57$
$\theta=0.05$	$0.68$	$0.82$	$0.91$	$\mathbf{0.95}$	$0.93$	$0.85$	$0.74$	$0.57$

Appendix

Table 55: F1: full 48-point accuracy surface from the exploratory scan. Row =

\theta

(descending), Column =

\gamma

. Peak entries per row are in bold . Global peak:

(\theta,\gamma)=(0.05,0.75)

0.95

Figure 48: F1: bandpass RoPE exploratory phase diagram. Rows are center frequencies $\theta\!\in\!\{0.05,0.10,0.25,0.50,1.00,2.00\}$ (top to bottom in this ordering); columns are momentum couplings $\gamma\!\in\!\{0.0,0.25,0.5,0.75,1.0,1.5,2.0,3.0\}$ . The brightest cell is $(0.05,0.75)=0.95$ . The bottom-right corner ( $\gamma=3.0$ ) collapses uniformly to $\sim 0.55$ regardless of $\theta$ , indicating that very large momentum coupling dominates the signal globally.

Parameter	Value
Architecture	nano single-block (identical to F1)
$V$ , $d_{\mathrm{model}}$ , $H$ , $d_{\mathrm{head}}$	$128$ , $64$ , $4$ , $16$
Chain length $L$	$8$ KV pairs
Training	$2000$ samples, batch $128$ , $8$ epochs, AdamW lr $10^{-3}$
RoPE family	Bandpass, $b=0.2$
$\theta$ grid (15 log-spaced)	from $0.03$ to $3.0$ rad

Appendix

Table 56: F2 configuration. Architecture identical to F1; sweep grid substantially denser, three seeds per cell.

$\gamma$	mean acc	std	$\gamma$	mean acc	std
$0.00$	$0.356$	$0.002$	$0.90$	$0.593$	$0.041$
$0.10$	$0.393$	$0.005$	$1.00$	$0.581$	$0.046$
$0.20$	$0.428$	$0.004$	$1.20$	$0.545$	$0.053$
$0.30$	$0.472$	$0.008$	$1.65$	$0.484$	$0.050$
$0.40$	$0.511$	$0.015$	$2.10$	$0.436$	$0.033$
$0.50$	$0.546$	$0.021$	$2.55$	$0.407$	$0.034$

Appendix

Table 57: F2

\gamma

-sweep at the peak frequency

\theta=0.0300

, three-seed mean and standard deviation. Inverted-U with peak at

\gamma=0.80

$\theta$	peak accuracy	at $\gamma$
$0.030$	$0.599$	$0.80$
$0.080$	$0.57$	$0.70$
$0.300$	$0.43$	$0.4$ – $1.0$
$3.000$	$0.49$ (wrap-around)	$0.80$

Appendix

Table 58: F2 cross-section by frequency: peak accuracy at four representative

\theta

values. The peak is monotone decreasing with

\theta

for

\theta\!\leq\!0.3

Figure 49: F2: low-pass induction heatmap with iso-accuracy contours. $\theta$ is on a log scale, $\gamma$ is linear. The $0.5$ -contour (yellow) encloses a region around $(\theta,\gamma)\!\in\!([0.03,0.2],[0.4,1.5])$ — the high-fidelity induction zone. The upper band at $\theta\!\in\![1,3]$ is the noise-corrupted regime.

Figure 50: F2: accuracy-versus- $\gamma$ slices at four $\theta$ . Peak height monotone decreasing with $\theta$ for $\theta\!\leq\!0.3$ ; $\theta=3.0$ shows the wrap-around recovery.

Figure 51: F2: accuracy-versus- $\theta$ slices at five $\gamma$ . For every $\gamma>0$ , accuracy falls sharply around $\theta\!\approx\!0.2$ — the empirical cut-off frequency $\theta_{c}$ of the low-pass filter.

Parameter	Value
Architecture	nano single-block (identical to F1, F2)
$d_{\mathrm{model}}$ , $H$ , $d_{\mathrm{head}}$	$64$ , $4$ , $16$
Vocabulary, chain length	$V=128$ , $L=8$
RoPE family	Monochromatic — single scalar $\theta$ for all $d/2$ dimensions
Wavelength grid (6)	$\{2,4,8,16,32,64\}$ positions
Corresponding $\theta$ values	$\{3.14,1.57,0.78,0.39,0.19,0.09\}$ rad

Appendix

Table 59: F3 configuration: monochromatic RoPE control.

wavelength	$\theta$ (rad)	$\gamma{=}0.1$	$\gamma{=}0.5$	$\gamma{=}1.0$	$\gamma{=}2.0$	$\gamma{=}4.0$
$2$	$3.14$	$0.46$	$\mathbf{0.62}$	$0.60$	$0.42$	$0.33$
$4$	$1.57$	$0.46$	$0.49$	$\mathbf{0.51}$	$0.40$	$0.27$
$8$	$0.78$	$0.48$	$\mathbf{0.53}$	$0.50$	$0.41$	$0.30$
$16$	$0.39$	$\mathbf{0.51}$	$\mathbf{0.51}$	$\mathbf{0.51}$	$0.42$	$0.29$
$32$	$0.19$	$0.48$	$\mathbf{0.55}$	$0.52$	$0.40$	$0.31$
$64$	$0.09$	$0.46$	$0.59$	$\mathbf{0.60}$	$0.41$	$0.28$

Appendix

Table 60: F3: monochromatic RoPE accuracy by wavelength (positions),

\theta

(radians), and

\gamma

. Peak per row in bold .

Figure 52: F3: monochromatic phase diagram. Two peaks visible at $(\lambda=2,\gamma=0.5)=0.62$ and $(\lambda=64,\gamma=1.0)=0.60$ . The $\gamma=4.0$ column collapses to $0.27$ – $0.33$ , consistent with F1 and F2’s universal high- $\gamma$ collapse.

Parameter	Value
Architecture	nano single-block
$V$ , $d_{\mathrm{model}}$ , $H$ , $d_{\mathrm{head}}$	$128$ , $64$ , $4$ , $16$
Chain length $L$	$8$ KV pairs (sequence length $17$ )
Training	$2000$ samples, batch $128$ , $10$ epochs, AdamW lr $10^{-3}$ , wd $10^{-2}$
RoPE family	Bandpass, $b=0.2$
$\theta$ grid (20 log-spaced)	from $0.02$ to $\pi\approx 3.14$ rad

Appendix

Table 61: G configuration.

$\theta$ (rad)	$\theta$ (deg)	$\gamma^{*}$	peak acc	baseline	gain	noise
$0.0200$	$1.1$	$0.80$	$0.778$	$0.447$	$+0.331$	$0.0200$
$0.0261$	$1.5$	$0.80$	$0.776$	$0.446$	$+0.330$	$0.0261$
$0.0341$	$2.0$	$0.80$	$0.771$	$0.445$	$+0.326$	$0.0341$
$0.0444$	$2.5$	$0.80$	$0.769$	$0.447$	$+0.321$	$0.0444$
$0.0580$	$3.3$	$0.80$	$0.761$	$0.451$	$+0.310$	$0.0580$
$0.0757$	$4.3$	$0.80$	$0.747$	$0.455$	$+0.292$	$0.0757$

Appendix

Table 62: G: optimal

\gamma^{*}

, peak accuracy, baseline accuracy, momentum gain, and noise level for all 20 tested

\theta

values. This is the canonical empirical table of the Low-Pass Induction Filter.

Figure 53: G main figure. (A) Full heatmap (log $\theta$ , linear $\gamma$ ) with the clean induction zone at low $\theta$ . (B) Inverted-U slice plots at $5$ representative $\theta$ . (C) Scatter of momentum gain versus rotational noise with the fitted line and the $r=-0.679$ , $p=9.9\!\times\!10^{-4}$ annotation. (D) Empirical $\gamma^{*}(\theta)$ (scatter, colored by peak accuracy) against the $\gamma^{*}\propto 1/\sin(\theta/2)$ theory curve. Theory overestimates $\gamma^{*}$ at low $\theta$ ; empirical $\gamma^{*}$ sits uniformly at $\sim 0.80$ for $\theta<0.2$ .

Figure 54: G theory validation panel. (A) Hamiltonian decomposition: the $2\sin(\theta/2)$ noise curve (red, shaded) versus the unit signal curve (green), with the low-pass-regime boundary annotated. (B) SNR curves for five $\gamma$ : at high $\gamma$ , SNR falls below unity at modest $\theta$ . (C) Experimental noise–gain scatter with fit $y=-0.110x+0.245$ . (D) Textual summary.

Figure 55: G supplementary analysis. (A) Phase diagram with iso-accuracy contours. (B) Box plots of accuracy by frequency band (Low/Mid/High); the low band is strongly right-shifted. (C) Variance analysis: std versus mean accuracy colored by $\log_{10}\theta$ . (D) Peak accuracy versus $\theta$ (sharp fall-off around $\theta=0.2$ ). (E) Momentum benefit per frequency band (bars). (F) Top- $10$ frequencies summary — all at $\gamma^{*}=0.8$ with peak accuracies from $0.778$ down to $0.612$ .

Parameter	Value
Architecture	3-layer transformer with pre-LN, GELU MLP
$d_{\mathrm{model}}$ , $H$ , $d_{\mathrm{head}}$ , $d_{\mathrm{ff}}$	$128$ , $4$ , $32$ , $256$
Vocabulary	$V=200$
Chain length $L$	$\in[8,12]$ KV pairs (random per example)
Max sequence length	$100$ tokens
Dataset	$5000$ train / $1000$ test (fresh samples)

Appendix

Table 63: H configuration. Three-layer transformer, larger

d_{\mathrm{model}}

, longer chain length,

60

training epochs.

RoPE type	$\theta_{\min}$	$\theta_{\max}$	range ratio
Single-frequency	$0.100000$	$0.100000$	$1.0\!\times$
Bandpass	$0.080000$	$0.120000$	$1.5\!\times$
Multi-frequency	$0.000178$	$1.000000$	$5{,}623.4\!\times$

Appendix

Table 64: H frequency spectrum properties at

d_{k}=32

(so

16

frequency slots).

	Single-frequency			Bandpass			Multi-frequency
$\gamma$	acc (%)	SEM	$\Delta$	acc (%)	SEM	$\Delta$	acc (%)	SEM	$\Delta$
$0.0$	$11.1$	$0.3$	base	$10.8$	$0.3$	base	$11.6$	$0.2$	base
$0.3$	$44.5$	$0.6$	$+33.4$	$43.8$	$1.0$	$+33.0$	$66.6$	$0.8$	$+55.1$
$0.5$	$72.1$	$0.4$	$+61.0$	$72.2$	$0.8$	$+61.4$	$89.7$	$0.6$	$+78.2$
$0.7$	$82.7$	$0.5$	$+71.6$	$82.3$	$0.2$	$+71.5$	$94.7$	$0.4$	$+83.2$
$1.0$	$\mathbf{86.8}$	$\mathbf{0.2}$	$\mathbf{+75.7}$	$\mathbf{87.5}$	$\mathbf{0.3}$	$\mathbf{+76.7}$	$\mathbf{96.2}$	$\mathbf{0.2}$	$\mathbf{+84.7}$

Appendix

Table 65: H test accuracy and momentum gain (over

\gamma=0

baseline) for each ( RoPE type,

\gamma

) cell, three-seed mean

\pm

SEM. Per-family baseline and peak highlighted.

Figure 56: H: RoPE frequency design space. (A/B/C) Per-dimension frequency spectra for the three families. Single-frequency (A) shows $16$ identical bars at $\theta=0.1$ (no escape routes); Bandpass (B) shows a linear ramp from $0.08$ to $0.12$ (limited escape); Multi-frequency (C, log scale) shows the exponential spread from $\sim 10^{-4}$ to $10^{0}$ (many escape routes). (D) Accuracy versus $\gamma$ for all three families, all inverted-U, all peaking at $\gamma=1.0$ , with multi-frequency cleanly above the others across the whole $\gamma$ range. (E) Theoretical-framework text panel.

Figure 57: H curve shapes. Per-family curves with SEM shading and peak annotations. All three are inverted-U with peak at $\gamma=1.0$ , but the multi-frequency curve is shifted upward by $\sim 9\%$ at every $\gamma>0$ — the Escape Routes advantage manifests as a level shift, not a shape change.

Task	$\gamma=0$ baseline	$\gamma=0.5$ momentum	$\Delta$
Associative Recall ( $\nabla$ )	$14.0\%$	$50.8\%$	$+36.8$ pp
Variable Tracking ( $\nabla$ )	$59.0\%$	$54.4\%$	$-4.6$ pp
Global Counting ( $\int$ )	$60.8\%$	$28.6\%$	$-32.2$ pp

Appendix

Table 66: Mechanistic-visualization paired training,

\theta=0.03

\gamma\!\in\!\{0,0.5\}

, single seed.

$\theta\backslash\gamma$	$0.0$	$0.2$	$0.4$	$0.6$	$0.8$	$1.0$
$0.01$	$13.2$	$24.0$	$56.8$	$72.6$	$\mathbf{76.0}$	$72.4$
$0.03$	$14.0$	$21.4$	$42.6$	$56.4$	$\mathbf{58.0}$	$54.0$
$0.05$	$14.8$	$19.8$	$33.8$	$\mathbf{42.4}$	$40.8$	$39.6$
$0.10$	$14.0$	$16.2$	$22.0$	$\mathbf{26.6}$	$24.0$	$19.8$
$0.20$	$12.6$	$13.0$	$13.2$	$\mathbf{13.4}$	$12.4$	$13.0$
$0.30$	$13.4$	$12.8$	$12.6$	$12.4$	$12.4$	$9.4$

Appendix

Table 67: Heatmap: Associative Recall accuracy (%) by

(\theta,\gamma)

, single seed,

36

training runs. Row maxima in bold .

Task	$\theta$	$0.0$	$0.2$	$0.4$	$0.6$	$0.8$	$1.0$	$1.2$
Associative Recall	$0.03$	$14.0$	$21.4$	$42.6$	$56.4$	$\mathbf{58.0}$	$54.0$	$45.0$
Associative Recall	$0.30$	$\mathbf{13.4}$	$12.8$	$12.6$	$12.4$	$12.4$	$9.4$	$9.2$
Variable Tracking	$0.03$	$\mathbf{59.0}$	$57.4$	$54.6$	$50.8$	$49.6$	$48.4$	$49.6$
Variable Tracking	$0.30$	$\mathbf{56.4}$	$53.2$	$52.8$	$51.6$	$47.8$	$48.6$	$47.6$
Global Counting	$0.03$	$\mathbf{60.8}$	$38.8$	$31.2$	$26.6$	$23.6$	$23.0$	$23.6$
Global Counting	$0.30$	$\mathbf{25.6}$	$24.4$	$23.6$	$22.4$	$21.6$	$21.2$	$20.8$

Appendix

Table 68: Accuracy versus

\gamma

at two frequencies for all three tasks. Row peaks in bold .

Figure 58: Mechanistic analysis. Panels (A)–(C): theory. (A) RoPE position encoding $\cos(\theta t)$ at four frequencies showing slow vs. fast rotation. (B) Discrete momentum $p_{t}=q_{t}-q_{t-1}$ versus the analytic derivative $-\theta\sin(\theta t)$ ; numerical and theoretical curves coincide modulo the discrete-step correction. (C) Transfer-function magnitude $|H(\omega)|=|1+\gamma(e^{-j\omega}-1)|$ for five $\gamma$ . Panels (D)–(F): attention maps, $\gamma=0$ versus $\gamma=0.5$ . (D) Associative Recall: sharper attention to the target position at $\gamma=0.5$ . (E) Variable Tracking: stronger diagonal structure at $\gamma=0.5$ . (F) Global Counting (negative control): attention becomes less uniform and therefore worse at counting. Panel (H): accuracy-versus- $\gamma$ curves. Panel (I): the $36$ -cell heatmap with white stars marking per-row optima.

Figure 59: $\gamma$ -progression on Associative Recall. Evolution of the attention map on a single held-out example as $\gamma$ is swept from $0$ to $1.0$ . Each panel header reports the accuracy of an independently-trained model at that $\gamma$ under $\theta=0.03$ . Accuracy rises from $14.0\%$ at $\gamma=0$ to $57.2\%$ at $\gamma=0.7$ before descending at $\gamma=1.0$ . The pattern visibly sharpens through $\gamma=0.7$ and begins to fragment at $\gamma=1.0$ .

Variant	Hops	Negation rate	Distractors	Baseline target
Original (pre-easing)	$6$ – $16$	$30$ – $50\%$	$2$ – $6$	$\sim 52\%$
v2 easing	$4$ – $9$	$30$ – $40\%$	$1$ – $3$	$\sim 59\%$
v3 easing	$2$ – $5$	$20$ – $30\%$	$0$ – $1$	$100\%$ ceiling
v4 (this section)	$3$ – $6$	$25$ – $30\%$	$1$ – $2$	target $70$ – $85\%$

Appendix

Table 69: Eased multi-hop configurations relative to the original (harder) and to prior easing attempts.

Difficulty	$\theta$	$\gamma=0$	$\gamma=0.3$	$\gamma=0.5$	$\gamma=0.7$	$\gamma=1.0$
Easy	$0.03$	$65.0$	$63.6$	$63.8$	$62.3$	$62.1$
Easy	$0.30$	$63.0$	$62.6$	$62.6$	$62.0$	$62.0$
Medium	$0.03$	$59.1$	$58.3$	$59.0$	$59.8$	$58.6$
Medium	$0.30$	$56.1$	$54.4$	$54.4$	$55.0$	$53.9$
Hard	$0.03$	$52.6$	$52.1$	$52.2$	$52.3$	$52.7$
Hard	$0.30$	$53.1$	$52.6$	$52.4$	$51.9$	$52.7$

Appendix

Table 70: All 90 runs of the multi-hop eased sweep. Each cell is mean accuracy (%) over three seeds.

Figure 60: Eased multi-hop results. Three panels, one per difficulty level, show accuracy against $\gamma$ with seed-variance error bars and the two theta values superimposed. The “Random” line at $50\%$ marks binary-task chance level. No configuration exceeds its $\gamma=0$ baseline by more than $+0.7$ pp.

Difficulty	$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$1.0$
Easy	$0.03$	$94.4$	$96.3$	$97.9$	$96.5$	$\mathbf{98.0}$
Easy	$0.30$	$93.9$	$94.4$	$\mathbf{95.0}$	$94.6$	$94.7$
Medium	$0.03$	$82.4$	$88.7$	$88.8$	$88.8$	$\mathbf{89.8}$
Medium	$0.30$	$81.1$	$81.2$	$\mathbf{81.5}$	$79.8$	$80.7$
Hard	$0.03$	$73.7$	$75.2$	$\mathbf{80.9}$	$74.0$	$80.5$
Hard	$0.30$	$65.2$	$70.1$	$\mathbf{71.4}$	$68.8$	$67.7$

Appendix

Table 71: Variable Tracking sweep, 3-seed means in %. Row maxima in bold .

Difficulty	$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$1.0$
Easy	$0.03$	$52.4$	$52.2$	$52.3$	$52.0$	$52.5$
Easy	$0.30$	$52.7$	$52.1$	$52.1$	$52.2$	$52.9$
Medium	$0.03$	$53.1$	$52.8$	$53.0$	$53.0$	$53.2$
Medium	$0.30$	$52.9$	$54.3$	$52.2$	$52.6$	$53.0$
Hard	$0.03$	$53.0$	$52.7$	$53.1$	$52.5$	—
Hard	$0.30$	—	—	—	—	—

Appendix

Table 72: Multi-Hop with Negations sweep, 3-seed means in %. All gains within seed noise.

Difficulty	$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$1.0$
Easy	$0.03$	$100.0$	$100.0$	$100.0$	$\mathbf{100.0}$	$99.7$
Easy	$0.30$	$90.6$	$78.4$	$70.4$	$54.7$	$62.3$
Medium	$0.03$	$85.4$	$86.1$	$85.2$	$\mathbf{97.4}$	$80.1$
Medium	$0.30$	$70.6$	$69.5$	$\mathbf{81.7}$	$58.0$	$22.0$
Hard	$0.03$	$12.8$	$\mathbf{73.4}$	$70.2$	$28.0$	$38.7$
Hard	$0.30$	$56.8$	$49.4$	$30.6$	$46.6$	$25.0$

Appendix

Table 73: Arithmetic CoT sweep, 3-seed means in %. Predict the final digit of the sum of two

n

-digit numbers. Row maxima in bold .

Difficulty	$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$1.0$
Easy	$0.03$	$31.7$	$31.6$	$31.6$	$31.8$	$31.8$
Easy	$0.30$	$31.7$	$31.4$	$31.6$	$32.0$	$31.6$
Medium	$0.03$	$25.3$	$24.8$	$24.8$	$25.1$	$25.2$
Medium	$0.30$	$25.4$	$24.9$	$24.7$	$25.4$	$25.4$
Hard	$0.03$	$20.4$	$20.8$	$20.7$	$20.5$	$20.6$
Hard	$0.30$	$20.7$	$21.0$	$20.7$	$20.8$	$20.5$

Appendix

Table 74: Global Counting sweep, 3-seed means in %. The

\int

-task’s expected behavior: no gain.

Task	Mechanism	Baseline	Peak	Relative gain
Variable Tracking	$\nabla$	$82.4\%$	$98.0\%$	$+19\%$
Multi-Hop w/ Negations	$\nabla$	$53.1\%$	$53.2\%$	$+0\%$
Arithmetic CoT	$\nabla$	$85.4\%$	$100.0\%$	$+17\%$
Global Counting	$\int$	$25.3\%$	$31.8\%$	$+1\%$

Appendix

Table 75: CoT cross-task summary at medium difficulty,

\theta=0.03

Task	$\gamma=0$	$0.5$	$0.7$	$0.9$	$1.2$
Arithmetic Carry	$0.888$	$0.917$	$0.957$	$0.966$	$\mathbf{0.970}$
List Reversal	$1.000$	$1.000$	$1.000$	$1.000$	$\mathbf{1.000}$
Parity	$0.486$	$\mathbf{0.515}$	$0.500$	$0.497$	$0.497$
Sorting (Min)	$\mathbf{0.997}$	$0.995$	$0.993$	$0.991$	$0.987$
Natural Induction	$0.127$	$0.578$	$0.796$	$0.870$	$\mathbf{0.906}$

Appendix

Table 76: Real-world reasoning main sweep at

\theta=0.03

. Accuracy values are fractions; each cell is the mean over five seeds. Row maxima in bold .

Figure 61: Real-world reasoning overview. (A) Inverted-U accuracy curves for all five tasks at $\theta=0.03$ . Natural Induction (orange) is the dramatic mover. (B) Peak momentum gains bar chart; Natural Induction’s $+0.74$ dwarfs everything else. (C) Heatmap of accuracy at $\theta=0.03$ across all $(\mathrm{task},\gamma)$ combinations. (D) Low-pass filter effect: gain against $\theta$ at optimal $\gamma\!\in\![0.5,1.0]$ , color per task. Natural Induction’s gain decays as $\theta$ increases, exactly as predicted. (E) Histogram of per-task optimal $\gamma$ across the five tasks — concentrating around $0.5$ – $0.7$ with Natural Induction extending to $1.2$ . (F) Summary text panel.

$\theta$	$\gamma=0.0$	$0.3$	$0.5$	$0.7$	$0.9$	$1.2$
$0.03$	$0.127$	$0.243$	$0.578$	$0.796$	$0.870$	$\mathbf{0.906}$
$0.10$	$0.179$	$0.249$	$0.533$	$0.769$	$0.858$	$\mathbf{0.923}$
$0.30$	$0.281$	$0.348$	$0.498$	$0.651$	$0.760$	$\mathbf{0.851}$
$1.00$	$0.178$	$0.163$	$0.160$	$0.152$	$0.161$	$\mathbf{0.158}$

Appendix

Table 77: Natural Induction: full

(\theta,\gamma)

grid, five-seed means. Row maxima in bold .

$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$0.9$	$1.2$
$0.03$	$0.888$	$0.896$	$0.917$	$0.957$	$0.966$	$\mathbf{0.970}$
$0.10$	$0.889$	$0.910$	$0.921$	$0.938$	$\mathbf{0.946}$	$0.930$
$0.30$	$0.898$	$0.927$	$0.934$	$0.944$	$\mathbf{0.946}$	$0.932$
$1.00$	$\mathbf{0.974}$	$0.973$	$0.963$	$0.953$	$0.947$	$0.930$

Appendix

Table 78: Arithmetic Carry: full

(\theta,\gamma)

grid, 5-seed means. Row maxima in bold .

Figure 62: Per-task inverted-U curves. Each panel sweeps $\gamma$ at four $\theta$ values (color scale dark purple at $\theta=0.03$ to yellow at $\theta=1.00$ ). The Natural Induction panel is the most striking: at $\theta=0.03$ the curve rises steeply from $12.7\%$ at $\gamma=0$ to $90.6\%$ at $\gamma=1.2$ , while at $\theta=1.00$ the curve is essentially flat around $16\%$ .

Task	Baseline	Peak	Peak $\gamma$	Peak $\theta$	Peak gain
Majority	$1.000$	$1.000$	$0.0$	$0.03$	$+0.000$
Induction	$0.275$	$0.846$	$1.2$	$0.10$	$+0.591$
Trajectory	$0.691$	$0.744$	$1.2$	$1.00$	$+0.040$
Dyck	$0.898$	$0.946$	$0.7$	$1.00$	$+0.016$

Appendix

Table 79: Multi-task validation main summary. Note: Majority baseline is

1.000

— the single-layer transformer with this small capacity solves the bag-of-tokens task cleanly.

$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$0.9$	$1.2$	$1.8$
$0.03$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$
$0.10$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$
$0.30$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$
$1.00$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$

Appendix

Table 80: Majority task: every one of the

28

cells (5 seeds each,

=140

runs) returns

1.000

accuracy. The negative control behaving exactly as theory predicts.

$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$0.9$	$1.2$	$1.8$
$0.03$	$0.152$	$0.243$	$0.397$	$0.577$	$0.701$	$\mathbf{0.787}$	$0.717$
$0.10$	$0.255$	$0.328$	$0.479$	$0.653$	$0.770$	$\mathbf{0.846}$	$0.801$
$0.30$	$0.354$	$0.438$	$0.531$	$0.611$	$\mathbf{0.648}$	$0.640$	$0.522$
$1.00$	$0.340$	$0.368$	$0.409$	$0.442$	$\mathbf{0.454}$	$0.453$	$0.444$

Appendix

Table 81: Induction task across the full

28

-cell grid, 5-seed means. Row maxima in bold .

$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$0.9$	$1.2$	$1.8$
$0.03$	$0.684$	$0.697$	$0.701$	$0.703$	$0.705$	$\mathbf{0.708}$	$0.697$
$0.10$	$0.688$	$0.697$	$0.704$	$0.708$	$0.709$	$\mathbf{0.709}$	$0.703$
$0.30$	$0.689$	$0.701$	$0.712$	$0.717$	$0.718$	$\mathbf{0.718}$	$0.704$
$1.00$	$0.704$	$0.725$	$0.734$	$0.740$	$0.743$	$\mathbf{0.744}$	$0.735$

Appendix

Table 82: Trajectory extrapolation across the full grid. The peak

\theta

1.00

, not in the low-pass region.

$\theta$	$\gamma=0$	$0.3$	$0.5$	$0.7$	$0.9$	$1.2$	$1.8$
$0.03$	$0.871$	$\mathbf{0.912}$	$0.905$	$0.900$	$0.896$	$0.885$	$0.826$
$0.10$	$0.886$	$\mathbf{0.917}$	$0.910$	$0.898$	$0.904$	$0.890$	$0.839$
$0.30$	$0.905$	$0.913$	$\mathbf{0.920}$	$0.915$	$0.904$	$0.915$	$0.843$
$1.00$	$0.930$	$0.942$	$\mathbf{0.946}$	$0.946$	$0.932$	$0.917$	$0.877$

Appendix

Table 83: Dyck language across the grid. Gentler U with peak near

\gamma\!\in\![0.3,0.5]

, clear over-rotation collapse at

\gamma=1.8

$t$ -statistic	$-2.910$
$p$ -value	$4.51\!\times\!10^{-3}$
Majority mean gain (at $\gamma>0$ )	$+0.000$
Non-majority mean gain (at $\gamma>0$ )	$+0.107$

Appendix

Table 84: Hypothesis test: is the Majority gain distribution significantly different from the pooled other-tasks distribution?

Figure 63: Multi-task validation main figure. (A) U-curves for all four tasks at $\theta=0.03$ : induction (green) rises sharply, trajectory (blue) and Dyck (red) rise modestly, majority (grey) is flat at $1.0$ . (B) Peak-gain bar chart with the majority bar at exactly zero (negative-control annotation). (C) Heatmap at $\theta=0.03$ . (D) Low-pass effect: gain-versus- $\theta$ at each task’s optimal $\gamma$ ; induction’s curve decays strongly with $\theta$ .

Figure 64: Per-task detailed analysis. Four subplots with U-curves at each of the four $\theta$ values. Majority’s subplot is rigid at $1.0$ ; induction’s is the cleanest U-family; trajectory’s U is shallow but consistent; Dyck’s is narrow with over-rotation damage at high $\gamma$ .

Figure 65: Task classification by momentum benefit. Scatter plots (peak gain, baseline) for each task; the green shaded region is the “momentum benefit zone” (peak gain $>0.05$ ); red is the “no-benefit zone.” Majority lands squarely in red; induction, trajectory and Dyck in green.

Chain $\backslash$ Vocab	$64$	$128$	$256$	$512$
$4$	$0.577$	$0.624$	$0.667$	$0.707$
$8$	$0.675$	$0.729$	$0.779$	$0.826$
$12$	$0.729$	$0.787$	$0.842$	$0.893$
$16$	$0.766$	$0.827$	$0.885$	$0.938$
$20$	$0.794$	$0.858$	$0.917$	$0.973$
$24$	$0.816$	$0.882$	$0.943$	$1.000$

Appendix

Table 85: Task difficulty

d(\ell,V)

for each

(\mathrm{chain\,length},\mathrm{vocab\,size})

combination. Higher is harder.

$\ell$	$V$	diff	baseline	peak	gain	best $\theta$	best $\gamma$
$4$	$64$	$0.577$	$0.651$	$1.000$	$+0.349$	$0.03$	$0.7$
$4$	$128$	$0.624$	$0.703$	$0.994$	$+0.291$	$0.03$	$0.9$
$4$	$256$	$0.667$	$0.653$	$0.911$	$+0.258$	$0.03$	$0.9$
$4$	$512$	$0.707$	$0.590$	$0.748$	$+0.158$	$0.03$	$0.7$
$8$	$64$	$0.675$	$0.482$	$0.999$	$+0.517$	$0.03$	$0.9$
$8$	$128$	$0.729$	$0.527$	$0.875$	$+0.348$	$0.03$	$0.9$

Appendix

Table 86: Per-

(\ell,V)

best case: best

(\theta,\gamma)

cell in each of

24

difficulty cells. Observe: best

\theta=0.03

for every low-difficulty cell but shifts to higher values at high difficulty; best

\gamma

drifts from

0.7

toward

1.2

–

1.8

as difficulty grows.

Bin	Difficulty range	Mean baseline acc	Mean momentum gain
Hard	$0.5$ – $0.7$	$0.652$	$+0.243\pm 0.119$
Very Hard	$0.7$ – $1.0$	$0.450$	$+0.167\pm 0.081$

Appendix

Table 87: Difficulty-binned momentum gain. Mean over all

(\theta,\gamma)

combinations within each chain-length/vocab bin.

Pair	Pearson $r$	$p$
Rotational noise $2\sin(\theta/2)$ vs gain	$-0.3719$	$2.88\!\times\!10^{-5}$
Task difficulty $d(\ell,V)$ vs gain	$-0.2470$	$6.54\!\times\!10^{-3}$
Baseline accuracy vs gain	$-0.0164$	$0.859$

Appendix

Table 88: Correlations between task-level quantities and momentum gain. Rows sorted by

|r|

Prediction	Result	Verdict
Low-pass filter (noise–gain $\rho<0$ )	$r=-0.372$ , $p=2.9\!\times\!10^{-5}$	PASS
Difficulty sweet spot (Hard $>$ Very Hard)	$F=12.04$ , $p=7.4\!\times\!10^{-4}$	PASS
Universal optimal $\gamma$ (clusters in $[0.7,1.2]$ )	median $\sim 0.9$	PASS

Appendix

Table 89: Three pre-registered predictions, each with its independent statistical test.

Figure 66: Multi-difficulty main validation figure. (A) Sweet-spot scatter: momentum gain versus task difficulty, colored by $\theta$ . Quadratic fit overlay (dashed red) is the inverted-U shape the sweet-spot hypothesis predicts. (B) Average-over- $\theta$ gain heatmap on the $(\mathrm{chain},\mathrm{vocab})$ grid. (C) Difficulty-regime U-curves. (D) Low-pass filter scatter: gain versus rotational noise, colored by difficulty; the $r=-0.372$ trend line is shown.

Figure 67: Phase diagrams by chain length (at $\theta=0.03$ ). Six subplots for chain lengths $\{4,8,12,16,20,24\}$ , each showing an $(\mathrm{vocab},\gamma)$ accuracy heatmap with the red star marking the optimum. As chain length grows, the optimum shifts diagonally — lower vocab, lower $\gamma$ — and peak accuracy falls (from $\approx 0.98$ at chain $4$ to $\approx 0.62$ at chain $24$ ).

Figure 68: Theoretical-validation figure. (A) Predicted benefit versus observed gain (normalized), color by $\theta$ . (B) Optimal $\gamma$ versus difficulty, colored by peak accuracy; horizontal dashed red line at $\gamma=0.8$ marks the theory-predicted optimum. Optimal $\gamma$ tightly clustered in $[0.7,1.2]$ across difficulties. (C) Low-pass effect stratified by difficulty bin. (D) Summary panel.

Section	Task	Class	Peak gain	Low- $\theta$ best?	PSA-consistent?
A18	Associative Recall	$\nabla$	$+0.368$	yes	yes
A18	Variable Tracking	$\nabla$	$-0.046$	—	neutral (narrow sweep)
A18	Global Counting	$\int$	$-0.322$	—	yes (harm as predicted)
A19	Multi-Hop Eased	$\nabla$	$+0.007$	—	null (task out of band)
A20	Variable Tracking	$\nabla$	$+0.156$	yes	yes
A20	Multi-Hop w/ Negations	$\nabla$	$+0.001$	—	null (task out of band)

Appendix

Table 90: Task-by-task verdict matrix across the reasoning arc. Each row is a distinct task; columns are class (

\nabla

\int

), peak gain in the best-case cell, whether low-

\theta

produced the best gain (low-pass-filter signature), and whether the outcome matches the PSA prediction.

Metric	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.7$ )	$\Delta$
$L_{\mathrm{new}}$	$8.0861$	$8.0713$	$-0.0147$
$L_{\mathrm{second}}$	$3.4099$	$3.4966$	$+0.0867$
$L_{\mathrm{rep}}$	$2.1103$	$2.2220$	$+0.1117$
$\Delta_{1\!-\!2}$	$4.6761$	$4.5747$	$-0.1014$
Train loss	$3.9424$	$4.0158$	$+0.0734$

Appendix

Table 91: Burstiness final metrics at step

10{,}000

\gamma=0.7

\beta=0

Hyp.	Statement	Measured	Verdict
H1	$\|\Delta L_{\mathrm{new}}\|<10\%$	$0.2\%$	PASS
H2	$L_{\mathrm{rep}}$ decreases ( $\Delta<0$ )	$+5.3\%$	FAIL
H3	$\Delta_{1\!-\!2}$ increases ( $\Delta>0$ )	$-0.1014$	FAIL

Appendix

Table 92: Hypothesis tests on the burstiness task.

Figure 69: Burstiness training trajectory. Training loss (top-left), $L_{\mathrm{new}}$ and $L_{\mathrm{rep}}$ (top-middle, right), first-second gap $\Delta_{1\!-\!2}$ (bottom-left), per-occurrence learning curve (bottom-middle), final bar-chart comparison (bottom-right). Baseline and momentum curves visibly interleaved throughout training on this $\int$ -task.

Figure 70: Learning curve by occurrence. The canonical visualization for this experiment. Baseline and momentum curves indistinguishable beyond the first occurrence. No “L-shape” for the momentum model, which would have been the signature of one-shot learning.

Metric	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.7$ )	$\Delta$
$L_{\mathrm{new}}$	$7.3450$	$7.3477$	$+0.0026$
$L_{\mathrm{second}}$	$2.1191$	$2.2813$	$+0.1622$
$L_{\mathrm{rep}}$	$1.6274$	$1.7687$	$+0.1413$
$\Delta_{1\!-\!2}$	$5.2259$	$5.0664$	$-0.1595$

Appendix

Table 93: Structural ICL without anchoring: final metrics.

\gamma=0.7

Hyp.	Statement	Measured	Verdict
H1	$\|\Delta L_{\mathrm{new}}\|<15\%$	$0.0\%$	PASS
H2	$L_{\mathrm{rep}}$ decreases	$+8.7\%$	FAIL
H3	$\Delta_{1\!-\!2}$ increases	$-0.1595$	FAIL

Appendix

Table 94: Hypothesis verdicts.

Figure 71: Structural ICL without anchoring: training curves. The baseline edges ahead on $L_{\mathrm{rep}}$ and the momentum model loses the first-second gap. The opposite of the predicted behavior — which motivates the context-mismatch diagnosis in Section A27 .

Figure 72: Per-occurrence learning curve. Baseline (blue) below momentum (red) for all $k\!\geq\!1$ , consistently.

Metric	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.2$ )	$\Delta$
$L_{\mathrm{new}}$	$7.1529$	$7.1458$	$-0.0071$
$L_{\mathrm{second}}$	$3.4463$	$3.9337$	$+0.4874$
$L_{\mathrm{rep}}$	$1.6496$	$2.1261$	$+0.4765$
$\Delta_{1\!-\!2}$	$3.7066$	$3.2121$	$-0.4946$

Appendix

Table 95: Chained ICL without anchoring: final metrics (

L=10

\gamma=0.2

Hyp.	Statement	Measured	Verdict
H1	$\|\Delta L_{\mathrm{new}}\|<15\%$	$0.1\%$	PASS
H2	$L_{\mathrm{rep}}$ decreases	$+28.9\%$	FAIL
H3	$\Delta_{1\!-\!2}$ increases	$-0.4946$	FAIL
H4	Deep-link improvement $k>7$ , $\Delta>0$	$-0.3492$	FAIL

Appendix

Table 96: Hypothesis verdict.

Depth $k$	Baseline	Momentum	$\Delta$ (B $-$ M)	Winner
$0$	$7.1529$	$7.1458$	$+0.0071$	Momentum
$1$	$3.4463$	$3.9337$	$-0.4874$	Baseline
$2$	$1.7209$	$2.3737$	$-0.6528$	Baseline
$3$	$1.4872$	$2.0387$	$-0.5515$	Baseline
$4$	$1.3765$	$1.8821$	$-0.5056$	Baseline
$5$	$1.3035$	$1.7631$	$-0.4596$	Baseline

Appendix

Table 97: Chain-depth breakdown: momentum loses at every depth

1

through

14

. The only “win” is at depth

0

(first-occurrence position).

Figure 73: Chained ICL training curves (no anchor). Baseline (blue) consistently below momentum (red).

Figure 74: Per-depth loss (no anchor). Baseline wins at every depth $\geq 1$ ; the gap is largest at depths $2$ – $5$ (the early chain positions where induction-head formation is easiest).

Metric	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.2$ )	$\Delta$
$L_{\mathrm{new}}$	$7.0849$	$7.0803$	$-0.0046$
$L_{\mathrm{second}}$	$3.0544$	$2.8714$	$-0.1830$
$L_{\mathrm{rep}}$	$1.2785$	$1.2262$	$-0.0523$
$\Delta_{1\!-\!2}$	$4.0305$	$4.2089$	$+0.1784$

Appendix

Table 98: Anchored ICL final metrics (

L=10

\gamma=0.2

). Compare against Section A27 ’s Table 95 :

L_{\mathrm{rep}}

flips from

+28.9\%

-4.1\%

on a one-token intervention.

Hyp.	Statement	Measured	Verdict
H1	$\|\Delta L_{\mathrm{new}}\|<15\%$	$0.1\%$	PASS
H2	$L_{\mathrm{rep}}$ decreases	$-4.1\%$	PASS
H3	$\Delta_{1\!-\!2}$ increases	$+0.1784$	PASS

Appendix

Table 99: Hypothesis verdict (anchored).

Configuration	Baseline $L_{\mathrm{rep}}$	Momentum $L_{\mathrm{rep}}$	Verdict
No anchor (Section A27 )	$1.6496$	$2.1261$	$+29\%$ FAIL
Anchored (this section)	$1.2785$	$1.2262$	$-4.1\%$ PASS

Appendix

Table 100: The no-anchor / anchored transition. One-token intervention flips the sign of the momentum effect on the same underlying task.

Figure 75: Anchored ICL training curves. Momentum model consistently at or below baseline throughout training, particularly on $L_{\mathrm{rep}}$ . $\Delta_{1\!-\!2}$ widens in the expected direction.

Figure 76: No-anchor vs anchored side-by-side. Left: per-depth loss; right: aggregate $L_{\mathrm{rep}}$ . Anchoring flips the sign of the momentum effect at every depth.

Metric	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.2$ )	$\Delta$
$L_{\mathrm{new}}$	$6.9860$	$7.0202$	$+0.0342$
$L_{\mathrm{second}}$	$2.3598$	$1.2309$	$-1.1289$
$L_{\mathrm{rep}}$	$1.7451$	$0.8288$	$-0.9163$
$\Delta_{1\!-\!2}$	$4.6262$	$5.7893$	$+1.1632$

Appendix

Table 101:

L=30

anchored stress test final metrics.

L_{\mathrm{rep}}

drops by more than half — from

1.7451

0.8288

Hyp.	Statement	Measured	Verdict
H1	$\|\Delta L_{\mathrm{new}}\|<15\%$	$0.5\%$	PASS
H2	$L_{\mathrm{rep}}$ decreases	$-52.5\%$	PASS
H3	$\Delta_{1\!-\!2}$ increases	$+1.1632$	PASS
H4	Gap larger than at $L=10$ ( $\Delta>0.0523$ )	$+0.9163$	PASS

Appendix

Table 102: Hypothesis verdict,

L=30

. All four pass.

Depth $k$	Baseline	Momentum	$\Delta$ (B $-$ M)	Winner
$0$	$6.9860$	$7.0202$	$-0.0342$	Baseline
$1$	$2.3598$	$1.2309$	$+1.1289$	Momentum
$2$	$1.5645$	$0.6515$	$+0.9131$	Momentum
$3$	$1.4266$	$0.5334$	$+0.8931$	Momentum
$4$	$1.4771$	$0.5769$	$+0.9002$	Momentum
$5$	$1.2770$	$0.5402$	$+0.7368$	Momentum

Appendix

Table 103: Per-depth loss on

L=30

anchored chains. Momentum wins at every depth

1

through

19

(only depth

0

goes to baseline, by

0.03

nats). Deep-chain improvement alone

+0.133

; average

+0.406

. Mirror image of Section A27 ’s negative-per-depth story.

Figure 77: $L=30$ stress-test training trajectory. $L_{\mathrm{rep}}$ separation is large and widens throughout training, the momentum model finishing at less than half the baseline’s loss.

Figure 78: Three-panel stress-test analysis. Left: per-depth loss, “momentum wins everywhere” signature. Middle: smoothed-delta plot. Right: the exponential-vs-linear decay interpretation — the baseline exhibits the predicted $p^{L}$ curve shape while the momentum model’s loss grows more slowly (closer to linear in depth).

Metric	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.2$ )	$\Delta$ (M $-$ B)
$L_{\mathrm{new}}$	$7.1026$	$7.2029$	$+0.1003$
$L_{\mathrm{rep}}$	$1.1446$	$1.1910$	$+0.0464$
$\Delta_{1\!-\!2}$	$5.9580$	$6.0119$	$+0.0539$
Improvement $(L_{\mathrm{rep}}^{B}-L_{\mathrm{rep}}^{M})/L_{\mathrm{rep}}^{B}$ :			$-4.1\%$

Appendix

Table 104: ICL metrics on anchored

L=30

chains with embedding-level momentum (notebook A). Despite using the correct transfer function in mathematical form, placing the shear at the embedding layer regresses the result.

Figure 79: Notebook A: training curves. Training loss, $L_{\mathrm{rep}}$ , and $\Delta_{1\!-\!2}$ are near-identical for baseline and momentum; the momentum model is slightly worse at the end.

Quantity	Baseline ( $\gamma=0$ )	Momentum ( $\gamma=0.2$ )
Sensitivity range	$[0.08,37.8]$	$[0.08,38.5]$
DC component	$98.98$	$95.93$
Nyquist component	$38.07$	$31.81$
DC / Nyquist ratio	$2.60$	$3.015$

Appendix

Table 105: Notebook A: spectral biopsy at Layer

0

. The DC/Nyquist ratio increases from

2.60

(baseline) to

3.015

(momentum) — the opposite of what a high-pass filter should produce.

Layer	DC (baseline)	Nyq (baseline)	DC (momentum)	Nyq (momentum)
$0$	$96.46$	$37.24$	$98.99$	$31.99$
$1$	$200.40$	$43.83$	$168.71$	$36.72$
$2$	$457.82$	$62.70$	$221.18$	$33.87$
$3$	$844.17$	$98.21$	$600.45$	$88.68$

Appendix

Table 106: Notebook A: layer-wise DC and Nyquist evolution. In the baseline, DC grows rapidly with depth (96

\to

844); the momentum model also grows, less aggressively but in the same direction — the opposite of what a working high-pass filter should produce at the shear’s intended location.

Hyp.	Statement	Measured	Verdict
H(a)	baseline is low-pass	DC/Nyq $=2.60$	PASS
H(b)	momentum is high-pass	DC/Nyq $=3.015$	FAIL
H(c)	momentum improves ICL	$-4.1\%$ regression	FAIL

Appendix

Table 107: Notebook A: hypothesis scorecard.

Figure 80: Notebook A: Bode-plot diagnostics. Top row: sensitivity vectors and their FFTs for baseline and momentum. Bottom row: FFT magnitudes and the ratio. The predicted high-pass response does not emerge in this configuration; the DC-to-Nyquist ratio rises in the wrong direction.

Metric	§ A29 Base	§ A29 Mom	B Base	B Mom
$L_{\mathrm{new}}$	$6.9860$	$7.0202$	$6.9860$	$7.0202$
$L_{\mathrm{rep}}$	$1.7451$	$0.8288$	$1.7451$	$0.8288$
$\Delta_{1\!-\!2}$	$4.6262$	$5.7893$	$4.6262$	$5.7893$
$L_{\mathrm{rep}}$ improvement	$-52.5\%$		$-52.5\%$

Appendix

Table 108: Notebook B: reproduction of the Section A29 main result. Baseline and momentum

L_{\mathrm{new}}

L_{\mathrm{rep}}

, and

\Delta_{1\!-\!2}

are reproduced to machine precision because notebook B shares that section’s configuration; see § A30.4 on why notebook A does not.

Quantity	Baseline	Momentum
Spectral entropy $H$	$6.2159$	$6.2354$
$\Delta$ -entropy (M $-$ B)	$+0.0196$
Theory–experiment correlation	$\mathbf{r=0.9865}$

Appendix

Table 109: Notebook B: spectral metrics on the attention-score spectrum. Theory–experiment correlation is the headline number: at

r=0.9865

, the measured AC-power spectrum at each frequency

\omega

99\%

correlated with

|H(\omega)|^{2}=|1+\gamma(1-e^{-j\omega})|^{2}

Figure 81: Notebook B: the Bode plot of emergence. Six panels: (top-left) baseline attention spectrum showing the diffuse low-pass structure; (top-middle) PSA spectrum with clear peaks; (top-right) the spectral ratio $\log(S_{\mathrm{mom}}/S_{\mathrm{base}})$ overlaid on the theoretical $|H|^{2}$ curve — they match; bottom row: further diagnostics including the theory-experiment correlation curve.

Figure 82: Notebook B: training dynamics. Both baseline and momentum follow similar training-loss trajectories, but the decomposed metrics ( $L_{\mathrm{rep}}$ , $\Delta_{1\!-\!2}$ ) show the expected separation: baseline consistently above momentum on $L_{\mathrm{rep}}$ and below on $\Delta_{1\!-\!2}$ .

Parameter	Value
Architecture	12-layer, 12-head GPT-style decoder
$d_{\mathrm{model}}$	$768$
Vocabulary	$8{,}192$
Block size	$512$
Total parameters	$91.7$ M
RoPE base	$10{,}000$

Appendix

Table 110: Section A31 configuration.

$\gamma$	Fluency loss	Logic loss	$\|\det J-1\|^{\dagger}$	Energy ratio $R$	$\kappa(J)$
† saturated: $1.0000$ here means $\det J\approx 0$ , not $\det J\approx 1$ . See § A31.4 .
$0$	$7.9926$	$0.6934$	$1.0000$	$0.552$	$5.3\!\cdot\!10^{2}$
$10^{-4}$	$7.9838$	$0.6934$	$1.0000$	$0.544$	$4.0\!\cdot\!10^{2}$
$2{\cdot}10^{-4}$	$7.9910$	$0.6934$	$1.0000$	$0.450$	$2.0\!\cdot\!10^{7}$ (spike)
$5{\cdot}10^{-4}$	$7.9795$	$0.6934$	$1.0000$	$0.461$	$1.6\!\cdot\!10^{2}$
$10^{-3}$	$7.9672$	$0.6934$	$1.0000$	$0.543$	$2.0\!\cdot\!10^{3}$

Appendix

Table 111: Per-

\gamma

final values after

10{,}000

training steps. The

|\det J-1|

column is flat at

1.0000

across all

13

runs, which means

\det J\approx 0

: the projected Jacobian is numerically singular and the statistic is saturated (§ A31.4 ). It is retained for completeness and carries no inference. The condition number

\kappa

exhibits intermittent spikes, up to

10^{8}

, and is recorded for completeness; since the probe from which it is computed is degenerate, no mechanistic reading of those spikes — as phase transitions in learning or as anything else — is offered here.

Figure 83: The 13-point $\gamma$ -sweep at 91.7M parameters. Learning curves (top row), Jacobian and energy diagnostics (middle rows), and the Jacobian condition number over training (bottom). The leakage-free energy ratio is the panel that carries inference; the Jacobian panels are saturated (§ A31.4 ) and the condition-number spikes are displayed without interpretation.

$\gamma$	emerged	steps to $50\%$ , by seed	mean	$\bar{E}$
$0.00$	$2/3$	$2900,\ 2500,\ \mathrm{NE}$	$2700$	$13.1$
$0.05$	$1/2$	$\mathrm{NE},\ 1750$	$1750$	$9.7$
$0.10$	$1/2$	$\mathrm{NE},\ 1550$	$1550$	$21.2$
$0.20$	$0/2$	$\mathrm{NE},\ \mathrm{NE}$	—	$64.7$
$0.40$	$3/3$	$1000,\ 2200,\ 2150$	$1783$	$12.0$
$0.50$	$3/3$	$1000,\ 1000,\ 2400$	$1467$	$6.9$

Appendix

Table 112: Single-layer induction emergence at

d_{k}=128

N=12

91.3

M parameters. Rows at

\gamma\leq 0.40

come from the width sweep, rows at

\gamma\geq 0.50

from the supercritical sweep; the

\gamma=0.00

and

0.40

rows carry a third seed contributed by the latter. “NE” = did not reach

50\%

within

3000

steps.

\bar{E}

is the mean final leakage-free energy ratio of § A31 . The two-seed rows are two-seed because sweep 1 was stopped at

49

80

runs by design, not because runs were discarded.

Figure 84: The emergence band at

d_{k}=128

N=12

91.3

M parameters. Top: steps to

50\%

induction accuracy, one marker per seed, bar at the mean; crosses at the top mark runs that never emerged within

3000

steps. Middle: emerged fraction. Bottom: final leakage-free energy ratio

E

, log scale, green for emerged and red for failed. Shading marks the band observed at the constant learning rate

3\!\times\!10^{-4}

used throughout ; its upper edge is a property of that step size and not of the operator (§ A31.11 ). Vertical lines:

\gamma_{c}^{T_{3}T_{4}}=0.413

(dotted), the depth-law prediction

\gamma^{*}(12)=0.680

(dash-dot), and

\gamma_{c}^{T_{4}}=0.787

(dashed). The

T_{4}

law’s second root at

2.245

(Remark 65 ) lies off the displayed range.

arm	outcome	$E_{\max}$ range	steps to $50\%$	verdict
A	$0/1$	$151.8$	—	pinned
B	$0/3$	$36.0$ – $106.4$	—	pinned
C	$\mathbf{2/3}$	$\mathbf{0.94}$ – $\mathbf{1.26}$	$\mathbf{1350,\ 1750}$	third still descending
D	$1/2$	$15.6$ – $27.7$	$1650$	second at $0.449$ and rising
E	$1/2$	$0.54$ – $0.74$	$1300$	second still descending

Appendix

Table 113: The decisive experiment. “descending” marks a run that had not reached

50\%

at its budget but whose induction loss was still falling; “pinned” marks one flat at chance.

E_{\max}

is the peak leakage-free energy ratio over the run.

Figure 85: The upper cutoff is optimiser conditioning. Left: peak energy ratio

E_{\max}

by arm, log scale. Every run above

E=30

is pinned at chance; every run below it emerges or is still descending. Right: steps to

50\%

induction accuracy; markers at the top mark runs that did not emerge within budget. Open grey squares are the constant-LR runs of Table 112 at the same couplings, shown for reference.

Model	Params	Layers	$d_{\mathrm{model}}$	Heads	Momentum
Goliath	406M	24	$1024$	$16$	none
David	163M	12	$768$	$12$	$\gamma=0.8$
Goliath Jr	163M	12	$768$	$12$	none

Appendix

Table 114: R 1 : model configurations. All three models use RoPE with base

10{,}000

and block size

512

. “David” adds momentum at

\gamma=0.8

Model	Params	Momentum	Fluency loss	Logic loss	Wall-clock
Goliath	$406$ M	none	$10.7745$	$0.6935$	$245.4$ min
David	$163$ M	$\gamma=0.8$	$10.8214$	$0.6933$	$93.3$ min
Goliath Jr	$163$ M	none	$10.7863$	$0.6934$	$88.2$ min

Appendix

Table 115: R 1 : final losses after

10{,}000

steps.

Hyp.	Statement	Margin	Verdict
H1	David ( $163$ M+mom) beats Goliath ( $406$ M) on fluency	$-0.4\%$	FAIL
H2	Goliath beats David on logic	$-0.0\%$	FAIL (tied at chance)
H3	David beats Goliath Jr at equal size	$-0.3\%$	FAIL

Appendix

Table 116: R 1 : hypothesis verdicts. All three hypotheses fail; all margins are

<0.5\%

Figure 86: R 1 : training curves. Fluency loss (left), logic loss (middle), and combined training loss (right) over $10{,}000$ training steps. All three fluency curves stay near the initial $\ln(50{,}257)\approx 10.82$ plateau throughout training. Logic loss is stuck at $\ln(2)=0.6931$ from step $\sim 500$ onward.

Patch	Before (R 1 )	After (R 2 )	Rationale
Batch size	$8$	$64$	Reduce gradient variance $\sim 2.83\times$
Vocabulary	$50{,}257$	$8{,}192$	Increase $P(\text{token recurrence})\sim 6\times$
Learning rate	$3\!\times\!10^{-4}$	$6\!\times\!10^{-4}$	$\sqrt{B_{\mathrm{new}}/B_{\mathrm{old}}}$ scaling

Appendix

Table 117: R 2 : the three patches relative to R 1 .

Model	Params	Layers	$d_{\mathrm{model}}$	Heads	Momentum
Goliath	$319$ M	$24$	$1024$	$16$	none
David	$98$ M	$12$	$768$	$12$	$\gamma=0.8$
Goliath Jr	$98$ M	$12$	$768$	$12$	none

Appendix

Table 118: R 2 : revised model configurations.

Model	Params	Momentum	Fluency loss	Logic loss	Wall-clock
Goliath	$319$ M	none	$7.7732$	$0.6932$	$1565.1$ min
David	$98$ M	$\gamma=0.8$	$8.1538$	$0.6933$	$592.2$ min
Goliath Jr	$98$ M	none	$7.9205$	$0.6934$	$542.1$ min

Appendix

Table 119: R 2 : final losses after

10{,}000

steps with patches. Fluency does escape the plateau (drops from

9.03

7.77

–

8.15

), confirming that the patches are working at the SNR level.

Hyp.	Statement	Margin	Verdict
H1	David ( $98$ M+mom) beats Goliath ( $319$ M) on fluency	$-4.9\%$	FAIL
H2	Goliath ties or beats David on logic	$+0.0\%$	PASS
H3	David beats Goliath Jr at equal size	$-2.9\%$	FAIL

Appendix

Table 120: R 2 : hypothesis verdicts (patched). H2 passes narrowly because all three models tie at chance on logic; H1 and H3 fail with margins between

3

and

5\%

Source	Task	Class	Peak gain	Low- $\theta$ best?	PSA-consistent?
§ A18	Associative Recall	$\nabla$	$+0.368$	yes	yes
§ A18	Variable Tracking	$\nabla$	$-0.046$	—	neutral (narrow sweep)
§ A18	Global Counting	$\int$	$-0.322$	—	yes (harm as predicted)
§ A19	Multi-Hop eased	$\nabla$	$+0.007$	—	null (out of band)
§ A20	Variable Tracking	$\nabla$	$+0.156$	yes	yes
§ A20	Multi-Hop w/ neg.	$\nabla$	$+0.001$	—	null (out of band)

Appendix

Table 121: Task-by-task verdict matrix for the Tranche 2 reasoning and ICL programs. Class:

\nabla

(derivative-class) or

\int

(integral-class, order-invariant). “Low-

\theta

best?” indicates whether low-frequency RoPE produced the largest gain, the empirical signature of the Low-Pass Induction Filter.

the claim of the body	Anchored in	Verdict	Evidence
Theorem 11 (four-term decomposition)	§ A2 , § A8 S4–5	PASS	NumPy verification $+$ $T_{4}$ -dominance at all $\gamma>0$
Theorem 27 (uniqueness in $\mathcal{C}_{\mathrm{PSA}}$ )	§ A3 , § A10 C15	PASS	C2 high-pass, FP-29 symmetric advantage
Cor. 41 (post- RoPE placement)	§ A2 , § A30	PASS	directional: $+4.1\%$ vs $-52.5\%$ ; runs unmatched
Cor. 41 , eq. ( 38 ) (Coriolis error vs. $\theta$ )	§ A8 S12	PASS ∗	directional slope $+0.307$ ; band-4 anomaly noted
Lemma 4 ( $M_{\gamma}$ is exact $\gamma$ -flow)	C6	PASS	exact operator residual, scale-independent; no training-scale evidence is claimed (§ A31.4 )
Thm. 38 (ii) (invariant conserved)	§ A10 C9	PASS ∗	flat on stationary inputs; causal-task confound

Appendix

Table 122: Audit of the claims of the body tested in these appendices. “Anchored in” gives the section number where the empirical content lives; “Verdict” summarizes the outcome, with PASS/CHECK/null tags as defined in the validation-suite framework of Section A8 .

Transformer Attention

Transformer attention lets models relate tokens by weighting information from other positions, enabling context-dependent sequence and multimodal processing.

See all 320 papers

Oct 7, 2026

HySPE: Positional Encoding via Symplectic Dual Shears

We introduce Hyperbolic Symplectic Positional Encoding (HySPE), grounding positional attention in non-compact symplectic transformations. While canonical Rotary Position Embedding (RoPE) parameterizes the compact, elliptic branch of

\Sp(2,\R)

via rotations, HySPE operationalizes its hyperbolic branch via a damped symmetric composition of dual shears, yielding a conformally symplectic contraction with two spectral decay rates per channel pair. To eliminate the exponential representation drift inherent to naive absolute factorizations, we diagonalize the operator in its invariant eigenbasis and introduce blockwise coordinate rebasing with adaptive centered execution. This guarantees length-independent numerical bounds while matching cached RoPE forward latency (7.21,ms on an RTX 4090). On TinyShakespeare, HySPE-UltraLong maintains an invariant perplexity of 4.810 up to

16\times

zero-shot extrapolation (

L=4096

), whereas RoPE degrades to 131.198. Scaled to a 51M-parameter subword Transformer on WikiText-103 (

L_{\text{train}}=512

), HySPE closely matches RoPE in-domain while robustly extrapolating to length 8192, reducing tail perplexity by 83.9% over RoPE. While these controlled experiments establish HySPE's extrapolation robustness and numerical stability, evaluating its scaling behavior on large-scale foundation models remains an important direction for future investigation.

Zhongping Ji

Positional Encoding Length Generalization Rotary Positional Embeddings

Ask PDF

Oct 20, 2026

RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies

Head-dimension compression can reduce the memory required by the key-value (KV) cache, but existing methods often reconstruct full-dimensional representations during inference, adding significant decoding FLOPs overhead. We formalize the design considerations for converting pretrained attention into a natively reduced-dimensional form that reduces both KV-cache storage and attention computation using a calibration set. Specifically, we consider reductions that (1) apply a shared projection to queries and keys so that they remain in a common reduced feature space, (2) derive this projection jointly from query and key activations to better preserve their interaction, and (3) are compatible with the pretrained RoPE transformation. To obtain a reconstruction-free realization of this reduction, we restrict it to coordinate selection and theoretically show that RoPE compatibility then holds if and only if complete rotary pairs are retained. Building on this characterization, we introduce RACE, a post-training method that selects rotary pairs using their joint query-key activation energy. We theoretically show that this criterion minimizes a separable, calibration-dependent upper bound on post-RoPE query-key product approximation error. The resulting reductions are folded into the model weights and require neither runtime reconstruction nor specialized kernels. Across three model families and different compression levels, RACE outperforms structured pruning and remains competitive with reconstruction-based methods on language-modeling, zero-shot, and long-context tasks. At 50% KV-cache retention and 256K context, RACE reduces total inference FLOPs by 47.8%, yielding 1.79x higher throughput, 43.2% lower time-to-first-token (TTFT), and 28.1% lower allocated GPU memory in comparison to an uncompressed model.

Irene Tenison, Abdulrahman Alabdulkareem, Daniel Karl I. Weidele +2

MIT CSAIL · MIT-IBM Computing Research Lab

KV-Cache Compression Transformer Attention LLM Inference Acceleration LLM Pruning

Ask PDF

Jul 7, 2026

Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

The pre-softmax score of an attention head is a bilinear form

score(i,j) = x_i^T M x_j

in a learned operator

M = W_q^T W_k

. Because M is generally non-symmetric, hence non-normal, it has a complex eigenspectrum and non-orthogonal eigenvectors, the regime where non-Hermitian and random-matrix tools apply. We ask what this spectrum encodes, at three levels for previous-token and induction circuits. Statically, across seven pretrained models spanning three positional schemes, the strongest previous-token heads are spectrally rotational under RoPE and non-rotational, or content-like, where position enters outside QK (learned-absolute and ALiBi); the model-level separation is perfect at every top-k examined (exact permutation

p=0.029

), and zeroing the per-frequency RoPE phase

Im(M_t)

eliminates induction on a pre-identified previous-token head in all three RoPE models. Dynamically, over public Pythia checkpoints every head originates at the random-matrix (Ginibre) null; the rotational signature emerges with the behavior, not before it, and the population-median suppression that yields the final profile follows circuit formation, so the profile is a consolidated fingerprint, not a precursor. Causally, and at toy scale, no spectral channel is necessary: constrained two-layer training reroutes around every ban with capability intact, albeit at a significant formation delay (four pre-registered contrasts,

q_BH <= 0.016

). The cost structure exposes each scheme's default: imposing symmetry slows learned-absolute models by a factor of 2.9, whereas a RoPE head with a fully symmetric static M still routes directionally via the phase channel, impossible under absolute positions. Within the settings examined, the positional scheme sets the default spectral algebra of an attention head's solution: a fingerprint sculpted after function, not a hard constraint upon it.

Li Hengyu

Institute for Solid State Physics, The University of Tokyo

Rotary Positional Embeddings Attention Head Analysis Induction Heads Multi-Head Attention

Ask PDF

Abstract

Transformer Attention

Explore similar work

HySPE: Positional Encoding via Symplectic Dual Shears

RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies

Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention