Nonconformal Distortion and Local Orientation Reversal in the Fourier Circuit of a Modular-Addition Transformer

Eduardo Fuentes; Gonzalo Riera; Héctor Allende-Cid

cs.AIOct 30, 2026

Nonconformal Distortion and Local Orientation Reversal in the Fourier Circuit of a Modular-Addition Transformer

Authors: Eduardo Fuentes, Gonzalo Riera, Héctor Allende-Cid

Organizations: AI Mechanisms Lab, Chile · WriteWise, Chile · Facultad de Matemáticas, Pontificia Universidad Católica de Chile, Santiago, Chile · Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS, Germany ·

Abstract

We study the local differential geometry of a Fourier-based modular-addition circuit in a one-layer transformer whose algorithm has previously been reverse-engineered. For each of the five key Fourier frequencies, we restrict the model's multilayer perceptron (MLP) contribution to a corresponding Fourier input plane and readout, and measure its directional Beltrami coefficient under continuous input perturbations. Whereas nondegenerate restrictions of ideal complex multiplication are conformal, the trained maps exhibit nonconformal distortion (median $|μ|=0.304$ -- $0.446$ ), spatial organization in the Beltrami argument, and regions of local orientation reversal. The median across frequencies decreases from $0.939$ at initialization to $0.390$ after training. The key-frequency planes also show lower median distortion than most random-plane controls, although their distributions overlap and the coordinate frames differ in conditioning. Across 500 checkpoints in one training run, most of the aggregate reduction in Beltrami magnitude occurs before the principal increase in the measured Fourier-circuit formation. At the individual-input level, Beltrami magnitude shows negligible association with the correct-logit margin ( $|ρ|\leq0.026$ ). Late-checkpoint measurements across six runs show comparable distortion for most selected frequencies. These results provide a quantitative characterization of local deformation in a frequency-projected component of an identified neural circuit, complementary to its algorithmic reconstruction.

Figures & tables

Frequency $k$	Median $\|\mu\|$	Median $\sigma_{1}/\sigma_{2}$	Orientation preserved
14	0.3468	2.0617	93.8%
35	0.3902	2.2684	92.6%
41	0.3036	1.8718	93.8%
42	0.4455	2.6070	78.5%
52	0.4001	2.2798	94.9%

Table 1: Directional Beltrami distortion in the five key frequencies at epoch 40,000 (Regime A,

16\times 16

grid). The singular-value ratio includes orientation-reversing points.

Figure 2: Pointwise directional Beltrami field of the trained transformer (epoch 40,000; Regime A; $128\times 128$ torus grid). (A, B) $|\mu|$ for $k=14$ and $k=42$ on the orientation-preserving locus; dark gray indicates $|\mu|\geq 1$ (1.27% and 21.45%, respectively). (C, D) $\arg\mu$ on the same locus using a cyclic colormap. Sharp boundaries reflect the piecewise-analytic structure induced by ReLU activation regions.

Block	Condition	Median $\|\mu\|$	Orientation preserved
A	Gaussian bilinear reference (population)	1.000	0.500
A	Untrained weights, epoch 0	0.939	0.548
A	Trained weights, epoch 40,000	0.390	0.907
B	Five key-frequency planes	0.302–0.435	0.903
B	Twenty random control planes	0.697	0.799

Table 2: Analytical and empirical controls. Block A uses the Regime A

16\times 16

grid; Block B uses a separate

12\times 12

plane-comparison experiment. Summary statistics have different per-plane aggregation rules; definitions and comparability checks are provided in Appendix C.

Figure 3: Median

|\mu|

for the five highest Fourier-norm frequencies of each of six training runs, measured at epochs 40,000 and 40,100. Companion dilatation and band-sensitivity panels appear in Appendix D.

Figure 4: Geometric distortion, Fourier-circuit formation, and model behavior. (A) Median

|\mu_{k}|

for five key frequencies and their median at all 500 checkpoints (Regime A,

16\times 16

grid). (B) Fraction of pair-centered logit variance explained by the five key frequencies at those checkpoints. Dotted lines mark epochs 476, 2,390, and 9,830; the epoch axis is logarithmic above 100. (C) Spearman correlations of

|\mu_{k}|

with correct-logit margins for 8,939 test pairs at epoch 9,400, alongside the embedding-norm baseline; shading indicates the 95th percentile of

|\rho|

under 1,000 permutations.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

$k$	Median $K$ ( $\|\mu\|<1$ )	Median $\sigma_{1}/\sigma_{2}$	Difference	Excluded (%)
14	2.0437	2.0617	$+0.018$	6.25
35	2.0932	2.2684	$+0.175$	7.42
41	1.8648	1.8718	$+0.007$	6.25
42	2.1472	2.6070	$+0.460$	21.48
52	2.1582	2.2798	$+0.122$	5.08

Appendix

Table B1: Censored and uncensored median dilatation in Regime A (epoch 40,000;

16\times 16

grid). The median of

K

uses only points with

|\mu|<1

; the median singular-value ratio uses the full grid.

Figure B1: Orientation reversal and angular organization across the five key Fourier frequencies. (A) Percentage of evaluation points with

|\mu|\geq 1

on the

128\times 128

Regime A grid. (B) Observed concentration of the differences in

\arg\mu

between neighboring grid points on the

16\times 16

Regime A grid, restricted to

|\mu|<1

, compared with the mean and 95th percentile of a spatial permutation null (1,000 permutations).

$k$	Median $\|\mu^{(a)}\|$	95% bootstrap CI	Fraction $\|\mu\|<1$
14	0.3223	$[0.3035,\,0.3481]$	1.000
35	0.4432	$[0.3112,\,0.6419]$	0.875
41	0.3077	$[0.2942,\,0.3212]$	1.000
42	0.4525	$[0.3632,\,0.8923]$	0.675
52	0.3550	$[0.2642,\,0.6046]$	0.975

Appendix

Table B2: Directional Beltrami distortion around real input embeddings (Regime B). Each frequency is evaluated at 40 sampled input pairs; confidence intervals are percentile-bootstrap intervals for the median from 2,000 replicates. The derivative directions remain interventional.

$k$	Global axial resultant $\overline{R}$	Observed neighboring concentration	Permutation null mean	Permutation null 95th percentile
14	0.0134	0.9800	0.0412	0.0815
35	0.2019	0.7866	0.0524	0.1003
41	0.0124	0.8561	0.0420	0.0806
42	0.0722	0.8407	0.0470	0.0919
52	0.1592	0.7715	0.0447	0.0859

Appendix

Table B3: Angular organization of the directional Beltrami field in Regime A (epoch 40,000). The global axial resultant summarizes angular concentration across the grid, whereas neighboring concentration measures spatial organization of differences in

\arg\mu

on the

16\times 16

grid, restricted to

|\mu|<1

, with toroidal neighbors in both directions. The spatial-permutation null uses 1,000 permutations.

$k$	Same pattern (%)	Median jump same	Median jump changed	99th pct. same	99th pct. changed	Top 1% jumps with change	Spearman $\rho$
14	62.8	0.0000	0.0558	0.0001	1.8951	1.000	0.731
42	22.8	0.0261	0.0355	0.1354	0.3589	1.000	0.275

Appendix

Table B4: ReLU activation-pattern changes and local jumps of the complex directional Beltrami coefficient in Regime A (epoch 40,000;

128\times 128

grid). For each

k

, 32,768 directed toroidal neighboring pairs (one per grid point in each of two directions) are compared. Jumps are

|\mu(x)-\mu(y)|

; “same” and “changed” refer to equality of the feedforward ReLU activation patterns at the two points.

Diagnostic	Key Fourier planes	Random control planes	$k=31$
Number of planes	5	20	1
Median $\|\mu\|$	0.302–0.435 (range)	0.697 (median of medians)	1.219
Orientation preserved	0.903 (mean)	0.799 (median)	0.000
Input-frame $\kappa[C\ S]$	1.003–1.027 (range)	1.3445 median; [1.048, 2.672]	Not reported
Readout-frame $\kappa[u\ v]$	1.01–1.03 (range)	1.05–2.79 (range)	Not reported
Key-subspace energy overlap	Not applicable	0.0947 mean; 0.0944 median; [0.054, 0.142]	Not reported

Appendix

Table C1: Control-plane comparability and diagnostic summaries (epoch 40,000;

12\times 12

grid). Key frequencies are

k\in\{14,35,41,42,52\}

. The sixth frequency

k=31

is shown separately; its orientation sign depends on a declared, externally unanchored convention.

Figure C1: Scale sensitivity of the key Fourier planes and the twenty norm-matched random controls (epoch 40,000;

12\times 12

grid). The multiplier

m

ranges from

0.50

2.00

: random control input frames are normalized to

ms_{\star}

(

s_{\star}=0.51423

), while each key-frequency input frame is multiplied by

m

relative to its original scale. (A) Each gray point is one random plane’s median

|\mu|

; dark diamonds are the medians of the twenty per-plane medians. Horizontal jitter is only for visibility. (B) Median

|\mu|

for each of the five fixed key-frequency planes across scale multipliers; the dashed curve repeats the random-plane median-of-medians for comparison. Lines connect measurements of the same frequency at different scale multipliers. At

m\leq 1.00

, two to three individual random planes fall within the key-frequency range; none do at

m=1.50

2.00

. The control-group median remains higher at every tested multiplier.

		Epoch 40,000			Epoch 40,100
Seed	$k$	Med. $\|\mu\|$	Med. $K$	$f_{+}$	Med. $\|\mu\|$	Med. $K$	$f_{+}$
1	3	1.0010	221	0.4961	0.9982	830	0.7539
	6	0.2312	1.546	0.8828	0.2224	1.397	0.8711
	23	0.4983	2.816	0.8164	0.4584	2.356	0.7305
	35	0.9076	20.472	0.9688	1.1050	—	0.0000
	39	0.3192	1.936	0.9883	0.3544	1.969	0.9492

Appendix

Table D1: Complete per-seed directional Beltrami measurements. Five independently selected frequencies per seed, evaluated at epochs 40,000 and 40,100 (Regime A,

16\times 16

grid). The

K

column is the median on

|\mu|<1

;

f_{+}

is the fraction of grid points in that locus.

$k$	$M_{k}(0)$	$M_{k}(40{,}000)$	$F_{k}(500)$	$F_{k}(2{,}300)$	$F_{k}(2{,}400)$
14	0.4758	0.3468	$-0.102$	0.746	0.745
35	1.1289	0.3902	0.859	0.963	0.966
41	2.2779	0.3036	0.957	0.950	0.948
42	0.9394	0.4455	0.693	0.777	0.773
52	0.4035	0.4001	$47.8^{\dagger}$	$43.9^{\dagger}$	$42.4^{\dagger}$

Appendix

Table E1: Geometric change and Fourier-circuit formation at reference checkpoints. (A)

M_{k}(t)=\operatorname{med}|\mu_{k}|

on the Regime A

16\times 16

grid, and signed fractions

F_{k}(t)

defined in Eq. 1 . Epoch 2,300 is the last saved checkpoint strictly before generalization onset (epoch 2,390); epoch 500 is the sensitivity cutoff, and epoch 2,400 is the nearest saved checkpoint afterward. (B) Fraction of pair-centered logit variance explained by the five key frequencies at the same checkpoints.

$k$	Spearman $\rho$	Null 95th $\|\rho\|$	Permutation $p$	Exceeds null?
14	$-0.0260$	0.0206	0.010	Yes
35	$-0.0146$	0.0208	0.166	No
41	$-0.0070$	0.0212	0.507	No
42	$-0.0224$	0.0196	0.026	Yes
52	$-0.0058$	0.0208	0.582	No

Appendix

Table E2: Association of directional Beltrami magnitude with correct-logit margin. Spearman correlations for the five key frequencies at epoch 9,400 (8,939 held-out pairs; no exclusions). The null threshold is the frequency-specific 95th percentile of

|\rho|

over 1,000 label permutations; the permutation

p

-values are two-sided and unadjusted across frequencies.

Neural Representation Geometry

Neural representation geometry studies how learned features are arranged and how that structure relates to model behavior and performance.

See all 264 papers

Jul 8, 2026

Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits

Transformers have demonstrated a remarkable ability to learn algorithmic reasoning, yet mechanistic analyses have mostly focused on globally invertible operations such as cyclic addition and group composition. In this work, we investigate how small transformers learn modular integer multiplication over composite moduli, a fundamentally non-invertible operation due to the presence of zero-divisors. We propose the monoid extension: a localized generalization of Group Composition via Representation (GCR) that suggests the learned computation does not rely on a single global representation space. Instead, the model partitions the input space into local hierarchical algebraic regions, where group-like structure survives and Fourier mechanisms can be applied. In transformers trained on square-free modular multiplication, we find that embeddings organize around these regions, attention exhibits class-sensitive routing and low-rank write directions, and local character features explain a large fraction of the model's output logits. Our results suggest that representation-theoretic mechanisms previously identified for group operations can extend beyond groups to more general structures.

Zitong Andrew Chen, Junaid Hasan, Akhil Srinivasan +2

1UW Math AI Lab, University of Washington.

Transformer Interpretability Transformer Attention Modular Arithmetic LLM Mathematical Reasoning

Ask PDF

Aug 5, 2026

A Counterexample to Fourier Alignment in Single-Neuron Modular Addition

We give a negative solution to MAIS-O60. We first construct an example in which an initially active ReLU neuron becomes completely inactive in finite time and thereafter remains frozen at a limit whose Fourier energy is equally distributed among all nonzero real frequency classes. The counterexample holds on an open set of initial conditions and therefore occurs with positive probability under Gaussian initialization. An appendix prepared by GPT-5.6 Sol strengthens the counterexample by showing that the same failure can occur for every Clarke trajectory from an open set of initial conditions, under the convention

\mathrm{ReLU}'(0)=0

, for smooth dead-zone approximations of ReLU, and for fixed-step full-batch gradient descent. Thus, single-frequency alignment is not a general consequence of training a single neuron on modular addition.

Gautam Neelakantan Memana

Shallow Neural Networks ReLU Neural Networks Frequency-Domain Feature Learning Modular Addition Neural Network Training Dynamics

Ask PDF

Oct 8, 2026

Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks

Neural networks trained on modular addition tasks often develop Fourier-structured representations that support exact generalization. While prior work has identified these Fourier circuits, the mechanism by which gradient-based training selects them from the data distribution remains unclear. We address this question using probability signatures, which express leading gradient interactions through conditional statistics of the training distribution. For modular addition, these signatures are cyclic shift operators and are diagonalized by the discrete Fourier transform, yielding approximately decoupled Fourier-mode dynamics. This explains the emergence of Fourier sparsity, frequency matching, and phase alignment. The same framework resolves a puzzle under label noise: corrupted examples can show faster early loss decrease than clean examples, despite lacking a coherent generalization rule. We show that noise increases conditional label collisions, strengthening early shared-coordinate reinforcement. Finally, this method can be applied to other operators. Taking XOR as an example, we observed the predicted frequency in experiments.

Yunji Wang, Junjie Yao, Linyu Liu +2

School of Mathematical Sciences, Shanghai Jiao Tong University · Huawei Technologies Ltd · Shanghai University of Finance and Economics +1

Neural Network Training Dynamics Frequency-Domain Feature Learning Modular Addition

Ask PDF

Abstract

Neural Representation Geometry

Explore similar work

Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits

A Counterexample to Fourier Alignment in Single-Neuron Modular Addition

Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks