Nonconformal Distortion and Local Orientation Reversal in the Fourier Circuit of a Modular-Addition Transformer

Eduardo Fuentes; Gonzalo Riera; Héctor Allende-Cid

cs.AIOct 30, 2026

Nonconformal Distortion and Local Orientation Reversal in the Fourier Circuit of a Modular-Addition Transformer

Authors: Eduardo Fuentes, Gonzalo Riera, Héctor Allende-Cid

Organizations: AI Mechanisms Lab, Chile · WriteWise, Chile · Facultad de Matemáticas, Pontificia Universidad Católica de Chile, Santiago, Chile · Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS, Germany ·

Abstract

We study the local differential geometry of a Fourier-based modular-addition circuit in a one-layer transformer whose algorithm has previously been reverse-engineered. For each of the five key Fourier frequencies, we restrict the model's multilayer perceptron (MLP) contribution to a corresponding Fourier input plane and readout, and measure its directional Beltrami coefficient under continuous input perturbations. Whereas nondegenerate restrictions of ideal complex multiplication are conformal, the trained maps exhibit nonconformal distortion (median $|μ|=0.304$ -- $0.446$ ), spatial organization in the Beltrami argument, and regions of local orientation reversal. The median across frequencies decreases from $0.939$ at initialization to $0.390$ after training. The key-frequency planes also show lower median distortion than most random-plane controls, although their distributions overlap and the coordinate frames differ in conditioning. Across 500 checkpoints in one training run, most of the aggregate reduction in Beltrami magnitude occurs before the principal increase in the measured Fourier-circuit formation. At the individual-input level, Beltrami magnitude shows negligible association with the correct-logit margin ( $|ρ|\leq0.026$ ). Late-checkpoint measurements across six runs show comparable distortion for most selected frequencies. These results provide a quantitative characterization of local deformation in a frequency-projected component of an identified neural circuit, complementary to its algorithmic reconstruction.

Figures & tables

Frequency $k$	Median $\|\mu\|$	Median $\sigma_{1}/\sigma_{2}$	Orientation preserved
14	0.3468	2.0617	93.8%
35	0.3902	2.2684	92.6%
41	0.3036	1.8718	93.8%
42	0.4455	2.6070	78.5%
52	0.4001	2.2798	94.9%

Table 1: Directional Beltrami distortion in the five key frequencies at epoch 40,000 (Regime A,

16\times 16

grid). The singular-value ratio includes orientation-reversing points.

Figure 2: Pointwise directional Beltrami field of the trained transformer (epoch 40,000; Regime A; $128\times 128$ torus grid). (A, B) $|\mu|$ for $k=14$ and $k=42$ on the orientation-preserving locus; dark gray indicates $|\mu|\geq 1$ (1.27% and 21.45%, respectively). (C, D) $\arg\mu$ on the same locus using a cyclic colormap. Sharp boundaries reflect the piecewise-analytic structure induced by ReLU activation regions.

Block	Condition	Median $\|\mu\|$	Orientation preserved
A	Gaussian bilinear reference (population)	1.000	0.500
A	Untrained weights, epoch 0	0.939	0.548
A	Trained weights, epoch 40,000	0.390	0.907
B	Five key-frequency planes	0.302–0.435	0.903
B	Twenty random control planes	0.697	0.799

Table 2: Analytical and empirical controls. Block A uses the Regime A

16\times 16

grid; Block B uses a separate

12\times 12

plane-comparison experiment. Summary statistics have different per-plane aggregation rules; definitions and comparability checks are provided in Appendix C.

Figure 3: Median

|\mu|

for the five highest Fourier-norm frequencies of each of six training runs, measured at epochs 40,000 and 40,100. Companion dilatation and band-sensitivity panels appear in Appendix D.

Figure 4: Geometric distortion, Fourier-circuit formation, and model behavior. (A) Median

|\mu_{k}|

for five key frequencies and their median at all 500 checkpoints (Regime A,

16\times 16

grid). (B) Fraction of pair-centered logit variance explained by the five key frequencies at those checkpoints. Dotted lines mark epochs 476, 2,390, and 9,830; the epoch axis is logarithmic above 100. (C) Spearman correlations of

|\mu_{k}|

with correct-logit margins for 8,939 test pairs at epoch 9,400, alongside the embedding-norm baseline; shading indicates the 95th percentile of

|\rho|

under 1,000 permutations.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

$k$	Median $K$ ( $\|\mu\|<1$ )	Median $\sigma_{1}/\sigma_{2}$	Difference	Excluded (%)
14	2.0437	2.0617	$+0.018$	6.25
35	2.0932	2.2684	$+0.175$	7.42
41	1.8648	1.8718	$+0.007$	6.25
42	2.1472	2.6070	$+0.460$	21.48
52	2.1582	2.2798	$+0.122$	5.08

Appendix

Table B1: Censored and uncensored median dilatation in Regime A (epoch 40,000;

16\times 16

grid). The median of

K

uses only points with

|\mu|<1

; the median singular-value ratio uses the full grid.

Figure B1: Orientation reversal and angular organization across the five key Fourier frequencies. (A) Percentage of evaluation points with

|\mu|\geq 1

on the

128\times 128

Regime A grid. (B) Observed concentration of the differences in

\arg\mu

between neighboring grid points on the

16\times 16

Regime A grid, restricted to

|\mu|<1

, compared with the mean and 95th percentile of a spatial permutation null (1,000 permutations).

$k$	Median $\|\mu^{(a)}\|$	95% bootstrap CI	Fraction $\|\mu\|<1$
14	0.3223	$[0.3035,\,0.3481]$	1.000
35	0.4432	$[0.3112,\,0.6419]$	0.875
41	0.3077	$[0.2942,\,0.3212]$	1.000
42	0.4525	$[0.3632,\,0.8923]$	0.675
52	0.3550	$[0.2642,\,0.6046]$	0.975

Appendix

Table B2: Directional Beltrami distortion around real input embeddings (Regime B). Each frequency is evaluated at 40 sampled input pairs; confidence intervals are percentile-bootstrap intervals for the median from 2,000 replicates. The derivative directions remain interventional.

$k$	Global axial resultant $\overline{R}$	Observed neighboring concentration	Permutation null mean	Permutation null 95th percentile
14	0.0134	0.9800	0.0412	0.0815
35	0.2019	0.7866	0.0524	0.1003
41	0.0124	0.8561	0.0420	0.0806
42	0.0722	0.8407	0.0470	0.0919
52	0.1592	0.7715	0.0447	0.0859

Appendix

Table B3: Angular organization of the directional Beltrami field in Regime A (epoch 40,000). The global axial resultant summarizes angular concentration across the grid, whereas neighboring concentration measures spatial organization of differences in

\arg\mu

on the

16\times 16

grid, restricted to

|\mu|<1

, with toroidal neighbors in both directions. The spatial-permutation null uses 1,000 permutations.

$k$	Same pattern (%)	Median jump same	Median jump changed	99th pct. same	99th pct. changed	Top 1% jumps with change	Spearman $\rho$
14	62.8	0.0000	0.0558	0.0001	1.8951	1.000	0.731
42	22.8	0.0261	0.0355	0.1354	0.3589	1.000	0.275

Appendix

Table B4: ReLU activation-pattern changes and local jumps of the complex directional Beltrami coefficient in Regime A (epoch 40,000;

128\times 128

grid). For each

k

, 32,768 directed toroidal neighboring pairs (one per grid point in each of two directions) are compared. Jumps are

|\mu(x)-\mu(y)|

; “same” and “changed” refer to equality of the feedforward ReLU activation patterns at the two points.

Diagnostic	Key Fourier planes	Random control planes	$k=31$
Number of planes	5	20	1
Median $\|\mu\|$	0.302–0.435 (range)	0.697 (median of medians)	1.219
Orientation preserved	0.903 (mean)	0.799 (median)	0.000
Input-frame $\kappa[C\ S]$	1.003–1.027 (range)	1.3445 median; [1.048, 2.672]	Not reported
Readout-frame $\kappa[u\ v]$	1.01–1.03 (range)	1.05–2.79 (range)	Not reported
Key-subspace energy overlap	Not applicable	0.0947 mean; 0.0944 median; [0.054, 0.142]	Not reported

Appendix

Table C1: Control-plane comparability and diagnostic summaries (epoch 40,000;

12\times 12

grid). Key frequencies are

k\in\{14,35,41,42,52\}

. The sixth frequency

k=31

is shown separately; its orientation sign depends on a declared, externally unanchored convention.

Figure C1: Scale sensitivity of the key Fourier planes and the twenty norm-matched random controls (epoch 40,000;

12\times 12

grid). The multiplier

m

ranges from

0.50

2.00

: random control input frames are normalized to

ms_{\star}

(

s_{\star}=0.51423

), while each key-frequency input frame is multiplied by

m

relative to its original scale. (A) Each gray point is one random plane’s median

|\mu|

; dark diamonds are the medians of the twenty per-plane medians. Horizontal jitter is only for visibility. (B) Median

|\mu|

for each of the five fixed key-frequency planes across scale multipliers; the dashed curve repeats the random-plane median-of-medians for comparison. Lines connect measurements of the same frequency at different scale multipliers. At

m\leq 1.00

, two to three individual random planes fall within the key-frequency range; none do at

m=1.50

2.00

. The control-group median remains higher at every tested multiplier.

		Epoch 40,000			Epoch 40,100
Seed	$k$	Med. $\|\mu\|$	Med. $K$	$f_{+}$	Med. $\|\mu\|$	Med. $K$	$f_{+}$
1	3	1.0010	221	0.4961	0.9982	830	0.7539
	6	0.2312	1.546	0.8828	0.2224	1.397	0.8711
	23	0.4983	2.816	0.8164	0.4584	2.356	0.7305
	35	0.9076	20.472	0.9688	1.1050	—	0.0000
	39	0.3192	1.936	0.9883	0.3544	1.969	0.9492

Appendix

Table D1: Complete per-seed directional Beltrami measurements. Five independently selected frequencies per seed, evaluated at epochs 40,000 and 40,100 (Regime A,

16\times 16

grid). The

K

column is the median on

|\mu|<1

;

f_{+}

is the fraction of grid points in that locus.

$k$	$M_{k}(0)$	$M_{k}(40{,}000)$	$F_{k}(500)$	$F_{k}(2{,}300)$	$F_{k}(2{,}400)$
14	0.4758	0.3468	$-0.102$	0.746	0.745
35	1.1289	0.3902	0.859	0.963	0.966
41	2.2779	0.3036	0.957	0.950	0.948
42	0.9394	0.4455	0.693	0.777	0.773
52	0.4035	0.4001	$47.8^{\dagger}$	$43.9^{\dagger}$	$42.4^{\dagger}$

Appendix

Table E1: Geometric change and Fourier-circuit formation at reference checkpoints. (A)

M_{k}(t)=\operatorname{med}|\mu_{k}|

on the Regime A

16\times 16

grid, and signed fractions

F_{k}(t)

defined in Eq. 1 . Epoch 2,300 is the last saved checkpoint strictly before generalization onset (epoch 2,390); epoch 500 is the sensitivity cutoff, and epoch 2,400 is the nearest saved checkpoint afterward. (B) Fraction of pair-centered logit variance explained by the five key frequencies at the same checkpoints.

$k$	Spearman $\rho$	Null 95th $\|\rho\|$	Permutation $p$	Exceeds null?
14	$-0.0260$	0.0206	0.010	Yes
35	$-0.0146$	0.0208	0.166	No
41	$-0.0070$	0.0212	0.507	No
42	$-0.0224$	0.0196	0.026	Yes
52	$-0.0058$	0.0208	0.582	No

Appendix

Table E2: Association of directional Beltrami magnitude with correct-logit margin. Spearman correlations for the five key frequencies at epoch 9,400 (8,939 held-out pairs; no exclusions). The null threshold is the frequency-specific 95th percentile of

|\rho|

over 1,000 label permutations; the permutation

p

-values are two-sided and unadjusted across frequencies.

Neural Representation Geometry

Neural representation geometry studies how learned features are arranged and how that structure relates to model behavior and performance.

See all 264 papers

Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits
Jul 8, 2026Zitong Andrew Chen, Junaid Hasan, Akhil Srinivasan +2Transformer Interpretability Transformer Attention
1UW Math AI Lab, University of Washington.
A Counterexample to Fourier Alignment in Single-Neuron Modular Addition
Aug 5, 2026Gautam Neelakantan MemanaShallow Neural Networks ReLU Neural Networks
Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks
Oct 8, 2026Yunji Wang, Junjie Yao, Linyu Liu +2Neural Network Training Dynamics Frequency-Domain Feature Learning
School of Mathematical Sciences, Shanghai Jiao Tong University · Huawei Technologies Ltd · Shanghai University of Finance and Economics +1

Jul 8, 2026J/K move · Enter open · S save

Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits

Zitong Andrew Chen, Junaid Hasan, Akhil Srinivasan, Hemkesh Bandi+1

1UW Math AI Lab, University of Washington.

Transformer Interpretability Transformer Attention Modular Arithmetic LLM Mathematical Reasoning

Open paper AskPDF arXiv

Transformers have demonstrated a remarkable ability to learn algorithmic reasoning, yet mechanistic analyses have mostly focused on globally invertible operations such as cyclic addition and group composition. In this work, we investigate how small transformers learn modular integer multiplication over composite moduli, a fundamentally non-invertible operation due to the presence of zero-divisors. We propose the monoid extension: a localized generalization of Group Composition via Representation (GCR) that suggests the learned computation does not rely on a single global representation space. Instead, the model partitions the input space into local hierarchical algebraic regions, where group-like structure survives and Fourier mechanisms can be applied. In transformers trained on square-free modular multiplication, we find that embeddings organize around these regions, attention exhibits class-sensitive routing and low-rank write directions, and local character features explain a large fraction of the model's output logits. Our results suggest that representation-theoretic mechanisms previously identified for group operations can extend beyond groups to more general structures.

Abstract

Neural Representation Geometry

Explore similar work

Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits

A Counterexample to Fourier Alignment in Single-Neuron Modular Addition

Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks