Complex-valued Phase-Coherent Transformers

Leona Hioki

cs.LGSep 18, 2026

Complex-valued Phase-Coherent Transformers

Authors: Leona Hioki

Abstract

Complex-valued Transformers have inherited softmax attention over the raw complex inner product. Outside natively complex domains this standard form stays near chance, and no complex attention had been shown to correct it. We show that the match must be a scaled cosine score: L2-normalise queries and keys, so the score reads their cosine similarity and ignores their magnitudes, and hold that score at order-one scale. With this the same models train on four diagnostic tasks under two different gates; without the normalisation they stay at chance on ListOps and Needle under both gates and fall far below on the other two, and a normalised score placed at too small a scale fails as well. The resulting family of phase-coherent Transformers (\PCT) matches or exceeds the strongest real-valued baseline across long-range memory, positional retrieval, hierarchical reasoning, frequency-domain classification and physical complex signals; it shows no degradation up to depth 20; and its loss decreases log-linearly over a 61-fold range of parameters. A member of the family, complex screening combined with a phase-coherent recurrence, is the first genuinely complex-valued neural network to solve Path-X, with 91.6% of its trainable parameters complex-valued against 38.2% for S4. We record these as signs of generalisation not previously seen in complex-valued neural networks.

Figures & tables

task (chance)	sigmoid gate	row-softmax gate
Copy Memory (0.10)	1.000 $\,\pm\,$ 0.000 / 0.151 $\,\pm\,$ 0.017	1.000 $\,\pm\,$ 0.000 / 0.820 $\,\pm\,$ 0.183
ListOps (0.10)	0.498 $\,\pm\,$ 0.016 / 0.134 $\,\pm\,$ 0.006	0.526 $\,\pm\,$ 0.007 / 0.134 $\,\pm\,$ 0.006
Needle (0.016)	1.000 $\,\pm\,$ 0.000 / 0.016 $\,\pm\,$ 0.001	1.000 $\,\pm\,$ 0.000 / 0.000 †
FFT-MNIST (0.10)	0.498 $\,\pm\,$ 0.003 / 0.381 $\,\pm\,$ 0.018	0.507 $\,\pm\,$ 0.026 / 0.384 $\,\pm\,$ 0.013

Table 1: Accuracy with / without query–key normalisation, under two gates. † Single 30K-step run at this configuration; fifteen further runs of the same unnormalised cell at other lengths and budgets all score

\leq 0.25

gate	Copy	ListOps	FFT-MNIST	Needle
sigmoid ( PCT )	1.000 $\,\pm\,$ 0.000	0.498 $\,\pm\,$ 0.016	0.498 $\,\pm\,$ 0.003	1.000 $\,\pm\,$ 0.000
L2-softmax	1.000 $\,\pm\,$ 0.000	0.526 $\,\pm\,$ 0.007	0.507 $\,\pm\,$ 0.026	1.000 $\,\pm\,$ 0.000
screening	1.000 $\,\pm\,$ 0.000	0.506 $\,\pm\,$ 0.011	0.693 $\,\pm\,$ 0.060	1.000 ‡

Table 2: The three gates of the family on the scaled cosine score (same configurations as Table 1 ). ‡ Single seed at this configuration.

gate	Copy: $b{=}-\log N\to b{=}0$	FFT-MNIST: $b{=}-\log N\to b{=}0$
ReLU	0.102 $\,\pm\,$ 0.001 $\to$ 1.000 $\,\pm\,$ 0.000	0.314 $\,\pm\,$ 0.014 $\to$ 0.674 $\,\pm\,$ 0.033
clamped ReLU	0.102 $\,\pm\,$ 0.001 $\to$ 1.000 $\,\pm\,$ 0.000	0.314 $\,\pm\,$ 0.014 $\to$ 0.618 $\,\pm\,$ 0.095
squared ( $s^{2}$ )	0.998 $\,\pm\,$ 0.003 $\to$ 1.000 $\,\pm\,$ 0.000	0.219 $\,\pm\,$ 0.010 $\to$ 0.589 $\,\pm\,$ 0.074
cubed ( $s^{3}$ )	1.000 $\,\pm\,$ 0.000 $\to$ 1.000 $\,\pm\,$ 0.000	0.228 $\,\pm\,$ 0.015 $\to$ 0.485 $\,\pm\,$ 0.094
sigmoid (control)	1.000 $\,\pm\,$ 0.000 $\to$ 1.000 $\,\pm\,$ 0.000	0.498 $\,\pm\,$ 0.003 $\to$ 0.516 $\,\pm\,$ 0.071

Table 3: Accuracy before and after moving the gate bias to

0

variant	property changed	Copy	ListOps	FFT-MNIST
sigmoid ( PCT )	—	1.000 $\,\pm\,$ 0.000	0.498 $\,\pm\,$ 0.016	0.498 $\,\pm\,$ 0.003
L2-softmax	keys compete ( $\sum_{j}\alpha_{ij}=1$ )	1.000 $\,\pm\,$ 0.000	0.526 $\,\pm\,$ 0.007	0.507 $\,\pm\,$ 0.026
screening	multiplicative screening	1.000 $\,\pm\,$ 0.000	0.506 $\,\pm\,$ 0.011	0.693 $\,\pm\,$ 0.060
complex-valued gate a	gate rotates each contribution	1.000 $\,\pm\,$ 0.000	0.298 $\,\pm\,$ 0.043	0.471 $\,\pm\,$ 0.039
complex-normalised softmax b	normaliser rotates tokens	0.909 $\,\pm\,$ 0.157	0.429 $\,\pm\,$ 0.043	0.416 $\,\pm\,$ 0.028
Holographic-style c	phase-interference decay, value rotation	1.000 $\,\pm\,$ 0.000	0.483 $\,\pm\,$ 0.016	0.536 $\,\pm\,$ 0.073

Table 4: Gate variants on the scaled cosine score.

{}^{a}\;\alpha_{ij}=\sigma(s_{ij}+b)\,e^{is_{ij}}

. b Weights

e^{z_{ij}}/\sum_{k}e^{z_{ik}}

with complex logits

z_{ij}=\langle\bar{q}_{i},\bar{k}_{j}\rangle\sqrt{d_{h}}

. c Row softmax over

s_{ij}\,e^{-\alpha|\Delta\phi_{ij}|}

with

\Delta\phi_{ij}=\arg\langle q_{i},k_{j}\rangle

and learnable

\alpha

, values rotated by

e^{i\Delta\phi_{ij}}

, at the score scale of Section 2.1 ; on Needle all three seeds reach

1.000

by step 2,500 (in-loop evaluation; the runs ended before the final evaluation).

task	real sigmoid, normalised	real softmax, normalised	complex counterparts
Copy $d{=}1000$	1.000 $\,\pm\,$ 0.000	1.000 $\,\pm\,$ 0.000	1.000 / 1.000
Needle $L{=}2048$	0.317 $\,\pm\,$ 0.170 (max 0.50)	0.259 $\,\pm\,$ 0.133 (max 0.40)	1.000 / 1.000

Table 5: Real-valued cells with normalised queries and keys (

N{=}3

task	evaluation	r_softmax	r_sigmoid	r_screen	c_softmax	PCT	c_screen
Copy $d{=}500$ (mid)	synthetic	0.445	0.945	1.000	0.092	1.000	0.528
Copy $d{=}2000$ (mid)	synthetic	0.100	0.083	1.000	0.100	1.000	0.683
Needle $L{=}2048$ (mid)	synthetic, $N{=}1^{\ast}$	0.000	0.000	1.000	0.000	1.000	1.000
ListOps $L{=}1024$	synthetic	0.146	0.177	0.698	0.104	0.854	0.833
LRA-Text 4K	held-out	0.627	0.627	0.632	0.628	0.665	0.635
LRA-Image (CIFAR)	held-out	0.222	0.224	0.318	0.232	0.372	0.341

Table 6: Six cells across nine rows (means over three seeds; per-seed values in the public archive). “mid” rows are equal-width (dim 256, depth 6). ∗ Single seed for every cell except PCT (

N{=}3

, all seeds at 1.000). The synthetic rows are final-step readings of the original sweep (one batch of 32 sequences); the same cells re-measured with a deterministic 2,048-sample evaluation at the Table 1 configuration appear in Tables 1 – 2 and Sections 2.4 – 2.7 , and no claim in the text rests on a synthetic-row margin smaller than

0.1

task	c_softmax (unnormalised)	PCT
Copy $d{=}500$ (mid)	0.092	1.000
Copy $d{=}2000$ (mid)	0.100	1.000
Needle $L{=}2048$ (mid)	0.000	1.000
ListOps $L{=}1024$	0.104	0.854
LRA-Text 4K	0.628	0.665
LRA-Image (CIFAR)	0.232	0.372

Table 7: PCT against the unnormalised complex softmax, from Table 6 . The cells share projections, values, feed-forward and training; they differ only in the normalisation of the score.

task	r_softmax	r_sigmoid	r_screen	PCT
Copy $d{=}500$ (mid)	0.445	0.945	1.000	1.000
Copy $d{=}2000$ (mid)	0.100	0.083	1.000	1.000
Needle $L{=}2048$ (mid)	0.000	0.000	1.000	1.000
ListOps $L{=}1024$	0.146	0.177	0.698	0.854
LRA-Text 4K	0.627	0.627	0.632	0.665
LRA-Image (CIFAR)	0.222	0.224	0.318	0.372

Table 8: PCT against the three real cells, from Table 6 . Real cells use dim 184 (head dimension 46) against dim 128 (head dimension 32) for PCT , so both sides hold the same number of stored scalars (Section 3.1 ); the mid rows are equal-width.

depth	params	30K steps ( $N{=}3$ )	100K (seed 0)
2	0.40M	0.623 $\,\pm\,$ 0.007	0.636
4	0.80M	0.623 $\,\pm\,$ 0.014	—
6	1.19M	0.634 $\,\pm\,$ 0.011	0.674
10	1.98M	0.654 $\,\pm\,$ 0.015	0.697
14	2.77M	0.620 $\,\pm\,$ 0.040	—
20	3.96M	0.633 $\,\pm\,$ 0.006	0.650

Table 9: Left: depth robustness of PCT at fixed width 128 (ListOps

L{=}1024

N{=}3

, 2,048-sample evaluation), with a 100K-step continuation of seed 0. Right: scaling at constant aspect ratio

\mathrm{dim}/\mathrm{depth}=32

, 60K steps,

N{=}3

model	Path-X
Transformer; Reformer, Performer, Linformer, BigBird, Luna-256	chance ( $\approx$ 50)
S4-v1	88.10
DSS	89.72
S4D-LegS	91.95
complex screening + phase-coherent recurrence (ours)	92.71 $\,\pm\,$ 0.89
S4D-Inv	92.80

Table 10: Rule-compliant Path-X test accuracy (%). Ours is the mean over three seeds

\,\pm\,

sample s.d., a first-pass run. Baselines: S4-v1/S4-LegS and MEGA from Ma et al. (2023); S4D and DSS from Gu et al. (2022); S5 and LRU from Orvieto et al. (2023).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

cell	LR $10^{-3}$	LR $3\cdot 10^{-3}$	LR $10^{-2}$	$b{=}8$ , $d{=}2000$	$b{=}32$ , $d{=}2000$	$b{=}256$ , $d{=}500$
real_softmax	0.37	0.69	0.31	0.04	0.10	0.05
real_sigmoid	0.07	1.00	0.22	0.04	0.10	0.07
real_screen	0.55	0.70	1.00	0.69	1.00	0.79
complex_softmax	0.09	0.39	0.35	0.08	0.08	0.06
PCT	1.00	1.00	1.00	1.00	1.00	1.00
complex_screen	0.39	1.00	0.08	0.03	0.53	0.31

Appendix

Table 11: Copy accuracy across learning rates (left) and effective batch sizes (right).

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 540 papers

Complex-Valued Phase-Coherent Transformer
May 11, 2026Leona HiokiSoftmax Attention Complex-Valued Neural Networks
When do complex-valued neural networks help? A study of representation, geometry, and optimization
May 26, 2026Ashutosh KumarNeural Network Optimization Complex-Valued Neural Networks
Owl Autonomous Imaging, Inc.
Perturbing the Phase: Analyzing Adversarial Robustness of Complex-Valued Neural Networks
Feb 6, 2026Florian Eilers, Christof Duhme, Xiaoyi JiangAdversarial Attacks Neural Network Robustness
Department of Computer Science University of Münster Münster, Germany

May 11, 2026J/K move · Enter open · S save

Complex-Valued Phase-Coherent Transformer

Leona Hioki

Softmax Attention Complex-Valued Neural Networks Transformer Attention Attention Mechanisms Sequence Modeling

Open paper AskPDF arXiv

Complex-valued Transformers have largely inherited softmax attention from real-valued architectures. However, row-normalised token competition is not necessarily aligned with phase-preserving computation. In this paper, we introduce the Phase-Coherent Transformer (PCT), which applies a real-valued, element-independent, smooth gate to L2-normalised complex query-key similarities. PCT replaces token competition with token-non-competing attention and is designed to preserve phase information across layers. Across mid-scale benchmarks spanning long-range memory, hierarchical long-range reasoning, positional retrieval, phase-based memory and superposition, and image classification, PCT shows strong generalisation across task categories. Under parameter-fair comparison, PCT consistently outperforms both the standard softmax Transformer and its direct complex-valued counterpart. Moreover, even on tasks traditionally considered difficult for complex-valued neural networks, such as NIAH and LRA-Text, PCT remains competitive with Multiscreen, the strongest real-valued NN baseline in our comparison. Experiments introducing gates that deliberately violate the PCT conditions show that the design is not incidental: smooth gates that preserve negatively aligned phase components remain strong, whereas gates that delete such components collapse on long-range retrieval, and gates whose outputs become excessively large suffer clear performance degradation. PCT also shows no depth-related accuracy collapse across the tested depth range. These results support introducing multi-layer phase-coherent structure into attention as a promising design principle for achieving generalisation in complex-valued Transformers.

Abstract

Transformer

Explore similar work

Complex-Valued Phase-Coherent Transformer

When do complex-valued neural networks help? A study of representation, geometry, and optimization

Perturbing the Phase: Analyzing Adversarial Robustness of Complex-Valued Neural Networks