Selective Synchronization Attention

Hasi Hays

cs.LGFeb 16, 2026

Selective Synchronization Attention

Authors: Hasi Hays

Organizations: Department of Chemical Engineering, University of Arkansas, Fayetteville, AR 72701, USA

Abstract

The Transformer architecture has become the foundation of modern deep learning, yet its core self-attention mechanism suffers from quadratic computational complexity and lacks grounding in biological neural computation. We propose Selective Synchronization Attention (SSA), a novel attention mechanism that replaces the standard dot-product self-attention with a closed-form operator derived from the steady-state solution of the Kuramoto model of coupled oscillators. In SSA, each token is represented as an oscillator characterized by a learnable natural frequency and phase; the synchronization strength between token pairs, determined by a frequency-dependent coupling and phase-locking condition, serves as the attention weight. This formulation provides three key advantages: (i) natural sparsity arising from the phase-locking threshold, whereby tokens with incompatible frequencies automatically receive zero attention weight without explicit masking; (ii) unified positional-semantic encoding through the natural frequency spectrum, eliminating the need for separate positional encodings; and (iii) a single-pass, closed-form computation that avoids iterative ODE integration, with all components (coupling, order parameter, synchronization) derived from the oscillatory framework. We instantiate SSA within the Oscillatory Synchronization Network (OSN), a drop-in replacement for the Transformer block. Analysis of the synchronization matrices reveals non-uniform, head-diverse coupling patterns even at initialization, demonstrating a stronger architectural inductive bias than the approximately uniform attention produced by randomly initialized Transformers.

Figures & tables

Figure 1: Selective Synchronization Attention: conceptual illustration for the sentence “The cat sat on the mat.” (a) Phase space: tokens are represented as oscillators on the phase circle. Tokens with similar natural frequencies (

\omega_{1}

\omega_{2}

) synchronize into coherent clusters (red and blue), enabling information exchange via solid connections. Adjacent tokens from different clusters (“sat” and “on”) remain desynchronized (

S_{ij}=0

, dashed line), producing natural sparsity. (b) Frequency space: the two clusters occupy distinct regions, separated by the phase-locking threshold

|\Delta\omega|>KrJ

. (c) Synchronization matrix

\mathbf{S}

: the resulting block-diagonal structure, with strong within-cluster synchronization and zero cross-cluster weights.

Figure 2: Architecture comparison. Left: Standard Transformer block with multi-head dot-product attention and explicit positional encoding. Right: OSN block with Multi-Frequency Synchronization Heads using the closed-form phase-alignment operator. The blocks share identical input-output interfaces (

\mathbb{R}^{N\times D}\to\mathbb{R}^{N\times D}

), enabling drop-in replacement.

Figure 3: Single-block GPU benchmark (NVIDIA A100, Google Colab). Left: Throughput (K tokens/sec) vs. sequence length. Center: Forward pass latency (ms). Right: Peak GPU memory (MB). Batch sizes adapted per sequence length (

B=8

for

N\leq 512

;

B=4,2,1

for

N=1024,2048,4096

). Configuration:

D=512

H=8

k=64

for sparse variant. Each measurement averaged over 50 trials with 10 warmup iterations.

		K tok/s ( $\uparrow$ )			Memory ( $\downarrow$ )
$N$	$B$	TF	D	S	TF	D	S
128	8	1033	665	638	313	345	351
256	8	1527	1118	788	353	481	493
512	8	1636	950	700	481	993	1017
1024	4	1334	593	457	609	1633	1657
2048	2	977	350	277	865	2913	2937

Table 1: Single-block GPU benchmark (NVIDIA A100).

N

= sequence length,

B

= batch size. TF = Transformer, D = OSN dense, S = OSN sparse (

k\!=\!64

). Throughput in K tokens/sec (

\uparrow

); peak memory in MB (

\downarrow

Figure 4: Synchronization matrices $\mathbf{S}^{(h)}$ from eight Multi-Frequency Synchronization Heads for a randomly initialized OSN block ( $D=512$ , $H=8$ , NVIDIA A100). Each panel shows the first $128\times 128$ tokens of a 512-length sequence. The diagonal reflects self-synchronization ( $S_{ii}=1$ ), while off-diagonal values encode the frequency-dependent coupling and phase-locking condition. Different heads exhibit distinct synchronization profiles, demonstrating multi-frequency head diversity.

Figure 5: Distribution of the empirical order parameter

r

across eight synchronization heads, computed from 100 random input samples (

N=256

D=512

, NVIDIA A100). The order parameter concentrates around

r\approx 0.847

with low variance (

\sigma\approx 0.002

), demonstrating stable emergent phase coherence consistent with the Kuramoto formulation.

Explore similar work

Kuramoto Attention: Synchronizing Self-Attention on the Torus
Jun 10, 2026Joshua NunleyDynamical Systems Self-Attention
Department of Informatics, Luddy School of Informatics, Computing, and Engineering Cognitive Science Program Indiana University Bloomington
Attention as Frustrated Synchronization
Jun 17, 2026Joshua NunleyOscillatory Neural Networks Autoregressive Language Modeling
Cognitive Science Program Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington
Attention by Synchronization in Coupled Oscillator Networks
Jun 10, 2026Fabio Pasqualetti, Taosha GuoSoftmax Attention Dynamical Systems
Electrical Engineering and Computer Science University of California, Irvine

Jun 10, 2026J/K move · Enter open · S save

Kuramoto Attention: Synchronizing Self-Attention on the Torus

Joshua Nunley

Department of Informatics, Luddy School of Informatics, Computing, and Engineering Cognitive Science Program Indiana University Bloomington

Dynamical Systems Self-Attention

Open paper AskPDF arXiv

Transformer models are increasingly used as computational models of cognition and neural representation, so the mechanism implemented by self-attention is of interest beyond engineering performance. A complementary tradition in cognitive science models coordination, binding, and memory through dynamical interactions such as oscillator synchrony; we bring this mechanism into self-attention by introducing the Kuramoto Attention layer, whose value update is a synchronization step. Each token carries a bank of phase oscillators, so its hidden state lives on a high-dimensional torus. The attention weights form an adaptive coupling graph, and using the raw phase states as values makes the value update exactly the Kuramoto coupling direction for fixed attention weights. The softmax selects which oscillators couple, while the value path moves each token toward the attention-weighted circular mean of the tokens it selects. We train Kuramoto Attention on enwiki8 and CodeParrot against parameter-matched RoPE and SwiGLU transformers. At 5M parameters on CodeParrot, it improves on the transformer by both median and mean, with mean gaps of 0.012 validation and 0.010 test bits per byte. At 5M on enwiki8, all six runs have lower validation/test medians than the transformer and all-seed means within 0.01 BPC; five of six also form a tight lower-mean cluster. At 1M, it trails by about 0.02 BPC on enwiki8 and by 0.013-0.015 bits per byte on CodeParrot. Ablations and phase diagnostics show how the layer's synchronization and geometry-motivated components shape model performance. The result is a self-attention mechanism whose learned computation can be read directly as adaptive synchronization on phase states.

Abstract

Explore similar work

Kuramoto Attention: Synchronizing Self-Attention on the Torus

Attention as Frustrated Synchronization

Attention by Synchronization in Coupled Oscillator Networks