Neural language models process sequences of words, but the mathematical operations inside them are insensitive to the order in which words appear. Positional encodings are the component added to remedy this. Despite their importance, positional encodings have been designed largely by trial and error, without a mathematical theory of what they ought to do. This paper develops such a theory. Four results are established. First, any Transformer without a positional signal cannot solve any task sensitive to word order (Necessity Theorem). Second, training assigns distinct vector representations to distinct sequence positions at every global minimiser, under mild and verifiable conditions (Positional Separation Theorem). Third, the best achievable approximation to an information-optimal encoding is constructed via classical multidimensional scaling (MDS) on the Hellinger distance between positional distributions; the quality of any encoding is measured by a single number, the stress (Proposition 5, Algorithm 1). Fourth, the optimal encoding has effective rank r = rank(B) <= n-1 and can be represented with r(n+d) parameters instead of nd (minimal parametrisation result). Appendix A develops a proof of the Monotonicity Conjecture within the Neural Tangent Kernel (NTK) regime for masked language modelling (MLM) losses, sequence classification losses, and general losses satisfying a positional sufficiency condition, through five lemmas. Experiments on SST-2 and IMDB with BERT-base confirm the theoretical predictions and reveal that Attention with Linear Biases (ALiBi) achieves much lower stress than the sinusoidal encoding and Rotary Position Embedding (RoPE), consistent with a rank-1 interpretation of the MDS encoding under approximate shift-equivariance.
Figures & tables
Condition
Intuitive meaning
When it may fail
(H1) Non-stationarity
Different positions attract different token types
Completely stationary corpora (uniform positional marginals); never occurs in natural language
(H2) Order sensitivity
Reordering tokens changes the correct answer
Bag-of-words tasks; for such tasks PE is unnecessary by definition
(H3) Non-degeneracy
M does not collapse differences in mean embeddings
Degenerate initialisations of WQ or WK ; holds with probability one at standard initialisations
Table 1: The three conditions of Theorem 4 : intuitive meaning and when each may fail.
Encoding
Rank r
Stress
Parameters
Saving vs free
Sinusoidal
d=128
18.98
0 (fixed)
—
PMDS , r=1
1
0.281
160
96.1%
PMDS , r=2
2
0.060
320
92.2%
PMDS , r=3
3
0.047
480
88.3%
PMDS , r=7
7
0.020
1,120
72.7%
PMDS (full)
31
0.000
4,960
−21.1%
Table 2: Rank–stress–parameter trade-off on the synthetic corpus ( n=32 , ∣V∣=200 , d=128 ). Sinusoidal has zero trainable parameters but high stress. The low-rank MDS encoding achieves much lower stress with far fewer parameters than a free matrix.
Encoding
Stress
PMDS (Algorithm 1 )
0.009
Sinusoidal ( Vaswani et al., 2017 )
2.248
Random initialisation
24.805
Table 3: Stress on synthetic corpus ( n=32 , d=16 ). Lower is better.
Figure 1: Synthetic corpus. Left: Hellinger distance matrix dH(μi,μj) ; block structure reflects the three positional regimes. Right: eigenspectrum of B ; two dominant eigenvalues indicate low intrinsic dimensionality.
Figure 2: Synthetic corpus. Left: PMDS in its first two dimensions (coloured by position index); the three regimes separate cleanly. Right: stress of three encodings; PMDS achieves 241× lower stress than sinusoidal.
Encoding
SST-2
IMDB
PMDS (Algorithm 1 )
≈0
≈0
ALiBi ( Press et al., 2022 )
0.563
3.051
Sinusoidal ( Vaswani et al., 2017 )
272
1,134
RoPE ( Su et al., 2024 )
279
1,140
Random initialisation
1,337
4,618
Table 4: Stress of positional encodings ( d=768 ) on SST-2 ( n=128 ) and IMDB ( n=256 ). Lower is better. PMDS achieves exact isometry on both corpora ( rank(B)≤d ).
Figure 3: Stress of five positional encodings on SST-2 and IMDB ( d=768 ). ALiBi achieves much lower stress than sinusoidal and RoPE on both corpora; PMDS is zero by construction.
Figure 4: Stress vs embedding dimension d (left) and cumulative variance explained by the top- d eigenvectors of B (right) for SST-2 (top) and IMDB (bottom). PMDS reaches zero at d=rank(B) ; sinusoidal and RoPE stress grows exponentially with d .
Figure 5: Minimum pairwise separation mini=j∥pi∗−pj∗∥ during training on SST-2. Both curves remain strictly positive throughout, consistent with Theorem 4 . The scratch model starts at 0.70 and remains flat; the pre-trained model starts at 0.35 and remains flat. The difference is explained by initialisation geometry, not by training dynamics.
Figure 6: Monotonicity violation rate for three encodings on SST-2. PMDS achieves 22.0% , well below the 50% random baseline; the pre-trained P∗ achieves 25.4% . The scratch P∗ ( 48.2% ) is near-random, consistent with insufficient training.
Figure 7: Pairwise distances in three encodings vs Hellinger distances on SST-2. Left: PMDS ( r=1.000 , by construction). Centre: P∗ from scratch ( r=0.036 , near-random). Right: P∗ pre-trained ( r=0.168 , moderate positive correlation).
Figure 8: Layer-wise stress of sinusoidal PE projected through M(l) on SST-2. The scratch model has near-zero stress at all layers (untrained attention matrices do not distort PE geometry). The pre-trained model shows a sharp peak at layer 3 ( ≈5,000 ), followed by a plateau at layers 4–11 ( 1,100 – 1,900 ) and a final rise at layer 12 ( ≈2,500 ).
Positional encoding (PE) underpins how permutation-invariant Transformers represent sequence order, yet how positional information is processed and stored remains poorly understood. Modern PE methods such as RoPE still struggle on tasks such as long-context understanding or retrieval \cite{chen-etal-2025-hope}. Hence, a better understanding of the internal positional mechanism could help design better PE. Building on evidence that positional and semantic signals occupy nearly orthogonal subspaces in trained Transformers, we modify an encoder Transformer to process three explicitly disentangled streams: semantic, absolute positional (AP) and relative positional (RP), and confine the masked-language-modeling (MLM) objective to the semantic stream. This decoupling enables a clean mechanistic study and yields three take-aways. (1) The isolated AP subspace spontaneously collapses into a low-frequency two-dimensional manifold that captures the structure of the document; (2) Attention heads specialize into structure and semantic-oriented groups, with RP exclusively supporting the latter; (3) Standard positional encodings do not robustly retain macroscopic structure: RoPE and RP only weakly encode it, and entangled AP loses it in the final layers under MLM pressure. The disentangled approach preserves positional encoding, which improves linguistic representation on 49 of the 65 linguistic phenomena of the Flash-Holmes probing benchmark.
Pierre-Antoine Lequeu, Camille Barboule, Benjamin Piwowarski
Sorbonne Université, CNRS, ISIR, Paris, France · Orange Innovation
Standard positional encodings for transformers - sinusoidal and rotary (RoPE) - treat every position as equally local: they encode where a token is, but not how far its positional influence should extend. We propose that the Morlet wavelet, which simultaneously minimises uncertainty in position and frequency, is the natural basis for positional encoding, and introduce Morlet Positional Encoding (MoPE): each embedding dimension learns its own frequency and locality bandwidth from data. The main theoretical result is a unification: sinusoidal PE and the RoPE correlation kernel both emerge as limiting cases of MoPE when locality is switched off (sigma_i -> infinity). The phase of MoPE recovers the RoPE rotation angle exactly; the amplitude adds a learned Gaussian locality kernel that standard encodings lack. Empirically, MoPE combined with Energy-Gated Attention achieves +0.119 improvement over standard attention on TinyShakespeare, outperforming either component alone. Analysis of the learned parameters reveals that all 128 frequency-bandwidth pairs converge to the wavelet admissibility boundary - an empirical observation consistent with a companion result on energy gating, suggesting a reproducible property of character-level language signals that warrants further investigation.
Sequence learning reduces to similarity-based retrieval over a temporally indexed representation space, a constraint on any sequence model, not a property of a specific architecture. We show that a spiking Sparse Distributed Memory sequence machine (2007) and the transformer (2017) independently instantiate the same five functional operations (encoding, context maintenance, associative retrieval, storage, and decoding), with cosine similarity as the shared retrieval primitive in both. We formalise a Phase-Latency Isomorphism showing that sinusoidal positional phase and spike timing are linearly related, and prove that dot product attention is invariant to this mapping up to a global scale factor on the positional component (Lemma 1). Empirically, frequency-compressed positional encoding fails to converge on a positionally demanding copy task, while a learned rank-based embedding matches or exceeds sinusoidal encoding, indicating that the critical property for positional representation is distance discriminability under dot-product similarity, not sinusoidal form. Time, phase, and rank are three instantiations of the same computational primitive, an ordered index whose structure survives similarity-based retrieval.