Neural language models process sequences of words, but the mathematical operations inside them are insensitive to the order in which words appear. Positional encodings are the component added to remedy this. Despite their importance, positional encodings have been designed largely by trial and error, without a mathematical theory of what they ought to do. This paper develops such a theory. Four results are established. First, any Transformer without a positional signal cannot solve any task sensitive to word order (Necessity Theorem). Second, training assigns distinct vector representations to distinct sequence positions at every global minimiser, under mild and verifiable conditions (Positional Separation Theorem). Third, the best achievable approximation to an information-optimal encoding is constructed via classical multidimensional scaling (MDS) on the Hellinger distance between positional distributions; the quality of any encoding is measured by a single number, the stress (Proposition 5, Algorithm 1). Fourth, the optimal encoding has effective rank r = rank(B) <= n-1 and can be represented with r(n+d) parameters instead of nd (minimal parametrisation result). Appendix A develops a proof of the Monotonicity Conjecture within the Neural Tangent Kernel (NTK) regime for masked language modelling (MLM) losses, sequence classification losses, and general losses satisfying a positional sufficiency condition, through five lemmas. Experiments on SST-2 and IMDB with BERT-base confirm the theoretical predictions and reveal that Attention with Linear Biases (ALiBi) achieves much lower stress than the sinusoidal encoding and Rotary Position Embedding (RoPE), consistent with a rank-1 interpretation of the MDS encoding under approximate shift-equivariance.
Figures & tables
Condition
Intuitive meaning
When it may fail
(H1) Non-stationarity
Different positions attract different token types
Completely stationary corpora (uniform positional marginals); never occurs in natural language
(H2) Order sensitivity
Reordering tokens changes the correct answer
Bag-of-words tasks; for such tasks PE is unnecessary by definition
(H3) Non-degeneracy
M does not collapse differences in mean embeddings
Degenerate initialisations of WQ or WK ; holds with probability one at standard initialisations
Table 1: The three conditions of Theorem 4 : intuitive meaning and when each may fail.
Encoding
Rank r
Stress
Parameters
Saving vs free
Sinusoidal
d=128
18.98
0 (fixed)
—
PMDS , r=1
1
0.281
160
96.1%
PMDS , r=2
2
0.060
320
92.2%
PMDS , r=3
3
0.047
480
88.3%
PMDS , r=7
7
0.020
1,120
72.7%
PMDS (full)
31
0.000
4,960
−21.1%
Table 2: Rank–stress–parameter trade-off on the synthetic corpus ( n=32 , ∣V∣=200 , d=128 ). Sinusoidal has zero trainable parameters but high stress. The low-rank MDS encoding achieves much lower stress with far fewer parameters than a free matrix.
Encoding
Stress
PMDS (Algorithm 1 )
0.009
Sinusoidal ( Vaswani et al., 2017 )
2.248
Random initialisation
24.805
Table 3: Stress on synthetic corpus ( n=32 , d=16 ). Lower is better.
Figure 1: Synthetic corpus. Left: Hellinger distance matrix dH(μi,μj) ; block structure reflects the three positional regimes. Right: eigenspectrum of B ; two dominant eigenvalues indicate low intrinsic dimensionality.
Figure 2: Synthetic corpus. Left: PMDS in its first two dimensions (coloured by position index); the three regimes separate cleanly. Right: stress of three encodings; PMDS achieves 241× lower stress than sinusoidal.
Encoding
SST-2
IMDB
PMDS (Algorithm 1 )
≈0
≈0
ALiBi ( Press et al., 2022 )
0.563
3.051
Sinusoidal ( Vaswani et al., 2017 )
272
1,134
RoPE ( Su et al., 2024 )
279
1,140
Random initialisation
1,337
4,618
Table 4: Stress of positional encodings ( d=768 ) on SST-2 ( n=128 ) and IMDB ( n=256 ). Lower is better. PMDS achieves exact isometry on both corpora ( rank(B)≤d ).
Figure 3: Stress of five positional encodings on SST-2 and IMDB ( d=768 ). ALiBi achieves much lower stress than sinusoidal and RoPE on both corpora; PMDS is zero by construction.
Figure 4: Stress vs embedding dimension d (left) and cumulative variance explained by the top- d eigenvectors of B (right) for SST-2 (top) and IMDB (bottom). PMDS reaches zero at d=rank(B) ; sinusoidal and RoPE stress grows exponentially with d .
Figure 5: Minimum pairwise separation mini=j∥pi∗−pj∗∥ during training on SST-2. Both curves remain strictly positive throughout, consistent with Theorem 4 . The scratch model starts at 0.70 and remains flat; the pre-trained model starts at 0.35 and remains flat. The difference is explained by initialisation geometry, not by training dynamics.
Figure 6: Monotonicity violation rate for three encodings on SST-2. PMDS achieves 22.0% , well below the 50% random baseline; the pre-trained P∗ achieves 25.4% . The scratch P∗ ( 48.2% ) is near-random, consistent with insufficient training.
Figure 7: Pairwise distances in three encodings vs Hellinger distances on SST-2. Left: PMDS ( r=1.000 , by construction). Centre: P∗ from scratch ( r=0.036 , near-random). Right: P∗ pre-trained ( r=0.168 , moderate positive correlation).
Figure 8: Layer-wise stress of sinusoidal PE projected through M(l) on SST-2. The scratch model has near-zero stress at all layers (untrained attention matrices do not distort PE geometry). The pre-trained model shows a sharp peak at layer 3 ( ≈5,000 ), followed by a plateau at layers 4–11 ( 1,100 – 1,900 ) and a final rise at layer 12 ( ≈2,500 ).