Transformer Attention

In Transformer-based architectures, unmasked self-attention without positional information is permutation-equivariant with respect to the input sequence's tokens. To impose sequential order, token positions are typically encoded using a scheme with either fixed or learnable parameters. We introduce Hyperbolic Positional Encoding (HyPE), a method that utilizes hyperbolic functions' properties to encode tokens' relative positions. This approach biases the attention mechanism without the necessity of storing the $O(L^2)$ values of the bias tensor, with $L$ being the length of the input sequence. HyPE leverages preliminary concatenation operations and matrix multiplications, encoding relative distances by indirectly incorporating biases into the softmax computation. This design supports gradient backpropagation through the encoding and can be used with FlashAttention-2 under the width and scaling conditions below. We analytically demonstrate that, by careful hyperparameter selection, HyPE can approximate the causal attention bias of ALiBi \citep{press2021train} on a bounded range; model generalization beyond the pretraining length remains to be evaluated. The experimental evaluation of HyPE is proposed as a direction for future research.