Memory-Efficient Attention

Slim attention shrinks the context memory size by 2x for transformer models with MHA (multi-head attention), which can speed up inference by up to 2x for large context windows. Slim attention is an exact, mathematically identical implementation of the standard attention mechanism and therefore doesn't compromise model accuracy. In other words, slim attention losslessly compresses the context memory by a factor of 2. For encoder-decoder transformers, the context memory size can be reduced even further: For the Whisper models for example, slim attention reduces the context memory by 8x, which can speed up token generation by 5x for batch size 64 for example. And for the T5-11B model for example, the memory can be reduced by 32x because its MHA projection dimension is larger than the embedding dimension. See https://github.com/OpenMachine-ai/transformer-tricks for code and more transformer tricks, and https://www.youtube.com/watch?v=uVtk3B6YO4Y for this paper's YouTube video.

In Transformer-based architectures, unmasked self-attention without positional information is permutation-equivariant with respect to the input sequence's tokens. To impose sequential order, token positions are typically encoded using a scheme with either fixed or learnable parameters. We introduce Hyperbolic Positional Encoding (HyPE), a method that utilizes hyperbolic functions' properties to encode tokens' relative positions. This approach biases the attention mechanism without the necessity of storing the $O(L^2)$ values of the bias tensor, with $L$ being the length of the input sequence. HyPE leverages preliminary concatenation operations and matrix multiplications, encoding relative distances by indirectly incorporating biases into the softmax computation. This design supports gradient backpropagation through the encoding and can be used with FlashAttention-2 under the width and scaling conditions below. We analytically demonstrate that, by careful hyperparameter selection, HyPE can approximate the causal attention bias of ALiBi \citep{press2021train} on a bounded range; model generalization beyond the pretraining length remains to be evaluated. The experimental evaluation of HyPE is proposed as a direction for future research.