Polynomial Attention

Polynomial attention is a novel approach to improve the efficiency and expressiveness of attention mechanisms, primarily within transformer architectures. Current research focuses on developing polynomial-based alternatives to the computationally expensive softmax function used in standard attention, often incorporating techniques like polynomial sketching to achieve linear time complexity. This research aims to address the quadratic complexity bottleneck of traditional attention, enabling the training and deployment of larger models capable of handling longer sequences in applications ranging from natural language processing to solving partial differential equations. The resulting speed improvements and potential for enhanced model performance have significant implications for various fields requiring large-scale sequence modeling.