Relative Positional Encoding

Relative positional encoding (RPE) aims to improve the performance and generalization capabilities of transformer-based models by explicitly incorporating information about the relative positions of tokens within a sequence, addressing limitations of absolute positional encodings. Current research focuses on developing novel RPE methods, including those based on orthogonal polynomials, hyperbolic functions, and multiple kernel learning, to enhance length extrapolation, improve efficiency, and reduce positional bias in various applications such as natural language processing, computer vision, and time series analysis. These advancements are significant because they enable more robust and efficient processing of longer sequences and improve model performance across diverse tasks, impacting fields ranging from machine translation to medical image analysis.

Papers

July 18, 2023

Linearized Relative Positional Encoding
Zhen Qin, Weixuan Sun, Kaiyue Lu, Hui Deng, Dongxu Li, Xiaodong Han, Yuchao Dai, Lingpeng Kong, Yiran Zhong
Linear Transformer Relative Positional Encoding

June 14, 2023

Research on Named Entity Recognition in Improved transformer with R-Drop structure
Weidong Ji, Yousheng Zhang, Guohui Zhou, Xu Wang
Entity Recognition Transformer Architecture DH Research Transformer Encoder Relative Positional Encoding

May 26, 2023

March 8, 2023

RADAM: Texture Recognition through Randomized Aggregated Encoding of Deep Activation Maps
Leonardo Scabini, Kallil M. Zielinski, Lucas C. Ribas, Wesley N. Gonçalves, Bernard De Baets, Odemir M. Bruno
Relative Positional Encoding Texture Recognition Texture Representation Texture Analysis

February 21, 2023

Generic Dependency Modeling for Multi-Party Conversation
Weizhou Shen, Xiaojun Quan, Ke Yang
Transformer Based Language Model Multi Party Relative Positional Encoding Speech Utterance

February 3, 2023

Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers
Krzysztof Marcin Choromanski, Shanda Li, Valerii Likhosherstov, Kumar Avinava Dubey, Shengjie Luo, Di He, Yiming Yang, Tamas Sarlos, Thomas Weingarten, Adrian Weller
LeArning Abstract Transformer Megatron Decepticons Linear Attention Fourier Transform Linear Transformer Relative Positional Encoding Low Rank Attention

February 1, 2023

Reverse Ordering Techniques for Attention-Based Channel Prediction
Valentina Rizzello, Benedikt Böck, Michael Joham, Wolfgang Utschick
Seq2seq Model Sequence to Sequence Model Relative Positional Encoding Channel Prediction

September 9, 2022

SUPER-Rec: SUrrounding Position-Enhanced Representation for Recommendation
Taejun Lim, Siqu Long, Josiah Poon, Soyeon Caren Han
Effective Recommendation Item Representation Relative Positional Encoding Positional Embeddings Rating Scale

July 15, 2022

Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP
Zhicai Wang, Yanbin Hao, Xingyu Gao, Hao Zhang, Shuo Wang, Tingting Mu, Xiangnan He
Vision Transformer Multi Layer Token Level Multi Head Self Attention Relative Positional Encoding MLP Architecture Token Mixing

May 26, 2022

Your Transformer May Not be as Powerful as You Expect
Shengjie Luo, Shanda Li, Shuxin Zheng, Tie-Yan Liu, Liwei Wang, Di He
Transformer Based Universal Approximation Relative Positional Encoding Position Detection Transformer

May 10, 2022

KeypointNeRF: Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypoints
Marko Mihajlovic, Aayush Bansal, Michael Zollhoefer, Siyu Tang, Shunsuke Saito
3D Human Body Model Relative Positional Encoding Spatial Encoding Volumetric Avatar

February 23, 2022

FastRPB: a Scalable Relative Positional Encoding for Long Sequence Tasks
Maksim Zubkov, Daniil Gavrilov
Transformer Model Linear Transformer Relative Positional Encoding Sequence Task Shift Invariant Kernel

January 30, 2022

GRPE: Relative Positional Encoding for Graph Transformer
Wonpyo Park, Woonggi Chang, Donggeon Lee, Juntae Kim, Seung-won Hwang
Representation Learning Transformer Architecture Graph Transformer Positional Encoding Relative Positional Encoding Position Aware Graph

November 12, 2021

PESTO: Switching Point based Dynamic and Relative Positional Encoding for Code-Mixed Languages
Mohsin Ali, Kandukuri Sai Teja, Sumanth Manduru, Parth Patwa, Amitava Das
Word Embeddings Exceptional Point Code Mixed Relative Positional Encoding

Relative Positional Encoding

Papers

Linearized Relative Positional Encoding

Research on Named Entity Recognition in Improved transformer with R-Drop structure

Randomized Positional Encodings Boost Length Generalization of Transformers

Improving Position Encoding of Transformers for Multivariate Time Series Classification

RADAM: Texture Recognition through Randomized Aggregated Encoding of Deep Activation Maps

Generic Dependency Modeling for Multi-Party Conversation

Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers

Reverse Ordering Techniques for Attention-Based Channel Prediction

SUPER-Rec: SUrrounding Position-Enhanced Representation for Recommendation

Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP

Your Transformer May Not be as Powerful as You Expect

KeypointNeRF: Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypoints

FastRPB: a Scalable Relative Positional Encoding for Long Sequence Tasks

GRPE: Relative Positional Encoding for Graph Transformer

PESTO: Switching Point based Dynamic and Relative Positional Encoding for Code-Mixed Languages