Relative Positional Encoding

Relative positional encoding (RPE) aims to improve the performance and generalization capabilities of transformer-based models by explicitly incorporating information about the relative positions of tokens within a sequence, addressing limitations of absolute positional encodings. Current research focuses on developing novel RPE methods, including those based on orthogonal polynomials, hyperbolic functions, and multiple kernel learning, to enhance length extrapolation, improve efficiency, and reduce positional bias in various applications such as natural language processing, computer vision, and time series analysis. These advancements are significant because they enable more robust and efficient processing of longer sequences and improve model performance across diverse tasks, impacting fields ranging from machine translation to medical image analysis.

Papers

October 28, 2024

HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation
Yuhan Chen, Ang Lv, Jian Luan, Bin Wang, Wei Liu
Positional Encoding Extrapolation Framework Relative Positional Encoding Context Awareness

July 25, 2024

Relating the Seemingly Unrelated: Principled Understanding of Generalization for Generative Models in Arithmetic Reasoning Tasks
Xingcheng Xu, Zibo Zhao, Haipeng Zhang, Yanqing Yang
Generative Model Strong Generalization General Principle Relative Positional Encoding Arithmetic Task Generalization Technique

July 3, 2024

PosMLP-Video: Spatial and Temporal Relative Position Encoding for Efficient Video Recognition
Yanbin Hao, Diansong Zhou, Zhicai Wang, Chong-Wah Ngo, Meng Wang
Vision Transformer Spatial Domain Video Recognition Relative Positional Encoding Video LMMs Temporal Positional Encoding

July 1, 2024

Eliminating Position Bias of Language Models: A Mechanistic Approach
Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, Heng Ji
Language Model Relative Positional Encoding Mechanistic Model Causal Attention

June 14, 2024

LieRE: Generalizing Rotary Position Encodings
Sophie Ostmeier, Brian Axelrod, Michael E. Moseley, Akshay Chaudhari, Curtis Langlotz
Lie Group Relative Positional Encoding Dimensional Input 3D Object Classification Rotary Position

May 23, 2024

DAPE: Data-Adaptive Positional Encoding for Length Extrapolation
Chuanyang Zheng, Yihang Gao, Han Shi, Minbin Huang, Jingyao Li, Jing Xiong, Xiaozhe Ren, Michael Ng, Xin Jiang, Zhenguo Li, Yu Li
Positional Encoding Relative Positional Encoding Length Extrapolation

April 29, 2024

PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Models
Arpit Aggarwal
Positional Encoding Relative Positional Encoding Baseline Transformer

March 26, 2024

MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation
Weiguo Gao
Relative Positional Encoding Length Extrapolation Cartesian Product Extrapolation

March 21, 2024

KeyPoint Relative Position Encoding for Face Recognition
Minchul Kim, Yiyang Su, Feng Liu, Anil Jain, Xiaoming Liu
Face Recognition Affine Transformation Relative Positional Encoding Image Alignment Affine Transform

February 22, 2024

Comparing Graph Transformers via Positional Encodings
Mitchell Black, Zhengchao Wan, Gal Mishne, Amir Nayyeri, Yusu Wang
Graph Transformer Positional Encoding Relative Positional Encoding

January 29, 2024

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang, Liwei Wang, Jingjing Xu, Zhi Zhang, Hongxia Yang, Di He
Positional Encoding Bird Specie Inter Frame Relative Positional Encoding Length Extrapolation STONE Deterioration Pattern

October 30, 2023

HyPE: Attention with Hyperbolic Biases for Relative Positional Encoding
Giorgio Angelotti
Attention Mechanism Human Attention Input Sequence Data Encoding Transformer Based Architecture Relative Positional Encoding

October 6, 2023

Functional Interpolation for Relative Positions Improves Long Context Transformers
Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, Srinadh Bhojanapalli
Transformer Architecture Gradient Approximation Relative Positional Encoding Context Transformer Text Benchmark General Transformer Position Encoding Relative Position

October 3, 2023

RoFormer for Position Aware Multiple Instance Learning in Whole Slide Image Classification
Etienne Pochet, Rami Maroun, Roger Trullo
Deep Learning Self Attention Multiple Instance Learning Computational Pathology Instance Learning Whole Slide Image Classification Relative Positional Encoding

August 31, 2023

Decoupled Local Aggregation for Point Cloud Learning
Binjie Chen, Yunzhou Xia, Yu Zang, Cheng Wang, Jonathan Li
Point Cloud lightWeight Network Relative Positional Encoding Point Cloud Learning Point Feature Local Aggregation

August 30, 2023

LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models
Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, Sinong Wang
Large Language Model Long Context Length Generalization Relative Positional Encoding

August 9, 2023

Optimizing a Transformer-based network for a deep learning seismic processing workflow
Randy Harsuko, Tariq Alkhalifah
Deep Learning Transformer Based Network Seismic Data Relative Positional Encoding

August 8, 2023

V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection
Yichao Shen, Zigang Geng, Yuhui Yuan, Yutong Lin, Ze Liu, Chunyu Wang, Han Hu, Nanning Zheng, Baining Guo
Point Cloud 3D Object Detection 3D Object Relative Positional Encoding Group DETR V2 RGB D Datasets Lw Detr

August 7, 2023

RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling
Herman Sugiharto, Aradea, Husni Mubarok
Attention Module Multi Head Attention Multi Head Relative Positional Encoding Natural Language Model

July 19, 2023

Exploring Transformer Extrapolation
Zhen Qin, Yiran Zhong, Hui Deng
Receptive Field Long Sequence Language Modeling Task Relative Positional Encoding Length Extrapolation Transformer Length Extrapolation

Relative Positional Encoding

Papers

HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation

Relating the Seemingly Unrelated: Principled Understanding of Generalization for Generative Models in Arithmetic Reasoning Tasks

PosMLP-Video: Spatial and Temporal Relative Position Encoding for Efficient Video Recognition

Eliminating Position Bias of Language Models: A Mechanistic Approach

LieRE: Generalizing Rotary Position Encodings

DAPE: Data-Adaptive Positional Encoding for Length Extrapolation

PoPE: Legendre Orthogonal Polynomials Based Position Encoding for Large Language Models

MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation

KeyPoint Relative Position Encoding for Face Recognition

Comparing Graph Transformers via Positional Encodings

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation

HyPE: Attention with Hyperbolic Biases for Relative Positional Encoding

Functional Interpolation for Relative Positions Improves Long Context Transformers

RoFormer for Position Aware Multiple Instance Learning in Whole Slide Image Classification

Decoupled Local Aggregation for Point Cloud Learning

LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models

Optimizing a Transformer-based network for a deep learning seismic processing workflow

V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection

RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling

Exploring Transformer Extrapolation