Position Embeddings

Position embeddings are crucial components in transformer-based models, encoding the positional information of input elements to overcome the inherent permutation invariance of self-attention mechanisms. Current research focuses on improving the efficiency and effectiveness of position embeddings, particularly within object detection, 3D scene understanding, and natural language processing tasks, exploring novel architectures like Relation-DETR and methods such as absolute window position embedding and dynamic position encoding. These advancements aim to enhance model performance, address issues like slow convergence and context awareness limitations, and improve robustness to variations in input sequence length and data augmentation strategies. Ultimately, refined position embedding techniques are vital for advancing the capabilities of transformer models across diverse applications.

Papers

September 29, 2024

PEAR: Position-Embedding-Agnostic Attention Re-weighting Enhances Retrieval-Augmented Generation with Zero Inference Overhead
Tao Tan, Yining Qian, Ang Lv, Hongzhan Lin, Songhao Wu, Yongbo Wang, Feng Wang, Jingtong Wu, Xin Lu, Rui Yan
Large Language Model Retrieval Augmented Generation Attention Weight Context Awareness Position Embeddings

July 16, 2024

Relation DETR: Exploring Explicit Position Relation Prior for Object Detection
Xiuquan Hou, Meiqin Liu, Senlin Zhang, Ping Wei, Badong Chen, Xuguang Lan
Contrastive Learning Object Detection Detection Transformer Position Embeddings Relation Aware

December 9, 2023

EipFormer: Emphasizing Instance Positions in 3D Instance Segmentation
Mengnan Zhao, Lihe Zhang, Yuqiu Kong, Baocai Yin
Point Cloud 3D Instance Segmentation Instance Query Position Embeddings

November 9, 2023

Window Attention is Bugged: How not to Interpolate Position Embeddings
Daniel Bolya, Chaitanya Ryali, Judy Hoffman, Christoph Feichtenhofer
Computer Vision ImageNet 1k Window Attention Position Embeddings Local Window

May 8, 2023

A Frustratingly Easy Improvement for Position Embeddings via Random Padding
Mingxu Tao, Yansong Feng, Dongyan Zhao
Pre Trained Language Model Extractive Question Answering Transformer Based Pre Trained Language Positional Embeddings Zero Padding Position Embeddings Simultaneous Improvement

April 17, 2023

Transformer with Selective Shuffled Position Embedding and Key-Patch Exchange Strategy for Early Detection of Knee Osteoarthritis
Zhe Wang, Aladine Chetouani, Mohamed Jarraya, Didier Hans, Rachid Jennane
Vision Transformer Data Augmentation Transformer Based Early Detection Data Augmentation Technique Knee Osteoarthritis Position Embeddings Certifiable Patch Defense

March 17, 2023

CAPE: Camera View Position Embedding for Multi-View 3D Object Detection
Kaixin Xiong, Shi Gong, Xiaoqing Ye, Xiao Tan, Ji Wan, Errui Ding, Jingdong Wang, Xiang Bai
3D Object Detection Multi View 3D Position Embeddings

January 9, 2023

Deep Planar Parallax for Monocular Depth Estimation
Haoqian Liang, Zhichao Li, Ya Yang, Naiyan Wang
Monocular Depth Estimation Depth Prediction Position Embeddings Planar Parallax

December 11, 2022

Focal-PETR: Embracing Foreground for Efficient Multi-Camera 3D Object Detection
Shihao Wang, Xiaohui Jiang, Ying Li
Background Generation 3D Feature Multi Camera 3D Object Detection Position Embeddings Local Image Feature

December 10, 2022

Position Embedding Needs an Independent Layer Normalization
Runyi Yu, Zhennan Wang, Yinhuai Wang, Kehan Li, Yian Zhao, Jian Zhang, Guoli Song, Jie Chen
Vision Transformer Product Specific Position Information Layer Normalization Token Embeddings Position Embeddings

July 15, 2022

Position Prediction as an Effective Pretraining Strategy
Shuangfei Zhai, Navdeep Jaitly, Jason Ramapuram, Dan Busbridge, Tatiana Likhomanenko, Joseph Yitan Cheng, Walter Talbott, Chen Huang, Hanlin Goh, Joshua Susskind
Transformer Based Supervised Autoencoder Pre Training BERT Model Position Prediction Position Embeddings

May 25, 2022

Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers
Bin Ren, Yahui Liu, Yue Song, Wei Bi, Rita Cucchiara, Nicu Sebe, Wei Wang
Vision Transformer PRIvacy Leakage Gradient Inversion Attack Jigsaw Puzzle Position Embeddings Absolute Pose Regressors

April 18, 2022

Dynamic Position Encoding for Transformers
Joyce Zheng, Mehdi Rezagholizadeh, Peyman Passban
Transformer Megatron Decepticons Neural Machine Translation Recurrent Network Position Embeddings Position Encoding Thought Empowers Transformer

March 14, 2022

XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understanding
Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, Liqing Zhang
Visually Rich Document Multimodal Network Position Embeddings Position Encoding

March 10, 2022

PETR: Position Embedding Transformation for Multi-View 3D Object Detection
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, Jian Sun
End to End Product Specific Position Information Image Feature Character Transformation Multi View 3D Multi View 3D Object Detection Position Embeddings

February 7, 2022

HeadPosr: End-to-end Trainable Head Pose Estimation using Transformer Encoders
Naina Dhingra
Human Head Encoder Side Transformer Encoder Transformer Encoders Head Pose Estimation Position Embeddings Prediction Head