Positional Embeddings

Positional embeddings are crucial for encoding the sequential or spatial order of data within neural networks, enabling models to understand context and relationships beyond simple feature values. Current research focuses on improving the efficiency and effectiveness of positional embeddings, particularly for handling very long sequences in large language models and adapting them to irregular data structures like point clouds and irregular time series. These advancements are significantly impacting various fields, enhancing performance in tasks ranging from machine translation and recommendation systems to medical image analysis and 3D point cloud processing. The development of more robust and flexible positional embeddings is a key area of ongoing investigation, driving improvements in model accuracy and scalability.

Papers

October 14, 2024

A Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformers
Alex Stein, Samuel Sharpe, Doron Bergman, Senthil Kumar, C. Bayan Bruss, John Dickerson, Tom Goldstein, Micah Goldblum
Transformer Megatron Decepticons Event Description Event Sequence Temporal Embeddings Event Prediction Positional Embeddings Auto Regressive Autoregressive LLM

October 5, 2024

Correlation-Aware Select and Merge Attention for Efficient Fine-Tuning and Context Length Extension
Ning Wang, Zekun Li, Tongxin Bai, Guoqi Li
Large Language Model Long Sequence Sparse Attention Context Length Selection Policy Positional Embeddings

September 30, 2024

Continuous-Time Linear Positional Embedding for Irregular Time Series Forecasting
Byunghyun Kim, Jae-Gil Lee
Time Series Forecasting Positional Embeddings Irregular Time Series Sampled Time Series

September 16, 2024

Anatomical Positional Embeddings
Mikhail Goncharov, Valentin Samokhin, Eugenia Soboleva, Roman Sokolov, Boris Shirokikh, Mikhail Belyaev, Anvar Kurmukov, Ivan Oseledets
Voxel Wise Positional Embeddings Voxel Based

August 21, 2024

Towards Inducing Document-Level Abilities in Standard Multilingual Neural Machine Translation Models
Varun Gumma, Pranjal A. Chitale, Kalika Bali
Neural Machine Translation Multilingual Neural Machine Translation Versatile Approach Positional Embeddings Sinusoidal Activation Pre Trained Neural Machine Translation

July 12, 2024

Pre-training Point Cloud Compact Model with Partial-aware Reconstruction
Yaohua Zha, Yanzi Wang, Tao Dai, Shu-Tao Xia
Point Cloud Masked Modeling Positional Embeddings

July 3, 2024

Learning Positional Attention for Sequential Recommendation
Fan Luo, Juan Zhang, Shenghui Xu
Self Attention Sequential Recommendation Attention Model Positional Embeddings Position Aware Attention

May 30, 2024

Contextual Counting: A Mechanistic Study of Transformers on a Quantitative Task
Siavash Golkar, Alberto Bietti, Mariel Pettee, Michael Eickenberg, Miles Cranmer, Keiya Hirashima, Geraud Krawezik, Nicholas Lourie, Michael McCabe, Rudy Morel, Ruben Ohana, Liam Holden Parker, Bruno Régaldo-Saint Blancard, Kyunghyun Cho, Shirley Ho
Transformer Megatron Decepticons Causal Transformer Positional Embeddings Causal Attention Mechanistic Study Rotary Position Quantitative Explanation Based Counting

May 29, 2024

Contextual Position Encoding: Learning to Count What's Important
Olga Golovneva, Tianlu Wang, Jason Weston, Sainbayar Sukhbaatar
Positional Encoding Borda Counting Positional Embeddings Position Encoding

May 23, 2024

Configuring Data Augmentations to Reduce Variance Shift in Positional Embedding of Vision Transformers
Bum Jun Kim, Sang Woo Kim
Vision Transformer Data Augmentation Data Augmentation Method Positional Embeddings

February 21, 2024

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, Mao Yang
Large Language Model K TOKEN LLM Fine Tuning Long Text Positional Embeddings Context Window LLM in Context Synthetic Fibre Rope

November 1, 2023

Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation
Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky
Transformer Language Model Positional Embeddings Attention Alignment Transformer Length Extrapolation

September 7, 2023

DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions
Haochen Wang, Junsong Fan, Yuxi Wang, Kaiyou Song, Tong Wang, Zhaoxiang Zhang
Vision Transformer Pre Trained Vision Transformer Attention Based Model Positional Embeddings Self Supervised Pretext Task

September 1, 2023

Robust Point Cloud Processing through Positional Embedding
Jianqiao Zheng, Xueqian Li, Sameera Ramasinghe, Simon Lucey
Point Cloud Point Cloud Classification Point Cloud Processing Positional Embeddings Point Cloud Transformer Point Embeddings

August 14, 2023

A Novel Ehanced Move Recognition Algorithm Based on Pre-trained Models with Positional Embeddings
Hao Wen, Jie Wang, Xiaodong Qiao
Pre Trained Model Positional Embeddings Motion Recognition

June 27, 2023

SPDER: Semiperiodic Damping-Enabled Object Representation
Kathan Shah, Chawin Sitawarin
Neural Network Architecture Audio Representation Image Representation Neural Implicit Representation Positional Embeddings Periodic Phenomenon

May 23, 2023

Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
Ta-Chung Chi, Ting-Han Fan, Li-Wei Chen, Alexander I. Rudnicky, Peter J. Ramadge
Self Attention Transformer Language Model Positional Embeddings Frozen Language Model Latent Position

May 8, 2023

December 21, 2022

Reconstruction Probing
Najoung Kim, Jatin Khilnani, Alex Warstadt, Abed Qaddoumi
Language Model Contextual Representation Linear Probing Positional Embeddings First Learning Based Reconstructability Predictor