cs.LGNov 5, 2026

ConvPE: Convolutional Position Embedding Improves Transformers

Authors: Zehao Wang

Organizations: Nanjing University

Abstract

ConvPE (Convolutional Position Embedding) improves both the performance and efficiency of Transformer models in language modeling, long-context extension, and vision. It injects a windowed, content-dependent bias into attention scores through a lightweight causal convolution over a low-dimensional projection of the hidden states. As a lightweight plug-in, ConvPE is compatible with RoPE, attention masks, and KV concatenation. It also supports high-dimensional position embeddings via high-dimensional convolutions. In a reproduction, ConvPE outperforms five 1D baselines and eight 2D baselines. The code is publicly available at https://github.com/wzhagi/convpe.

Figures & tables

Explore similar work

CardsList
  1. RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

    May 15, 2026Yufeng Du, Phillip Harris, Minyang Tian +5Rotary Positional EmbeddingsSelf-Attention