Self-Attention

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 195

All topics

Interpolation between Convolution and Attention via K-Nearest Neighbors
May 31, 2026Mingi KangSelf-Attention Convolutional Neural Networks
Bowdoin College
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
May 29, 2026Harry Jake Cunningham, Nicola Muca CironeTransformer Interpretability Feature Attribution
Department of Computer Science, University College London, London, UK. · Cartesia.AI, San Francisco, US.
VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion
May 28, 2026Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral +4Video Diffusion Models Self-Attention
1Virginia Tech · 2fal
Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables
May 28, 2026Masaaki Imaizumi, Masanori Koyama, Noboru Isobe +1Transformer Self-Attention
The University of Tokyo, Tokyo, Japan · RIKEN Advanced Intelligene Project, Tokyo, Japan · Kyoto University, Kyoto, Japan
Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics
May 28, 2026Matthew Smart, Soumya Ganguly, Nilava Metya +2Self-Attention Empirical Bayes
Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ, USA · Department of Mathematics, Rutgers University, Piscataway, NJ, USA · Department of Physics and Astronomy, Rutgers University, Piscataway, NJ, USA +1
Parallax: Parameterized Local Linear Attention for Language Modeling
May 27, 2026Yifei Zuo, Dhruv Pai, Zhichen Zeng +3Language Model Pretraining Self-Attention
Northwestern University · Tilde Research · University of Washington
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
May 27, 2026Xiuying Wei, Caglar GulcehreMemory-Augmented Language Models Self-Attention
EPFL, CLAIRE Lausanne, Switzerland
PLS in the Mirror of Self-Attention
May 27, 2026Jiangsheng, YouRecursive Least Squares Self-Attention
Jason
Meta-Attention: Bayesian Per-Token Routing for Efficient Transformer Inference
May 27, 2026Alan FerrariEfficient Transformer Inference Self-Attention
Knowledge Lab AG Zürich, Switzerland
UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training
May 26, 2026Keqi Deng, Shaoshi Ling, Ruchao Fan +1Self-Attention Long-Context Language Model Inference
Microsoft, USA
Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective
May 26, 2026Hyunmin Cho, Woo Kyoung Han, Kyong Hwan JinSelf-Attention Transformer Attention
Department of Electrical Engineering, Korea University, Seoul, South Korea.
AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers
May 26, 2026Semi Lee, Hyejin Go, Hyesong ChoiToken Merging Self-Attention
Electronic Engineering · Soongsil University · Seoul, South Korea
Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
May 25, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusion
May 25, 2026Tuna Tuncer, Felix Becker, Thomas PfeilVideo Diffusion Models Self-Attention
1Technical University of Munich · 2Tensordyne
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
May 25, 2026Xintong Yang, Hao Gu, Binxing Xu +6Memory-Augmented Language Models Self-Attention
The Hong Kong University of Science and Technology · Zhejiang University
Quaternion Self-Attention with Shared Scores
May 24, 2026Shogo Yamauchi, Tohru Nitta, Hideaki TamoriSelf-Attention Attention Mechanisms
The Asahi Shimbun Company, Tokyo, Japan · Tokyo Woman’s Christian University, Tokyo, Japan.
Interdomain Attention: Beyond Token-Level Key-Value Memory
May 23, 2026Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin +4Self-Attention Long-Context Language Modeling
Approaching I/O-optimality for Approximate Attention
May 22, 2026Pál András Papp, Aleksandros Sobczyk, Anastasios ZouziasSelf-Attention Efficient Attention
Computing Systems Lab, Huawei Technologies, Zurich, Switzerland
Improved Belief-Attention in Vision Task
May 22, 2026Guoqiang ZhangVisual Attention Self-Attention
University of Exeter
Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
May 22, 2026Yongzhong XuTransformer Interpretability Circuit Discovery
ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
May 21, 2026Joe SharrattMixed-Precision Quantization Self-Attention
Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity
May 21, 2026Hangyue Zhao, Paul Caillon, Erwan Fagnou +1Self-Attention Structured Sparsity
ESPCI PSL, Paris, France · LAMSADE, Université Paris Dauphine - PSL, Paris, France
ASAP: Attention Sink Anchored Pruning
May 21, 2026Jaehyuk Lee, Hanyoung Kim, Yanggee Kim +1Self-Attention Efficient ViTs
Department of Mathematics Korea University Seoul, Republic of Korea · Korea University Seoul, Republic of Korea
Tensor Cache: Eviction-conditioned Associative Memory for Transformers
May 21, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +2Memory-Augmented Neural Networks Self-Attention
Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA
Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
May 21, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
Manifold-Guided Attention Steering
May 20, 2026Ian Li, Kapilesh Guruprasad, Raunak Sengupta +3Language Model Steering Self-Attention
University of California, San Diego
EntmaxKV: Support-Aware Decoding for Entmax Attention
May 20, 2026Gonçalo Duarte, Miguel Couceiro, Marcos V. TrevisoSelf-Attention KV Caching
Instituto Superior Técnico, Universidade de Lisboa. · ELLIS Unit Lisbon. · INESC-ID. +1
Towards Understanding Self-Pretraining for Sequence Classification
May 20, 2026Omar Coser, Loredana Zollo, Paolo Soda +1Self-Attention Masked Language Modeling
PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Models
May 20, 2026Yanyi Lyu, Letian Chen, Futing Sun +3Self-Attention Diffusion Language Model Inference
Harbin Institute of Technology (Shenzhen)
Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention
May 19, 2026Wenhu Zhang, Yiming Wu, Huanyu Wang +6Self-Attention Long-Context Language Modeling
The Hong Kong University of Science and Technology · The University of Hong Kong · Zhejiang University +1

May 31, 2026, cs.CVJ/K move · Enter open · S save

Interpolation between Convolution and Attention via K-Nearest Neighbors

Mingi Kang

Bowdoin College

Self-Attention Convolutional Neural Networks k-Nearest Neighbors

Open paper AskPDF arXiv

The shift from Convolutional Neural Networks to Transformers has reshaped computer vision, yet these two architectural families are typically viewed as fundamentally distinct. Convolutional Neural Networks are defined by spatially local convolution operations, while Transformers rely on global self-attention. We argue that convolution and self-attention, despite their apparent differences, can be unified within a single k-nearest neighbor aggregation framework. The critical insight is that both operations are special cases of neighbor selection and weighted aggregation. Convolution selects neighbors by spatial proximity while self-attention selects by feature similarity, revealing that they lie on a continuous spectrum rather than representing categorically different computations. We introduce Convolutional Nearest Neighbors (ConvNN), a unified framework that formalizes this connection. ConvNN exactly recovers standard and depthwise convolution by restricting neighbor selection to normalized spatial coordinates, and exactly recovers self-attention and its sparse variants, including KVT-attention, by replacing spatial proximity with scaled dot-product similarity. Beyond these special cases, ConvNN serves as a drop-in replacement for both convolution and attention layers, enabling systematic exploration of the intermediate spectrum between local and global aggregation through configurable similarity functions, neighbor selection strategies, positional encodings, and aggregation kernels.

Momentum

Related topics

Latest papers 195