Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

Compressed Sensing for Capability Localization in Large Language Models
Feb 11, 2026Anna Bair, Yixuan Even Xu, Mingjie Sun +1Attention Head Analysis LLM Interpretability
Carnegie Mellon University
Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
Feb 11, 2026Keston Aquino-MichaelsTransformer Self-Attention
No Way Labs
FGAA-FPN: Foreground-Guided Angle-Aware Feature Pyramid Network for Oriented Object Detection
Feb 11, 2026Jialin MaMulti-Scale Feature Fusion Object Detection
International school of BUPT, Beijing University Of Posts And Telecommunications, Beijing,China.
Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference
Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3Self-Attention KV Caching
University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong +1
ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
Feb 8, 2026Jitai Hao, Qiang Huang, Yaowei Wang +2Self-Attention KV Caching
Harbin Institute of Technology (Shenzhen), China
Quantum Attention by Overlap Interference: Predicting Classical and Many-Body Quantum Sequences
Feb 6, 2026Alessio Pecilli, Matteo RosatiSelf-Attention Transformer Attention
Dipartimento di Ingegneria Civile, Informatica e delle Tecnologie Aeronautiche, Universit`a degli Studi Roma Tre, Via della Vasca Navale 79, 00146 Rome, Italy
FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion
Feb 5, 2026Zhuokun Chen, Jianfei Cai, Bohan ZhuangVideo Diffusion Models Self-Attention
1Monash University · 2Zhejiang University
Poly-attention: a general scheme for higher-order self-attention
Feb 2, 2026Sayak Chakrabarti, Toniann Pitassi, Josh AlmanSelf-Attention Compositional Reasoning
Computer Science, Columbia University, New York, NY 10027.
STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
Feb 2, 2026Weikang Meng, Liangyu Huo, Yadan Luo +4LLM Inference Acceleration Linear Attention
SMULL Group, Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · UQMM Lab, University of Queensland +1
Efficient Neural Controlled Differential Equations via Attentive Kernel Smoothing
Feb 2, 2026Egor Serov, Ilya Kuleshov, Alexey ZaytsevEfficient Neural Network Inference Neural Controlled Differential Equations
Applied AI Institute · Risk AI Research Lab
Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching
Jan 28, 2026Fengrui Zuo, Zhiwei Ke, Yiming Liu +3Diffusion Language Model Inference Token Pruning
School of Computer Science and Technology, University of Science and Technology of China · Suzhou Institute of Advanced Research, University of Science and Technology of China
RayRoPE: Projective Ray Positional Encoding for Multi-view Attention
Jan 21, 2026Yu Wu, Minsik Jeon, Jen-Hao Rick Chang +2Multi-View 3D Reconstruction Novel View Synthesis
Carnegie Mellon University · Apple
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
Jan 17, 2026Xingyue Huang, Xueying Ding, Mingxuan Ju +3Self-Attention Long-Context Language Modeling
University of Oxford · Carnegie Mellon University · Snap Inc.
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
Jan 11, 2026Jaewon Sok, Jewon Yeom, Seonghyeon Park +2LLM Pruning Attention Head Analysis
Department of Rural Systems Engineering, Seoul National University · Graduate School of Data Science, Seoul National University · Department of Aerospace Engineering, Seoul National University
Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
Dec 18, 2025Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra +1Self-Attention Long-Context Language Model Inference
Microsoft Research India.
Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics
Dec 14, 2025Jingdi Lei, Di Zhang, Soujanya PoriaDynamical Systems Linear Attention
Nanyang Technological University · Fudan University
Block Sparse Flash Attention
Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU Acceleration Self-Attention
Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel
Data-Free Pruning of Self-Attention Layers in LLMs
Dec 3, 2025Dhananjay Saikumar, Blesson VargheseLLM Pruning LLM Compression
School of Computer Science University of St Andrews United Kingdom
Selective Rotary Position Embedding
Nov 21, 2025Sajad Movahedi, Timur Carstensen, Arshia Afzal +3Softmax Attention Rotary Positional Embeddings
ELLIS Institute Tübingen1, LIONS, EPFL2, University of Freiburg3, Max-Planck-Institute for Intelligent Systems4, Prior Labs5
Predicting the Emergence of Induction Heads in Language Model Pretraining
Nov 21, 2025Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan SchneiderLanguage Model Pretraining In-Context Learning
Department of Linguistics, Georgetown University.
Unifying Convolution and Attention via Convolutional Nearest Neighbors
Nov 18, 2025Mingi Kang, Jeová Farias Sales Rocha NetoHybrid CNN-Transformer Architectures Self-Attention
Bowdoin College
Controllably Efficient Language Models
Nov 7, 2025Jatin Prakash, Aahlad Puli, Rajesh RanganathEfficient Transformer Inference Long-Context Language Modeling
New York University
Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers
Oct 28, 2025Rabin AdhikariAttention Head Analysis Transformer
Saarland University 66123 Saarbrücken
NOSA: Native and Offloadable Sparse Attention
Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache Offloading GPU Acceleration
NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. · OpenBMB, China. · BUPT, Beijing, China. +1
Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
Oct 4, 2025Ruoxi Yu, Haotian Jiang, Jingpu Cheng +3Softmax Attention Attention Head Analysis
Center for Data Science, Peking University · Institute for Functional Intelligent Materials, National University of Singapore · Department of Mathematics, National University of Singapore +1
Accelerating Attention with Basis Decomposition
Oct 2, 2025Jialin ZhaoSelf-Attention Transformer Attention
Department of Computer Science Tsinghua University
Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractors
Oct 1, 2025Yen-Shan Chen, Sian-Yao Huang, Cheng-Lin Yang +1Language Model Steering RAG Poisoning Attacks
CyCraft AI Lab, Taiwan · National Taiwan University
RefAM: Attention Magnets for Zero-Shot Referral Segmentation
Sep 26, 2025Anna Kukleva, Enis Simsar, Alessio Tonioni +4Zero-Shot Learning Image Segmentation
Max Planck Institute for Informatics, SIC · ETH Zürich · Google +1
Rotary Position Encodings for Graphs
Sep 26, 2025Isaac Reid, Arijit Sehanobish, Cederik Höfs +7Rotary Positional Embeddings Graph Representation Learning
University of Cambridge · Google DeepMind · Independent Researcher +3
Cross-Attention is Half Explanation in Speech-to-Text Models
Sep 22, 2025Sara Papi, Dennis Fucci, Marco Gaido +2Transformer Interpretability Cross-Attention
Fondazione Bruno Kessler, Italy

Feb 11, 2026, cs.CLJ/K move · Enter open · S save

Compressed Sensing for Capability Localization in Large Language Models

Anna Bair, Yixuan Even Xu, Mingjie Sun, J. Zico Kolter

Carnegie Mellon University

Attention Head Analysis LLM Interpretability Language Modeling Compressed Sensing Attention Mechanisms

Open paper AskPDF arXiv

Large language models (LLMs) exhibit a wide range of capabilities, including mathematical reasoning, code generation, and linguistic behaviors. We show that Transformer architectures contain small subsets of attention heads that are necessary for certain capabilities. Zeroing out as few as five task-specific heads can degrade performance by up to

60\%

on standard benchmarks measuring the capability of interest, while largely preserving performance on unrelated tasks. We introduce a compressed sensing-based method that exploits the sparsity of these heads to identify them via strategic knockouts and a small number of model evaluations. We validate these findings across Llama and Qwen models ranging from 1B to 14B parameters and a diverse set of capabilities including mathematical abilities and code generation, revealing a modular organization in which specialized capabilities are dependent on sparse, functionally distinct components. Overall, our results suggest that capability localization is a general organizational principle of Transformer language models, with implications for interpretability, model editing, and AI safety. Code is released at https://github.com/locuslab/llm-components.

Momentum

Related topics

Latest papers 378