Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding
Apr 27, 2026Wang Fan, Wei Cao, Xi Zha +5Self-Attention Long-Context Language Model Inference
Fudan University
Kwai Summary Attention Technical Report
Apr 27, 2026Chenglong Chu, Guorui Zhou, Guowang Zhang +35Self-Attention KV Caching
Kuaishou
Nautile-370M: Spectral Memory Meets Attention in a Small Reasoning Model
Apr 27, 2026Maixent ChenebauxSelf-Attention Small Language Models
Trickstr
Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models
Apr 27, 2026Amogh Sheth, Biruk Assefa, Yi Wen Huang +2LLM Pruning Self-Attention
Edison Academy Magnet School · State University of New York College at Plattsburgh · The University of Texas at Austin +2
ELSA: Exact Linear-Scan Attention for Fast and Memory-Light Vision Transformers
Apr 26, 2026Chih-Chung Hsu, Xin-Di Ma, Wo-Ting Liao +1Efficient Transformer Inference Softmax Attention
Advanced Computer Vision Laboratory, National Yang Ming Chiao Tung University
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Mixture-of-Experts Language Models Long-Context Language Model Inference
Transformer as an Euler Discretization of Score-based Variational Flow
Apr 26, 2026Huadong LiaoRepresentation Learning Transformer
HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
Apr 26, 2026Peize He, Yaodi Luo, Xiaoqian Liu +7Audio Token Compression Self-Attention
1EPIC Lab, Shanghai Jiao Tong University · 2DAIL Tech · 3Northeastern University +2
Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers
Apr 26, 2026Giansalvo CirrincioneTransformer Interpretability Representation Collapse
aLaboratoire LTI, Université de Picardie Jules Verne, Chemin du Thil, 80025 Amiens, France
Different Strokes for Different Folks: Writer Identification for Historical Arabic Manuscripts
Apr 24, 2026Hamza A. Abushahla, Ariel Justine N. Panopio, Layth Al-Khairulla +1Document Understanding Historical Document Analysis
Department of Computer Science and Engineering, American University of Sharjah, Sharjah 26666, United Arab Emirates
HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models
Apr 24, 2026Abhinaba BasuTransformer Attention Language Modeling
National Institute of Electronics and Information Technology (NIELIT)
Dissociating Decodability and Causal Use in Bracket-Sequence Transformers
Apr 24, 2026Aryan Sharma, Cutter Dawes, Shivam RavalTransformer Interpretability Self-Attention
1Yale University · 2Independent · 3Harvard University
Shared Lexical Task Representations Explain Behavioral Variability In LLMs
Apr 23, 2026Zhuonan Yang, Jacob Xiaochen Li, Francisco Piedrahita Velez +5Prompt Sensitivity LLM Prompting
Brown University · MIT · Northeastern
Evaluating Post-hoc Explanations of the Transformer-based Genome Language Model DNABERT-2
Apr 23, 2026Isabel Kurth, Paulo Yanez Sarmiento, Bernhard Y. RenardTransformer Interpretability Attention Mechanisms
Hasso Plattner Institute University of Potsdam
The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
Apr 23, 2026Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3Efficient Transformer Inference Self-Attention
Harvard University
Attention-Augmented YOLOv8 with Ghost Convolution for Real-Time Vehicle Detection in Intelligent Transportation Systems
Apr 22, 2026Syed Sajid Ullah, Muhammad Zunair Zamir, Ahsan Ishfaq +1Intelligent Transportation Systems Object Detection
School of Energy and Electrical Engineering, Chang’an University, Xi’an, China · School of Information Engineering, Chang’an University, Xi’an, China
Simplified Sparse Attention via Gist Tokens
Apr 22, 2026Yuzhen Mao, Michael Y. Li, Emily B. FoxSelf-Attention Long-Context Language Model Inference
Stanford University
Cognitive Alignment At No Cost: Inducing Human Attention Biases For Interpretable Vision Transformers
Apr 21, 2026Ethan KnightsTransformer Interpretability Visual Attention
Cambridge, UK
Tracing Relational Knowledge Recall in Large Language Models
Apr 21, 2026Nicholas Popovič, Michael FärberLinear Probing Factual Knowledge in Language Models
TU Dresden & ScaDS.AI Dresden/Leipzig, Germany
Super Apriel: One Checkpoint, Many Speeds
Apr 21, 2026SLAM Labs, :, Oleksiy Ostapenko +13LLM Inference Acceleration Attention Mechanisms
SLAM Labs · ServiceNow Research · University of Toronto +4
The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content
Apr 21, 2026Yuqi Zhang, Di ZhangRetrieval-Augmented Generation Self-Attention
Xi’an Jiaotong-Liverpool University
Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling
Apr 21, 2026Weijie Zhao, Mingquan Liu, Bolun Wang +4Transformer Language Model Scaling Laws
Luxi Tech, Shenzhen, China · The School of Computer Science, China University of Geosciences, 388 Lumo Road, Wuhan, 430074, Hubei, China.
LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit
Apr 21, 2026Manav PandeyLLM Sycophancy LLM Alignment
Georgia Institute of Technology
FG $^2$ -GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control
Apr 21, 2026Pingwei Sun, Yuxuan Hu, Jianchao Tan +6Long-Context Modeling Linear Attention
Meituan · Beijing, China
DeltaSeg: Tiered Attention and Deep Delta Learning for Multi-Class Structural Defect Segmentation
Apr 20, 2026Enrique Hernandez Noguera, Md Meftahul Ferdaus, Elias Ioup +1Attention Mechanisms Semantic Segmentation
University of New Orleans, New Orleans, LA, USA · Center for Geospatial Sciences, Naval Research Laboratory, Hancock County, MS, USA
Sessa: Selective State Space Attention
Apr 20, 2026Liubomyr HorbatkoSelf-Attention Long-Context Language Modeling
Attention-ResUNet for Automated Fetal Head Segmentation
Apr 20, 2026Ammar Bhilwarawala, Mainak BandyopadhyayUltrasound Image Segmentation Medical Image Segmentation
School of Computer Engineering, KIIT Deemed to be University, Bhubaneswar, India
Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
Apr 20, 2026Yujie Chen, Tailai Chen, Yifeng Gao +4Self-Attention Long-Context Language Model Inference
Shanghai Jiao Tong University · University of California, San Diego · Carnegie Mellon University
DSAINet: An Efficient Dual-Scale Attentive Interaction Network for General EEG Decoding
Apr 20, 2026Zhiyuan Ma, Zeyuan Li, Zihao Qiu +6Cross-Subject EEG Decoding EEG Decoding
School of Biomedical Engineering, Tsinghua Laboratory of Brain and Intelligence, and IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China · School of Basic Medical Sciences and Tsinghua Laboratory of Brain and Intelligence, Tsinghua University, Beijing 100084, China · Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing 100084, China +1
HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads
Apr 19, 2026Juyuan Wang, Chenxing Wang, Yuchen Fang +8Attention Head Analysis Self-Attention
Weixin Group, Tencent, China · 2South China University of Technology, Guangzhou, China

Apr 27, 2026, cs.ARJ/K move · Enter open · S save

Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding

Wang Fan, Wei Cao, Xi Zha, Kedi Ma+4

Fudan University

Self-Attention Long-Context Language Model Inference LLM Inference Acceleration Sparse Attention Attention Mechanisms

Open paper AskPDF arXiv

Long contexts improve capabilities of large language models but pose serious hardware challenges: compute and memory footprints grow linearly with sequence length. Particularly, the decoding phase continuously accesses massive KV cache, dramatically increasing bandwidth and computing pressure. Existing accelerators are primarily designed and evaluated for short contexts. They suffer from significant performance degradation when processing long contexts. To bridge this gap, we identify the major bottleneck and present a hardware accelerator for long context attention decoding via hardware-software co-design. On the software side, we propose dual-compression dynamic sparse attention. It combines ultra-low-precision quantization with feature sparsity to minimize prediction overhead. A hardware-friendly approximate Top-K selection further reduces filter complexity from

O(n \log k)

O(n)

. On the hardware side, we deeply optimize compute and memory access to tackle bottlenecks from intricate interplay between sparse attention and long contexts, and establish a performance model to derive the optimal co-design scheme. The resulting hardware adopts a fully pipelined parallel architecture and achieves

O(n)

efficiency even for long sequences. Experiments show that our design delivers

3.82\times

speedup and

74.19\times

energy efficiency over A100. Compared to SOTA accelerators, this is the first ASIC accelerator that efficiently supports long context inference, with at least

3.5\times

higher throughput and

2.08\times

better energy efficiency.

Momentum

Related topics

Latest papers 378