Local Attention

Follow this topic Ask about this topic

Local attention limits or favors interactions within neighborhoods, using methods such as sliding or dilated windows, blockwise connections, and learned locality biases. It is studied in language and vision transformers, spiking networks, video generation, and neural operators, with evaluations ranging from language-model perplexity to image segmentation and scientific-domain generalization. Local patterns can reduce attention and cache costs while improving spatial representations, and hybrid designs combine local layers with global connections or memory to retain long-range information. A central tension is choosing locality that saves computation without losing useful global context; strict locality may also help compositional generalization, but its benefits depend on the task and architecture.

Latest papers

11 papers0 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining
Aug 12, 2026Qiuwu Chen, Zimo Liu, Yuchen Li +8Language Model Pretraining Memory-Augmented Language Models
AIGCode · South China University of Technology · Pazhou Laboratory
Rethinking Attention Locality in Spiking Transformers
Aug 9, 2026Zeqi Zheng, Zizheng Zhu, Yuping Yan +3Spiking Transformers Spiking Neural Networks
1Zhejiang University · 2Westlake University · 3Peking University
TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention
Aug 3, 2026Avni Mittal, Avinash Anand, Ashutosh Kumar +7Neural Cellular Automata Autoregressive Language Modeling
Independent Researcher · Singapore Institute of Technology, Singapore · IIIT Delhi, India +2
On Locality and Length Generalization in Visual Reasoning
Jul 10, 2026Pulkit Madan, Sanjay Haresh, Reza Ebrahimi +3Visual Reasoning Length Generalization
Qualcomm AI Research*
Towards Memory-Efficient Autoregressive Video Generation via Instance-Specific Parametric Absorption
Jul 1, 2026Xiaomeng Fu, Jia Li, Yiming Hu +5Long-Horizon Video Generation Streaming Video Generation
Institute of Information Engineering, Chinese Academy of Sciences · AMAP, Alibaba Group · The University of Hong Kong
Zero-shot generalization of transformer neural operators to larger domains
Jun 12, 2026Armand de Villeroché, Sibo Cheng, Vincent Le Guen +5Transformer Neural Operator Local Attention
CEREA, ENPC, EDF R&D, Institut Polytechnique de Paris, Île-de-France, France · SINCLAIR AI Laboratory, Saclay, Île-de-France, France · EDF R&D, Île-de-France, France +1
Post-Boundary Bridge: Must Local Attention Go Global Between Global Layers?
Jun 1, 2026Zhibo YangEfficient Attention Local Attention
Ocean University of China
Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention
May 12, 2026Lingdong Li, Hangming Zhang, Qiang YuSpiking Transformers Efficient ViTs
School of Future Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300354, China · School of Artificial Intelligence, Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin 300354, China
Characterizing the Expressivity of Local Attention in Transformers
May 1, 2026Jiaoda Li, Ryan CotterellTransformer Expressivity Transformer
Neural Dynamics Self-Attention for Spiking Transformers
Mar 9, 2026Dehao Zhang, Fukai Guo, Shuai Wang +6Spiking Transformers Spiking Neural Networks
University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · The Chinese University of Hong Kong (Shenzhen)
Locality-Attending Vision Transformer
Mar 5, 2026Sina Hajimiri, Farzad Beizaee, Fereshteh Shakeri +3Vision Transformer Semantic Segmentation
ÉTS Montreal, LIVIA, ILLS

Aug 12, 2026J/K move · Enter open · S save

LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining

Qiuwu Chen, Zimo Liu, Yuchen Li, Ying Sun+7

AIGCode · South China University of Technology · Pazhou Laboratory

Language Model Pretraining Memory-Augmented Language Models Knowledge Augmentation for Language Models Decoder-Only Language Models Local Attention Attention Mechanisms

Open paper AskPDF arXiv

Large language models (LLMs) have achieved remarkable breakthroughs across various applications. However, their architectures remain inefficient in pretraining due to two main limitations: (i) self-attention lacks an explicit inductive bias for locality, leading to redundant modeling of sequence-internal local information; (ii) mixture-of-experts (MoE) implicitly couples knowledge storage with computational pathways, hindering flexible access to sequence-external global knowledge. To overcome these limitations, we propose LoKiFormer, a novel LLM architecture that augments the standard decoder with two dedicated modules: 1) Local Fusion Attention (LFA), which incorporates a convolutional fusion to attention, explicitly capturing local patterns and allowing the attention to operate on more informative representations; 2) Knowledge Memory Module (KMM), which introduces a parametric key-value memory that explicitly stores global knowledge in addressable slots, decoupling storage from computation and enabling direct knowledge retrieval. Together, these modules enable LoKiFormer to achieve more efficient and effective integration of information at both levels. Experimental results show that LoKiFormer converges 1.33x faster in pre-training than baseline models, underscoring its superiority over existing LLM architectures.