Sliding-Window Attention

Follow this topic Ask about this topic

Latest papers 23

3 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

All topics

FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation
Oct 12, 2026Zekun Shi, Alastair James Arthur PriceLong-Context Language Modeling Long-Context Modeling
National University of Singapore · Sea AI Lab, Singapore · University of Toronto
Backend-Agnostic Sparse Attention for Fast High-Resolution Visual Generation
Oct 6, 2026Liao Ma, Jiayi Song, Yunfeng Wu +2Video Diffusion Models Diffusion Transformer
School of Artificial Intelligence, Shanghai Jiao Tong University · School of Data Science, Fudan University · School of Computing and Data Science, The University of Hong Kong
How Local Mixing Encodes Relative Position in Global NoPE Attention
Sep 29, 2026Cutter Dawes, Nick Alonso, Tom Figliolia +1Gated Attention Positional Encoding
Zyphra Research
Thinking Outside the Box: Retention and Transmission of Information in Sliding-Window KV Inference
Sep 28, 2026Timothy DeLise, Seth CromelinLong-Context Language Model Inference KV-Cache Reuse
Sliding-window beats linear attention
Aug 28, 2026Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1KV Caching LLM Inference Acceleration
Microsoft Applied Sciences Group (ASG) · Independent
Prefix Sliding for efficient test-time scaling
Aug 26, 2026Niklas Muennighoff, Zhengyang Wang, Zeyi Chen +15LLM Inference Efficiency Test-Time Scaling
Stanford University · University of California at Santa Barbara · University of Washington +1
Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures
Aug 25, 2026Molka Chkir, Syed Muhammad Danish, Jos Höll +1LLM Inference Efficiency Long-Context Language Model Inference
Algoma University, Brampton, Canada · School of Informatics, Reutlingen University, Reutlingen
Maglev: Sliding Recurrent Memory
Aug 3, 2026Bo Liu, Qiang LiuMemory-Augmented Language Models Next-Token Prediction
The University of Texas at Austin
Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers
Jul 26, 2026Dongseong Hwang, Prasanth Yadla, Kaan Elgin +8TTS Synthesis Streaming TTS Synthesis
Apple
Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context
Jul 23, 2026Alagappan ValliappanMulti-Token Prediction Long-Context Language Model Inference
NVIDIA
NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation
Jun 26, 2026Qiong Tang, Xiangkun Hu, Xiangyang Liu +2Long-Context Language Model Inference Sliding-Window Attention
FARS
Unlimited OCR Works
Jun 22, 2026Youyang Yin, Huanhuan Liu, YY +14Efficient Transformer Inference Long-Context Language Model Inference
Baidu Inc.
ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation
Jun 16, 2026Yao Chen, Yinqi Yang, Junyuan Shang +6LLM Inference Acceleration Sparse Attention
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Baidu Inc.
Rethinking the Role of Efficient Attention in Hybrid Architectures
Jun 13, 2026Ziqing Qiao, Yinuo Xu, Chaojun Xiao +6Attention Head Analysis Long-Context Modeling
1Tsinghua University · 2OpenBMB
Architecture-Aware Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoning
Jun 10, 2026Kai Liu, Peijie Dong, Xinchen Xie +5RL for Language Model Reasoning LLM Inference Acceleration
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Hong Kong University of Science and Technology (Guangzhou) · Shanghai AI Laboratory
Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete
Jun 1, 2026Qian Li, Xinyu Mao, Shang-Hua TengTransformer Expressivity Transformer
Shenzhen Research Institute of Big Data · University of Southern California
Periodic RoPE for Infinite Context LLMs
May 27, 2026Simin HuoRotary Positional Embeddings Long-Context Language Modeling
Shanghai Jiao Tong University
Gated-SwinRMT: Unifying Swin Windowed Attention with Retentive Manhattan Decay via Input-Dependent Gating
Apr 7, 2026Dipan Maity, Suman Mondal, Arindam RoyVision Transformer Gated Attention
Student Kolkata, West Bengal, India · Department of Computer Science Yogoda Satsanga Palpara Mahavidyalaya West Bengal, India · Department of Computer Science & Application Prabhat Kumar College Contai West Bengal, India
BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs
Apr 7, 2026Abbas Ghaddar, Ivan Kobyzev, Boxing Chen +1Hybrid Attention LLM Inference Acceleration
Huawei Noah’s Ark Lab, Montreal Research Center, Canada
Stochastic Attention: Connectome-Inspired Randomized Routing for Expressive Linear-Time Attention
Apr 1, 2026Zehao Jin, Yanan SuiEfficient Attention Sliding-Window Attention
Tsinghua University
Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers
Mar 27, 2026Yusheng Zhao, Hourun Li, Bohan Wu +5Hybrid Attention Long-Context Language Modeling
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University, Beijing, China · Contribution during internship at Huawei Technologies Co., Ltd. · Huawei Technologies Co., Ltd.
Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation
Mar 24, 2026ByeongCheol Lee, Hyun Seok Seong, Sangeek Hyun +3Open-Vocabulary Semantic Segmentation Sliding-Window Attention
Sungkyunkwan University
WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models
Mar 17, 2026Hanna Lee, Tan Dat Nguyen, Jaehoon Kang +1TTS Synthesis Self-Attention
Korea Advanced Institute of Science and Technology, Republic of Korea · Sungkyunkwan University, Republic of Korea

Oct 12, 2026J/K move · Enter open · S save

FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation

Zekun Shi, Alastair James Arthur Price

National University of Singapore · Sea AI Lab, Singapore · University of Toronto

Long-Context Language Modeling Long-Context Modeling Sparse Attention Sliding-Window Attention

Open paper AskPDF arXiv

Linear-time sequence models typically store long-context information in a recurrent state. A fixed-size state limits how much information can be stored, while hybrid architectures that add full attention for better recall incur quadratic sequence cost. Alternatively, sparse attention with continued pretraining (CPT) can extend the effective context length. Generalization beyond the trained length remains challenging, however, and CPT can be costly. We introduce FreeSolo, a native sparse attention mechanism with a short-range sliding window branch and a long-range branch that selects keypoint cells for direct token attention. Its linear-time variant, FS-L, learns to prune the prefix before query-dependent selection through short CPT. FS-B searches balanced trees with

O(N\log^2 N)

algorithmic cost including radix-sort construction. FreeSolo demonstrates better zero-shot length extrapolation than the baselines without long-context fine-tuning.