Sliding-Window Attention

Ask about this topic

Overview

Sliding-window attention studies how to process long sequences by letting each token attend mainly to a fixed, nearby window, which can make computation and KV-cache use scale more gently with sequence length. Approaches include causal windows with attention sinks, shifted windows for cross-window communication, and hybrid models that reserve some layers or heads for global attention; distillation, reinforcement learning, and recurrent memory are also used to retain information beyond the window. The method is applied in language modeling, reasoning, speech generation, document processing, and vision, with evaluations spanning long-context retrieval, reasoning benchmarks, and generation quality. Its central tension is that smaller windows improve efficiency but can weaken long-range recall, so research asks which information or global pathways are needed to preserve quality and support length extrapolation.

Latest papers

CardsList

24 papers3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

  1. Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech

    Oct 14, 2026Jeseung Lim, Jiseong Woo, Heeseung KimLong-Form TTS SynthesisTTS Synthesis

    Department of Physics, University of Seoul, Seoul, Republic of Korea · Department of AI, University of Seoul, Seoul, Republic of Korea

  2. FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation

    Oct 12, 2026Zekun Shi, Alastair James Arthur PriceLong-Context Language ModelingLong-Context Modeling

    National University of Singapore · Sea AI Lab, Singapore · University of Toronto

  3. Backend-Agnostic Sparse Attention for Fast High-Resolution Visual Generation

    Oct 6, 2026Liao Ma, Jiayi Song, Yunfeng Wu +2Video Diffusion ModelsDiffusion Transformer

    School of Artificial Intelligence, Shanghai Jiao Tong University · School of Data Science, Fudan University · School of Computing and Data Science, The University of Hong Kong

  4. How Local Mixing Encodes Relative Position in Global NoPE Attention

    Sep 29, 2026Cutter Dawes, Nick Alonso, Tom Figliolia +1Gated AttentionPositional Encoding

    Zyphra Research

  5. Thinking Outside the Box: Retention and Transmission of Information in Sliding-Window KV Inference

    Sep 28, 2026Timothy DeLise, Seth CromelinLong-Context Language Model InferenceKV-Cache Reuse

  6. Sliding-window beats linear attention

    Aug 28, 2026Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1Linear AttentionKV Caching

    Microsoft Applied Sciences Group (ASG) · Independent

  7. Prefix Sliding for efficient test-time scaling

    Aug 26, 2026Niklas Muennighoff, Zhengyang Wang, Zeyi Chen +15LLM Inference EfficiencyTest-Time Scaling

    Stanford University · University of California at Santa Barbara · University of Washington +1

  8. Understanding the Energy Scaling of Large Language Model Inference Across Context Lengths and Attention Architectures

    Aug 25, 2026Molka Chkir, Syed Muhammad Danish, Jos Höll +1Energy-Aware LLM InferenceLLM Inference Efficiency

    Algoma University, Brampton, Canada · School of Informatics, Reutlingen University, Reutlingen

  9. Maglev: Sliding Recurrent Memory

    Aug 3, 2026Bo Liu, Qiang LiuMemory-Augmented Language ModelsNext-Token Prediction

    The University of Texas at Austin

  10. Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers

    Jul 26, 2026Dongseong Hwang, Prasanth Yadla, Kaan Elgin +8TTS SynthesisStreaming TTS Synthesis

    Apple

  11. Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context

    Jul 23, 2026Alagappan ValliappanMulti-Token PredictionLong-Context Language Model Inference

    NVIDIA

  12. NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation

    Jun 26, 2026Qiong Tang, Xiangkun Hu, Xiangyang Liu +2Long-Context Language Model InferenceSliding-Window Attention

    FARS

  13. Unlimited OCR Works

    Jun 22, 2026Youyang Yin, Huanhuan Liu, YY +14Efficient Transformer InferenceLong-Context Language Model Inference

    Baidu Inc.

  14. ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation

    Jun 16, 2026Yao Chen, Yinqi Yang, Junyuan Shang +6LLM Inference AccelerationSparse Attention

    Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Baidu Inc.

  15. Rethinking the Role of Efficient Attention in Hybrid Architectures

    Jun 13, 2026Ziqing Qiao, Yinuo Xu, Chaojun Xiao +6Attention Head AnalysisLong-Context Modeling

    1Tsinghua University · 2OpenBMB

  16. Architecture-Aware Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoning

    Jun 10, 2026Kai Liu, Peijie Dong, Xinchen Xie +5RL for Language Model ReasoningLLM Inference Acceleration

    Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Shanghai AI Laboratory · Hong Kong University of Science and Technology (Guangzhou)

  17. Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete

    Jun 1, 2026Qian Li, Xinyu Mao, Shang-Hua TengTransformer ExpressivityTransformer

    Shenzhen Research Institute of Big Data · University of Southern California

  18. Periodic RoPE for Infinite Context LLMs

    May 27, 2026Simin HuoRotary Positional EmbeddingsLong-Context Language Modeling

    Shanghai Jiao Tong University

  19. Gated-SwinRMT: Unifying Swin Windowed Attention with Retentive Manhattan Decay via Input-Dependent Gating

    Apr 7, 2026Dipan Maity, Suman Mondal, Arindam RoyVision TransformerGated Attention

    Student Kolkata, West Bengal, India · Department of Computer Science Yogoda Satsanga Palpara Mahavidyalaya West Bengal, India · Department of Computer Science & Application Prabhat Kumar College Contai West Bengal, India

  20. BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs

    Apr 7, 2026Abbas Ghaddar, Ivan Kobyzev, Boxing Chen +1Hybrid AttentionLLM Inference Acceleration

    Huawei Noah’s Ark Lab, Montreal Research Center, Canada

  21. Stochastic Attention: Connectome-Inspired Randomized Routing for Expressive Linear-Time Attention

    Apr 1, 2026Zehao Jin, Yanan SuiEfficient AttentionSliding-Window Attention

    Tsinghua University

  22. Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers

    Mar 27, 2026Yusheng Zhao, Hourun Li, Bohan Wu +5Hybrid Transformer ArchitecturesHybrid Attention

    State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University, Beijing, China · Contribution during internship at Huawei Technologies Co., Ltd. · Huawei Technologies Co., Ltd.

  23. Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation

    Mar 24, 2026ByeongCheol Lee, Hyun Seok Seong, Sangeek Hyun +3Open-Vocabulary Semantic SegmentationSliding-Window Attention

    Sungkyunkwan University

  24. WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models

    Mar 17, 2026Hanna Lee, Tan Dat Nguyen, Jaehoon Kang +1TTS SynthesisSelf-Attention

    Korea Advanced Institute of Science and Technology, Republic of Korea · Sungkyunkwan University, Republic of Korea