Hybrid Attention

Ask about this topic

Overview

Hybrid attention studies how to combine full softmax attention, which supports precise access to context, with more efficient mechanisms such as gated linear attention, recurrent memory, sliding windows, and sparse retrieval. Models allocate these mechanisms across layers, heads, or tokens, using approaches such as architecture search, distillation of pretrained Transformers, and learned or training-free routing. Evaluations commonly test language modeling, reasoning, and long-context retrieval on benchmarks such as RULER, NIAH, and LongBench, alongside inference speed and memory use. The central challenge is preserving reliable long-range recall while reducing computation and cache costs; the best allocation and layer ordering can depend on the task and hardware.

Latest papers

CardsList

79 papers22 papers in the last four weeks, up 47% on the four weeks before. 0.1% of all new papers.

  1. Layerwise Repacking: Variable-Length Hybrid-Attention Training without an Extra Redistribution Pass

    Oct 26, 2026Jiahao Zhang, Junwei Feng, Dongliang Huang +3Communication-Efficient Distributed TrainingHybrid Attention

    KlingAI Research Beijing, China

  2. MixPT: Point Transformer with Interleaved Window and Global Attention

    Oct 22, 2026Minkyun Seo, Hyungtae Lim, Jaesik ParkPoint TransformerPoint Cloud Understanding

    Seoul National University · Massachusetts Institute of Technology

  3. Competing Learning in Hybrid Architectures

    Oct 19, 2026Xingyu Shen, Yingfa Chen, Ziqing Qiao +3Hybrid Language ModelsHybrid Attention

    Department of Science and Technology, Tsinghua University, Beijing, China

  4. Improving the Expressive Power of Hybrid Sequence Models via Multi-Hop Retrieval

    Oct 18, 2026Yam Eitan, Gilad Yehudai, Haggai MaronMulti-Hop RetrievalHybrid Attention

    Technion – Israel Institute of Technology · NVIDIA

  5. Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers

    Oct 16, 2026Bingxin Xing, Yu Zhang, Jiatong Ding +2Diffusion Model Inference AccelerationSparse Attention

    Xi’an Jiaotong University · Xiamen University · Shanghai Jiao Tong University

  6. Predicting Live Birth After Fresh Embryo Transfer in IVF/ICSI: Differential ICM/Trophectoderm Morphology Encoding and a Hybrid Attention--Logistic Regression Ensemble

    Oct 15, 2026Abdul Raheem Shahzad, Jinxin Chen, Ming Zhang +1Clinical Outcome PredictionTabular Prediction

    School of Computer Science, Wuhan University, Wuhan, China · Department of Obstetrics and Gynecology, Zhongnan Hospital of Wuhan University, Wuhan, China · Reproductive Medical Center, Zhongnan Hospital of Wuhan University, Wuhan, Hubei, P.R. China, and also with the Clinical Medicine Research Center of Prenatal Diagnosis and Birth Health in Hubei Province, Wuhan, Hubei, P.R. China

  7. Kwai Summary Attention V2 Technical Report

    Oct 13, 2026OneRec Team, Chenglong Chu, Guohong Mu +30KV-Cache CompressionHybrid Attention

  8. The Effect of Layer Ordering in Hybrid Architectures

    Oct 13, 2026John Cooper, Mingchen Ma, Jiaqi Mo +4Hybrid Transformer ArchitecturesHybrid Attention

    Department of Computer Sciences, University of Wisconsin-Madison · Department of Computer Sciences, John Hopkins University

  9. GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs

    Oct 10, 2026Junjie Li, Jiong Lou, Jie LiKV-Cache EvictionKV-Cache Compression

    Shanghai Jiao Tong University

  10. PANDA: Prior-guided Attentional Dual-path Architecture

    Oct 9, 2026Rishabh Jain, Federico Belotti, Stefano Coniglio +2Hypergraph Neural NetworksHigher-Order GNNs

    University of Cambridge, UK · University of Bergamo, Italy · Independent Researcher

  11. HART: Hybrid Attention-RWKV Time series Foundation Model

    Oct 9, 2026Qingwen Lin, Boyan Xu, Shifeng Xie +3Time Series ForecastingTime-Series Foundation Models

    School of Computer Science, Guangdong University of Technology · Huawei Noah’s Ark Lab · LIPADE, Université Paris Cité, Paris, France +2

  12. Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position

    Oct 7, 2026Xiaoran Liu, Ziwei He, Xipeng QiuLong-Context Language ModelingHybrid Attention

    Fudan University · Shanghai Innovation Institute · OpenMOSS Team

  13. PHBA: Prefix-State Hybrid Block Attention

    Oct 6, 2026Ruijie Li, Jiaxi Hu, Shiyu Wang +1Long-Context ModelingLinear Attention

    The Hong Kong University of Science and Technology (Guangzhou) · Independent researcher

  14. Hybrid Latent Attention for Looped Language Models

    Oct 6, 2026Yuhan Chen, Siyuan Zhang, Nan Wang +2Long-Context Language Model InferenceLLM Inference Acceleration

    Virginia Tech · Independent Researcher

  15. Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs

    Oct 5, 2026Hyunji Lee, Joykirat Singh, Zaid Khan +5Long-Context Language ModelingRecurrent Transformers

    UNC Chapel Hill · The University of Texas at Austin · Mila +1

  16. HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models

    Oct 5, 2026Zhuokun Chen, Feng Chen, Xi Lin +4World Model LearningLong-Horizon Video Generation

    Monash University · The University of Adelaide · Zhejiang University

  17. The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends

    Sep 30, 2026Zhentao Tan, Jingyi Shen, Yanbo Li +3Attention MechanismsPersistent Memory for Language Models

    Alibaba Token Hub, Alibaba Group

  18. Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contexts

    Sep 30, 2026Dong Gyun Kang, Megha Thukral, Kwangsoo KimAttention Head AnalysisOOD Generalization

    College of Computing, Georgia Institute of Technology · Department of Transdisciplinary Medicine, Seoul National University Hospital

  19. Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval

    Sep 29, 2026Eric Enouen, Sainyam GalhotraIn-Context RetrievalSemantic Search

    Cornell University

  20. Cropland PAtteRNS: Parallel Dimensional Attention Networks and Attention to Dataset Disparity for Crop Segmentation in Satellite Imagery Time Series Data

    Sep 29, 2026Joseph Metcalfe, Sara Sharifzadeh, Fabio CaraffiniRemote Sensing Image SegmentationAgricultural Remote Sensing

    Department of Computer Science, Swansea University, Swansea, SA1 8EN, United Kingdom

  21. How Local Mixing Encodes Relative Position in Global NoPE Attention

    Sep 29, 2026Cutter Dawes, Nick Alonso, Tom Figliolia +1Gated AttentionPositional Encoding

    Zyphra Research

  22. Multilinguality in Hybrid Attention LLMs

    Sep 28, 2026Lucas Bandarkar, Junlin Hu, Chenyuan Yang +2Multilingual Language ModelsCross-Lingual Representation Alignment

    University of California, Los Angeles · Fudan University

  23. WorldAttention: An Efficient Attention Architecture for Interactive Video World Models

    Sep 28, 2026Zeyu Zhang, Jinyuan Mao, Dakai An +6Real-Time Interactive Video GenerationLong-Horizon Video Generation

    DAMO Academy, Alibaba Group · Zhejiang University · Hong Kong University of Science and Technology +2

  24. PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention

    Sep 27, 2026Kunming Shao, Jierun Chen, Yanli Wang +4Self-AttentionLLM Inference Acceleration

    The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · Sun Yat-sen University +1

  25. Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMs

    Sep 27, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +6Long-Context Language Model InferenceLLM Inference Acceleration

    Institute of Artificial Intelligence, China Telecom (TeleAI) China · Shanghai Jiao Tong University · Individual Researcher +1

  26. SketchSSM: Write to the Full State, Read from a Compact Sketch

    Sep 27, 2026Omin Kwon, JoongWon Shin, Minseo Kim +3LLM Inference AccelerationLinear Attention

    Seoul National University · UC Berkeley · KAIST

  27. The Key Handoff: Retrieval in Hybrid Language Models

    Sep 26, 2026Kaan Kale, Oguzhan Baser, Sriram VishwanathHybrid Language ModelsHybrid Transformer-State Space Model Architectures

    Georgia Institute of Technology, USA · University of Texas at Austin, USA

  28. Empowering Hybrid Attention Models on NPUs

    Sep 26, 2026Yinyuan Zhang, Daliang Xu, Xiaolong Huang +4LLM Inference AccelerationOn-Device Inference

    Peking University · Beijing University of Posts and Telecommunications

  29. On-Policy Attention Linearization

    Sep 25, 2026Arian Raje, Anupam Nayak, Anthony Fei +3Linear AttentionHybrid Transformer Architectures

    Carnegie Mellon University · Cornell University

  30. Where Compute Matters: Heterogeneous Attention for Efficient Video Diffusion

    Sep 25, 2026Olga Zatsarynna, Denis Korzhenkov, Juergen Gall +2Video Diffusion ModelsDiffusion Model Inference Acceleration

    Qualcomm AI Research · University of Bonn