Hybrid Attention

Ask about this topic

Momentum

12 papers in the last four weeks, up 71% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 46

All topics
CardsList

  1. Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position

    Oct 7, 2026Xiaoran Liu, Ziwei He, Xipeng QiuLong-Context Language ModelingHybrid Attention

    Fudan University · Shanghai Innovation Institute · OpenMOSS Team

  2. PHBA: Prefix-State Hybrid Block Attention

    Oct 6, 2026Ruijie Li, Jiaxi Hu, Shiyu Wang +1Long-Context ModelingLinear Attention

    The Hong Kong University of Science and Technology (Guangzhou) · Independent researcher

  3. Hybrid Latent Attention for Looped Language Models

    Oct 6, 2026Yuhan Chen, Siyuan Zhang, Nan Wang +2Long-Context Language Model InferenceLLM Inference Acceleration

    Virginia Tech · Independent Researcher

  4. Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs

    Oct 5, 2026Hyunji Lee, Joykirat Singh, Zaid Khan +5Long-Context Language ModelingRecurrent Transformers

    UNC Chapel Hill · The University of Texas at Austin · Mila +1

  5. HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models

    Oct 5, 2026Zhuokun Chen, Feng Chen, Xi Lin +4World Model LearningLong-Horizon Video Generation

    Monash University · The University of Adelaide · Zhejiang University

  6. Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contexts

    Sep 30, 2026Dong Gyun Kang, Megha Thukral, Kwangsoo KimAttention Head AnalysisOOD Generalization

    College of Computing, Georgia Institute of Technology · Department of Transdisciplinary Medicine, Seoul National University Hospital

  7. Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval

    Sep 29, 2026Eric Enouen, Sainyam GalhotraIn-Context RetrievalSemantic Search

    Cornell University

  8. Cropland PAtteRNS: Parallel Dimensional Attention Networks and Attention to Dataset Disparity for Crop Segmentation in Satellite Imagery Time Series Data

    Sep 29, 2026Joseph Metcalfe, Sara Sharifzadeh, Fabio CaraffiniRemote Sensing Image SegmentationAgricultural Remote Sensing

    Department of Computer Science, Swansea University, Swansea, SA1 8EN, United Kingdom

  9. How Local Mixing Encodes Relative Position in Global NoPE Attention

    Sep 29, 2026Cutter Dawes, Nick Alonso, Tom Figliolia +1Gated AttentionPositional Encoding

    Zyphra Research

  10. Multilinguality in Hybrid Attention LLMs

    Sep 28, 2026Lucas Bandarkar, Junlin Hu, Chenyuan Yang +2Multilingual Language ModelsCross-Lingual Representation Alignment

    University of California, Los Angeles · Fudan University

  11. WorldAttention: An Efficient Attention Architecture for Interactive Video World Models

    Sep 28, 2026Zeyu Zhang, Jinyuan Mao, Dakai An +6Real-Time Interactive Video GenerationLong-Horizon Video Generation

    DAMO Academy, Alibaba Group · Zhejiang University · Hong Kong University of Science and Technology +2

  12. PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention

    Sep 27, 2026Kunming Shao, Jierun Chen, Yanli Wang +4Self-AttentionLLM Inference Acceleration

    The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · Sun Yat-sen University +1

  13. Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMs

    Sep 27, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +6Long-Context Language Model InferenceLLM Inference Acceleration

    Institute of Artificial Intelligence, China Telecom (TeleAI) China · Shanghai Jiao Tong University · Individual Researcher +1

  14. SketchSSM: Write to the Full State, Read from a Compact Sketch

    Sep 27, 2026Omin Kwon, JoongWon Shin, Minseo Kim +3LLM Inference AccelerationLinear Attention

    Seoul National University · UC Berkeley · KAIST

  15. HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing

    Sep 22, 2026Jianyu Wei, Yizhao Gao, Qihao Zhang +12Long-Context RetrievalKV-Cache Management

    LLM-Core Xiaomi

  16. A Hybrid Attention Model Learning Unified Time-aware Patch Representation for Irregular Multivariate Time Series Forecasting

    Sep 19, 2026Li Lin, Zhihao Lin, Qi Zhang +3Irregular Time-Series ModelingMultivariate Time Series Forecasting

    Southeast University, China · Nanyang Technological University, Singapore · Timecho Ltd., China

  17. Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation

    Sep 17, 2026Haocheng Xi, Yiming Xie, Hexu Zhao +8Video Diffusion ModelsStreaming Video Generation

    University of California, Berkeley · Impossible, Inc. · University of Texas at Austin

  18. Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design

    Sep 2, 2026Runlin Shi, Bojian Yin, Guoqi LiRotary Positional EmbeddingsLinear Attention

    Institute of Automation, Chinese Academy of Sciences, Beijing, China

  19. DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving

    Aug 31, 2026Yanqi Yu, Pingwei Sun, Jianchao Tan +4KV CachingMemory-Efficient Inference

    East China Normal University · Meituan · South China University of Technology

  20. Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs

    Aug 31, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +2LLM Inference AccelerationPrefix Caching

    Institute of Artificial Intelligence, China Telecom (TeleAI) · Shanghai Jiao Tong University

  21. Ask Self, Ask Others: Relation Is All You Need

    Aug 20, 2026Yuting Ge, Pengju Yang, Mingkai NieSelf-AttentionDecoder-Only Language Models

    Department of Electrical Engineering City University of Hong Kong · Jilin University · National University of Singapore

  22. Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus

    Aug 12, 2026Zunhai Su, Bohan Sun, Xialie Zhuang +8Linear AttentionHybrid Attention

    Startlux · Tsinghua University · University of Chinese Academy of Sciences +3

  23. Hybrid Gated Attention

    Aug 12, 2026Zekun Zhou, Ruobing Xie, Lanrui Wang +1Low-Rank Matrix DecompositionGated Attention

    Tencent Hunyuan · Peking University

  24. Linearized 2-Simplicial Attention

    Aug 10, 2026Aritra Das, Dhruman Gupta, Debayan GuptaLinear AttentionAttention Mechanisms

    1Truth Audit Labs

  25. Retrofitting Linear Attention into Diffusion Language Models

    Aug 6, 2026Jinha Kim, Younghun Roh, Jaeyeon KimBlockwise Diffusion Language ModelsDiffusion Language Model Inference

    Apple · *Work done at MIT · Google DeepMind +1

  26. Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

    Aug 3, 2026Li Wang, Yi Su, Xiabao Wu +9Speculative DecodingLLM Inference Acceleration

    State Key Laboratory for Novel Software Technology, Nanjing University · Ant Group · Shanghai Jiao Tong University +1

  27. Hybrid Attention Estimation Pipeline for Adaptive HRI Using an Expressive Robotic Head

    Jul 31, 2026Pablo Moraes, Monica Rodriguez, Christopher Peters +4Human-Robot InteractionHybrid Attention

    Technological University of Uruguay, Uruguay · Ostfalia University of Applied Sciences, Germany

  28. LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs

    Jul 31, 2026Yirui Liu, Ruoling Qi, Longwen Wang +5KV CachingLLM Inference Acceleration

    Institute of Artificial Intelligence, China Telecom (TeleAI) · Shanghai Jiao Tong University · Xi’an Jiaotong University +1

  29. Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

    Jul 30, 2026Chongjian Ge, Hanwen Jiang, Tianyu Wang +9Video Diffusion ModelsDiffusion Transformer

    Adobe Research

  30. A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

    Jul 2, 2026Wanyun CuiLong-Context RetrievalMemory-Augmented Language Models

    Shanghai University of Finance and Economics