Overview

Hybrid attention studies how to combine full softmax attention, which supports precise access to context, with more efficient mechanisms such as gated linear attention, recurrent memory, sliding windows, and sparse retrieval. Models allocate these mechanisms across layers, heads, or tokens, using approaches such as architecture search, distillation of pretrained Transformers, and learned or training-free routing. Evaluations commonly test language modeling, reasoning, and long-context retrieval on benchmarks such as RULER, NIAH, and LongBench, alongside inference speed and memory use. The central challenge is preserving reliable long-range recall while reducing computation and cache costs; the best allocation and layer ordering can depend on the task and hardware.

Latest papers

CardsList

85 papers8 papers in the last four weeks, down 71% on the four weeks before. 0.1% of all new papers.

  1. Beyond K/V Caches: Cross-Model State Handoff in Hybrid Linear-Attention LLMs

    Nov 5, 2026Jiacheng Cai, Tao Zhang, Siqi Liang +2KV-Cache TransferLLM Inference Acceleration

  2. Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining

    Nov 4, 2026João Eduardo Batista, Emil Vatai, Mohamed WahibLLM Inference AccelerationLong-Context Language Model Inference

  3. Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models

    Oct 29, 2026Dmytro Babarytskyi, Oleksandr BabarytskyiRecurrent Language ModelsHybrid Attention

  4. MixPT: Point Transformer with Interleaved Window and Global Attention

    Oct 22, 2026Minkyun Seo, Hyungtae Lim, Jaesik ParkPoint TransformerPoint Cloud Understanding

  5. Competing Learning in Hybrid Architectures

    Oct 19, 2026Xingyu Shen, Yingfa Chen, Ziqing Qiao +3Hybrid Language ModelsHybrid Attention

  6. Improving the Expressive Power of Hybrid Sequence Models via Multi-Hop Retrieval

    Oct 18, 2026Yam Eitan, Gilad Yehudai, Haggai MaronMulti-Hop RetrievalHybrid Attention

  7. Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers

    Oct 16, 2026Bingxin Xing, Yu Zhang, Jiatong Ding +2Diffusion Model Inference AccelerationSparse Attention

  8. Predicting Live Birth After Fresh Embryo Transfer in IVF/ICSI: Differential ICM/Trophectoderm Morphology Encoding and a Hybrid Attention--Logistic Regression Ensemble

    Oct 15, 2026Abdul Raheem Shahzad, Jinxin Chen, Ming Zhang +1Clinical Outcome PredictionTabular Prediction

  9. Kwai Summary Attention V2 Technical Report

    Oct 13, 2026OneRec Team, Chenglong Chu, Guohong Mu +30KV-Cache CompressionHybrid Attention

  10. The Effect of Layer Ordering in Hybrid Architectures

    Oct 13, 2026John Cooper, Mingchen Ma, Jiaqi Mo +4Hybrid Transformer ArchitecturesHybrid Attention

  11. PANDA: Prior-guided Attentional Dual-path Architecture

    Oct 9, 2026Rishabh Jain, Federico Belotti, Stefano Coniglio +2Hypergraph Neural NetworksHigher-Order GNNs

  12. HART: Hybrid Attention-RWKV Time series Foundation Model

    Oct 9, 2026Qingwen Lin, Boyan Xu, Shifeng Xie +3Time Series ForecastingTime-Series Foundation Models

  13. Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position

    Oct 7, 2026Xiaoran Liu, Ziwei He, Xipeng QiuLong-Context Language ModelingHybrid Attention

  14. PHBA: Prefix-State Hybrid Block Attention

    Oct 6, 2026Ruijie Li, Jiaxi Hu, Shiyu Wang +1Long-Context ModelingLinear Attention

  15. Hybrid Latent Attention for Looped Language Models

    Oct 6, 2026Yuhan Chen, Siyuan Zhang, Nan Wang +2Long-Context Language Model InferenceLLM Inference Acceleration

  16. Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs

    Oct 5, 2026Hyunji Lee, Joykirat Singh, Zaid Khan +5Long-Context Language ModelingRecurrent Transformers

  17. HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models

    Oct 5, 2026Zhuokun Chen, Feng Chen, Xi Lin +4World Model LearningLong-Horizon Video Generation

  18. Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contexts

    Sep 30, 2026Dong Gyun Kang, Megha Thukral, Kwangsoo KimAttention Head AnalysisOOD Generalization

  19. Cropland PAtteRNS: Parallel Dimensional Attention Networks and Attention to Dataset Disparity for Crop Segmentation in Satellite Imagery Time Series Data

    Sep 29, 2026Joseph Metcalfe, Sara Sharifzadeh, Fabio CaraffiniRemote Sensing Image SegmentationAgricultural Remote Sensing

  20. How Local Mixing Encodes Relative Position in Global NoPE Attention

    Sep 29, 2026Cutter Dawes, Nick Alonso, Tom Figliolia +1Gated AttentionPositional Encoding

  21. Multilinguality in Hybrid Attention LLMs

    Sep 28, 2026Lucas Bandarkar, Junlin Hu, Chenyuan Yang +2Multilingual Language ModelsCross-Lingual Representation Alignment

  22. WorldAttention: An Efficient Attention Architecture for Interactive Video World Models

    Sep 28, 2026Zeyu Zhang, Jinyuan Mao, Dakai An +6Real-Time Interactive Video GenerationLong-Horizon Video Generation

  23. PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention

    Sep 27, 2026Kunming Shao, Jierun Chen, Yanli Wang +4Self-AttentionLLM Inference Acceleration

  24. Just Let Linear States Forget the Distant Past: Prefix Caching via Suffix Replay for Hybrid LLMs

    Sep 27, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +6Long-Context Language Model InferenceLLM Inference Acceleration