Hybrid Attention

Hybrid attention studies how to combine full softmax attention, which supports precise access to context, with more efficient mechanisms such as gated linear attention, recurrent memory, sliding windows, and sparse retrieval. Models allocate these mechanisms across layers, heads, or tokens, using approaches such as architecture search, distillation of pretrained Transformers, and learned or training-free routing. Evaluations commonly test language modeling, reasoning, and long-context retrieval on benchmarks such as RULER, NIAH, and LongBench, alongside inference speed and memory use. The central challenge is preserving reliable long-range recall while reducing computation and cache costs; the best allocation and layer ordering can depend on the task and hardware.