Challenging Quadratic Attention–A Holistic View on the Rise of Alternative Language Model Architectures
Organizations: Social Computing Group Technical University of Munich Boltzmannstraße 3, 85748, Garching, Germany
Abstract
Transformers have dominated sequence processing tasks for the past seven years–most notably language modeling. However, the inherent quadratic complexity of their attention mechanism remains a significant bottleneck as context length increases. We review and distill the recent efforts to overcome this bottleneck, including advances in (sub-quadratic) attention variants, recurrent neural networks, state space models, and hybrid architectures. We critically analyze approaches regarding compute and memory complexity, benchmark results, and fundamental limitations to assess whether the dominance of pure-attention transformers may soon be challenged, which we consider possible, particularly in domain-specific and edge-device applications.
Figures & tables
| Method | Training | Inference | |||
| Time | Space | Parallel | Time | Space | |
| FFT-Convolution | Yes | ||||
| RNN | No | 2 2 2 Assuming the sequence has been processed already, only necessary once | |||
| Vanilla Transformer | Yes | ||||
| LSH (Reformer) | Yes | ||||
| FAVOR+ (Performer) | Yes | ||||
| Model | Benchmark Selection | |||||||
|---|---|---|---|---|---|---|---|---|
| 0.7-1.5B | Size | MMLU | LMB | ARC-E | ARC-C | Wino. | Hella. | PIQA |
| Titans-MAG | 760M | - | 41.0 | 68.2 | 36.2 | 52.9 | 48.9 | 70.3 |
| Griffin | 1B | 29.5 | - | 67.0 | 36.9 | 65.2 | 67.2 | 77.4 |
| Llama3.2* | 1B | 32.1 | 63.0 | - | - | 60.7 | 63.7 | - |
| GLA | 1.3B | - | 46.9 | 57.2 | 26.6 | 53.9 | 49.8 | 71.8 |
| HGRN2 | 1.3B | - | 49.4 | 58.1 | 28.1 | 52.3 | 51.8 | 71.4 |
Efficient Attention
Efficient attention reduces the compute, memory traffic, and cache costs of Transformer attention while preserving useful long-range interactions.
See all 150 papers