Bilateral Log-Linear Attention
Organizations: Oak Ridge National Laboratory
Abstract
Sequence models represent past context at different resolutions, using token-level keys and values, fixed-size recurrent states, or hierarchies of compressed summaries. Full attention supports strong associative recall but requires quadratic work during training and prefill and a linearly growing key-value cache during inference. Linear attention and state-space models reduce these costs through fixed-size recurrent memory, but this compression can limit recall. Log-linear attention offers a middle ground by maintaining logarithmically many state matrices. However, its hierarchy favors finer resolution near the current query while compressing earlier context into coarser summaries. To complement this allocation of resolution, we introduce bilateral log-linear attention, which pairs the existing hierarchy with a second hierarchy anchored at the sequence origin. By combining their reads, the two branches provide finer access to both early and recent context while remaining causal. We derive recurrent and chunkwise parallel formulations for the added branch, preserving log-linear training and prefill work, logarithmic decoding work per token, and logarithmic persistent state storage at fixed feature dimensions and chunk size. In multi-query associative recall, this design raises mean overall test accuracy from 76.37% to 96.12% over forward log-linear attention across multiple training seeds. For pretrained language models under approximate parameter matching, it raises the mean score on six selected RULER needle-in-a-haystack tasks from 26.97% to 50.03% at a context length of 6144 tokens. In the same setting, bilateral attention also outperforms forward log-linear attention on seven of eight downstream retrieval tasks, including question answering and information extraction.
Figures & tables
| Mechanism | Training time | Decode time per token | Decode memory |
|---|---|---|---|
| Softmax attention | |||
| Linear attention w/wo. scalar decay | |||
| Log-linear attention w/wo. scalar decay |
| Forward branch | Reverse branch | Bilateral | |
|---|---|---|---|
| Training/prefill work | |||
| Decode work per token | |||
| State matrices per head | |||
| Matrix-state elements per head |
| Perplexity | LM-Eval | RULER-NIAH | Retrieval | |||||
| Model (layers) | Parameters (M) | WikiText-2 | LAMBADA | avg. | 2048 | 4096 | 6144 | avg. |
| Layer-matched (12 layers) | ||||||||
| Mamba-2 (12) | 338.1 | 31.37 | 39.66 | 40.92 | 31.37 | 27.23 | 12.79 | 18.76 |
| Log-linear (12) | 350.5 | 30.23 | 33.78 | 42.23 | 43.65 | 40.52 | 27.04 | 21.46 |
| Bilateral (12) | 365.6 | 30.21 | 33.20 | 41.51 | 63.18 | 59.14 | 50.03 | 25.16 |
| Transformer (12) | 275.7 | 32.53 | 40.75 | 40.81 | 67.42 | 61.28 | 59.77 | 29.76 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Operation | Arithmetic work |
|---|---|
| Apply source gates | |
| Form chunk summaries | |
| Scan summaries and obtain bucket totals | |
| Local causal query–key/value products | |
| Read incoming chunk states | |
| Read completed states at later queries |
| Model | Trainable parameters | Selected peak learning rate |
|---|---|---|
| Mamba-2 | 580,656 | 0.00316227766 |
| Log-linear | 591,216 | 0.01 |
| Bilateral | 604,896 | 0.00316227766 |
| Transformer | 655,680 | 0.01 |
| Model | Overall | ||||
|---|---|---|---|---|---|
| Mamba-2 | |||||
| Log-linear | |||||
| Bilateral | |||||
| Transformer |
| Training seed | Mamba-2 | Log-linear | Bilateral | Transformer |
|---|---|---|---|---|
| 10 | 14.47 | 88.71 | 99.01 | 99.49 |
| 11 | 78.13 | 81.78 | 96.86 | 99.17 |
| 12 | 83.17 | 78.14 | 96.77 | 99.53 |
| 13 | 69.52 | 45.54 | 87.38 | 99.59 |
| 14 | 80.69 | 80.85 | 97.01 | 99.67 |
| 15 | 83.68 | 80.84 | 98.99 | 99.25 |
| Model | Layers | Parameters | Configuration note |
|---|---|---|---|
| Layer-matched (12 layers) | |||
| Mamba-2 | 12 | 338,147,520 | One state matrix per head |
| Log-linear | 12 | 350,549,952 | Forward hierarchy |
| Bilateral | 12 | 365,608,320 | Shared, normalized, gated |
| Transformer | 12 | 275,682,816 | MLP width 2048 |
| Approximately parameter-matched | |||
| Model | Wiki. PPL | Lamb. PPL | Lamb. | PIQA | Hella. | Wino. | ARC-E | ARC-C | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Layer-matched (12 layers) | |||||||||
| Mamba-2 | 31.37 | 39.66 | 27.93 | 63.71 | 33.83 | 50.20 | 45.71 | 24.15 | 40.92 |
| Log-linear | 30.23 | 33.78 | 30.53 | 64.20 | 34.13 | 51.38 | 46.97 | 26.19 | 42.23 |
| Bilateral | 30.21 | 33.20 | 31.13 | 62.79 | 34.15 | 51.46 | 45.92 | 23.63 | 41.51 |
| Transformer | 32.53 | 40.75 | 30.56 | 61.70 | 31.64 | 52.01 | 44.57 | 24.40 | 40.81 |
| Approximately parameter-matched | |||||||||
| Model | S-NIAH-1 | S-NIAH-2 | S-NIAH-3 | MK-NIAH-1 | MQ-NIAH | MV-NIAH | Avg. |
|---|---|---|---|---|---|---|---|
| Layer-matched (12 layers) | |||||||
| Mamba-2 | 77.80 | 13.40 | 33.60 | 26.60 | 19.15 | 17.65 | 31.37 |
| Log-linear | 99.80 | 68.20 | 11.20 | 34.40 | 24.05 | 24.25 | 43.65 |
| Bilateral | 100.00 | 100.00 | 70.60 | 49.60 | 28.80 | 30.10 | 63.18 |
| Transformer | 99.80 | 100.00 | 65.60 | 59.20 | 39.45 | 40.45 | 67.42 |
| Approximately parameter-matched | |||||||
| Model | S-NIAH-1 | S-NIAH-2 | S-NIAH-3 | MK-NIAH-1 | MQ-NIAH | MV-NIAH | Avg. |
|---|---|---|---|---|---|---|---|
| Layer-matched (12 layers) | |||||||
| Mamba-2 | 58.40 | 41.80 | 1.80 | 22.20 | 20.20 | 19.00 | 27.23 |
| Log-linear | 99.40 | 62.40 | 8.00 | 30.20 | 21.20 | 21.95 | 40.52 |
| Bilateral | 99.80 | 98.60 | 55.40 | 49.60 | 23.65 | 27.80 | 59.14 |
| Transformer | 98.40 | 100.00 | 47.80 | 64.80 | 29.80 | 26.90 | 61.28 |
| Approximately parameter-matched | |||||||
| Model | S-NIAH-1 | S-NIAH-2 | S-NIAH-3 | MK-NIAH-1 | MQ-NIAH | MV-NIAH | Avg. |
|---|---|---|---|---|---|---|---|
| Layer-matched (12 layers) | |||||||
| Mamba-2 | 38.60 | 6.60 | 6.20 | 12.00 | 5.15 | 8.20 | 12.79 |
| Log-linear | 94.80 | 15.00 | 5.60 | 17.40 | 15.30 | 14.15 | 27.04 |
| Bilateral | 99.80 | 89.80 | 23.60 | 45.80 | 20.90 | 20.25 | 50.03 |
| Transformer | 100.00 | 100.00 | 42.80 | 60.40 | 27.80 | 27.65 | 59.77 |
| Approximately parameter-matched | |||||||
| Model | SWDE | SQuAD | FDA | TriviaQA | DROP | NQ-512 | NQ-1024 | NQ-2048 | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Layer-matched (12 layers) | |||||||||
| Mamba-2 | 20.61 | 28.18 | 10.62 | 38.86 | 16.05 | 13.78 | 11.53 | 10.44 | 18.76 |
| Log-linear | 24.93 | 31.20 | 15.43 | 39.75 | 17.87 | 16.63 | 13.62 | 12.28 | 21.46 |
| Bilateral | 29.79 | 33.51 | 32.76 | 39.57 | 17.44 | 17.83 | 15.93 | 14.47 | 25.16 |
| Transformer | 37.44 | 33.71 | 62.07 | 39.10 | 17.68 | 18.06 | 16.06 | 13.93 | 29.76 |
| Approximately parameter-matched | |||||||||