The Kernel Reality Check: Benchmarking and Distilling Efficient Attention at Scale
Organizations: Concordia University · Mila – Quebec AI Institute · Laval University · University of Tübingen
Abstract
Kernelized attention methods, which aim to replace the complexity of softmax attention with alternatives, have been the focus of intense research in the last five years. Despite growing interest, systematic evaluations and fair comparisons across approaches and scales are still missing. We analyze kernelized attention in two halves and highlight prominent limitations on each. On efficiency, FlashAttention achieves lower memory consumption and faster wall-clock time than SOTA kernelized methods during training and in vision transformers across most sequence lengths (with inference-time memory in causal language models the only clear exception), contradicting the core assumption for pure kernelized attention used as a drop-in replacement for softmax. We also discover that reported efficiency gains often fail to reproduce against modern baselines. On quality, current kernelized methods leave gaps of 2-10 points relative to softmax under a unified distillation protocol, and model scala alone does not close these gaps. We show this gap is not intrinsic to kernelization: LARA++, a principled extension of LARA that approximates softmax via importance sampling with adaptive proposal distributions, recovers up to 99% of softmax accuracy. We establish that the gap is closable; however, the computational overhead remains substantial, leaving the efficiency wall intact. Code is available at https://github.com/firatoncel/KernelRealityCheckNeurIPS2026
Figures & tables
| DeiT-S/224 | DeiT-B/224 | DeiT-B/384 | ||||
|---|---|---|---|---|---|---|
| Acc | Acc | Acc | ||||
| Softmax | 79.8 | — | 81.8 | — | 82.9 | — |
| Linear attention | ||||||
| Hedgehog | 77.7 | 79.7 | 80.4 | |||
| DiJiang | 71.9 | 74.5 | 72.8 | |||
| LARA | 77.4 | 80.0 | 81.0 | |||
| DeiT-S/224 | DeiT-B/224 | DeiT-B/384 | train ms/step | |
|---|---|---|---|---|
| Softmax | 79.8 / 1.00 | 81.8 / 1.00 | 82.9 / 1.00 | — |
| Hedgehog | 77.7 / 0.41 | 79.7 / 0.71 | 80.4 / 0.52 | 294.6 |
| DiJiang | 71.9 / 0.31 | 74.5 / 0.48 | 72.8 / 0.48 | 274.8 |
| LARA | 77.4 / 0.35 | 80.0 / 0.49 | 81.0 / 0.58 | 289.6 |
| LARA++ | 78.5 / 0.35 | 80.8 / 0.49 | 82.1 / 0.58 | 352.1 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Attention Type | # Landmarks | Acc. (%) |
|---|---|---|
| Softmax | ||
| LARA (chunking) | ||
| LARA (chunking) | ||
| LARA (random) | ||
| LARA++ (sampled) | ||
| LARA++ |
| Method | Seed 0 | Seed 42 | Seed 23 | Mean Std |
|---|---|---|---|---|
| Hedgehog | 77.7 | 77.6 | 77.6 | |
| DiJiang | 71.9 | 71.8 | 71.8 | |
| LARA | 77.4 | 77.5 | 77.2 | |
| LARA++ (sampled) | 78.2 | 77.7 | 77.7 | |
| LARA++ (deterministic) | 78.5 | 78.3 | 78.2 |
| Softmax | Flash | Hedgehog | DiJiang | Hedgehog-FLA | DiJiang-FLA | |
|---|---|---|---|---|---|---|
| ViT-S inf | 269.7 / 4294 | 69.0 / 302 | 82.5 / 347 | 82.0 / 327 | 82.2 / 421 | 82.4 / 342 |
| ViT-B inf | 499.5 / 8740 | 115.2 / 754 | 128.7 / 841 | 131.7 / 816 | 131.3 / 983 | 133.2 / 846 |
| ViT-L inf | 1351.5 / 12398 | 363.7 / 1748 | 390.0 / 1826 | 402.9 / 1887 | 396.5 / 2015 | 407.0 / 1926 |
| ViT-S train | 1205.7 / 33341 | 144.9 / 2830 | 156.6 / 4935 | 157.0 / 4085 | 158.5 / 4960 | 158.4 / 4085 |
| ViT-B train | 2445.3 / 66843 | 357.4 / 5844 | 366.4 / 9996 | 381.0 / 8323 | 369.7 / 9972 | 382.6 / 8323 |
| ViT-L train | OOM | 1134.0 / 15261 | 1143.7 / 26162 | 1194.6 / 21727 | 1150.4 / 26162 | 1199.0 / 21727 |
| Softmax | Flash | Hedgehog | DiJiang | Hedgehog-FLA | DiJiang-FLA | |
|---|---|---|---|---|---|---|
| GPT2-S train @4K | 315.7 / 15051 | 111.6 / 3264 | 633.7 / 24682 | 392.6 / 13867 | 130.6 / 4930 | 131.0 / 4201 |
| GPT2-M train @4K | 801.7 / 37067 | 312.1 / 8733 | 1152.8 / 65681 | 820.7 / 37033 | 373.2 / 13175 | 383.6 / 11257 |
| GPT2-L train @4K | 1643.3 / 67973 | 716.6 / 16850 | OOM | 1635.6 / 69961 | 808.9 / 25179 | 838.7 / 21630 |
| GPT2-S prefill @16K | 2961.9 / 28680 | 215.2 / 2886 | 804.2 / 1325 | 549.6 / 1106 | 304.0 / 1583 | 304.8 / 1256 |
| GPT2-M prefill @16K | 5645.2 / 41239 | 621.0 / 7014 | 1704.0 / 2534 | 1373.2 / 2291 | 820.6 / 2866 | 839.8 / 2491 |
| GPT2-L prefill @16K | 10872.8 / 55828 | 1413.9 / 13180 | 3321.7 / 4569 | 2691.6 / 4372 | 1719.8 / 4984 | 1761.0 / 4621 |
| Model | Softmax | LARA (32) | LARA++ (16) | LARA++ (32) |
|---|---|---|---|---|
| DINOv2-S | 81.4 / 13.6 | 77.1 / 5.5 | 77.4 / 4.9 | 79.1 / 5.5 |
| DINOv2-B | 84.5 / 18.6 | 81.8 / 9.4 | 82.0 / 8.9 | 83.1 / 9.4 |
| DINOv2-L | 86.4 / 22.1 | 84.8 / 12.2 | 84.7 / 11.0 | 85.5 / 12.2 |
| Method | ms/step | train MB | forward() ms | forward() MB |
|---|---|---|---|---|
| Hedgehog-FLA | 294.6 | 19315 | 85.23 | 3331 |
| DiJiang-FLA | 274.8 | 16390 | 68.87 | 2958 |
| LARA | 289.6 | 15961 | 75.15 | 3313 |
| LARA++ | 352.1 | 16106 | 147.81 | 3330 |
| Model | LARA fwd (no clustering) | LARA++ fwd | Clustering (isolated) | Clustering share |
|---|---|---|---|---|
| DINOv2-S | 3.6ms | 8.9ms | 5.4ms | 60% |
| DINOv2-B | 9.8ms | 17.2ms | 8.4ms | 49% |
| DINOv2-L | 14.1ms | 24.5ms | 10.6ms | 43% |
| DINOv2-g | 9.9ms | 24.5ms | 14.5ms | 59% |
| Method | Forward FLOPs (rel. softmax) | Kernel launches | DRAM traffic proxy (MB) |
|---|---|---|---|
| Softmax | 77.6G ( ) | 3 | 168 |
| LARA (32) | 71.3G ( ) | 62 | 676 |
| LARA++ (32) | 72.9G ( ) | 344 | 2109 |
| Task | DINOv2-S | DINOv2-B | DINOv2-L |
|---|---|---|---|
| IN1K top-1 | 81.4 / 79.1 / 77.4 | 84.5 / 83.1 / 82.0 | 86.4 / 85.5 / 84.7 |
| VOC2012 mIoU | 83.4 / 80.1 / 77.4 | 85.2 / 83.5 / 81.9 | 85.1 / 85.2 / 83.0 |
| ADE20K mIoU | 44.9 / 38.2 / 34.4 | 47.7 / 42.5 / 39.1 | 48.3 / 48.4 / 41.7 |
| Model | Dataset | BS | Dist. LR | # Dist. Ep. | FT LR | # FT Ep. |
|---|---|---|---|---|---|---|
| DeiT-small | ImageNet-1K | 256 | 3e-5 | 1 12 | 2e-6 | 5 |
| DeiT-base | ImageNet-1K | 256 | 3e-5 | 1 12 | 2e-6 | 5 |
| DeiT-base-384 | ImageNet-1K | 64 | 2e-5 | 0.25 12 | 2e-6 | 3 |
| Model | Dataset | BS | Dist. LR | # Dist. Steps | FT LR | # FT Steps |
|---|---|---|---|---|---|---|
| Small | YFCC15M | 32 | 2e-5 | 2500 12 | 3e-6 | 2500 8 |
| Base | YFCC15M | 32 | 2e-5 | 2500 12 | 3e-6 | 2500 8 |
| Large | YFCC15M | 32 | 2e-5 | 2500 12 | 3e-6 | 2500 8 |
| Model | Dataset | BS | FT LR | # Epochs |
|---|---|---|---|---|
| Small/Base/Large | ADE20k | 16 | 3e-4 | 2 |
| Small/Base/Large | VOC2012 | 16 | 5e-4 | 10 |
| Model | Dataset | BS | Dist. LR | # Dist. Ep. | FT LR | # FT Ep. |
|---|---|---|---|---|---|---|
| DeiT-small | ImageNet-1K | 32 | 0.02 | 2 | 2e-6 | 8 |
| DeiT-base | ImageNet-1K | 32 | 0.02 | 2 | 2e-6 | 8 |
| DeiT-base-384 | ImageNet-1K | 16 | 0.02 | 0.25 2 | 2e-6 | 0.25 8 |
| Model | Dataset | BS | Dist. LR | # Dist. Ep. | FT LR | # FT Ep. |
|---|---|---|---|---|---|---|
| DeiT-small | ImageNet-1K | 256 | 0.001 | 2 | 5e-5 | 8 |
| DeiT-base | ImageNet-1K | 256 | 0.001 | 2 | 5e-5 | 8 |
| DeiT-base-384 | ImageNet-1K | 64 | 0.001 | 0.25 2 | 5e-5 | 0.25 8 |
| Softmax | LARA++ (32) | ||
|---|---|---|---|
| IN1K top-1 | 86.5 | 86.0 | |
| VOC2012 mIoU | 86.70 | 85.99 |
Efficient Attention
Efficient attention reduces the compute, memory traffic, and cache costs of Transformer attention while preserving useful long-range interactions.
See all 149 papers