Not All Attention Is Equal: A Quantitative Survey of the EEI Trade-off

Aditya Singh

cs.LGAug 16, 2026

Not All Attention Is Equal: A Quantitative Survey of the EEI Trade-off

Authors: Aditya Singh

Abstract

Attention mechanisms have driven machine learning for a decade, from neural machine translation to language models that do general-purpose reasoning. This survey covers four connected threads: their formulation for sequence-to-sequence tasks, adaptation to computer vision, efficiency innovations that address the quadratic bottleneck, and advances in interpretability. We define three criteria: efficiency, expressiveness, and interpretability, and compare twenty-one methods using an EEI scoring framework. Scores come from a single rater with an assumed +/-1-point perturbation range. A deterministic Monte Carlo analysis with 200,000 samples shows that, under this perturbation model, rank changes of more than one position occur in 67-70% of samples on average. A rank-matched null model reproduces a similar stability profile, so the results support coarse tier-level comparisons rather than fine-grained rankings. The survey traces attention from Bahdanau-Luong alignment through the Transformer and into vision architectures. It reviews fixed and learned sparse attention, linear attention, IO-aware exact algorithms including FlashAttention, and state-space alternatives including Mamba. It also covers induction heads, superposition, and the attention-SSM duality. We further provide a structured narrative review, a benchmark synthesis with cross-study caveats, a five-problem research gap analysis, and a 2015-2026 evolution timeline. We conclude by framing attention research as an expansion of the efficiency-expressiveness-interpretability frontier and identifying future directions including unified efficiency benchmarks, learned routing for hybrid architectures, length generalization, and scalable mechanistic interpretability.

Figures & tables

Panel of Figure 1 — Figure 1: Roadmap of attention evolution. The field progressed from foundational sequence-to-sequence mechanisms through vision adaptations, an efficiency-focused era producing diverse sub-quadratic strategies, and into the current hybrid era where adaptive routing and attention-SSM convergence dominate. Dashed arrows indicate convergent lines of research. Arrows trace thematic lineage rather than strict chronology or direct code inheritance.

	Vision	FlashAttn	Mamba/SSM	Quant. EEI	2025–26
Tay et al. (2022)	×	×	×	×	×
This survey	✓	✓	✓	✓	✓

Table 1: Comparison with prior attention surveys. “EEI” denotes the Efficiency–Expressiveness–Interpretability framework introduced here. Checkmarks indicate substantive coverage: for Vision, a dedicated section comparing ViT, DETR, and/or Swin; for the other columns, dedicated coverage of the listed topic. Tay et al. (2022) discusses vision applications in passing but lacks a dedicated vision-transformer section.

Source	Retrieved	Source-level retained
arXiv	$\approx$ 820	$\approx$ 48
IEEE Xplore	$\approx$ 140	$\approx$ 15
ACL Anthology	$\approx$ 210	$\approx$ 22
ACM Digital Library	$\approx$ 95	$\approx$ 9
Total retrieved	$\approx$ 1,265	—
Unique after dedup.	$\approx$ 1,123	$\approx$ 94

Table 2: Author-maintained literature-search summary by source. Database counts are author-maintained approximate retrieval counts; records may overlap across databases (each row lists the number of records retrieved from that source). After deduplication, approximately 1,123 unique records remained for screening; counts are not independently auditable because the underlying export and deduplication logs were not retained. “Source-level retained” counts refer to approximately 94 papers that survived title/abstract screening; these per-source figures are not additive across rows because the same paper may be retained from multiple sources. The unique count of 94 was reconstructed after deduplication; of these, 63 were included in the final synthesis after full-text review.

Panel of Figure 2 — Figure 2: Literature flow diagram, inspired by the PRISMA 2009 reporting convention but using reconstructed author-maintained counts rather than an independently reproducible systematic-review record. Database search returned approximately 1,265 records; after duplicate removal, approximately 1,123 unique records were screened on title/abstract. Approximately 94 records retained after screening (Table 2 ) proceeded to full-text review, from which 31 were excluded, yielding 63 papers in the final synthesis. The screening log is unavailable, so the diagram documents scope rather than evidence of a fully reproducible systematic review.

Panel of Figure 3 — Figure 3: The EEI (Efficiency–Expressiveness–Interpretability) score map. Each bubble plots a method by Efficiency ( $x$ -axis) and Expressiveness ( $y$ -axis), while bubble area encodes Interpretability. Colours identify the manuscript families: full attention, sparse/routed/window, linear/kernel, IO-aware/KV-efficient, structured sequence, and adaptive. Bubble centers reflect the assigned ordinal scores; small display-only offsets and leader lines separate overlapping methods. A red P marks each non-dominated Efficiency–Expressiveness score pair, $(8,10)$ and $(10,8)$ , and the gray arrow indicates the desired direction for future work. Scores are rubric-derived ordinal judgments, not measured benchmark values.

Method	E	Ex	I	Family
Attention mechanisms
Transformer (Vaswani et al., 2017)	4	10	6	Full Attention
GQA (Ainslie et al., 2023)	7	9	6	Dense (KV-efficient)
MQA (Shazeer, 2019)	7	8	6	Dense (KV-efficient)
Longformer (Beltagy et al., 2020)	8	7	7	Sparse/Window
BigBird (Zaheer et al., 2020)	7	8	7	Sparse/Window

Table 4: EEI scores for representative attention methods. Scores are single-rater estimates with

\pm 1

assumed perturbation range; see Section 2 (Score Consistency and Limitations) for the perturbation sensitivity results. † Method is an arXiv preprint (mid-2026) without independent reproduction; see Section 7.7 and Table 15 . Rows are grouped by architectural layer; the panel is a design-space comparison, so composite-score comparisons are most meaningful within a layer and illustrative across layers.

Method	E	Ex	I	LRA Avg.	Speedup
Transformer	4	10	6	54.39	1.0 $\times$
FlashAttention-2/3	8	10	6	—	$\sim 2.0\times$ *
GQA	7	9	6	—	—
MQA	7	8	6	—	—
PagedAttention	8	10	6	—	—
Ring Attention	7	10	6	—	—

Table 5: EEI scores versus empirical benchmark metrics. LRA Avg. values are from Table 12 where available (“—” indicates no published value; otherwise the value is reported directly in this table); speedup values are from Table 11 where available.

Method	Equal	Retrieval	Deployment
Transformer	69%	66%	26%
FlashAttention-2/3	65%	51%	77%
GQA	78%	80%	82%
MQA	77%	81%	79%
PagedAttention	65%	51%	77%
Ring Attention	74%	57%	82%

Table 6: Monte Carlo score-perturbation sensitivity: fraction of samples in which a method’s average-tie rank changes by more than one position, rounded to whole percentage points (Monte Carlo standard error per cell

\approx 0.1

pp). Null expectation (rank-matched): mean instability under the same perturbation process applied to 100 replicates of score triplets drawn from the distinct values observed on each axis (E: [4, 6, 7, 8, 9, 10]; Ex: [6, 7, 8, 9, 10]; I: [5, 6, 7, 8]), (100 reps, 50000 samples each, seed 43), averaged at each method’s baseline rank. Excess: observed mean minus null expectation; values near zero indicate that rank instability follows from perturbation magnitude and rank censoring rather than the specific score configuration.

Panel of Figure 4 — Figure 4: Three foundational attention variants: additive (Bahdanau), multiplicative (Luong), and scaled dot-product (Vaswani).

Variant	Score Function	Complexity	Parameters	Expressiveness	Hardware
Bahdanau (Additive)	$v^{T}\tanh(Ws+Uh)$	$O(L_{\mathrm{dec}}L_{\mathrm{enc}}d)$	High (W,U,v)	High: learned nonlinear	Low: no single batched $QK^{\top}$
Luong (Multiplicative)	$s^{T}h$	$O(L_{\mathrm{dec}}L_{\mathrm{enc}}d)$	None	Moderate: linear	High: dot product
Vaswani (Scaled Dot-Product)	$QK^{T}/\sqrt{d_{k}}$	$O(L^{2}d)$	None	Moderate + scaling	Very High: tensor cores

Table 7: Comparison of foundational attention variants.

Panel of Figure 5 — Figure 5: Scaled dot-product attention computation flow.

Panel of Figure 6 — Figure 6: Bidirectional (full) vs. causal (masked) attention patterns.

Category	Subcategory	Representative Works	Mechanism	Complexity
Dense	Full Softmax	Vaswani et al., 2017	$QK^{\top}$ pairwise	$O(L^{2}d_{\mathrm{model}})$
	Multi-Head	Vaswani et al., 2017	$h$ projected heads	$O(L^{2}d_{\mathrm{model}})$
	GQA/MQA	Ainslie et al., 2023; Shazeer, 2019	Shared KV heads	$O(L^{2}d_{\mathrm{model}})$ comp.; $O(g_{\mathrm{KV}}Ld_{k})$ cache
Sparse	Fixed Window	Beltagy et al., 2020; Liu et al., 2021	$w$ -token local window	$O(Lwd_{\mathrm{model}})$
	Fixed Strided	Child et al., 2019	Every- $k$ pattern	$O(L\sqrt{L}\,d_{\mathrm{model}})$
	Hybrid Global	Beltagy et al., 2020; Zaheer et al., 2020	Window + random + global	$O(L(w+r_{\mathrm{random}}+n_{\mathrm{global}})d_{\mathrm{model}})$

Table 8: Comprehensive taxonomy of sequence-efficiency methods. The seven top-level families are intentionally heterogeneous: they classify the primary source of efficiency or architectural change rather than enforcing a single ontological level (e.g., IO-aware is an implementation strategy, sparse is an attention pattern, structured sequence is an alternative operator family, and hybrid is an architectural composition). Here

d_{\mathrm{model}}=hd_{k}

g_{\mathrm{KV}}

is the number of KV groups,

m

is the kernel-feature dimension,

d_{v}

is the value dimension,

K

is the learned-group count,

B

is block size,

s

is the selected-block count,

r_{\mathrm{random}}

and

n_{\mathrm{global}}

are the random- and global-token counts, and

n_{A},n_{S},n_{R}

count attention, SSM, and recurrent layers. Constant factors and linear-projection costs are suppressed.

Panel of Figure 7 — Figure 7: Taxonomy of efficient sequence-mixing methods (abridged; the seven families and leaf groupings follow Table 8 ).

Method	Complexity	Quality vs Full Attn	Best Regime	Hardware	Production Ready
Fixed Sparse	$O(L)$ or $O(L\sqrt{L})$	Moderate loss on long-range retrieval	Long docs (4K–32K)	Broad GPU support	High (HF integration)
Learnable Sparse	$O(L^{3/2})$ (k-means routing)	Near-full quality trained; degrades on unseen patterns	Very long (32K+)	Broad GPU support	Low (niche)
Linear Attention	$O(L)$	Approx. error can grow with $L$ ; weak on recall	$>$ 8K tokens	Broad GPU support; no custom kernels	Medium (xFormers)
IO-Aware (FA1/2/3)	$O(L^{2})$ compute, $O(L)$ HBM	Exact (no approximation)	All; sweet spot $<$ 32K	Ampere+ / Hopper+ FP8	Very High (SDPA)
SSMs (Mamba)	$O(L)$ time, $O(1)$ memory	Competitive perplexity; limited direct retrieval evidence	Long gen (32K+)	GPU-specific custom kernel	Medium (Mamba repo)
Hybrid (SSM+attention, e.g., Jamba, Samba)	Quadratic prefill; $O(L)$ decoding (per token)	Variable; can match full attn on mixed workloads	Mixed workloads	Broad GPU support	Medium (Jamba, RWKV eco.)

Table 9: Comparison of efficient attention methods. Complexity classifications follow the method-specific sources cited in Table 8 .

Use Case	Primary Recommendation	Alternative	Key Constraint
RAG / Retrieval-Augmented Gen.	FlashAttention + GQA	Adaptive Sparsity	Application-specific retrieval accuracy threshold
Long-Doc Summarization (8K–32K)	FlashAttention + GQA	Longformer / BigBird	Budget for quality loss
Code Completion (autoregressive)	GQA / MQA + PagedAttention	FlashAttention	KV cache size critical at 128K+
Video Understanding (many tokens)	Swin / Window Attention	Perceiver cross-attn	Token count #1 constraint
On-Device Inference	Mamba / SSM	GQA + INT8 quant	Memory footprint limited
Real-Time Streaming	Mamba-2 (SSD)	RWKV	Constant $O(1)$ state required

Table 10: Practitioner use-case mapping: recommended method families by application profile.

Method	Speedup	vs. Baseline	Condition	Source
FlashAttention	2.0–3.0 $\times$	PyTorch attn	A100-40GB, 1K, FP16 (up to 3 $\times$ at 128–2K)	Dao et al., 2022
FlashAttention-2	$\sim 2.0\times$	FA1	A100-80GB, 8K, FP16	Dao, 2024
FlashAttention-3	$1.5\text{--}2.0\times$	FA2	H100, 8K, FP16 (FP8: $\sim 1.6\times$ )	Shah et al., 2024
Focus †	2.0 $\times$	Full attn. (w/ FA)	H100-80GB	Yao et al., 2026
DashAttention †	up to 3.3 $\times$	FA3	GH200, decoding, up to 93.75% sparsity	Huang et al., 2026
Mamba	up to $3\times$ ; $4$ – $5\times$	conv. SSMs; same-size Transformer	A100, up to 32K	Gu & Dao, 2024

Table 11: Reported wall-clock speedups from the literature. Conditions vary by hardware, sequence length, and precision; see original papers for full details.

Method	LRA Avg.	Val. PPL	Throughput	Source
Full Attn. (Transformer)	54.39	—	1.0 $\times$ (baseline)	LRA benchmark (Tay et al., 2021)
Longformer	53.46	—	—	LRA benchmark (Tay et al., 2021)
BigBird	55.01	—	—	LRA benchmark (Tay et al., 2021)
Sparse Transformer	51.24	—	—	LRA benchmark (Tay et al., 2021)
Reformer	50.67	—	—	LRA benchmark (Tay et al., 2021)
Performer	51.41	—	—	LRA benchmark (Tay et al., 2021)

Table 12: Benchmark performance comparison across representative methods. LRA Avg. values follow the official Long Range Arena protocol (Tay et al., 2021); RWKV’s value is from its own LRA evaluation (Peng et al., 2023a). Val. PPL reports the validation-set language-modeling perplexity (lower is better) as reported in each cited paper; the single reported value is Mamba’s Pile validation perplexity, ‡ ). Throughput reports the speedup claimed in each original paper relative to the baseline and hardware stated there; the bases differ across rows (details in Table 11 ). “—” indicates that no published result was available for that benchmark configuration.

Category	$N$	LRA	Speedup	Median Ex
Full Attention	1	54.39	1.0 $\times$ [—]	10 [—]
IO-Aware (FlashAttn-2/3)	1	—	$\sim 2.0\times$ (FA-2 vs FA-1) [—]	10 [—]
Sparse Pattern	4	52.6 [50.67, 55.01]	—	7 [7–8]
Linear / Kernel	2	51.4 [51.36, 51.41]	—	6 [6–6]
Structured SSM	2	—	3 $\times$ (vs. conv. SSM); 6 $\times$ (vs. FA-2) ∗	7.5 [7–8]
Long Convolution	1	—	5 $\times$ @ 8K; 2 $\times$ vs. FA-2; 100 $\times$ at 64K	7 [—]

Table 13: Quantitative cross-study synthesis by synthesis category. Bracketed values denote the observed within-category range; LRA mean is computed only over methods in the family that report LRA results (Footnote † ). These ranges are descriptive, not inferential: they summarize the reported values for the small number of methods per category (

n=1

–4) and are not estimates of statistical uncertainty in the underlying EEI construct.

Era	Dominant Trend	Representative Methods	Key Shift
2015–2019	Foundation	Bahdanau, Luong, Transformer, BERT	Attention establishes sequence modeling
2019–2020	Sparse/Window	Sparse Transformer, Longformer	First sub-quadratic attention alternatives
2020	Sparse + Linear / Kernel	Reformer, Performer, Linformer, BigBird	Sub-quadratic scaling through sparsity, hashing, and approximation
2021–2023	IO-Aware / KV-Efficient	FlashAttention, GQA, PagedAttention	Hardware-co-design with limited quality trade-offs
2023–2024	State-Space/Recurrent	Mamba, Hyena, RWKV, xLSTM	$O(L)$ or $O(L\log L)$ via recurrence or long convolution
2025–2026	Adaptive + Next-Gen SSMs	Focus, DashAttention, Mamba-3	Learned routing recovers near-full Ex; Mamba-3 improves SSM state tracking

Table 14: Chronological trend analysis of attention research. Each era is defined by the dominant architectural paradigm and key methodological innovations; eras are thematic sub-eras, not chronologically exclusive, and trend labels are thematic groupings rather than literal family strings from Table 8 (e.g., GQA, a Dense KV-efficient method, sits in the IO-Aware era because KV-compression co-evolved with hardware co-design).

Method	Code Available	Checkpoints	Benchmarks Used	Reproduced
Transformer (Vaswani)	PyTorch/TF core	All major hubs	WMT14	Yes, by $>$ 1K papers
FlashAttention-2	PyTorch core (SDPA)	N/A (kernel only)	GPT-3 1.3B/2.7B (2K/8K context)	Yes, multiple impl.
FlashAttention-3	GitHub (Dao et al.)	N/A (kernel only)	kernel benchmarks (H100 SXM5, seq 512–16K), FP8 numerical error	Yes, H100 required
GQA	HF Transformers	Llama-2/3, Mistral	CNN/DailyMail, WMT, TriviaQA	Yes, in production
MQA	PyTorch/HF core	N/A (arch. pattern)	WMT14	Yes, in production
PagedAttention	vLLM (production)	N/A (kernel only)	ShareGPT, Alpaca traces	Yes, in production

Table 15: Reproducibility audit of the scored panel and selected contextual methods. Code and checkpoint availability, as verified by the author on 7 Aug 2026.

Panel of Figure 8 — Figure 8: Evolution of selected attention and sequence-modeling systems (2015–2026). Foundational period established the core architecture; the sparse/linear era introduced sub-quadratic alternatives; the current hybrid era blends systems co-design, SSMs, and learned sparsity. The timeline is a selected chronology of representative methods and system landmarks (including model-scale entries such as GPT-4), not an exhaustive listing of every surveyed method. Eras are thematic sub-eras, not chronologically exclusive. Node years denote the cited publication/release year; venue years may differ for methods whose initial public release preceded formal publication (e.g., Bahdanau et al. (2015) was posted to arXiv in 2014; BERT is cited to Devlin et al. (2019) but was first released in October 2018; Mamba and FlashAttention-2 are commonly cited by their 2024 venue years but were first released in December 2023 and July 2023, respectively). Horizontal offsets within a year are display-only and do not encode publication date. Year positions on the axis are schematic rather than proportional to elapsed time; years in which no surveyed method debuted are omitted.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Method	E	Ex	I	Evidence (score $\to$ evidence type)
Transformer	4	10	6	$4\!\to\!$ C (quadratic), $10\!\to\!$ C (exact), $6\!\to\!$ M (head attribution)
FlashAttention-2/3	8	10	6	$8\!\to\!$ C+H (IO-aware tiling), $10\!\to\!$ C (exact), $6\!\to\!$ M (same as full attn)
GQA	7	9	6	$7\!\to\!$ C+H (KV cache reduction), $9\!\to\!$ C (shared-KV, minor loss), $6\!\to\!$ M (shared proj.)
MQA	7	8	6	$7\!\to\!$ C+H (full KV sharing), $8\!\to\!$ B (slightly lower quality at matched size), $6\!\to\!$ M (shared proj.)
PagedAttention	8	10	6	$8\!\to\!$ H (block-level serving), $10\!\to\!$ C (exact), $6\!\to\!$ M (same as full attn)
Ring Attention	7	10	6	$7\!\to\!$ H (distributed overhead), $10\!\to\!$ C (exact), $6\!\to\!$ M (same as full attn)

Appendix

Table 16: EEI Score Evidence Toolkit. Each score is supported by at least one evidence type; “C” = complexity analysis, “B” = published benchmark, “H” = hardware/kernel measurement, “M” = mechanistic/interpretability analysis.

Attention Mechanisms

Attention mechanisms let neural networks weight information from other tokens or regions according to relevance, enabling context-dependent processing.

See all 513 papers

Mar 12, 2026

Why Attend to Everything? Focus is the Key

Standard attention scales quadratically with sequence length. Efficient attention methods reduce this O(n^2) cost, but when retrofitted into pretrained models, they often degrade perplexity, downstream accuracy, or both. We introduce Focus, a method that learns which token pairs matter. Focus adds a small set of learnable centroids--as few as 148K parameters per layer--that act as gates: only token pairs belonging to the same centroid group attend to each other over long ranges. Focus is composable: it can be added to any pretrained model by training only the centroids while keeping all original weights frozen. Experiments show that composing Focus onto pretrained models yields zero degradation on downstream benchmarks across model sizes from 124M to 70B parameters and five attention architectures. Surprisingly, sparse Focus attention outperforms full attention at 124M scale (30.3 vs. 31.4 perplexity) and matches full attention when trained from scratch at 7B scale (13.82 vs. 13.89). Focus is also fast: top-k group membership gives a 2x speedup with better quality than the original pretrained model. Using our FlashAttention decomposition, Focus achieves an 8.6x speedup at 1M tokens without custom kernels.

Hengshuai Yao, Xing Chen, Ahmed Murtadha +8

Sapient Intelligence · University of Alberta · Tsinghua University

Attention Mechanisms Efficient Attention Attention Approximation Sparse Attention Long-Context Modeling

Ask PDF

May 7, 2026

Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers

Transformers usually expose one inference cost per trained model, while deployed systems often need multiple cost-quality operating points. We study Budgeted Attention Allocation, a monotone head-gating mechanism conditioned on a requested attention budget. Dense warm-starting is important for stability: on a robust synthetic sequence task, one budgeted model reaches 99.7% accuracy at 0.303 estimated attention cost and 100.0% accuracy at 0.504 cost. On held-out AG News with a custom word-level transformer, hard-gate adaptation turns soft cost control into measured single-thread CPU speed, reaching 82.1% accuracy with 1.28x speedup at budget 0.50. In pretrained BERT-Mini AG News, budgeted structural pruning reaches 87.6% accuracy with 1.20x speedup at budget 0.50; a validation-ranked zero-shot dense post-hoc structural baseline reaches 86.1%, and one recovery epoch raises that per-budget specialist to 87.9%. On DBpedia14, BERT-Mini budgeted gates reach 97.4% at exact budget 0.50 versus 96.6% for dense full attention. Static fixed-budget gates and recovered dense specialists remain strong. The contribution is therefore not universal dominance, but a reproducible feasibility study of one controllable checkpoint across budgets that can trade attention cost for accuracy and be converted into measured structural speedups on small CPU benchmarks.

Amrit Nidhi

Independent Researcher

Efficient Transformer Inference Self-Attention Gated Attention Cost-Aware Inference Efficient Attention Multi-Head Attention

Ask PDF

May 9, 2026

Architecture, Not Scale: Circuit Localization in Large Language Models

Mechanistic interpretability assumes that circuit analysis becomes harder as models scale. We challenge this assumption by showing that the attention architecture matters more than parameter count. Studying three circuit types across Pythia and Qwen2.5, we find that grouped query attention produces circuits that are far more concentrated and mechanistically stable than standard multi-head attention at comparable scales. The same concentration pattern holds across indirect object identification, induction heads, and factual recall. Within a single architecture family (Qwen2.5), factual recall circuits undergo a discrete phase transition above a critical scale, collapsing to a single bottleneck rather than degrading gradually. These findings suggest that some architectural choices make large models more tractable to study and that interpretability difficulty is not a fixed consequence of model size.

Sohan Venkatesh

Manipal Institute of Technology Bengaluru.

Transformer Interpretability Grouped-Query Attention Circuit Discovery Attention Head Analysis Mechanistic Interpretability Circuits in Language Models

Ask PDF

Abstract

Attention Mechanisms

Explore similar work

Why Attend to Everything? Focus is the Key

Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers

Architecture, Not Scale: Circuit Localization in Large Language Models