MANAR (Memory-augmented Attention with Navigational Abstract Conceptual Representation), contextualization layer generalizes standard multi-head attention (MHA) by instantiating the principles of Global Workspace Theory (GWT). While MHA enables unconstrained all-to-all communication, it lacks the functional bottleneck and global integration mechanisms hypothesized in cognitive models of consciousness. MANAR addresses this by implementing a central workspace through a trainable memory of abstract concepts and an Abstract Conceptual Representation (ACR). The architecture follows a two-stage logic that maps directly to GWT mechanics: (i) an integration phase, where retrieved memory concepts converge to form a collective "mental image" (the ACR) based on input stimuli; and (ii) a broadcasting phase, where this global state navigates and informs the contextualization of individual local tokens. We demonstrate that efficient linear-time scaling is a fundamental architectural byproduct of instantiating GWT functional bottleneck, as routing global information through a constant-sized ACR resolves the quadratic complexity inherent in standard attention. MANAR is a compatible re-parameterization of MHA with identical semantic roles for its projections, enabling knowledge transfer from pretrained transformers via weight-copy and thus overcoming the adoption barriers of structurally incompatible linear-time alternatives. MANAR enables non-convex contextualization, synthesizing representations that provably lie outside the convex hull of input tokens - a mathematical reflection of the creative synthesis described in GWT. Empirical evaluations confirm that MANAR matches or exceeds strong baselines across language (GLUE score of 85.1), vision (83.9% ImageNet-1K), and speech (2.7% WER on LibriSpeech), positioning it as an efficient and expressive alternative to quadratic attention.
Figures & tables
Figure 1: Geometric interpretation of contextualization with and without retrieved memory concepts.
Figure 2: High level architecture of MANAR
Model
MNLI
QNLI
RTE
MRPC
QQP
STS-B
CoLA
SST
Avg.
BERT [ 18 ]
84.0 / 84.4
89.0
61.0
86.3
89.1
89.5
57.3
93.0
80.7
data2vec [ 6 ]
83.2/83.0
90.9
67.0
90.2
89.1
87.2
62.2
91.8
82.7
RoBERTa [ 36 ]
84.4 /84.1
90.4
75.3
89.8
89.6
89.3
57.4
92.3
83.4
MANAR-484.64.128
83.7/83.4
90.3
71.9
90.6
89.1
90.3
59.7
94.4
83.8
MANAR-16K.128.128 †
84.0 / 84.6
91.2
73.6
91.7
90.2
90.5
63.1
94.1
85.1
Table 1: GLUE development set results (%). Higher is better.
Model
Image Size
Top-1 Acc. (%)
Small Models
DeiT-S [ 49 ]
2242
79.8
Vim-S [ 57 ]
2242
80.3
MANAR-256.32.96-S
2242
80.7
MANAR-4K.32.96-S †
2242
81.6
Base Models
Table 2: Comparison of different backbone architectures on ImageNet-1K.
Model
dev-clean
dev-other
test-clean
test-other
wav2vec2.0 [ 7 ]
2.7
7.9
3.4
8.0
HuBERT [ 25 ]
2.7
7.8
3.4
8.1
data2vec [ 6 ]
2.2
6.4
2.8
6.8
MANAR-256.64.128
2.3
6.7
2.9
6.8
MANAR-4K.128.128 †
2.0
6.4
2.7
6.4
Table 3: Word Error Rate (WER; %) on LibriSpeech standard dev/test sets. All models use the same 12-layer Transformer encoder. Decoding uses the official 4-gram language model [ 24 ] . Lower is better.
Figure 3: Comparing the time and space complexity of MANAR against vanilla MHA (left), and MANAR-based DeiT-S ( MANAR-256.32.96-S ) against vanilla DeiT-S (right). The upper-right time measurements are shown on a quadratic scale to account for the resolution-dependent quadratic cost.
Figure 4: Colormap is normalized to [78, 81.7] to highlight differences in the high-accuracy regime; values outside the range saturate.
Figure 5: Fraction of layer outputs lying outside the convex hull of input values (CHM) across encoder layers for image, speech, and language models.
Linear attention transformers have become a strong alternative to softmax attention due to their efficiency. However, linear attention tends to be less expressive and results in reduced accuracy compared to softmax attention. To bridge the accuracy gap between softmax attention and linear attention, we manipulate Mamba-2, a very strong linear attention variant. We first simplify Mamba-2 down to its most fundamental and important components, evaluating which specific choices make it most accurate. From this simplified Mamba variant (Mamba-2S), we improve the A-mask and increase the order of the hidden state, resulting in a method, which we call 2Mamba, that is nearly as accurate as softmax attention, yet much more memory efficient for long context lengths. We also investigate elements to Mamba-2 that help surpass softmax attention accuracy. Code is provided for all our experiments.
Gabriel Mongaras, Eric C. Larson
Lyle School of Engineering Southern Methodist University Dallas, TX 75205
Language models typically construct attention values from contextual hidden states, even when some of their content may be reusable across contexts. We investigate whether token-indexed memory can replace the dedicated value projection when complemented by contextual information. We propose Memory Attention (MA), which forms values by combining layer-specific token memory with contextual keys. The memory supplies token-specific representations, while the keys preserve context dependence. At inference, normalization can be folded into the memory tables, reducing value construction to lookup and addition. Token-indexed retrieval also enables CPU offloading with prefetching, reducing GPU parameter storage. Under matched training token budgets and with additional memory parameters, experiments across attention configurations show improved language modeling and average downstream performance.
As long-context language modeling becomes increasingly important, the cost of maintaining and attending to large Key/Value (KV) caches grows rapidly, becoming a major bottleneck in both training and inference. While prior works such as Multi-Query Attention (MQA) and Multi-Latent Attention (MLA) reduce memory by sharing or compressing KV features, they often trade off representation quality or incur runtime overhead. We propose Memory-Keyed Attention (MKA), a hierarchical attention mechanism that integrates multi-level KV caches (local, session, and long-term) and learns to route attention across them dynamically. We further introduce Route-Fused MKA (FastMKA), a broadcast-routed variant that fuses memory sources before attention computation for improved efficiency. Experiments on different sequence lengths show that FastMKA achieves a favorable accuracy-efficiency trade-off: comparable perplexity to MLA while achieving up to 5x faster training throughput and 1.8x lower evaluation latency. These results highlight MKA as a practical and extensible framework for efficient long-context attention.
Dong Liu, Yanxuan Yu, Ben Lengerich +1
University of California, Los Angeles, Los Angeles, California, USA · Columbia University, New York, New York, USA · University of Wisconsin-Madison, Madison, Wisconsin, USA