MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation

Zuher Jahshan; Ben Ben Ishay; Leonid Yavits

cs.AIMar 19, 2026

MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation

Authors: Zuher Jahshan, Ben Ben Ishay, Leonid Yavits

Organizations: Bar Ilan University, Ramat Gan, Israel

Abstract

MANAR (Memory-augmented Attention with Navigational Abstract Conceptual Representation), contextualization layer generalizes standard multi-head attention (MHA) by instantiating the principles of Global Workspace Theory (GWT). While MHA enables unconstrained all-to-all communication, it lacks the functional bottleneck and global integration mechanisms hypothesized in cognitive models of consciousness. MANAR addresses this by implementing a central workspace through a trainable memory of abstract concepts and an Abstract Conceptual Representation (ACR). The architecture follows a two-stage logic that maps directly to GWT mechanics: (i) an integration phase, where retrieved memory concepts converge to form a collective "mental image" (the ACR) based on input stimuli; and (ii) a broadcasting phase, where this global state navigates and informs the contextualization of individual local tokens. We demonstrate that efficient linear-time scaling is a fundamental architectural byproduct of instantiating GWT functional bottleneck, as routing global information through a constant-sized ACR resolves the quadratic complexity inherent in standard attention. MANAR is a compatible re-parameterization of MHA with identical semantic roles for its projections, enabling knowledge transfer from pretrained transformers via weight-copy and thus overcoming the adoption barriers of structurally incompatible linear-time alternatives. MANAR enables non-convex contextualization, synthesizing representations that provably lie outside the convex hull of input tokens - a mathematical reflection of the creative synthesis described in GWT. Empirical evaluations confirm that MANAR matches or exceeds strong baselines across language (GLUE score of 85.1), vision (83.9% ImageNet-1K), and speech (2.7% WER on LibriSpeech), positioning it as an efficient and expressive alternative to quadratic attention.

Figures & tables

Figure 1: Geometric interpretation of contextualization with and without retrieved memory concepts.

Figure 2: High level architecture of MANAR

Model	MNLI	QNLI	RTE	MRPC	QQP	STS-B	CoLA	SST	Avg.
BERT [ 18 ]	84.0 / 84.4	89.0	61.0	86.3	89.1	89.5	57.3	93.0	80.7
data2vec [ 6 ]	83.2/83.0	90.9	67.0	90.2	89.1	87.2	62.2	91.8	82.7
RoBERTa [ 36 ]	84.4 /84.1	90.4	75.3	89.8	89.6	89.3	57.4	92.3	83.4
MANAR-484.64.128	83.7/83.4	90.3	71.9	90.6	89.1	90.3	59.7	94.4	83.8
MANAR-16K.128.128 †	84.0 / 84.6	91.2	73.6	91.7	90.2	90.5	63.1	94.1	85.1

Table 1: GLUE development set results (%). Higher is better.

Model	Image Size	Top-1 Acc. (%)
Small Models
DeiT-S [ 49 ]	$224^{2}$	79.8
Vim-S [ 57 ]	$224^{2}$	80.3
MANAR-256.32.96-S	$224^{2}$	80.7
MANAR-4K.32.96-S †	$224^{2}$	81.6
Base Models

Table 2: Comparison of different backbone architectures on ImageNet-1K.

Model	dev-clean	dev-other	test-clean	test-other
wav2vec2.0 [ 7 ]	2.7	7.9	3.4	8.0
HuBERT [ 25 ]	2.7	7.8	3.4	8.1
data2vec [ 6 ]	2.2	6.4	2.8	6.8
MANAR-256.64.128	2.3	6.7	2.9	6.8
MANAR-4K.128.128 †	2.0	6.4	2.7	6.4

Table 3: Word Error Rate (WER; %) on LibriSpeech standard dev/test sets. All models use the same 12-layer Transformer encoder. Decoding uses the official 4-gram language model [ 24 ] . Lower is better.

Figure 3: Comparing the time and space complexity of MANAR against vanilla MHA (left), and MANAR-based DeiT-S ( MANAR-256.32.96-S ) against vanilla DeiT-S (right). The upper-right time measurements are shown on a quadratic scale to account for the resolution-dependent quadratic cost.

Figure 4: Colormap is normalized to [78, 81.7] to highlight differences in the high-accuracy regime; values outside the range saturate.

Figure 5: Fraction of layer outputs lying outside the convex hull of input values (CHM) across encoder layers for image, speech, and language models.

Explore similar work

2Mamba2Furious: Linear in Complexity, Competitive in Accuracy
Feb 19, 2026Gabriel Mongaras, Eric C. LarsonLinear Attention State Space Models
Lyle School of Engineering Southern Methodist University Dallas, TX 75205
Memory Attention
Sep 23, 2026Jiale KangMemory-Augmented Language Models LLM Inference Acceleration
MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning
Mar 21, 2026Dong Liu, Yanxuan Yu, Ben Lengerich +1Attention Mechanisms Long-Context Language Modeling
University of California, Los Angeles, Los Angeles, California, USA · Columbia University, New York, New York, USA · University of Wisconsin-Madison, Madison, Wisconsin, USA

Feb 19, 2026J/K move · Enter open · S save

2Mamba2Furious: Linear in Complexity, Competitive in Accuracy

Gabriel Mongaras, Eric C. Larson

Lyle School of Engineering Southern Methodist University Dallas, TX 75205

Linear Attention State Space Models

Open paper AskPDF arXiv

Linear attention transformers have become a strong alternative to softmax attention due to their efficiency. However, linear attention tends to be less expressive and results in reduced accuracy compared to softmax attention. To bridge the accuracy gap between softmax attention and linear attention, we manipulate Mamba-2, a very strong linear attention variant. We first simplify Mamba-2 down to its most fundamental and important components, evaluating which specific choices make it most accurate. From this simplified Mamba variant (Mamba-2S), we improve the A-mask and increase the order of the hidden state, resulting in a method, which we call 2Mamba, that is nearly as accurate as softmax attention, yet much more memory efficient for long context lengths. We also investigate elements to Mamba-2 that help surpass softmax attention accuracy. Code is provided for all our experiments.

Abstract

Explore similar work

2Mamba2Furious: Linear in Complexity, Competitive in Accuracy

Memory Attention

MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning