Drawing on recent breakthroughs in cellular neurobiology and detailed biophysical modeling linking neocortical pyramidal neurons to distinct mental-state regimes, this work introduces a mathematically grounded formulation showing how models (e.g., Transformers) can implement computational principles underlying awake imaginative thought to pre-select relevant information before attention is applied via triadic modulation loops among queries (Q), keys (K), and values (V).~Scalability experiments on ImageNet-1K, benchmarked against a standard Vision Transformer (ViT), demonstrate significantly faster learning with reduced computational demand (fewer heads, layers, and tokens), consistent with our prior findings in reinforcement learning and language modeling. The approach operates at approximately O(N) complexity with respect to the number of input tokens N.
Figures & tables
Figure 1: Integrated view of biologically inspired mechanisms: (a) Pyramidal two-point neuron; (b) Co4 triadic reasoning via Q – K – V interactions; (c–d) MOD function dynamics illustrating context-sensitive filtering; Y represents the FF signal, which is separated into relevant and irrelevant streams depending on the strength of C ; (e–f) MOD contours and vector fields across the R – C space. Surface plots are presented over a wider range of R and C to illustrate the global geometry of the cooperation landscape, while contour and vector-field visualizations focus on a smaller range to highlight local gradient flow and regime transitions near the origin. Variations in the strengths of R and C shift the system across distinct processing regimes analogous to the neurobiological AA, AD, and AD+Awake regimes, producing corresponding geometric deformations in gradient flow. By shaping representations prior to downstream readout, these modulation laws guide optimization along R–C interaction manifolds, reducing propagation through noisy or irrelevant directions.
Figure 2: Early training comparison between an attention-only Vision Transformer (ViT) ( Dosovitskiy, 2020 ) , trained from scratch, and a Co 4 machine endowed with intrinsic mental-state-dependent processing regimes analogous to awake imaginative processing ( Phillips et al., 2024 ; Graham et al., 2025 ) , which pre-select relevant information before attention is applied. The task is to identify a bird from the Mini-ImageNet dataset. Brightness indicates regions emphasized. In the ViT model, this is after attention. In contrast, Co 4 rapidly forms a coherent interpretation of the input, highlighting the top- k salient regions via internally generated awake imaginative regimes before attention is computed. Co 4 exhibits earlier and sharper activation over the semantically relevant object (bird), indicating more coherent internal inference. These findings raise questions about the necessity of deep attention stacks.
Figure 3: The figure visualizes the complete attention distribution over N input tokens: a single-layer Co 4 machine versus an attention-only ViT ( Dosovitskiy, 2020 ) , both trained on Mini-ImageNet for 30 epochs. The ViT exhibits more dispersed attention with less selective localization. In contrast, Co 4 demonstrates more centered, context-sensitive activation patterns, indicating stronger spatial coherence.
Figure 4: Co 4 versus Transformer on CIFAR-10, Tiny-ImageNet, and Mini-ImageNet, trained from scratch: (i–iii) performance of a single-layer model; (iv) inference runtime as a function of sequence length for a single layer (see A.6 for theoretical computational cost analysis); and (v–vi) layer-wise validation accuracy on Tiny-ImageNet and Mini-ImageNet.
Dataset
Model
L
Combination
Hidden
MLP
H
Params.
Acc.
ImageNet-1K
Co 4
8
Co 4
768
3076
1
53.4M
70.24%
Co 4
6
Co 4
768
3076
1
40.4M
67.30%
Co 4
6
4 Co 4 + 2 MHSA
768
3076
1
41.6M
75.01 %
Transformer
12
MHSA
768
3076
12
86M
74.80%
Mini-ImageNet
Co 4
1
Co 4
384
3076
1
2.0M
39.20%
Co 4
4
Co 4
384
3076
1
8M
45.40%
Table 1: Comparison between Transformer and Co 4 models across ImageNet-1K, Mini-ImageNet, Tiny-ImageNet, and CIFAR-10: L, H, Params, and Acc denote the number of layers, attention heads, model parameters, and accuracy, respectively.
Dataset
L
H
Ep.
Embedding
Acc.
Imagenet-1k
6
1
300
768
67.30%
6
6
300
768
67.81%
6
12
300
768
67.89%
Cifar10
4
1
30
384
80.30%
4
4
30
384
80.36%
Table 2: Effect of varying the number of attention heads on Co 4 performance: L, H, Ep, and Acc denote the number of layers, attention heads, epochs, and accuracy, respectively.
Table 3: Effect of different modulation equations on CIFAR-10 performance (4 layers, 1 head, 30 epochs, 384-dimensional embeddings).
Dataset
Model
L
Tok.
k
Acc.
BL
Tiny-ImageNet
Co 4
4
64
44
31%
–
Co 4
4
64
32
33%
–
Co 4
4
64
8
40%
–
Co 4
4
64
3
37%
–
ViT
4
64
8
32%
35%
Co 4
4
64
8
40%
–
Table 4: Top- k token selection study comparing Co 4 with Transformer (ViT) under identical architectural settings. Performance is shown for multiple values of k , together with the ViT baseline ( Dosovitskiy, 2020 ) : L, Tok, k , Acc, and BL denote the number of layers, total tokens, the top- k threshold, top- k accuracy, and baseline ViT accuracy, respectively.
Figure 5: Training results across CartPole (i–iii), PyBullet Ant (iv–v), CarRacing (visual input: 96×96×4 ) (vi), and Acrobot (vii), with heat maps (viii–ix) provided as empirical evidence. TM1 – TM4 denote alternative well-established TPN-inspired asynchronous MOD functions ( Kay and Phillips, 2020 ) (see A.3).
Table 10
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: An example of “thinking fast and slow” as discussed in ( Kahneman, 2011 ) , illustrates how solving a riddle can involve combining rapid, intuitive processing in high-level perceptual state (fast thinking) with more deliberate, reflective refinement in the awake thought state (slow thinking).
Figure 7: Co4 basic architecture: latent Q , K , and V tokens evolve via TPN-like circuits and triadic modulation under distal, proximal, and universal context. Enables scalable, context-sensitive reasoning at an approximate cost of O(N) .
Figure 8: (a) Raster plots for 150 neurons. (i) standard TPNs + BDSP (ii) and context-sensitive (CS)-TPNs + BDSP with cooperative MOD(R,C) dynamics. In CS-TPNS, a clear distinction in bursting probability is observed as the inputs transition from low–low (LL) to high–high (HH), corresponding to input combinations 00 , 01 , 10 , and 11 . (b) CS-TPNs fire substantially less frequently than TPNs alone and exhibit bursting primarily when R and C inputs are coherent.
Modern Vision-Language Models (VLMs) benefit from chain-of-thought prompting and test-time scaling, but these gains often come with prohibitive inference cost due to large visual contexts and long decoding chains. We view this cost through two coupled axes: Visual Context Scaling (VCS), which controls how much visual evidence is passed to the language model, and Visual Reasoning Scaling (VRS), which controls how much inference-time reasoning search is performed. Existing methods typically optimize one axis at a time, leaving the joint allocation of compute across these axes underexplored. We introduce Adaptive Visual Inference Scaling (AVIS), a lightweight policy that adapts both VCS and VRS per query. AVIS realizes VCS through Key Diversity Visual (KDV) pruning, a training-free O(N) key-based rule for removing redundant visual tokens before prefilling, and realizes VRS through adaptive self-consistency, using a learned difficulty predictor to select the number of reasoning rollouts. AVIS is deployment-friendly and compatible with shared-prefill inference, where all rollouts reuse a single prefilling pass and KV cache. Across diverse image and video reasoning benchmarks, AVIS improves the accuracy--compute trade-off relative to VCS-only and VRS-only baselines, and remains effective on top of RL post-trained VLMs while keeping compute and latency low.
Ahmadreza Jeddi, Minh Ngoc Le, Amirhossein Kazerouni +8
AI Center-Toronto, Samsung Electronics · University of Toronto · Vector Institute +1
Sparse Attention Vectors (SAVs) have emerged as an excellent training-free alternative to supervised finetuning or low-rank adaptation to improve the performance of Vision Language Models (VLMs). At their heart, SAVs select a few accurate attention heads for a task of interest and use them as classifiers, rather than relying on the model's prediction. In a similar spirit, we find that directly probing the raw activations of the VLM, in the form of scalar values, is sufficient to yield accurate classifiers on diverse visually grounded downstream tasks. Shifting focus from attention vectors to scalar activations dramatically increases the search space for accurate parameters, allowing us to find more discriminative neurons immediately from the first generated token. We call such activations Super Neurons (SNs). In this probing setting, we discover that enough SNs appear in the shallower layers of the large language model to allow for extreme early exiting from the first layer of the model at the first generated token. Compared to the original network, SNs robustly improve the classification performance while achieving a speedup of up to 5.10x.
Pierre Musacchio, Jaeyi Jeong, Dahun Kim +1
Seoul National University · EPFL · Google Deepmind
MANAR (Memory-augmented Attention with Navigational Abstract Conceptual Representation), contextualization layer generalizes standard multi-head attention (MHA) by instantiating the principles of Global Workspace Theory (GWT). While MHA enables unconstrained all-to-all communication, it lacks the functional bottleneck and global integration mechanisms hypothesized in cognitive models of consciousness. MANAR addresses this by implementing a central workspace through a trainable memory of abstract concepts and an Abstract Conceptual Representation (ACR). The architecture follows a two-stage logic that maps directly to GWT mechanics: (i) an integration phase, where retrieved memory concepts converge to form a collective "mental image" (the ACR) based on input stimuli; and (ii) a broadcasting phase, where this global state navigates and informs the contextualization of individual local tokens. We demonstrate that efficient linear-time scaling is a fundamental architectural byproduct of instantiating GWT functional bottleneck, as routing global information through a constant-sized ACR resolves the quadratic complexity inherent in standard attention. MANAR is a compatible re-parameterization of MHA with identical semantic roles for its projections, enabling knowledge transfer from pretrained transformers via weight-copy and thus overcoming the adoption barriers of structurally incompatible linear-time alternatives. MANAR enables non-convex contextualization, synthesizing representations that provably lie outside the convex hull of input tokens - a mathematical reflection of the creative synthesis described in GWT. Empirical evaluations confirm that MANAR matches or exceeds strong baselines across language (GLUE score of 85.1), vision (83.9% ImageNet-1K), and speech (2.7% WER on LibriSpeech), positioning it as an efficient and expressive alternative to quadratic attention.