Cortical computation is strikingly low-dimensional: a handful of latent variables, carried in a neural population's activity, steer the higher-dimensional responses of individual neurons. Our aim is sample efficiency-models that decode well from limited data and at small parameter budgets. In a standard Transformer layer, the feed-forward block applies the same operator to every token. We suggest a von-Neumann inspired hypothesis of efficient computation as an alternative for neural decoding: a controller decodes an instruction and then executes a token-specific operator; the usual realization-a soft mixture of experts-only blends their outputs, not operators. We introduce a von-Neumann State-Space Transformer (VN-SST), a memory-augmented Transformer whose feed-forward block is a low-rank instruction bank: a shared base operator plus a small set of learned low-rank instructions, from which a per-token code synthesizes the weight matrix actually used at that token. The code is read from a low- dimensional projection of a carried state-space memory, so a slow latent trajectory acts as an instruction pointer-mirroring how low-dimensional dynamics may route cortical computation. On three motor-cortex neural-decoding benchmarks, VN-SST is far more data-efficient than a modern Transformer, each jointly predicting spikes and decoding behavior. This model wins by a wide margin on the scarcest benchmark, leads on the other two, and turns longer context into rising rather than falling accuracy. We evaluated that the network compresses a large instruction bank to a few bits per token, so program capacity acts as a control channel, not an accuracy lever. The same model is also more parameter-efficient on two small text benchmarks used for language modeling (LLMs), suggesting a generic mechanism.
Figures & tables
Benchmark
decode
Linear R2
Transformer R2
VN–SST R2
MC_RTT
finger vel.
-0.236
0.207
0.351
MC_Maze
hand vel.
0.549
0.655
0.716
Area2_Bump
hand vel.
0.561
0.632
0.696
Table 1: Joint neural codec, parameter scaling with limited data: peak behavioral–decoding R2 (mean of 3 seeds). “Linear” is a memoryless ridge decoder with short causal history.
Figure 1: Neural–codec parameter scaling with limited data. (a–c) Behavioral–decoding R2 vs. non–embedding parameters on the MC_RTT, MC_Maze, and Area2_Bump codecs, respectively, for the Transformer (blue) and VN–SST (green, “instruction bank”), with a linear reference. The two ladders are aligned to the same ∼64 – 270 K window (Transformer on widths 40 – 80 , VN–SST on 24 – 48 ), since VN–SST’s carried state and instruction bank add a fixed per–width overhead. Curves show the mean over 3 seeds. Away from the accuracy ceiling the architectures separate, with VN–SST leading on all three codecs; the margin is largest on the data–scarce MC_RTT codec in (a), where VN–SST reaches R2=0.35 against 0.21 for the Transformer and the memoryless linear decoder is not predictive ( R2<0 ).
Benchmark
data ( n )
Transformer
VN–SST
MC_RTT
2k → 8k bins
0.192 → 0.522
0.295 → 0.607
MC_Maze
14k → 58k bins
0.655 → 0.847
0.672 → 0.854
Area2_Bump
7k → 28k bins
0.599 → 0.778
0.678 → 0.795
Table 2: Data scaling at the fixed–depth ∼150 K decoding model ( 50 epochs, smoothed rates, mean of 3 seeds): decode R2 at the smallest → largest training budget (four recording–bin budgets per codec).
Figure 2: Data scaling at the fixed–depth ∼150 K decoding model (mean of 3 seeds). (a–c) Decode R2 vs. training bins on the MC_RTT, MC_Maze, and Area2_Bump codecs, respectively, for the Transformer (blue) and VN–SST (green).
Benchmark
seq. len. ( L )
Transformer
VN–SST
MC_RTT
16 → 128
0.612 @ 16
0.680 @ 128
MC_Maze
16 → 128
0.879 @ 32
0.878 @ 16
Area2_Bump
16 → 128
0.766 @ 32
0.779 @ 64
Table 3: Sequence–length scaling at the fixed–depth ∼150 K decoding model and full data, at a matched training budget (epochs scaled with L so the number of optimizer steps is held constant across context lengths; mean of 3 seeds): best decode R2 over L∈{16,32,64,128} and the L at which it peaks (each cell reads best R2 @ L ). On the scarce MC_RTT codec the two diverge—the Transformer’s decode gently declines as L grows ( 0.61→0.57 ) while VN–SST’s rises ( 0.60→0.68 ), converting longer context into accuracy through its carried state; on the data–rich codecs both plateau (VN–SST slightly higher and flatter, and it alone holds up at L=128 ).
Figure 3: Sequence–length scaling at fixed model and full data, at a matched update budget (epochs scaled with L ; mean of 3 seeds): decode R2 vs. training context length L (log 2 x –axis), on the MC_RTT (a), MC_Maze (b), and Area2_Bump (c) codecs. (a) On the scarce MC_RTT codec the two architectures cross over: the Transformer starts higher at L=16 but declines as the window grows ( 0.61→0.57 ), whereas VN–SST rises ( 0.60→0.68 ) and overtakes it—threading state across segments, it converts longer context into decoding accuracy while the Transformer’s wider receptive field does not help its short–horizon behavioral read–out. (b,c) On the data–rich MC_Maze and Area2_Bump codecs both are high and largely flat, with VN–SST slightly higher.
Benchmark
K
code bits
log2K
particip. ratio
decode R2 (range)
MC_RTT
32
2.55
5.0
3.4
0.635–0.664
MC_Maze
32
2.89
5.0
5.1
0.864–0.872
Area2_Bump
32
3.20
5.0
6.5
0.779–0.787
Table 4: Control–bits diagnostic. Given a program bank of size K=32 at a fixed model size, we collect the per–token instruction codes ct on validation data and report the spectral entropy of their covariance (bits of code variation) and the participation ratio (effective number of instructions used), alongside the range of decode R2 over K∈{1,…,32} . On every codec the model compresses a 32 –instruction bank to ≈2.5 – 3.2 bits / ≈3.4 – 6.5 effective operators—well below the log2K=5 ceiling—while decode R2 is essentially flat in K : program capacity is a control/compression knob, not an accuracy lever.
Figure 4: Program–manifold sweep at fixed model size. (a) Decode R2 is essentially flat and non–monotonic in the program count K —capacity is not the bottleneck here. (b,c) The per–token instruction code compresses: given K=32 instructions the network uses only ≈2.5 – 3.2 bits of code entropy and a participation ratio of ≈3.4 – 6.5 , bending progressively below the log2K ceiling—so the network uses only a few instructions, consistently across recordings.
Transformer
VN–SST
Corpus
best PPL
slope
best PPL
slope
tiny–Shakespeare
61
-0.23
49
-0.33
WikiText–2
52
-0.20
45
-0.26
Table 5: Additional modality check—sub–word language modeling. Validation perplexity (best over a ∼0.13 – 1.7 M non–embedding parameter ladder at fixed two–layer depth with width scaled, mean of 3 seeds, at the largest ( 3× ) data budget) and the fitted power–law slope vs. parameters, for the Transformer and VN–SST on byte–level BPE tiny–Shakespeare and WikiText–2. The instruction–bank model, unchanged from the neural experiments, is more parameter–efficient across the ladder on both corpora, attains the best perplexity on both, and has the steeper slope on both.
Figure 5: Additional modality check: language–model scaling at two data budgets. Validation perplexity vs. non–embedding parameters (log–log, with power–law fits) for the Transformer and VN–SST. Top row (a,b): the small “1 × ” budget; bottom row (c,d): the “3 × ” budget with three times as many tokens; tiny–Shakespeare in (a,c) and WikiText–2 in (b,d). The instruction–bank model—unchanged from the neural experiments—tracks below the Transformer across the ladder in every panel, has the steeper slope, and reaches the best perplexity. More data lowers both curves; VN–SST’s lead widens with data on tiny–Shakespeare and narrows on WikiText–2 (where both models saturate), but it stays ahead throughout.