Can the global-workspace account of transformer representations extend to state-space language models? We fit Jacobian lenses to the residual streams and recurrent states of Mamba-1, Mamba-2 and Mamba-3, using the original 1000-prompt recipe. Joint residual--state readouts improve recovery of known intermediate concepts over the residual lens on at least five of six task families in every tested Mamba checkpoint. On Mamba-2, state alone exceeds residual and logit lenses on all six families; a normalised joint readout improves on both components on five. Temporal maps and word-list experiments show earlier content remaining state-readable as residual visibility changes. We also propose sign-guarded steering, which improves target top-five success over coordinate exchange on matched verbal-report trials in five models. Recurrent state alone supports this verbal access. These gains do not extend consistently to relational answers: guarded edits often output the edited concept itself, and Mamba-3's joint edits can disrupt successful state-only redirection. Recurrent state thus provides a complementary carrier of workspace content, whose recovery, persistence and causal uses require separate measurements.
Figures & tables
Model
Params.
Blocks
dmodel
State blocks
Role
Mamba-1
2.8B
64
2560
63
recurrent comparison
Mamba-2
2.7B
64
2560
63
principal model
Mamba-3 SISO
1.5B
24
2048
23
recurrent comparison
Mamba-3 MIMO
1.5B
24
2048
23
readout comparison
Pythia
2.8B
32
2560
–
transformer control
Nemotron-3.5
30B/3B active
52
2688
23
hybrid control
Table 1: Models and paper-relevant coverage. “State blocks” counts source blocks with fitted state lenses; the final block is the fit target and has no state lens. The MIMO model has readout results but not the full state intervention suite.
Task
Edited concept (example)
Scored output
Verbal report
Category answer: Canada to Brazil
Brazil in next-token top five
Two-hop reasoning
Implicit intermediate: Brazil to Mexico
Spanish, the new intermediate’s answer
Flexible generalisation
Explicit argument: France to Canada
Ottawa, the function’s new answer
Table 2: The edited concept and the scored output are task dependent.
Model
Best state > residual
Plain joint > residual
State coverage
Mamba-1 2.8B
5/6
6/6
63 source blocks
Mamba-2 2.7B
6/6
6/6
63 source blocks
Mamba-3 SISO 1.5B
5/6
5/6
23 source blocks
Mamba-3 MIMO 1.5B
4/6
5/6
23 source blocks
Table 3: Number of task families (out of six) on which a readout has higher AUC than the within-model residual J-lens. “Best state” denotes the highest AUC among the evaluated state-lens variants for each task family. These are directional per-family comparisons, not claims that every difference is statistically significant.
Task
Residual Jh
State Jfull
Joint normalised
Permuted-state joint
Multihop
0.475
0.557
0.679
0.431
Multilingual
0.362
0.615
0.647
0.312
Order of ops.
0.460
0.795
0.807
0.515
Poetry
0.029
0.101
0.114
0.092
Association
0.058
0.237
0.231
0.117
Typo
0.303
0.706
0.714
0.350
Table 4: Mamba-2 intermediate recovery over all 63 source blocks, measured by normalised pass@ k AUC (not task accuracy). The normalised joint readout combines residual and state after separately normalising their transported representations. Bold marks the best non-oracle readout in each row.
Figure 1: Mamba-2 intermediate recovery over all 63 source blocks. State alone outperforms residual alone on all six families; the normalised joint readout exceeds both components on five, with association the exception. The permuted-state joint control destroys coordinate correspondence while preserving dimensionality. Values are normalised pass@ k AUC , not task accuracy.
Figure 2: Selected Mamba-2 rank maps across all 63 source blocks, using the post-update convention. Seattle remains top-ranked at state block 28 from the end of “Needle” through the final readout, while its residual visibility varies. In the emerald question, May becomes readable late in the residual stream and remains outside the state’s top 100 throughout. Both prompts are answered correctly.
Mamba-1
Mamba-2
Rank words held@5
7.5
6.7
Transported-state pursuit words
18.7
18.6
Table 5: Word-list recovery with the Jfull state lens over all 63 source blocks. Values are last-quartile means over comma positions in 80-word lists. Pursuit uses k=25 and threshold zero on the lens-transformed state in final-residual coordinates; its occupancy and rank-based words held@5 are distinct estimators.
Transformers
SSMs
Hybrid
Measurement
Qwen3.6 27B †
Pythia 2.8B
Mamba-1 2.8B
Mamba-2 2.7B
Mamba-3 SISO 1.5B
Nemotron 3.5
Verbal report Target top-5
85/124
242/399
85/446
114/335
207/728
130/621
Two-hop reasoning Requested answer
47/90
5/90
6/90
7/90
6/90
15/90
Flexible generalisation Requested answer
80/192
13/192
30/192
12/192
23/192
29/192
Introspection Injected token top-1
0/101
2/99
R : 0/99 S : 0/99
R : 0/99 S : 4/99
R : 6/101 S : 49/101
R : 0/99 S : 93/99
Directed modulation Focus / control / suppress hit@5
.38/.07/.07
.04/.07/.04
R : .01/.05/.04 S : .15/.15/.13
R : .01/.05/.04 S : .15/.20/.17
R : .00/.00/.01 S : .07/.09/.10
R : .02/.02/.02 S : .03/.05/.02
Table 6: Workspace properties across models. Our measurements in the first three rows use Xunit exchange: simultaneous residual–state edits ( RS ) for recurrent models and residual edits ( R ) for Pythia. Verbal report scores target top-five; the two suite tasks score the requested answer. Other rows retain their separate readout or injection protocols. All Qwen3.6-27B values are quoted from Wang and Reid (2026) and were not reproduced in this study.
Transformers
SSMs
Hybrid
Intervention
Qwen3.6 27B †
Pythia 2.8B
Mamba-1 2.8B
Mamba-2 2.7B
Mamba-3 SISO 1.5B
Nemotron 3.5
Table 6 result Coordinate exchange in this study
85/124
242/399
85/446
114/335
207/728
130/621
Sign-guarded steering
–
298/399
426/446
333/335
677/728
621/621
Table 7: Verbal report: sign-guarded steering compared directly with Table 6 . The first row repeats its verbal-report counts and model order. Both methods edit residual and state together in recurrent models and residual alone in Pythia. Success means the target enters the next-token top five.
Figure 3: Two-hop outputs under coordinate exchange ( X ) and sign-guarded steering ( G ). Panel A shows all 90 trials per model; panel B shows those answered correctly before editing. Colours distinguish the requested answer, the edited concept and other outputs. Both methods edit residual and state together in recurrent models, and residual alone in Pythia. The Mamba-3 callout compares state-only and joint steering. These are saved token predictions, not complete responses; Appendix E gives the counts, scoring rules and qualifications.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Any block (0–62)
Band (16–48)
Task
State
Residual
State
Residual
Multihop
0.557
0.475
0.444
0.313
Multilingual
0.615
0.362
0.601
0.276
Order of ops.
0.795
0.460
0.589
0.266
Poetry
0.101
0.029
0.078
0.014
Association
0.237
0.058
0.196
0.051
Appendix
Table 8: Mamba-2 state and residual AUC over all source blocks and over the fixed intervention band.
Task
State Jfull
Logit
Tuned
Permuted state
Multihop
0.557
0.379
0.441
0.021
Multilingual
0.615
0.449
0.446
0.056
Order of ops.
0.795
0.647
0.666
0.164
Poetry
0.101
0.049
0.036
0.049
Association
0.237
0.027
0.036
0.022
Typo
0.706
0.292
0.279
0.040
Appendix
Table 9: Mamba-2 normalised pass@ k AUC over all 63 source blocks. The Jfull state lens exceeds both residual baselines on all six families.
Task
WikiText J-lens
Old Pile J-lens
Logit
Tuned
Multihop
0.470
0.474
0.296
0.353
Multilingual
0.324
0.333
0.214
0.217
Order of ops.
0.736
0.748
0.583
0.674
Poetry
0.063
0.078
0.018
0.031
Association
0.008
0.014
0.001
0.002
Typo
0.377
0.369
0.287
0.239
Appendix
Table 10: Pythia-2.8B recovery AUC over all 31 source layers. The primary J-lens uses the conforming 1000-prompt WikiText recipe; the old 16-prompt Pile lens is a non-conforming sensitivity comparison.
Task
Original
Row zeroed
Change
Multihop
0.475
0.443
−0.0322
Multilingual
0.362
0.335
−0.0267
Order of ops.
0.460
0.417
−0.0436
Poetry
0.029
0.045
+0.0157
Association
0.058
0.056
−0.0013
Typo
0.303
0.304
+0.0008
Appendix
Table 11: Mamba-2 residual J-lens recovery, original and with output row 2107 zeroed. Changes use unrounded AUC values.
Model
Band blocks
Jfull
Jread
Clean
Mamba-1
33
1 [0.2, 6.0]
7 [3.8, 15.2]
20
Mamba-2
33
1 [0.2, 6.0]
–
25
Mamba-2, sparse
12
1 [0.2, 6.0]
1 [0.2, 6.0]
25
Mamba-3 SISO
13
12 [7.8, 21.9]
19 [14.0, 30.6]
27
Random / shuffled
matched
1 [0.2, 6.0] per run
–
Appendix
Table 12: Persistent state clamps: requested swap counts out of 90, with Wilson 95% intervals in brackets (percentage units). “Clean” counts correct original answers. The sparse Mamba-2 lens has 23 source blocks, 12 in the intervention band. Controls match Mamba-1/3 Jread and full-coverage Mamba-2 Jfull ; each of three random seeds and the shuffled-pair run gives the stated count.
Construction / outcome
Count
95% interval (%)
Residual exchange, all scorable
10/80
[6.9, 21.5]
Residual exchange, given clean correct
7/23
[15.6, 50.9]
Original pursuit, retained
22/37
[43.5, 73.7]
Row-zeroed pursuit, retained
12/37
[19.6, 48.5]
Coordinates 2–11, retained
22/37
[43.5, 73.7]
Random coordinates 2–21, retained
26/37
[54.2, 82.5]
Appendix
Table 13: Mamba-2 coordinate exchange and ten-atom pursuit ablations. Exchange success and retained correct answers are different outcomes. The two coordinate controls match the ablation comparison in Figure 4 .
Construction
State, all
State, clean correct
Both, all
Jread exchange
1/84 [0.2, 6.4]
0/29 [0.0, 11.7]
6/84 [3.3, 14.7]
Jread clamp
22/84 [18.0, 36.5]
14/29 [31.4, 65.6]
15/84 [11.1, 27.4]
Jfull clamp
13/84 [9.3, 24.7]
8/29 [14.7, 45.7]
14/84 [10.2, 26.1]
Appendix
Table 14: Mamba-3 protocol interventions using raw lens vectors, at blocks 6–18 and every prompt position. Counts include Wilson 95% intervals in brackets (percentage units). The final column applies both residual and state edits; for clamp rows, both use clamp transfer.
Model
Residual only
State only
Both
Mamba-1
10/90
1/90
11/90
Mamba-2
1/90
1/90
1/90
Mamba-3
3/90
12/90
14/90
Appendix
Table 15: Historical signed clamp ( N ) by intervention location: requested swaps in the 90-item workspace suite. State edits use Jfull and persist through the recurrence; all edits apply at every prompt position. Mamba-1 edits residual blocks 16–48 and 12 sampled state blocks within that band; Mamba-2 edits all 33 blocks in the band for both representations. Mamba-3 is the SISO checkpoint, with both representations edited at blocks 6–18. Comparisons within each row use the same prompts and scoring.
Figure 4: Construction-specific causal evidence. Panel A compares persistent state clamps and matched direction controls in the common 90-item workspace suite; the three random seeds and shuffled pairs each yield 1/90 and share a single plotted control marker. Panel B reports Mamba-2’s residual coordinate exchange under the separate protocol. Panel C compares Mamba-2 pursuit ablations and variance-matched coordinate controls; fewer retained answers indicate a larger effect. Denominators and constructions are shown explicitly and must not be pooled into one causal score.
Condition
Target rank
Top-1 word
Top-5 success
Clean
20
Canada
No
Residual clamp
1
Brazil
Yes
State clamp
16
France
No
Simultaneous clamps
1
Brazil
Yes
Matched random residual
23
Canada
No
Matched random state
20
Canada
No
Appendix
Table 16: Historical signed-clamp ( N ) outcome on Mamba-1 trial country-05-t00 ; the random rows are its original matched controls.
Table 18: Verbal report: all operators and interfaces. Counts use the stated n ; C is clean success under the swapped-target criterion, not original-answer competence. The final three columns list matched-norm control successes for seeds 0/1/2 separately. U and Rw denote unit and raw exchange. Mamba-3 verbal includes the replay-qualified measurements.
Model
Site
G−Xunit
Interval
G−Xraw
Interval
Mamba-1
R
+0.740
[+0.628, +0.844]
+0.733
[+0.620, +0.835]
Mamba-1
S
+0.581
[+0.499, +0.654]
+0.590
[+0.510, +0.662]
Mamba-1
RS
+0.765
[+0.672, +0.857]
+0.760
[+0.674, +0.847]
Mamba-2
R
+0.510
[+0.402, +0.617]
+0.507
[+0.392, +0.616]
Mamba-2
S
+0.767
[+0.704, +0.829]
+0.767
[+0.704, +0.831]
Mamba-2
RS
+0.654
[+0.547, +0.757]
+0.645
[+0.533, +0.750]
Appendix
Table 19: Verbal report: paired G−X success-rate differences and simultaneous intervals. Both exchange conventions retain the original 78-contrast family correction; these are not independent replications.
Model
Site
n
N
G
Xunit
Xraw
Mamba-1
R
446
436
376
34
28
Mamba-1
S
446
2
202
1
3
Mamba-1
RS
446
442
401
19
17
Mamba-2
R
335
26
258
74
77
Mamba-2
S
335
0
203
0
0
Mamba-2
RS
335
27
314
69
74
Appendix
Table 20: Descriptive top-one sensitivity on the same verbal-report trials. The primary endpoint remains top five.
Model
Site
n
C
N
G
U
Rw
G ctrl
U ctrl
Rw ctrl
Mamba-1
R
90
1
10
5
7
7
1/1/3
1/1/1
1/1/1
Mamba-1
S
90
1
1
1
1
1
1/1/1
1/1/1
1/1/1
Mamba-1
RS
90
1
11
4
6
6
1/2/3
1/1/1
1/1/1
Mamba-2
R
90
1
1
12
7
7
1/1/1
1/2/1
0/2/1
Mamba-2
S
90
1
1
2
1
1
1/1/1
1/1/1
1/1/1
Mamba-2
RS
90
1
1
6
7
5
1/1/1
1/2/1
0/3/1
Appendix
Table 21: Two-hop reasoning: all operators and interfaces. Counts use the stated n ; C is clean success under the swapped-target criterion, not original-answer competence. The final three columns list matched-norm control successes for seeds 0/1/2 separately. U and Rw denote unit and raw exchange. Mamba-3 verbal includes the replay-qualified measurements.
Model
Site
G−Xunit
Interval
G−Xraw
Interval
Mamba-1
R
-0.022
[-0.089, +0.056]
-0.022
[-0.089, +0.056]
Mamba-1
S
+0.000
[+0.000, +0.000]
+0.000
[+0.000, +0.000]
Mamba-1
RS
-0.022
[-0.089, +0.044]
-0.022
[-0.089, +0.044]
Mamba-2
R
+0.056
[-0.033, +0.144]
+0.056
[-0.011, +0.133]
Mamba-2
S
+0.011
[+0.000, +0.044]
+0.011
[+0.000, +0.044]
Mamba-2
RS
-0.011
[-0.100, +0.067]
+0.011
[-0.067, +0.078]
Appendix
Table 22: Two-hop reasoning: paired G−X success-rate differences and simultaneous intervals. Both exchange conventions retain the original 78-contrast family correction; these are not independent replications.
Model
Site
n
C
N
G
U
Rw
G ctrl
U ctrl
Rw ctrl
Mamba-1
R
192
4
37
18
32
33
2/2/3
6/4/3
6/4/3
Mamba-1
S
192
4
3
3
3
3
4/4/4
4/4/3
4/4/3
Mamba-1
RS
192
4
37
18
30
31
2/2/3
6/4/3
6/4/3
Mamba-2
R
192
2
2
8
12
12
0/0/0
2/1/3
2/2/3
Mamba-2
S
192
2
3
2
3
3
2/2/2
2/2/2
2/2/2
Mamba-2
RS
192
2
2
23
12
11
0/0/0
2/1/3
2/2/3
Appendix
Table 23: Flexible generalisation: all operators and interfaces. Counts use the stated n ; C is clean success under the swapped-target criterion, not original-answer competence. The final three columns list matched-norm control successes for seeds 0/1/2 separately. U and Rw denote unit and raw exchange. Mamba-3 verbal includes the replay-qualified measurements.
Model
Site
G−Xunit
Interval
G−Xraw
Interval
Mamba-1
R
-0.073
[-0.188, +0.047]
-0.078
[-0.193, +0.042]
Mamba-1
S
+0.000
[+0.000, +0.000]
+0.000
[+0.000, +0.000]
Mamba-1
RS
-0.062
[-0.172, +0.052]
-0.068
[-0.177, +0.052]
Mamba-2
R
-0.021
[-0.120, +0.078]
-0.021
[-0.115, +0.078]
Mamba-2
S
-0.005
[-0.026, +0.000]
-0.005
[-0.026, +0.000]
Mamba-2
RS
+0.057
[-0.036, +0.146]
+0.062
[-0.016, +0.141]
Appendix
Table 24: Flexible generalisation: paired G−X success-rate differences and simultaneous intervals. Both exchange conventions retain the original 78-contrast family correction; these are not independent replications.
Exchange failures
Edited-intermediate outputs
Model
Total
Retain clean first token
Coordinate exchange
Sign-guarded steering
Mamba-1
84
73
1
55
Mamba-2
83
69
2
37
Mamba-3 SISO
84
59
2
74
Nemotron
75
60
1
73
Pythia
85
72
3
72
Appendix
Table 25: Two-hop output diagnostics on 90 trials per model, at the Table 6 interfaces: simultaneous residual and state edits for recurrent models, residual edits for Pythia. Exchange-failure columns give the total and the subset retaining the clean first token. Edited-intermediate columns count outputs across all 90 trials under each method. Exchange uses the unit-vector convention.
Model
Edit
n
A
I
O
S
C
D
Mamba-1
Xunit
90
6
1
11
1
61
10
Mamba-1
G
90
4
55
0
0
7
24
Mamba-1
N
90
11
12
2
0
48
17
Mamba-1
Xraw
90
6
1
12
1
59
11
Mamba-2
Xunit
90
7
2
12
2
55
12
Mamba-2
G
90
6
37
0
0
21
26
Appendix
Table 26: Two-hop output partition: all trials. A = requested answer, I = edited intermediate, O = original answer, S = source intermediate, C/D = other failure with the same/changed clean first token. Categories are exclusive and exhaustive; original-answer scoring uses the saved second step after whitespace where required. No original-answer cases are unresolved. RS for recurrent models, R for Pythia.
Model
Edit
n
A
I
O
S
C
D
Mamba-1
Xunit
20
3
1
11
0
0
5
Mamba-1
G
20
3
12
0
0
0
5
Mamba-1
N
20
6
6
2
0
0
6
Mamba-1
Xraw
20
3
1
12
0
0
4
Mamba-2
Xunit
25
5
1
12
0
0
7
Mamba-2
G
25
4
10
0
0
0
11
Appendix
Table 27: Two-hop output partition: originally correct trials. A = requested answer, I = edited intermediate, O = original answer, S = source intermediate, C/D = other failure with the same/changed clean first token. Categories are exclusive and exhaustive; original-answer scoring uses the saved second step after whitespace where required. No original-answer cases are unresolved. RS for recurrent models, R for Pythia.
Model
Site
Edit
A
I
O
S
C
D
Mamba-1
R
N
10
12
2
0
48
18
Mamba-1
R
G
5
53
0
0
9
23
Mamba-1
R
Xunit
7
1
10
1
62
9
Mamba-1
R
Xraw
7
1
10
1
64
7
Mamba-1
S
N
1
0
20
1
64
4
Mamba-1
S
G
1
0
20
1
64
4
Appendix
Table 28: Two-hop output partitions at separate residual and state interfaces (90 trials per row). Categories A/I/O/S/C/D are as in Table 26 . Together with that table, these cover every N/G/X interface; Pythia has residual only.
Figure 5: Flexible generalisation at every measured interface. Unit exchange and guarded transfer are compared on the same 192 swaps per model; the right-hand counts in each panel give X/G. Requested-answer success (left) and first-token output of the edited argument (right) are distinct endpoints. The latter is a descriptive post-hoc diagnostic: every observed argument output fails the task criterion. The 192 swaps reuse 64 prompts three times.
Figure 6: Held-out Mamba-2 residual-Jacobian geometry across 200 WikiText positions. Exact same-position Jacobians are weakly aligned early and become more similar late (left). A shared top-16 output subspace captures much more derivative energy than the corresponding input subspace, especially early (right). These quantities diagnose context dependence; they do not explain the intervention results.
Model
Alignment
Own overlap
Residual energy
State energy
Random
Mamba-1
0.887
0.500
0.381
0.752
0.0062
Mamba-2
0.882
0.504
0.554
0.659
0.0063
Mamba-3 SISO
0.894
0.593
0.400
0.619
0.0079
Appendix
Table 29: Output geometry of the fitted residual and Jfull state lenses, averaged over the protocol band. Alignment is the mean squared cosine of the 16 closest principal pairs between the leading 64-dimensional subspaces. Own overlap is the direct top-16 subspace overlap, normalised by 16. Energy columns use the shared rank-16 projector; the random column gives residual-lens energy under rank-16 random projectors.
Model
Task
Full
Shared
Complement
Own
Random
Mamba-1
Multihop
0.554
0.204
0.534
0.211
0.019
Multilingual
0.710
0.106
0.666
0.100
0.042
Order of ops.
0.801
0.389
0.807
0.722
0.106
Poetry
0.115
0.039
0.113
0.043
0.058
Association
0.337
0.116
0.274
0.073
0.038
Typo
0.712
0.150
0.853
0.079
0.066
Appendix
Table 30: Normalised joint intermediate-recovery AUC over all source blocks, using post-update residual and Jfull state readouts. Shared and own retain 16 directions; own projects each component onto its own lens’s top-16 subspace. Random averages ten rank-16 projectors. Complement removes the shared directions and retains dmodel−16 dimensions; its comparison with full tests removal, rather than equal-rank compression.
Analysis checked
Assertions
State and joint readouts
974
Held-out residual Jacobian geometry
752
Held-out state Jacobian geometry
4077
Readout protocols and intervention controls
2214
Direction and fixed-budget controls
508
Coordinate exchange and numerical cross-checks
1234
Appendix
Table 31: Numerical consistency checks on the archived results. All listed checks passed. Counts refer to assertions, not independent experiments.
Mamba's recurrent state h_t is, by construction, a compressed summary of every token seen so far. This raises a tempting hypothesis: if we extract token-level outputs y_t at fixed patch boundaries, we obtain semantic sentence summaries for free, with no pooling head, no fine-tuning, and no [CLS] token. We test this hypothesis carefully. Across five benchmarks (SST-2, CoLA, MRPC, STS-B, IMDb), we compare four strategies for extracting frozen sentence representations from a pretrained Mamba-130M backbone under a strict frozen-feature probing protocol, using three random seeds where computationally feasible. The results do not support the hypothesis: patch boundary readouts do not consistently outperform simple mean pooling. We identify and quantify two structural pathologies: severe anisotropy (mean pairwise cosine similarity 0.9999, std 0.000044) and representational collapse in the raw final SSM state (MCC = 0.000 on CoLA across all three seeds, confirmed via confusion matrix). We further propose orthogonal injection, a modified recurrence that constrains new information per
Mamba models underperform Transformers on in-context retrieval. This gap is usually attributed to Mamba's fixed-size recurrent state, which loses information as the context grows. We challenge this assumption by focusing on query-first retrieval, a setup in which the query precedes the context and state capacity should not be the binding constraint. We show that even in this setting Mamba lags substantially behind Transformers, despite its selection mechanism, which was designed to distinguish relevant content from distractors. We then demonstrate that the Mamba selection gates have sufficient control over the state updates to influence the model output: intervening on specific gates alone can swing accuracy between extremes. However, as gates are input-dependent, they require the incoming token's representation to encode its relevance to the query. We therefore probe token representations across layers, and find that the relevance signal emerges only gradually with model depth, and that the gates begin to reflect it only several layers deeper. Crucially, we find that gates in retrieval-critical layers are not yet aligned with relevance. The selection mechanism is thus expressive enough to solve the task, but it commonly operates on a relevance signal that has not yet arrived. Mamba's retrieval bottleneck therefore extends beyond state capacity: even when capacity is sufficient, the delayed emergence of relevance information across depth often prevents successful retrieval.
Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While the current Transformer-based models deliver strong model quality, their quadratic compute and linear memory make inference expensive. This has spurred the development of sub-quadratic models with reduced linear compute and constant memory requirements. However, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice. Guided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule that enables richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation for better model performance without increasing decode latency. Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with Mamba-3's MIMO variant further improving accuracy by another 1.2 points for a total 1.8 point gain. Across state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half of its predecessor's state size. Our evaluations demonstrate Mamba-3's ability to advance the performance-efficiency Pareto frontier.
Aakash Lahoti, Kevin Y. Li, Berlin Chen +5
Carnegie Mellon University · Princeton University · Together AI +1