Residual Streams Read, Recurrent States Remember: The Global Workspace in Mamba Models

Wenlong Wang; Fergal Reid

cs.AISep 26, 2026

Residual Streams Read, Recurrent States Remember: The Global Workspace in Mamba Models

Authors: Wenlong Wang, Fergal Reid

Organizations: Fin AI Research

Abstract

Can the global-workspace account of transformer representations extend to state-space language models? We fit Jacobian lenses to the residual streams and recurrent states of Mamba-1, Mamba-2 and Mamba-3, using the original 1000-prompt recipe. Joint residual--state readouts improve recovery of known intermediate concepts over the residual lens on at least five of six task families in every tested Mamba checkpoint. On Mamba-2, state alone exceeds residual and logit lenses on all six families; a normalised joint readout improves on both components on five. Temporal maps and word-list experiments show earlier content remaining state-readable as residual visibility changes. We also propose sign-guarded steering, which improves target top-five success over coordinate exchange on matched verbal-report trials in five models. Recurrent state alone supports this verbal access. These gains do not extend consistently to relational answers: guarded edits often output the edited concept itself, and Mamba-3's joint edits can disrupt successful state-only redirection. Recurrent state thus provides a complementary carrier of workspace content, whose recovery, persistence and causal uses require separate measurements.

Figures & tables

Model	Params.	Blocks	$d_{\rm model}$	State blocks	Role
Mamba-1	2.8B	64	2560	63	recurrent comparison
Mamba-2	2.7B	64	2560	63	principal model
Mamba-3 SISO	1.5B	24	2048	23	recurrent comparison
Mamba-3 MIMO	1.5B	24	2048	23	readout comparison
Pythia	2.8B	32	2560	–	transformer control
Nemotron-3.5	30B/3B active	52	2688	23	hybrid control

Table 1: Models and paper-relevant coverage. “State blocks” counts source blocks with fitted state lenses; the final block is the fit target and has no state lens. The MIMO model has readout results but not the full state intervention suite.

Task	Edited concept (example)	Scored output
Verbal report	Category answer: Canada to Brazil	Brazil in next-token top five
Two-hop reasoning	Implicit intermediate: Brazil to Mexico	Spanish, the new intermediate’s answer
Flexible generalisation	Explicit argument: France to Canada	Ottawa, the function’s new answer

Table 2: The edited concept and the scored output are task dependent.

Model	Best state $>$ residual	Plain joint $>$ residual	State coverage
Mamba-1 2.8B	5/6	6/6	63 source blocks
Mamba-2 2.7B	6/6	6/6	63 source blocks
Mamba-3 SISO 1.5B	5/6	5/6	23 source blocks
Mamba-3 MIMO 1.5B	4/6	5/6	23 source blocks

Table 3: Number of task families (out of six) on which a readout has higher AUC than the within-model residual J-lens. “Best state” denotes the highest AUC among the evaluated state-lens variants for each task family. These are directional per-family comparisons, not claims that every difference is statistically significant.

Task	Residual $J^{h}$	State $J_{\mathrm{full}}$	Joint normalised	Permuted-state joint
Multihop	0.475	0.557	0.679	0.431
Multilingual	0.362	0.615	0.647	0.312
Order of ops.	0.460	0.795	0.807	0.515
Poetry	0.029	0.101	0.114	0.092
Association	0.058	0.237	0.231	0.117
Typo	0.303	0.706	0.714	0.350

Table 4: Mamba-2 intermediate recovery over all 63 source blocks, measured by normalised pass@

k

AUC (not task accuracy). The normalised joint readout combines residual and state after separately normalising their transported representations. Bold marks the best non-oracle readout in each row.

Figure 1: Mamba-2 intermediate recovery over all 63 source blocks. State alone outperforms residual alone on all six families; the normalised joint readout exceeds both components on five, with association the exception. The permuted-state joint control destroys coordinate correspondence while preserving dimensionality. Values are normalised pass@

k

AUC , not task accuracy.

Figure 2: Selected Mamba-2 rank maps across all 63 source blocks, using the post-update convention. Seattle remains top-ranked at state block 28 from the end of “Needle” through the final readout, while its residual visibility varies. In the emerald question, May becomes readable late in the residual stream and remains outside the state’s top 100 throughout. Both prompts are answered correctly.

	Mamba-1	Mamba-2
Rank words held@5	7.5	6.7
Transported-state pursuit words	18.7	18.6

Table 5: Word-list recovery with the

J_{\mathrm{full}}

state lens over all 63 source blocks. Values are last-quartile means over comma positions in 80-word lists. Pursuit uses

k=25

and threshold zero on the lens-transformed state in final-residual coordinates; its occupancy and rank-based words held@5 are distinct estimators.

	Transformers		SSMs			Hybrid
Measurement	Qwen3.6 27B †	Pythia 2.8B	Mamba-1 2.8B	Mamba-2 2.7B	Mamba-3 SISO 1.5B	Nemotron 3.5
Verbal report Target top-5	85/124	242/399	85/446	114/335	207/728	130/621
Two-hop reasoning Requested answer	47/90	5/90	6/90	7/90	6/90	15/90
Flexible generalisation Requested answer	80/192	13/192	30/192	12/192	23/192	29/192
Introspection Injected token top-1	0/101	2/99	$R$ : 0/99 $S$ : 0/99	$R$ : 0/99 $S$ : 4/99	$R$ : 6/101 $S$ : 49/101	$R$ : 0/99 $S$ : 93/99
Directed modulation Focus / control / suppress hit@5	.38/.07/.07	.04/.07/.04	$R$ : .01/.05/.04 $S$ : .15/.15/.13	$R$ : .01/.05/.04 $S$ : .15/.20/.17	$R$ : .00/.00/.01 $S$ : .07/.09/.10	$R$ : .02/.02/.02 $S$ : .03/.05/.02

Table 6: Workspace properties across models. Our measurements in the first three rows use

X_{\rm unit}

exchange: simultaneous residual–state edits (

RS

) for recurrent models and residual edits (

R

) for Pythia. Verbal report scores target top-five; the two suite tasks score the requested answer. Other rows retain their separate readout or injection protocols. All Qwen3.6-27B values are quoted from Wang and Reid (2026) and were not reproduced in this study.

	Transformers		SSMs			Hybrid
Intervention	Qwen3.6 27B †	Pythia 2.8B	Mamba-1 2.8B	Mamba-2 2.7B	Mamba-3 SISO 1.5B	Nemotron 3.5
Table 6 result Coordinate exchange in this study	85/124	242/399	85/446	114/335	207/728	130/621
Sign-guarded steering	–	298/399	426/446	333/335	677/728	621/621

Table 7: Verbal report: sign-guarded steering compared directly with Table 6 . The first row repeats its verbal-report counts and model order. Both methods edit residual and state together in recurrent models and residual alone in Pythia. Success means the target enters the next-token top five.

Figure 3: Two-hop outputs under coordinate exchange (

X

) and sign-guarded steering (

G

). Panel A shows all 90 trials per model; panel B shows those answered correctly before editing. Colours distinguish the requested answer, the edited concept and other outputs. Both methods edit residual and state together in recurrent models, and residual alone in Pythia. The Mamba-3 callout compares state-only and joint steering. These are saved token predictions, not complete responses; Appendix E gives the counts, scoring rules and qualifications.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

	Any block (0–62)		Band (16–48)
Task	State	Residual	State	Residual
Multihop	0.557	0.475	0.444	0.313
Multilingual	0.615	0.362	0.601	0.276
Order of ops.	0.795	0.460	0.589	0.266
Poetry	0.101	0.029	0.078	0.014
Association	0.237	0.058	0.196	0.051

Appendix

Table 8: Mamba-2 state and residual AUC over all source blocks and over the fixed intervention band.

Task	State $J_{\mathrm{full}}$	Logit	Tuned	Permuted state
Multihop	0.557	0.379	0.441	0.021
Multilingual	0.615	0.449	0.446	0.056
Order of ops.	0.795	0.647	0.666	0.164
Poetry	0.101	0.049	0.036	0.049
Association	0.237	0.027	0.036	0.022
Typo	0.706	0.292	0.279	0.040

Appendix

Table 9: Mamba-2 normalised pass@

k

AUC over all 63 source blocks. The

J_{\mathrm{full}}

state lens exceeds both residual baselines on all six families.

Task	WikiText J-lens	Old Pile J-lens	Logit	Tuned
Multihop	0.470	0.474	0.296	0.353
Multilingual	0.324	0.333	0.214	0.217
Order of ops.	0.736	0.748	0.583	0.674
Poetry	0.063	0.078	0.018	0.031
Association	0.008	0.014	0.001	0.002
Typo	0.377	0.369	0.287	0.239

Appendix

Table 10: Pythia-2.8B recovery AUC over all 31 source layers. The primary J-lens uses the conforming 1000-prompt WikiText recipe; the old 16-prompt Pile lens is a non-conforming sensitivity comparison.

Task	Original	Row zeroed	Change
Multihop	0.475	0.443	$-0.0322$
Multilingual	0.362	0.335	$-0.0267$
Order of ops.	0.460	0.417	$-0.0436$
Poetry	0.029	0.045	$+0.0157$
Association	0.058	0.056	$-0.0013$
Typo	0.303	0.304	$+0.0008$

Appendix

Table 11: Mamba-2 residual J-lens recovery, original and with output row 2107 zeroed. Changes use unrounded AUC values.

Model	Band blocks	$J_{\mathrm{full}}$	$J_{\mathrm{read}}$	Clean
Mamba-1	33	1 [0.2, 6.0]	7 [3.8, 15.2]	20
Mamba-2	33	1 [0.2, 6.0]	–	25
Mamba-2, sparse	12	1 [0.2, 6.0]	1 [0.2, 6.0]	25
Mamba-3 SISO	13	12 [7.8, 21.9]	19 [14.0, 30.6]	27
Random / shuffled	matched	1 [0.2, 6.0] per run		–

Appendix

Table 12: Persistent state clamps: requested swap counts out of 90, with Wilson 95% intervals in brackets (percentage units). “Clean” counts correct original answers. The sparse Mamba-2 lens has 23 source blocks, 12 in the intervention band. Controls match Mamba-1/3

J_{\mathrm{read}}

and full-coverage Mamba-2

J_{\mathrm{full}}

; each of three random seeds and the shuffled-pair run gives the stated count.

Construction / outcome	Count	95% interval (%)
Residual exchange, all scorable	10/80	[6.9, 21.5]
Residual exchange, given clean correct	7/23	[15.6, 50.9]
Original pursuit, retained	22/37	[43.5, 73.7]
Row-zeroed pursuit, retained	12/37	[19.6, 48.5]
Coordinates 2–11, retained	22/37	[43.5, 73.7]
Random coordinates 2–21, retained	26/37	[54.2, 82.5]

Appendix

Table 13: Mamba-2 coordinate exchange and ten-atom pursuit ablations. Exchange success and retained correct answers are different outcomes. The two coordinate controls match the ablation comparison in Figure 4 .

Construction	State, all	State, clean correct	Both, all
$J_{\mathrm{read}}$ exchange	1/84 [0.2, 6.4]	0/29 [0.0, 11.7]	6/84 [3.3, 14.7]
$J_{\mathrm{read}}$ clamp	22/84 [18.0, 36.5]	14/29 [31.4, 65.6]	15/84 [11.1, 27.4]
$J_{\mathrm{full}}$ clamp	13/84 [9.3, 24.7]	8/29 [14.7, 45.7]	14/84 [10.2, 26.1]

Appendix

Table 14: Mamba-3 protocol interventions using raw lens vectors, at blocks 6–18 and every prompt position. Counts include Wilson 95% intervals in brackets (percentage units). The final column applies both residual and state edits; for clamp rows, both use clamp transfer.

Model	Residual only	State only	Both
Mamba-1	10/90	1/90	11/90
Mamba-2	1/90	1/90	1/90
Mamba-3	3/90	12/90	14/90

Appendix

Table 15: Historical signed clamp (

N

) by intervention location: requested swaps in the 90-item workspace suite. State edits use

J_{\mathrm{full}}

and persist through the recurrence; all edits apply at every prompt position. Mamba-1 edits residual blocks 16–48 and 12 sampled state blocks within that band; Mamba-2 edits all 33 blocks in the band for both representations. Mamba-3 is the SISO checkpoint, with both representations edited at blocks 6–18. Comparisons within each row use the same prompts and scoring.

Figure 4: Construction-specific causal evidence. Panel A compares persistent state clamps and matched direction controls in the common 90-item workspace suite; the three random seeds and shuffled pairs each yield 1/90 and share a single plotted control marker. Panel B reports Mamba-2’s residual coordinate exchange under the separate protocol. Panel C compares Mamba-2 pursuit ablations and variance-matched coordinate controls; fewer retained answers indicate a larger effect. Denominators and constructions are shown explicitly and must not be pooled into one causal score.

Condition	Target rank	Top-1 word	Top-5 success
Clean	20	Canada	No
Residual clamp	1	Brazil	Yes
State clamp	16	France	No
Simultaneous clamps	1	Brazil	Yes
Matched random residual	23	Canada	No
Matched random state	20	Canada	No

Appendix

Table 16: Historical signed-clamp (

N

) outcome on Mamba-1 trial country-05-t00 ; the random rows are its original matched controls.

Model	Evaluation prompts	Valid clean prompts	Eligible target swaps
Mamba-1 2.8B	140	110	446
Mamba-2 2.7B	140	99	335
Mamba-3 SISO 1.5B	140	124	728
Nemotron-3.5 hybrid	140	134	621
Pythia-2.8B	140	107	399

Appendix

Table 17: Corrected verbal-report denominator funnel.

Model	Site	$n$	$N$	$G$	U	Rw	$G$ ctrl	U ctrl	Rw ctrl
Mamba-1	R	446	446	417	87	90	5/4/3	2/1/0	2/1/0
Mamba-1	S	446	32	290	31	27	0/0/0	0/0/0	0/0/0
Mamba-1	RS	446	446	426	85	87	7/6/5	2/0/1	2/0/1
Mamba-2	R	335	44	291	120	121	0/0/0	0/1/1	3/2/4
Mamba-2	S	335	0	258	1	1	0/0/0	0/0/0	0/0/0
Mamba-2	RS	335	47	333	114	117	1/4/0	0/1/0	4/2/5

Appendix

Table 18: Verbal report: all operators and interfaces. Counts use the stated

n

; C is clean success under the swapped-target criterion, not original-answer competence. The final three columns list matched-norm control successes for seeds 0/1/2 separately. U and Rw denote unit and raw exchange. Mamba-3 verbal includes the replay-qualified measurements.

Model	Site	$G-X_{\rm unit}$	Interval	$G-X_{\rm raw}$	Interval
Mamba-1	R	+0.740	[+0.628, +0.844]	+0.733	[+0.620, +0.835]
Mamba-1	S	+0.581	[+0.499, +0.654]	+0.590	[+0.510, +0.662]
Mamba-1	RS	+0.765	[+0.672, +0.857]	+0.760	[+0.674, +0.847]
Mamba-2	R	+0.510	[+0.402, +0.617]	+0.507	[+0.392, +0.616]
Mamba-2	S	+0.767	[+0.704, +0.829]	+0.767	[+0.704, +0.831]
Mamba-2	RS	+0.654	[+0.547, +0.757]	+0.645	[+0.533, +0.750]

Appendix

Table 19: Verbal report: paired

G-X

success-rate differences and simultaneous intervals. Both exchange conventions retain the original 78-contrast family correction; these are not independent replications.

Model	Site	$n$	$N$	$G$	$X_{\rm unit}$	$X_{\rm raw}$
Mamba-1	R	446	436	376	34	28
Mamba-1	S	446	2	202	1	3
Mamba-1	RS	446	442	401	19	17
Mamba-2	R	335	26	258	74	77
Mamba-2	S	335	0	203	0	0
Mamba-2	RS	335	27	314	69	74

Appendix

Table 20: Descriptive top-one sensitivity on the same verbal-report trials. The primary endpoint remains top five.

Model	Site	$n$	C	$N$	$G$	U	Rw	$G$ ctrl	U ctrl	Rw ctrl
Mamba-1	R	90	1	10	5	7	7	1/1/3	1/1/1	1/1/1
Mamba-1	S	90	1	1	1	1	1	1/1/1	1/1/1	1/1/1
Mamba-1	RS	90	1	11	4	6	6	1/2/3	1/1/1	1/1/1
Mamba-2	R	90	1	1	12	7	7	1/1/1	1/2/1	0/2/1
Mamba-2	S	90	1	1	2	1	1	1/1/1	1/1/1	1/1/1
Mamba-2	RS	90	1	1	6	7	5	1/1/1	1/2/1	0/3/1

Appendix

Table 21: Two-hop reasoning: all operators and interfaces. Counts use the stated

n

Model	Site	$G-X_{\rm unit}$	Interval	$G-X_{\rm raw}$	Interval
Mamba-1	R	-0.022	[-0.089, +0.056]	-0.022	[-0.089, +0.056]
Mamba-1	S	+0.000	[+0.000, +0.000]	+0.000	[+0.000, +0.000]
Mamba-1	RS	-0.022	[-0.089, +0.044]	-0.022	[-0.089, +0.044]
Mamba-2	R	+0.056	[-0.033, +0.144]	+0.056	[-0.011, +0.133]
Mamba-2	S	+0.011	[+0.000, +0.044]	+0.011	[+0.000, +0.044]
Mamba-2	RS	-0.011	[-0.100, +0.067]	+0.011	[-0.067, +0.078]

Appendix

Table 22: Two-hop reasoning: paired

G-X

success-rate differences and simultaneous intervals. Both exchange conventions retain the original 78-contrast family correction; these are not independent replications.

Model	Site	$n$	C	$N$	$G$	U	Rw	$G$ ctrl	U ctrl	Rw ctrl
Mamba-1	R	192	4	37	18	32	33	2/2/3	6/4/3	6/4/3
Mamba-1	S	192	4	3	3	3	3	4/4/4	4/4/3	4/4/3
Mamba-1	RS	192	4	37	18	30	31	2/2/3	6/4/3	6/4/3
Mamba-2	R	192	2	2	8	12	12	0/0/0	2/1/3	2/2/3
Mamba-2	S	192	2	3	2	3	3	2/2/2	2/2/2	2/2/2
Mamba-2	RS	192	2	2	23	12	11	0/0/0	2/1/3	2/2/3

Appendix

Table 23: Flexible generalisation: all operators and interfaces. Counts use the stated

n

Model	Site	$G-X_{\rm unit}$	Interval	$G-X_{\rm raw}$	Interval
Mamba-1	R	-0.073	[-0.188, +0.047]	-0.078	[-0.193, +0.042]
Mamba-1	S	+0.000	[+0.000, +0.000]	+0.000	[+0.000, +0.000]
Mamba-1	RS	-0.062	[-0.172, +0.052]	-0.068	[-0.177, +0.052]
Mamba-2	R	-0.021	[-0.120, +0.078]	-0.021	[-0.115, +0.078]
Mamba-2	S	-0.005	[-0.026, +0.000]	-0.005	[-0.026, +0.000]
Mamba-2	RS	+0.057	[-0.036, +0.146]	+0.062	[-0.016, +0.141]

Appendix

Table 24: Flexible generalisation: paired

G-X

success-rate differences and simultaneous intervals. Both exchange conventions retain the original 78-contrast family correction; these are not independent replications.

	Exchange failures		Edited-intermediate outputs
Model	Total	Retain clean first token	Coordinate exchange	Sign-guarded steering
Mamba-1	84	73	1	55
Mamba-2	83	69	2	37
Mamba-3 SISO	84	59	2	74
Nemotron	75	60	1	73
Pythia	85	72	3	72

Appendix

Table 25: Two-hop output diagnostics on 90 trials per model, at the Table 6 interfaces: simultaneous residual and state edits for recurrent models, residual edits for Pythia. Exchange-failure columns give the total and the subset retaining the clean first token. Edited-intermediate columns count outputs across all 90 trials under each method. Exchange uses the unit-vector convention.

Model	Edit	$n$	A	I	O	S	C	D
Mamba-1	Xunit	90	6	1	11	1	61	10
Mamba-1	G	90	4	55	0	0	7	24
Mamba-1	N	90	11	12	2	0	48	17
Mamba-1	Xraw	90	6	1	12	1	59	11
Mamba-2	Xunit	90	7	2	12	2	55	12
Mamba-2	G	90	6	37	0	0	21	26

Appendix

Table 26: Two-hop output partition: all trials. A = requested answer, I = edited intermediate, O = original answer, S = source intermediate, C/D = other failure with the same/changed clean first token. Categories are exclusive and exhaustive; original-answer scoring uses the saved second step after whitespace where required. No original-answer cases are unresolved. RS for recurrent models, R for Pythia.

Model	Edit	$n$	A	I	O	D
Mamba-1	Xunit	20	3	1	11	5
Mamba-1	G	20	3	12	0	5
Mamba-1	N	20	6	6	2	6
Mamba-1	Xraw	20	3	1	12	4
Mamba-2	Xunit	25	5	1	12	7
Mamba-2	G	25	4	10	0	11

Appendix

Table 27: Two-hop output partition: originally correct trials. A = requested answer, I = edited intermediate, O = original answer, S = source intermediate, C/D = other failure with the same/changed clean first token. Categories are exclusive and exhaustive; original-answer scoring uses the saved second step after whitespace where required. No original-answer cases are unresolved. RS for recurrent models, R for Pythia.

Model	Site	Edit	A	I	O	S	C	D
Mamba-1	R	N	10	12	2	0	48	18
Mamba-1	R	G	5	53	0	0	9	23
Mamba-1	R	Xunit	7	1	10	1	62	9
Mamba-1	R	Xraw	7	1	10	1	64	7
Mamba-1	S	N	1	0	20	1	64	4
Mamba-1	S	G	1	0	20	1	64	4

Appendix

Table 28: Two-hop output partitions at separate residual and state interfaces (90 trials per row). Categories A/I/O/S/C/D are as in Table 26 . Together with that table, these cover every N/G/X interface; Pythia has residual only.

Figure 5: Flexible generalisation at every measured interface. Unit exchange and guarded transfer are compared on the same 192 swaps per model; the right-hand counts in each panel give X/G. Requested-answer success (left) and first-token output of the edited argument (right) are distinct endpoints. The latter is a descriptive post-hoc diagnostic: every observed argument output fails the task criterion. The 192 swaps reuse 64 prompts three times.

Figure 6: Held-out Mamba-2 residual-Jacobian geometry across 200 WikiText positions. Exact same-position Jacobians are weakly aligned early and become more similar late (left). A shared top-16 output subspace captures much more derivative energy than the corresponding input subspace, especially early (right). These quantities diagnose context dependence; they do not explain the intervention results.

Model	Alignment	Own overlap	Residual energy	State energy	Random
Mamba-1	0.887	0.500	0.381	0.752	0.0062
Mamba-2	0.882	0.504	0.554	0.659	0.0063
Mamba-3 SISO	0.894	0.593	0.400	0.619	0.0079

Appendix

Table 29: Output geometry of the fitted residual and

J_{\mathrm{full}}

state lenses, averaged over the protocol band. Alignment is the mean squared cosine of the 16 closest principal pairs between the leading 64-dimensional subspaces. Own overlap is the direct top-16 subspace overlap, normalised by 16. Energy columns use the shared rank-16 projector; the random column gives residual-lens energy under rank-16 random projectors.

Model	Task	Full	Shared	Complement	Own	Random
Mamba-1	Multihop	0.554	0.204	0.534	0.211	0.019
	Multilingual	0.710	0.106	0.666	0.100	0.042
	Order of ops.	0.801	0.389	0.807	0.722	0.106
	Poetry	0.115	0.039	0.113	0.043	0.058
	Association	0.337	0.116	0.274	0.073	0.038
	Typo	0.712	0.150	0.853	0.079	0.066

Appendix

Table 30: Normalised joint intermediate-recovery AUC over all source blocks, using post-update residual and

J_{\mathrm{full}}

state readouts. Shared and own retain 16 directions; own projects each component onto its own lens’s top-16 subspace. Random averages ten rank-16 projectors. Complement removes the shared directions and retains

d_{\rm model}-16

dimensions; its comparison with full tests removal, rather than equal-rank compression.

Analysis checked	Assertions
State and joint readouts	974
Held-out residual Jacobian geometry	752
Held-out state Jacobian geometry	4077
Readout protocols and intervention controls	2214
Direction and fixed-budget controls	508
Coordinate exchange and numerical cross-checks	1234

Appendix

Table 31: Numerical consistency checks on the archived results. All listed checks passed. Counts refer to assertions, not independent experiments.

State Space Language Models

See all 19 papers

Lost in State Space: Probing Frozen Mamba Representations
Apr 30, 2026Bhagyashree Wagh, Akash SinghRepresentation Geometry Representation Collapse
University of Washington · University of Florida
Beyond State Capacity: Why Retrieval Breaks in Mamba
Oct 14, 2026Nir Yarden, Amit Ben-Artzy, Roy SchwartzIn-Context Retrieval Selective SSMs
The Hebrew University of Jerusalem
Mamba-3: Improved Sequence Modeling using State Space Principles
Mar 16, 2026Aakash Lahoti, Kevin Y. Li, Berlin Chen +5State Space Language Models Sequence Modeling
Carnegie Mellon University · Princeton University · Together AI +1

Apr 30, 2026J/K move · Enter open · S save

Lost in State Space: Probing Frozen Mamba Representations

Bhagyashree Wagh, Akash Singh

University of Washington · University of Florida

Representation Geometry Representation Collapse Sentence Embeddings State Representation Learning State Space Models Semantic Representation Learning

Open paper AskPDF arXiv

Mamba's recurrent state h_t is, by construction, a compressed summary of every token seen so far. This raises a tempting hypothesis: if we extract token-level outputs y_t at fixed patch boundaries, we obtain semantic sentence summaries for free, with no pooling head, no fine-tuning, and no [CLS] token. We test this hypothesis carefully. Across five benchmarks (SST-2, CoLA, MRPC, STS-B, IMDb), we compare four strategies for extracting frozen sentence representations from a pretrained Mamba-130M backbone under a strict frozen-feature probing protocol, using three random seeds where computationally feasible. The results do not support the hypothesis: patch boundary readouts do not consistently outperform simple mean pooling. We identify and quantify two structural pathologies: severe anisotropy (mean pairwise cosine similarity 0.9999, std 0.000044) and representational collapse in the raw final SSM state (MCC = 0.000 on CoLA across all three seeds, confirmed via confusion matrix). We further propose orthogonal injection, a modified recurrence that constrains new information per

Abstract

State Space Language Models

Explore similar work

Lost in State Space: Probing Frozen Mamba Representations

Beyond State Capacity: Why Retrieval Breaks in Mamba

Mamba-3: Improved Sequence Modeling using State Space Principles