We classify real and complex state-space models (SSMs) with linear transitions, distinguishing fixed-basis diagonal equivalence from input-output equivalence and tracking coefficient-conditioning windows under coordinate changes. We characterize diagonal equivalence through commuting transition families whose members are diagonalizable over the chosen field. With time-shared token-local matrix maps, we identify this class's state-mediated kernels with unnormalized scalar-mode linear attention with multiplicative masks. Mamba 3's multiple-input, multiple-output (MIMO) construction lifts one value per local interface; SSD-MIMO routes independent values for local retrieval rank above one. HybridMamba combines Mamba 1's per-coordinate decay, Mamba 2-style heads and Mamba 3's complex rotations and two-endpoint writes, mixing real and rotating state blocks with different decay laws. They recover Mamba 1-3 single-input, single-output (SISO) configurations as exact special cases. On five large H100 bfloat16 workloads, our fused GPU kernels improve geometric-mean training time over official Mamba 1/3 in matched SISO recoveries, with Mamba 2 near parity at SSM-core scope. Memory tradeoffs depend on the baseline and metric: saved-for-backward versus peak allocation. In small-model multi-token recall experiments, both designs outperform Mamba 1-3 on the hardest tier; higher routing ranks perform best in gated or phase-diverse configurations. In single-seed language models with about 500M parameters, both improve on Mamba 1/2 in test loss at 2B tokens, but the routing gains do not translate into lower language-model loss. Our best configurations approach Mamba 3 SISO through 10B tokens while remaining behind it and Mamba 3 MIMO in language-model loss.
Figures & tables
Figure 1: Local value routing, illustrated at equal total state width 4N . Left: four scalar interfaces, with Btr∈RN×1 and Ctr∈R1×N . Right: one SSD-MIMO group at R=O=4 supplies the same vector to four sibling states, with Btr∈RN×4 and Ctr∈R4×N , then combines their vector readouts as yt=∑rζr,tCtrhtr , with scalar sibling weights ζr,t . Both yield four output coordinates; parameter sharing and computation need not match; coefficient generation and feedthrough are omitted.
Figure 2: MQMKVAR recall combines MQAR ( Arora et al., 2023a ) with MAD’s fuzzy recall ( Poli et al., 2024 ) . Dashed outlines mark k11k8 , a proper prefix of k11k8k3 ; dotted outlines mark the longer key’s suffix k8k3 . Queries match keys from their first token. By convention, proper-prefix queries start their targets at the first padding token ∅ , which confirms key completion; other answers start at their last key token. Targets align with the input positions predicting them; “–” and all context positions carry no loss. Earlier answers never enter the query input. “ ⋯ ” elides context entries, queries and variable-length, loss-masked padding gaps between queries. Results: Section 6.1 ; protocol: Appendix E.1 .
Figure 3: Transition classes at common field F , state dimension S≥2 (for S=1 they coincide) and horizon L . Whole ellipses use window bound ℓ^ ; the shaded portions impose 1≤ℓ<ℓ^≤L on every class, so DF(ℓ,S)=DF(ℓ^,S)∩SF(ℓ,S) , and likewise for commuting families. Diagonal equivalence requires, and is guaranteed by, pairwise commutativity and individual diagonalizability over F of the entire realized transition family (Theorem 8 ).
Figure 4: Shared HybridMamba/SSD-MIMO mixer. Conv1d is one-dimensional convolution; RMSNorm is root-mean-square normalization. Dashed boundaries define the core input xt separately in each variant: (a) before the value projection, xt=zt , with coefficients from st=xt (default, Mamba 2/3 lineage); (b) after it, xt=vt , with coefficients from st=ut (Mamba 1 lineage). Boxes show step Δt , decay At (per coordinate in Mamba 1/HM, or head-shared), angle θt (with rotation), write Bt and read Ct ; gates, trapezoid weight and potential are omitted. The read adds Dtut . B/C normalization is on by default; output root-mean-square (RMS) normalization is off by default and on in the Mamba 2 preset.
Default; (η,p,n) , tied within complex pairs; no rate bias
HM dynamic
at=−act(Wast)
Per n or head; no bias/floor
HM potential
logα=Δa+cnj(μt−μt−1)
Bounded selected potential
HM flat dynamic
at=−max(act(Wast+ba),10−4)
Head/token; zero-initialized bias
Table 2: Decay laws. Rates a are negative, so the applied decay α=exp(Δa) lies in (0,1) (potential law: see text). Heavy-tail selection is the default for the SSD dynamic rate and the HM dynamic and per-head dynamic rates.
Target
Design
Matching settings
Mamba 1
HybridMamba
Real static per-coordinate decay; exponential Euler; state-branch coefficients; matched convolution and projections.
Mamba 2
SSD-MIMO
Scalar decay, scalar local interfaces; exponential Euler; joint drive/write/read convolution and output normalization.
Mamba 3 SISO
SSD-MIMO
Scalar local interfaces; matched rate law and rotation, two-endpoint write, write/read normalization and biases; convolution-free mixer.
Table 5: Mamba SISO configurations recovered in exact arithmetic after matching coefficients, projections, normalization and initial state.
Figure 5: Hard-tier held-out token accuracy at each swept peak LR, rates increasing from left to right within each configuration. Horizontal lines mark seed means; the selected eligible rate has a thick line and filled seed markers. Mamba 1’s 2/3 marks one diverged seed at 2×10−3 . The mechanism group shows rank-one SSD and real R2O2 with gates off. HM denotes A +dynamic; r/c denote real/complex.
Figure 6: Validation-loss curves, one seed per arm, without smoothing: (a) the eleven main-table configurations at constant LR up to 2 B tokens, with a 1 – 2 B zoom; (b) the five configurations continued from 2 B to 10 B with cosine decay. Fixed-window validation losses, distinct from the test-shard losses of Table 7 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Target
Parameter correspondence
Mamba 1
P=1 , step rank ⌈dmodel/16⌉ ; no B/C or output norm. Match input/output, convolution, B/C/step maps, Alog and head D .
Mamba 2
R=O=1 , one B/C group, no B/C norm or biases. Split packed projections and convolution; match step bias, Alog , head D and output normalization; no static rate bias.
Mamba 3 SISO
R=O=1 , no output norm; half of the state rows rotate (in pairs) with ω=π , ν=0 . V↦u , K,Q↦B,C ; match the rate activation (heavy-tail by default), logαt=Δtat and the step Δt that scales the write, B/C gains and biases, step bias, head D , output gate and angle/endpoint maps.
Appendix
Table 8: Parameter correspondences for the SISO presets in Table 5 . Mamba 3 MIMO has no exact preset (Corollary 22 ).
Shape
B
dinner
L
elements
MiB (bf16)
Balanced
16
4096
2048
128⋅220
256
Long
2
2048
32768
128⋅220
256
Wide
4
8192
4096
128⋅220
256
Batch
128
1024
1024
128⋅220
256
LM
6
2048
2048
24⋅220
48
LM
32
2048
2048
128⋅220
256
Appendix
Table 11: Benchmark shapes: batch B , drive width dinner , length L , elements BLdinner of one drive-sized tensor and its size in bf16. The four large shapes share 227 elements, 256 MiB per tensor. Table 4 uses the large shapes and the first language-model shape (LM); Table 12 uses all ten, with earlier-schedule HM measurements identified in its caption.
Model
Kernel time
Mixer time
Kernel peak
Mixer peak
HM-0, P=64
0.960
0.958
0.516
0.648
HM-75, P=64
0.981
0.993
0.486
0.554
HM-100, P=64
0.992
1.038
0.503
0.567
HM-0, P=32
0.963
0.962
0.525
0.652
HM-75, P=32
1.010
1.039
0.762
0.793
HM-100, P=32
1.025
1.078
0.830
0.850
Appendix
Table 20: HM backward-buffer ablation: geometric-mean optimized/unreduced-buffer ratios over Table 4 ’s five shapes ( <1 : lower time or peak). Times include forward and backward; Mixer is the eager full module. Both schedules use identical models and retained-for-backward storage. At P=32 , HM-75/100 mixer time rises 3.9% / 7.8% on geometric mean, up to 18% on one shape; m1exact is essentially unchanged. H100, bf16, N=128 .
Tier
Length
Key length
Value length
Interleave
Batch
Epochs
Updates
Easy
1024
1
1
0
64
32
50,016
Medium
1024
2–4
2–6
0.4
64
32
50,016
Hard
2048
5–12
5–12
0.8
32
16
50,000
Appendix
Table 25: Base training recipe by tier (peak LR 10−3 ); the hard sweep changes its peak LR. Each run uses one of seeds 101,202,303 . The metric is final held-out token accuracy: correct supervised value tokens divided by all supervised value tokens.
Configuration
R/O
Row partition and law
Gates, rotation and write
M1
1/1
128 real, coordinate-static
No map gates/rotation; Euler; M1 endpoints.
M2
1/1
128 real, head-static
No map gates/rotation; Euler; M2 endpoints.
M3
1/1
64 complex + 64 real, head-dynamic
No map gates; fixed frequency π , no read offset; two endpoints; M3 maps.
Attention
—
No recurrent state
One attention head; learned absolute positions.
R1O1-r
1/1
128 real, head-dynamic
Ungated; no rotation; two endpoints; SSD maps.
R1O1-c
1/1
64 complex + 64 real, head-dynamic
Ungated; default learned rotation; two endpoints; SSD maps.
Appendix
Table 26: Exact MQ row configurations. Counts denote real coordinates, so 64 complex rows mean 32 complex coordinates. All SSM rows use N=128 ; the text defines the endpoint recipes and rotation.
Configuration
Easy
Medium
Hard
Peak LR
10−3
10−3
5×10−4
10−3
2×10−3
M1 2 2 2 M1 at 2×10−3 : 48.9 (diverged seed counted as 0 ) / 73.3 (completed seeds, 2/3 ). The rate is ineligible.
99.9 99.9, 100.0, 100.0
89.3 88.8, 89.7, 89.4
67.1 69.2, 61.5, 70.6
69.6 72.8, 68.8, 67.3
73.3 † 74.1, 72.6, D
M2
100.0 100.0, 100.0, 100.0
91.7 88.7, 89.1, 97.3
42.0 66.0, 30.3, 29.6
36.4 52.0, 29.5, 27.8
39.9 46.9, 43.5, 29.4
M3
99.9 99.6, 100.0, 100.0
76.6 84.7, 58.1, 87.1
27.8 48.2, 14.3, 21.0
12.1 2.8, 2.2, 31.2
1.1 2.9, 0.1, 0.2
Attention
100.0 100.0, 100.0, 100.0
22.2 22.2, 22.2, 22.2
0.5 0.4, 0.3, 0.7
1.1 1.9, 0.9, 0.4
0.0 0.0, 0.0, 0.0
R1O1-r
100.0 100.0, 100.0, 100.0
89.6 89.7, 89.7, 89.4
67.0 63.9, 71.7, 65.4
62.3 71.6, 67.4, 47.9
35.4 15.0, 41.1, 50.1
Appendix
Table 27: Complete MQMKVAR results: held-out token accuracy (percent). Each cell shows the completed-seed mean above seeds 101, 202 and 303; all cells complete 3/3 attempts except † (2/3). Underlining marks the best eligible hard-tier mean over the three common rates; ties prefer 10−3 , then the smaller rate. D denotes the non-escaping, diverged M1 seed described in Appendix E.2 .
Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While the current Transformer-based models deliver strong model quality, their quadratic compute and linear memory make inference expensive. This has spurred the development of sub-quadratic models with reduced linear compute and constant memory requirements. However, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice. Guided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule that enables richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation for better model performance without increasing decode latency. Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with Mamba-3's MIMO variant further improving accuracy by another 1.2 points for a total 1.8 point gain. Across state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half of its predecessor's state size. Our evaluations demonstrate Mamba-3's ability to advance the performance-efficiency Pareto frontier.
Aakash Lahoti, Kevin Y. Li, Berlin Chen +5
Carnegie Mellon University · Princeton University · Together AI +1
State Space Models (SSMs) such as Mamba-2 offer linear-time inference but their memory footprint limits edge deployment. Prior ternary SSM work (Slender-Mamba) trains from scratch on 150B tokens; we show a pretrained checkpoint suffices, reducing the marginal token budget by 1,000x. Using grouped quantization-aware training (QAT) with knowledge distillation from a frozen FP16 teacher, we compress Mamba-2 1.3B to 3.61x (2,687 to 744 MB) and achieve 48.1% zero-shot accuracy (7-task average) in just 102M tokens (4 GPU-hours, single H100) -- approaching Bi-Mamba's 48.4% (within +/-0.9pp CI). This QAT-from-pretrained setting reveals zero-ratio collapse, a novel instability caused by learnable quantization scales that does not arise in from-scratch training. We further show that post-hoc correction strategies effective for Transformers fail for SSMs due to error accumulation through the recurrence. These results demonstrate that ternary SSMs do not require expensive from-scratch training: QAT from pretrained checkpoints with KD is a data-efficient alternative.
Ramprasath Ganesaraja, Sahil Dilip Panse, Swathika N
Mixture-of-experts (MoE) is a common approach for increasing parameter capacity, but applying MoE to state space model (SSM) token mixers can multiply the cost of the recurrent state update. We study how to introduce expert specialization into selective SSMs while preserving computational efficiency. We show that MoE--SSM can refer to two designs: (1) MoE over separated SSMs, which maintains multiple state trajectories and thus scales compute with the number of experts; and (2) MoE-parameterized SSM, which mixes experts in parameter space, maintains a single state trajectory, and evaluates the recurrence once. Our method, Switch Mamba (Swimba), follows the second design by routing over expert-produced SSM streams. Theoretically, we establish well-definedness and stability for MoE-parameterized SSMs and characterize the relationship between the two designs. Empirically, we evaluate Swimba on standard benchmark tasks and measure real-time throughput and latency. Under matched FLOPs, Swimba achieves slightly better average performance than the baseline, with a small slowdown in real-time latency and throughput. Overall, these results suggest that parameter-space MoE can increase SSM capacity while keeping the dominant recurrence cost fixed.
Zhixu Du, Krishna Teja Chitty-Venkata, Murali Emani +3
Duke University · Red Hat, Inc. · Argonne National Laboratory