Diagonal Equivalence in State Space Models: Characterization, Retrieval Rank, and Selective Extensions
Organizations: Institute of Mathematics, Technische Universität Berlin
Abstract
We classify real and complex state-space models (SSMs) with linear transitions, distinguishing fixed-basis diagonal equivalence from input-output equivalence and tracking coefficient-conditioning windows under coordinate changes. We characterize diagonal equivalence through commuting transition families whose members are diagonalizable over the chosen field. With time-shared token-local matrix maps, we identify this class's state-mediated kernels with unnormalized scalar-mode linear attention with multiplicative masks. Mamba 3's multiple-input, multiple-output (MIMO) construction lifts one value per local interface; SSD-MIMO routes independent values for local retrieval rank above one. HybridMamba combines Mamba 1's per-coordinate decay, Mamba 2-style heads and Mamba 3's complex rotations and two-endpoint writes, mixing real and rotating state blocks with different decay laws. They recover Mamba 1-3 single-input, single-output (SISO) configurations as exact special cases. On five large H100 bfloat16 workloads, our fused GPU kernels improve geometric-mean training time over official Mamba 1/3 in matched SISO recoveries, with Mamba 2 near parity at SSM-core scope. Memory tradeoffs depend on the baseline and metric: saved-for-backward versus peak allocation. In small-model multi-token recall experiments, both designs outperform Mamba 1-3 on the hardest tier; higher routing ranks perform best in gated or phase-diverse configurations. In single-seed language models with about 500M parameters, both improve on Mamba 1/2 in test loss at 2B tokens, but the routing gains do not translate into lower language-model loss. Our best configurations approach Mamba 3 SISO through 10B tokens while remaining behind it and Mamba 3 MIMO in language-model loss.
Figures & tables
| Family/law | Formula | Resolution/default |
|---|---|---|
| SSD dynamic | Default; head/token, optionally sibling; zero-initialized bias | |
| SSD static | Per ; spread rate ladder | |
| HM static | Default; , tied within complex pairs; no rate bias | |
| HM dynamic | Per or head; no bias/floor | |
| HM potential | Bounded selected potential | |
| HM flat dynamic | Head/token; zero-initialized bias |
| Target | Design | Matching settings |
|---|---|---|
| Mamba 1 | HybridMamba | Real static per-coordinate decay; exponential Euler; state-branch coefficients; matched convolution and projections. |
| Mamba 2 | SSD-MIMO | Scalar decay, scalar local interfaces; exponential Euler; joint drive/write/read convolution and output normalization. |
| Mamba 3 SISO | SSD-MIMO | Scalar local interfaces; matched rate law and rotation, two-endpoint write, write/read normalization and biases; convolution-free mixer. |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Target | Parameter correspondence |
|---|---|
| Mamba 1 | , step rank ; no B/C or output norm. Match input/output, convolution, B/C/step maps, and head . |
| Mamba 2 | , one B/C group, no B/C norm or biases. Split packed projections and convolution; match step bias, , head and output normalization; no static rate bias. |
| Mamba 3 SISO | , no output norm; half of the state rows rotate (in pairs) with , . , ; match the rate activation (heavy-tail by default), and the step that scales the write, B/C gains and biases, step bias, head , output gate and angle/endpoint maps. |
| Shape | elements | MiB (bf16) | |||
|---|---|---|---|---|---|
| Balanced | 16 | 4096 | 2048 | 256 | |
| Long | 2 | 2048 | 32768 | 256 | |
| Wide | 4 | 8192 | 4096 | 256 | |
| Batch | 128 | 1024 | 1024 | 256 | |
| LM | 6 | 2048 | 2048 | 48 | |
| LM | 32 | 2048 | 2048 | 256 |
| Model | Kernel time | Mixer time | Kernel peak | Mixer peak |
|---|---|---|---|---|
| HM-0, P=64 | 0.960 | 0.958 | 0.516 | 0.648 |
| HM-75, P=64 | 0.981 | 0.993 | 0.486 | 0.554 |
| HM-100, P=64 | 0.992 | 1.038 | 0.503 | 0.567 |
| HM-0, P=32 | 0.963 | 0.962 | 0.525 | 0.652 |
| HM-75, P=32 | 1.010 | 1.039 | 0.762 | 0.793 |
| HM-100, P=32 | 1.025 | 1.078 | 0.830 | 0.850 |
| Tier | Length | Key length | Value length | Interleave | Batch | Epochs | Updates |
|---|---|---|---|---|---|---|---|
| Easy | 1024 | 1 | 1 | 0 | 64 | 32 | 50,016 |
| Medium | 1024 | 2–4 | 2–6 | 0.4 | 64 | 32 | 50,016 |
| Hard | 2048 | 5–12 | 5–12 | 0.8 | 32 | 16 | 50,000 |
| Configuration | Row partition and law | Gates, rotation and write | |
|---|---|---|---|
| M1 | 128 real, coordinate-static | No map gates/rotation; Euler; M1 endpoints. | |
| M2 | 128 real, head-static | No map gates/rotation; Euler; M2 endpoints. | |
| M3 | 64 complex + 64 real, head-dynamic | No map gates; fixed frequency , no read offset; two endpoints; M3 maps. | |
| Attention | — | No recurrent state | One attention head; learned absolute positions. |
| R1O1-r | 128 real, head-dynamic | Ungated; no rotation; two endpoints; SSD maps. | |
| R1O1-c | 64 complex + 64 real, head-dynamic | Ungated; default learned rotation; two endpoints; SSD maps. |
| Configuration | Easy | Medium | Hard | ||
|---|---|---|---|---|---|
| Peak LR | |||||
| M1 2 2 2 M1 at : (diverged seed counted as ) / (completed seeds, ). The rate is ineligible. | 99.9 99.9, 100.0, 100.0 | 89.3 88.8, 89.7, 89.4 | 67.1 69.2, 61.5, 70.6 | 69.6 72.8, 68.8, 67.3 | 73.3 † 74.1, 72.6, D |
| M2 | 100.0 100.0, 100.0, 100.0 | 91.7 88.7, 89.1, 97.3 | 42.0 66.0, 30.3, 29.6 | 36.4 52.0, 29.5, 27.8 | 39.9 46.9, 43.5, 29.4 |
| M3 | 99.9 99.6, 100.0, 100.0 | 76.6 84.7, 58.1, 87.1 | 27.8 48.2, 14.3, 21.0 | 12.1 2.8, 2.2, 31.2 | 1.1 2.9, 0.1, 0.2 |
| Attention | 100.0 100.0, 100.0, 100.0 | 22.2 22.2, 22.2, 22.2 | 0.5 0.4, 0.3, 0.7 | 1.1 1.9, 0.9, 0.4 | 0.0 0.0, 0.0, 0.0 |
| R1O1-r | 100.0 100.0, 100.0, 100.0 | 89.6 89.7, 89.7, 89.4 | 67.0 63.9, 71.7, 65.4 | 62.3 71.6, 67.4, 47.9 | 35.4 15.0, 41.1, 50.1 |
State Space Models
State space models represent sequences through evolving latent states, enabling efficient temporal modeling, system identification, and forecasting.
See all 321 papers