Understanding Mamba-3 Through Oscillatory Dynamics

Shutian Liang; Yue Song

cs.LGOct 9, 2026

Understanding Mamba-3 Through Oscillatory Dynamics

Authors: Shutian Liang, Yue Song

Organizations: College of AI, Tsinghua University

Abstract

Mamba-3 introduces input-dependent rotations into recurrent state updates, but their contribution beyond state retention remains unclear. We interpret these ro- tations as underdamped second-order dynamics, providing a framework for in- vestigating the role of oscillation in sequence modeling. Controlled comparisons show that underdamped dynamics outperform non-oscillatory damping regimes on state-tracking tasks, while language-model pretraining demonstrates benefits for long-context prediction and length generalization. Interventions in pretrained models further establish the importance of learned phase, and state analysis sug- gests that rotation regulates how retained information combines with incoming up- dates. Taken together, these findings identify underdamped dynamics as a useful mechanism for recurrent computation and suggest that oscillation supports long- context processing by regulating how information accumulates.

Figures & tables

Figure 1: A geometric perspective on Mamba-3’s recurrent dynamics. (a) Mamba-2 scales the retained state, while Mamba-3 additionally rotates it. (b) With fixed coefficients and no input, decay–rotation produces a spiraling trajectory. (c) This geometry motivates our interpretation of Mamba-3 as an underdamped second-order system with oscillatory dynamics.

Regime	Eigenvalues	Dynamics
Underdamped ( $0<\zeta<1$ )	Complex conjugate	Oscillatory
Critically damped ( $\zeta=1$ )	Repeated real	Non-oscillatory
Overdamped ( $\zeta>1$ )	Distinct real	Non-oscillatory

Table 1: Damping regimes of a second-order dynamical system.

Model	Parity	Mod 3	Arith. w/o Brackets
Mamba-2	3.91	0.04	36.76
Mamba-3 w/o rotation	3.03	0.78	48.95
Mamba-3 w/ Static rotation	4.05	0.85	25.49
Overdamped	3.27	0.58	81.60
Critically Damped	3.22	0.32	1.97
Underdamped (Mamba-3)	100.0	95.26	95.68

Table 2: Scaled Accuracy (%) of state-tracking.

Figure 2: Accuracy changes after phase removal in contiguous layer windows.

Figure 3: PG-19 perplexity across context lengths for models with/without oscillation. The dotted line marks the 2K training context length.

Params	Dynamics	Synthetic		Summary		Single-doc QA		Multi-doc QA		Few-shot Learning			Coding		Avg.
Params	Dynamics	PC	PR	GR	MN	MQA	QA	2WM	HQA	SS	TR	TQA	LCC	RB	Avg.
187M	w/o oscillation	5.96	6.07	7.75	9.17	9.62	4.37	6.71	5.24	7.87	12.67	10.63	20.25	20.57	9.76
187M	w/ oscillation	5.56	6.64	8.93	9.25	10.02	4.13	6.25	4.34	9.02	8.67	12.47	18.32	20.36	9.53
444M	w/o oscillation	5.96	7.19	8.37	10.52	10.71	5.28	7.29	5.13	10.36	11.00	19.29	19.58	17.19	10.61
444M	w/ oscillation	3.57	5.95	8.50	10.23	11.38	5.06	8.79	5.95	19.61	18.33	21.56	28.58	24.31	13.22

Table 3: LongBench-E results for models with and without oscillatory dynamics. We compare the non-oscillatory variant and the default oscillatory configuration at two model scales.

Params	Dynamics	FW-Edu	LAMB.	LAMB.	HellaS.	PIQA	Arc-E	Arc-C	WinoGr.	OBQA	Average
Params	Dynamics	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc_n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$
187M	w/o oscillation	16.32	38.04	31.73	40.86	67.41	60.73	28.84	53.83	22.00	43.63
187M	w/ oscillation	16.18	32.97	33.59	41.24	67.03	61.28	25.85	51.62	23.40	43.43
444M	w/o oscillation	12.68	18.38	41.01	52.40	70.62	69.02	32.94	57.06	28.20	50.18
444M	w/ oscillation	12.60	17.19	42.83	52.65	71.65	70.45	35.15	56.83	28.00	51.08

Table 4: Standard language modeling and downstream evaluation after pretraining on 100B tokens of FineWeb-Edu. We compare Mamba-3 models with/without oscillatory dynamics at two scales.

Figure 4: Representative phase-channel trajectories in pretrained Mamba-3, showing oscillations at various timescales during language processing.

Model	Intervention	2K	16K	64K
SISO	Native	21.09	21.41	20.60
	Decay permutation	114.52 $(5.43\times)$	126.51 $(5.91\times)$	125.27 $(6.08\times)$
	Phase shuffle	50.24 $(2.38\times)$	96.05 $(4.49\times)$	149.37 $(7.25\times)$
	Phase removal	67.99 $(3.22\times)$	285.66 $(13.34\times)$	851.49 $(41.33\times)$
	Phase reversal	152.27 $\bm{(7.22\times)}$	597.91 $\bm{(27.93\times)}$	1526.52 $\bm{(74.10\times)}$
MIMO	Native	21.12	21.47	20.52

Table 5: PG-19 perplexity under phase interventions. Higher indicates more damaging.

Model	Length	Cosine Similarity		RMS Norm
Model	Length	Native	Remove	Native	Remove
SISO	16K	0.293	0.429	7.635	8.899
SISO	64K	0.294	0.474	7.621	9.437
MIMO	16K	0.241	0.488	10.073	12.022
MIMO	64K	0.242	0.499	10.078	12.047

Table 6: Hidden-state statistics under native phase and phase removal on PG-19.

Intervention	SISO	MIMO
Native	20.60	20.52
Remove $B$	186.28 $(9.04\times)$	100.41 $(4.89\times)$
Remove $C$	250.90 $(12.18\times)$	117.68 $(5.74\times)$
Remove both	42.79 $(2.08\times)$	59.46 $(2.90\times)$

Table 7: PG-19 perplexity under phase removal during the final 100 tokens.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Update rule	Dynamics
Linear Attention ( Katharopoulos et al., 2020 )	$S_{t}=S_{t-1}+k_{t}v_{t}^{\top}$	No Decay
DeltaNet ( Schlag et al., 2021 ; Yang et al., 2024 )	$S_{t}=H_{t}S_{t-1}+\beta_{t}k_{t}v_{t}^{\top}$	Decay with a preserved mode
Gated DeltaNet ( Yang et al., 2025 )	$S_{t}=\alpha_{t}H_{t}S_{t-1}+\beta_{t}k_{t}v_{t}^{\top}$	Non-oscillatory two-rate decay
KDA ( Team et al., 2025 ; Team et al., 2026 )	$S_{t}=H_{t}D_{t}S_{t-1}+\beta_{t}k_{t}v_{t}^{\top}$	Non-oscillatory modal decay
Complex KDA ( Siems et al., 2026 )	$S_{t}=H_{t}D_{t}S_{t-1}+\beta_{t}k_{t}v_{t}^{\top}$	Can admit oscillatory modes
MDN ( Huang et al., 2026 )	$\begin{aligned} M_{t}&=\mu_{t}M_{t-1}-\eta_{t}k_{t}e_{t}^{\top},\\ S_{t}&=\alpha_{t}S_{t-1}-\beta_{t}M_{t}\end{aligned}$	Can admit oscillatory modes

Appendix

Table 8: Linear attention models and their homogeneous dynamics within an interval. Delta-rule models use unit-norm keys. For DeltaNet, Gated DeltaNet, and KDA, we assume

\beta_{t}\in(0,1)

; the decay gates in Gated DeltaNet and KDA lie in

(0,1)

. Complex KDA allows

\beta_{t}\in[0,2]

and

\alpha_{t,i}\in[-1,1]

. MDN uses its constrained gate parameterization.

Notation	Meaning
$I$	Identity matrix
$S_{t}$	Recurrent state matrix
$q_{t}$	Query vector
$k_{t}$	Key vector
$v_{t}$	Value vector
$o_{t}$	Readout vector, $S_{t}^{\top}q_{t}$

Appendix

Table 9: Notation and definitions.

parameters	187M	444M
Model Architecture
Layers	12	24
$d_{\mathrm{model}}$	768	1024
$d_{\mathrm{state}}$	128	128
Expansion Factor	2	2
Head Dimension	64	64

Appendix

Table 10: Implementation details for from-scratch pretraining on FineWeb-Edu 100B tokens.

Abbr.	Task	Abbr.	Task
2WM	2WikiMQA	GR	GovReport
HQA	HotpotQA	LCC	Long Code Completion
MQA	MultiFieldQA-en	MN	MultiNews
PC	Passage Count	PR	PassageRetrieval-en
QA	Qasper	RB	RepoBench-P
SS	SAMSum	TR	TREC

Appendix

Table 11: Task-name abbreviations used for LongBench-E evaluation.

Figure 5: Task-aligned phase structure in selected two-dimensional state pairs. Input 0 induces little phase change, whereas input 1 induces a rotation close to

180^{\circ}

for parity and

120^{\circ}

for mod-3.

Figure 6: Perplexity across context lengths on CodeParrot, MATH-Hard, TriviaQA, and SlimPajama for models with and without oscillatory dynamics. Rows correspond to the 187M (top) and 444M (bottom) models. Dotted vertical lines mark the 2K training context length.

Figure 7: Validation perplexity throughout pretraining. Training trajectories of Mamba-3 187M and 444M with and without oscillatory dynamics. Insets show the final 20,000 training steps.

Figure 8: PG-19 perplexity throughout pretraining for Mamba-3 187M. PG-19 perplexity is evaluated at context lengths from 2K to 64K at different training checkpoints. The oscillatory variant achieves lower perplexity across all evaluated context lengths toward the end of training.

Figure 9: PG-19 perplexity throughout pretraining for Mamba-3 444M. PG-19 perplexity is evaluated at context lengths from 2K to 64K at different training checkpoints. The oscillatory variant achieves lower perplexity across all evaluated context lengths toward the end of training, with a more pronounced separation at longer contexts.

Sequence Modeling

Sequence modeling studies how to represent, predict, classify, and generate ordered data such as text, time series, events, and biological sequences.

See all 250 papers

Non-Commutative State Tracking with Input-Dependent Low-Rank Updates in Mamba-3
Sep 23, 2026Hiroki Fujii, Masaki YamakitaMamba State Tracking
Department of Systems and Control Engineering, Institute of Science Tokyo, Tokyo, Japan
Mamba-3: Improved Sequence Modeling using State Space Principles
Mar 16, 2026Aakash Lahoti, Kevin Y. Li, Berlin Chen +5State Space Language Models Sequence Modeling
Carnegie Mellon University · Princeton University · Together AI +1
CHASE: Cache-Hole-Adapted Skip Exit for Looped State-Space Language Models
Jul 11, 2026Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo +1Efficient Language Model Inference LLM Reasoning
Rikkyo University · The University of Tokyo

Sep 23, 2026J/K move · Enter open · S save

Non-Commutative State Tracking with Input-Dependent Low-Rank Updates in Mamba-3

Hiroki Fujii, Masaki Yamakita

Department of Systems and Control Engineering, Institute of Science Tokyo, Tokyo, Japan

Mamba State Tracking State Space Models

Open paper AskPDF arXiv

State tracking from sequential observations can require both retaining information and updating it by composing observed operations. We extend Mamba-3's diagonal transition with an input-dependent low-rank reflection term to support noncommutative state tracking, in which the order of operations matters. The rank-one update couples state coordinates along an input-dependent direction, enabling non-diagonal state transitions within a single Mamba-3 block. The extension preserves Mamba-3's exponential-trapezoidal discretization, rotary embeddings (RoPE), and readout. For training, we adapt chunkwise computation to parallelize the proposed recurrence within each chunk. Experiments cover group word problems with discrete inputs and a shell game with continuous observations, in which a policy is trained by behavioral cloning. Among the models selected for their strong performance under fixed timing, the proposed model maintains higher tracking success on longer swap sequences in the shell game with continuous observations and timing jitter. These experiments show that the proposed method achieves high accuracy on the evaluated non-commutative tracking tasks, improving on standard Mamba-3. The extension thus offers a Mamba-3-based approach to non-commutative state tracking.

Abstract

Sequence Modeling

Explore similar work

Non-Commutative State Tracking with Input-Dependent Low-Rank Updates in Mamba-3

Mamba-3: Improved Sequence Modeling using State Space Principles

CHASE: Cache-Hole-Adapted Skip Exit for Looped State-Space Language Models