MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models

Syed Ibrahim Omer; Ginny Y. Wong. Xiangyu Zhao

cs.LGOct 6, 2026

MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models

Authors: Syed Ibrahim Omer, Ginny Y. Wong. Xiangyu Zhao

Organizations: Department of Data Science City University of Hong Kong · NVIDIA AI Technology Center NVIDIA, Santa Clara, USA

Abstract

State Space Models (SSMs) offer an efficient alternative to Transformers for sequence modeling, yet conditioning pre-trained SSMs for iterative generation typically operates outside the recurrent operator, through input injection or activation modulation. While such mechanisms expose the model to conditioning information, they leave the underlying temporal dynamics fixed. We introduce MaRK (Markov-adapted Recurrent Kernels), a dynamic operator-conditioning framework that maps context vectors directly into bounded modulations of a frozen SSM's recurrence ( $A$ ), read-in ( $B$ ), read-out ( $C$ ), skip ( $D$ ), and discretization ( $Δ$ ) parameters. Viewed through the lens of LPV-SSM systems, MaRK induces a context-indexed family of Markov parameter sequences, allowing each diffusion timestep to reshape the model's input-output memory kernel. We instantiate MaRK on a frozen 111M-parameter Hydra SSM backbone and study three adapter geometries: Hypernet, Chebyshev polynomial, and Discrete Cosine Transform kernels. Since these adapters modify the Markov parameter sequence through low-rank auxiliary maps on the frozen backbone, parameter-efficient fine-tuning arises as a structural consequence of the adaptation mechanism itself, requiring only 6.3--11M trainable auxiliary parameters to transition from a bidirectional objective to an iterative diffusion regime. The bounded recurrence parameterization further yields an analytic Affine Quadratic Stability certificate for the modulated recurrence. Through synthetic LPV recovery experiments and Markov-operator diagnostics, we show that MaRK recovers coordinate-invariant temporal operators under matched assumptions and produces distinct, stable timestep-conditioned memory profiles. Empirically, the Chebyshev variant yields the strongest performance, achieving an average validation loss of 2.55, followed by the DCT (2.59) and Hypernet (3.77) geometries.

Figures & tables

Figure 1 : Stability margins ( $\varepsilon_{\ell}$ ) across all the layers of the pre-trained Hydra SSM at the extreme bounds of modulation by MaRK.

Panel of Figure 2 — Figure 2 : Markov Operator Error vs Dataset Size for synthetic Hypernet, Chebyshev, and DCT LPV recovery. Solid lines denote 15-seed means with normal-approximation 95% confidence intervals (shaded). Red dashed lines show an $O(N^{-1/2})$ reference normalized at the smallest dataset size.

Figure 3 : Head-resolved mean Markov parameter norm versus lag for Hypernet, Chebyshev, and DCT MaRK adapters at different diffusion timesteps $t$ . The dashed curve denotes the frozen Hydra backbone without timestep-conditioned operator modulation.

Kernel	Params	PTB	WikiText	Lambada	AG News	Arxiv	Avg.
Hypernet	6.3M	3.39 ± 0.31	3.73 ± 0.23	3.70 ± 0.17	3.91 ± 0.20	4.12 ± 0.10	3.77
Chebyshev	11.0M	2.42 ± 0.26	2.55 ± 0.16	2.40 ± 0.10	2.85 ± 0.14	2.55 ± 0.05	2.55
DCT	7.1M	2.48 ± 0.27	2.55 ± 0.14	2.47 ± 0.11	2.86 ± 0.14	2.61 ± 0.07	2.59

Table 1 : CART losses (

\mathcal{L}_{\text{CART}}\downarrow

) across 10 seeds (mean

\pm

95% CI,

1.96\times

SEM) and trainable adapter sizes across completed validation datasets. Bold = Best. Underline = Second Best.

Kernel	Full	All Except A	A Only	$\Delta$ Only	B,C Only	D Only	All Except $\Delta$	None
Hypernet	3.73 ± 0.26	3.91 ± 0.27	4.07 ± 0.29	4.04 ± 0.29	4.08 ± 0.29	3.78 ± 0.26	3.74 ± 0.26	3.99 ± 0.29
Chebyshev	2.55 ± 0.18	2.78 ± 0.19	3.44 ± 0.22	3.92 ± 0.26	3.06 ± 0.21	3.90 ± 0.26	2.56 ± 0.18	4.00 ± 0.27
DCT	2.55 ± 0.16	2.80 ± 0.16	3.24 ± 0.17	3.99 ± 0.24	3.05 ± 0.17	3.90 ± 0.24	2.59 ± 0.16	4.00 ± 0.24

Table 2 : Parameter ablation: CART validation loss (

\mathcal{L}_{\text{CART}}\downarrow

) on the WikiText dataset across 10 seeds (mean

\pm

95% CI,

1.96\times

SEM). "Full" modulates all five parameters; "None" modulates none (frozen Hydra). Bold = Best per kernel.

Variant	Latency (ms) seq=4096	Memory (MiB) seq=4096	Total Params (M)	Additional Params (M)	Overhead seq=512	Overhead seq=4096
Frozen Hydra	267.43 ± 0.06	4446.58 $\pm$ 0.06	111.68	0.00	-	-
Input Injection	270.86 ± 0.03	4455.25 $\pm$ 0.06	113.96	2.28	+3.6%	+1.3%
AdaLN Zero	276.84 ± 0.02	4474.52 $\pm$ 0.60	118.51	6.84	+7.6%	+3.5%
Hypernet	304.75 ± 0.03	4470.62 $\pm$ 0.06	117.95	6.27	+65.6%	+14.0%
Chebyshev	309.74 ± 0.10	4488.61 $\pm$ 0.06	122.65	10.97	+104.0%	+15.8%
DCT	307.28 ± 0.05	4473.93 $\pm$ 0.06	118.75	7.07	+88.7%	+14.9%

Table 3 : Inference performance and relative latency overhead over Frozen Hydra at batch size 8 on an NVIDIA A100-SXM4-80GB with the same frozen 23-layer backbone. Each variant: 15 seeds, 100 forward passes per seed, means

\pm

95% CI (t-distribution, 14df). Bold = MaRK (Ours)

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$t,\tau$	diffusion timestep and internal continuous time
$\mathbf{c}_{t}$	context/timestep embedding used as the LPV scheduling variable
$Q,Q^{\prime}$	base and modulated SSM parameters, $Q\in\{A,B,C,D,\Delta\}$
$A(\mathbf{c}_{t}),B(\mathbf{c}_{t})$	LPV-SSM matrix-valued functions for recurrence and read-in parameters
$C(\mathbf{c}_{t}),D(\mathbf{c}_{t})$	LPV-SSM matrix-valued functions for read-out and skip parameters
$x(\tau),u(\tau),y(\tau)$	LPV-SSM vector-valued functions for state, input and output

Appendix

Table 4 : Compact symbol guide for the main text.

Source	Ratio (%)
common-pile StackExchange	10.00
common-pile LibreTexts	1.67
common-pile YouTube	22.50
common-pile PubMed	7.50
common-pile CCCC	10.00
common-pile Project Gutenberg	5.00

Appendix

Table 5: Source mixture used to construct the Hydra train/validation corpus. Percentages are the relative shard-pool weights of the reported training runs and normalize at sampling.

Setting	Value
Trainable parameters	MaRK adapters and timestep embeddings only
Backbone	Frozen 23-layer Hydra-BERT SSM
Sequence length	4096 tokens
Epochs	3
Optimizer	Fused AdamW, $\beta=(0.9,0.999)$ , weight decay $0.01$
MaRK learning rate	$6.0\times 10^{-4}$

Appendix

Table 6: Training settings shared by all three MaRK kernels. Values are taken from the training configurations and the corresponding optimization implementation.

Kernel	Batch size	Warmup / total steps	Kernel-specific parameters
Hypernet	112	720 / 14,400	–
Chebyshev	112	720 / 14,400	Polynomial degree $=5$
DCT	112	720 / 14,400	Timepoints $=256$ , frequencies $=8$

Appendix

Table 7: Kernel-specific training settings. Shared kernel dimensions are rank

r=2

, adapter MLP dimension 256, and timestep embedding dimension 128.

Parameter	Value
$A$ -shift bound $\beta_{A}$	$\leq 0.5$
Scale and shift initial magnitude $\alpha,\beta$	$0.2$ (learned)
CART $p$ / scale	0.45 / 1.0

Appendix

Table 8: Modulation bounds and CART settings, identical across the three kernels.

Explore similar work

MuonSSM: Orthogonalizing State Space Models for Sequence Modeling
Jun 29, 2026Thai-Khanh Nguyen, Ngoc-Bich-Uyen Vo, Thieu N. Vo +2State Space Models Sequence Modeling
Faculty of Information Technology, Dainam University, Hanoi University of Science and Technology, Hanoi 10000, Vietnam · Faculty of Artificial Intelligence, Posts and Telecommunications Institute of Technology, Hanoi 10000, Vietnam · Department of Computer Science, University of Bath, Bath BA2 7AY, UK +1
Lag Operator SSMs: A Geometric Framework for Structured State Space Modeling
Dec 22, 2025Sutashu Tomonaga, Kenji Doya, Noboru MurataState Space Models State Space
Okinawa Institute of Science and Technology · Waseda University
SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning
Jun 24, 2026Omanshu ThapliyalEfficient Long-Context Inference Model Fine-Tuning
Hitachi America Ltd., Santa Clara, CA, USA.

Jun 29, 2026, cs.LGJ/K move · Enter open · S save

MuonSSM: Orthogonalizing State Space Models for Sequence Modeling

Thai-Khanh Nguyen, Ngoc-Bich-Uyen Vo, Thieu N. Vo, Tan M. Nguyen+1

Faculty of Information Technology, Dainam University, Hanoi University of Science and Technology, Hanoi 10000, Vietnam · Faculty of Artificial Intelligence, Posts and Telecommunications Institute of Technology, Hanoi 10000, Vietnam · Department of Computer Science, University of Bath, Bath BA2 7AY, UK+1

State Space Models Sequence Modeling Momentum

Open paper AskPDF arXiv

State space models (SSMs) have emerged as efficient linear-time alternatives to attention for long-sequence modeling. However, existing SSMs often suffer from instability and memory degradation over extended horizons due to poorly conditioned first-order updates and unbalanced update geometry. We introduce MuonSSM, a general framework that stabilizes SSM training by explicitly conditioning the geometry of memory updates rather than the recurrent transition matrix. MuonSSM augments SSMs with a momentum-based pathway and a lightweight Newton Schulz transformation on low-rank input injections, yielding bounded and spectrally conditioned updates while preserving parallel scan complexity. Theory shows that MuonSSM improves gradient propagation, mitigates spectral amplification, and enriches memory representations over long horizons. Extensive experiments across language, vision, and time-series benchmarks show consistent gains in accuracy, robustness, and long-context performance when integrated into diverse SSM backbones. These results establish geometric conditioning of updates as a principled pathway to stable, scalable sequence modeling.

Abstract

Explore similar work

MuonSSM: Orthogonalizing State Space Models for Sequence Modeling

Lag Operator SSMs: A Geometric Framework for Structured State Space Modeling

SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning