MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models

Syed Ibrahim Omer; Ginny Y. Wong. Xiangyu Zhao

cs.LGOct 6, 2026

MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models

Authors: Syed Ibrahim Omer, Ginny Y. Wong. Xiangyu Zhao

Organizations: Department of Data Science City University of Hong Kong · NVIDIA AI Technology Center NVIDIA, Santa Clara, USA

Abstract

State Space Models (SSMs) offer an efficient alternative to Transformers for sequence modeling, yet conditioning pre-trained SSMs for iterative generation typically operates outside the recurrent operator, through input injection or activation modulation. While such mechanisms expose the model to conditioning information, they leave the underlying temporal dynamics fixed. We introduce MaRK (Markov-adapted Recurrent Kernels), a dynamic operator-conditioning framework that maps context vectors directly into bounded modulations of a frozen SSM's recurrence ( $A$ ), read-in ( $B$ ), read-out ( $C$ ), skip ( $D$ ), and discretization ( $Δ$ ) parameters. Viewed through the lens of LPV-SSM systems, MaRK induces a context-indexed family of Markov parameter sequences, allowing each diffusion timestep to reshape the model's input-output memory kernel. We instantiate MaRK on a frozen 111M-parameter Hydra SSM backbone and study three adapter geometries: Hypernet, Chebyshev polynomial, and Discrete Cosine Transform kernels. Since these adapters modify the Markov parameter sequence through low-rank auxiliary maps on the frozen backbone, parameter-efficient fine-tuning arises as a structural consequence of the adaptation mechanism itself, requiring only 6.3--11M trainable auxiliary parameters to transition from a bidirectional objective to an iterative diffusion regime. The bounded recurrence parameterization further yields an analytic Affine Quadratic Stability certificate for the modulated recurrence. Through synthetic LPV recovery experiments and Markov-operator diagnostics, we show that MaRK recovers coordinate-invariant temporal operators under matched assumptions and produces distinct, stable timestep-conditioned memory profiles. Empirically, the Chebyshev variant yields the strongest performance, achieving an average validation loss of 2.55, followed by the DCT (2.59) and Hypernet (3.77) geometries.

Figures & tables

Figure 1 : Stability margins ( $\varepsilon_{\ell}$ ) across all the layers of the pre-trained Hydra SSM at the extreme bounds of modulation by MaRK.

Panel of Figure 2 — Figure 2 : Markov Operator Error vs Dataset Size for synthetic Hypernet, Chebyshev, and DCT LPV recovery. Solid lines denote 15-seed means with normal-approximation 95% confidence intervals (shaded). Red dashed lines show an $O(N^{-1/2})$ reference normalized at the smallest dataset size.

Figure 3 : Head-resolved mean Markov parameter norm versus lag for Hypernet, Chebyshev, and DCT MaRK adapters at different diffusion timesteps $t$ . The dashed curve denotes the frozen Hydra backbone without timestep-conditioned operator modulation.

Kernel	Params	PTB	WikiText	Lambada	AG News	Arxiv	Avg.
Hypernet	6.3M	3.39 ± 0.31	3.73 ± 0.23	3.70 ± 0.17	3.91 ± 0.20	4.12 ± 0.10	3.77
Chebyshev	11.0M	2.42 ± 0.26	2.55 ± 0.16	2.40 ± 0.10	2.85 ± 0.14	2.55 ± 0.05	2.55
DCT	7.1M	2.48 ± 0.27	2.55 ± 0.14	2.47 ± 0.11	2.86 ± 0.14	2.61 ± 0.07	2.59

Table 1 : CART losses (

\mathcal{L}_{\text{CART}}\downarrow

) across 10 seeds (mean

\pm

95% CI,

1.96\times

SEM) and trainable adapter sizes across completed validation datasets. Bold = Best. Underline = Second Best.

Kernel	Full	All Except A	A Only	$\Delta$ Only	B,C Only	D Only	All Except $\Delta$	None
Hypernet	3.73 ± 0.26	3.91 ± 0.27	4.07 ± 0.29	4.04 ± 0.29	4.08 ± 0.29	3.78 ± 0.26	3.74 ± 0.26	3.99 ± 0.29
Chebyshev	2.55 ± 0.18	2.78 ± 0.19	3.44 ± 0.22	3.92 ± 0.26	3.06 ± 0.21	3.90 ± 0.26	2.56 ± 0.18	4.00 ± 0.27
DCT	2.55 ± 0.16	2.80 ± 0.16	3.24 ± 0.17	3.99 ± 0.24	3.05 ± 0.17	3.90 ± 0.24	2.59 ± 0.16	4.00 ± 0.24

Table 2 : Parameter ablation: CART validation loss (

\mathcal{L}_{\text{CART}}\downarrow

) on the WikiText dataset across 10 seeds (mean

\pm

95% CI,

1.96\times

SEM). "Full" modulates all five parameters; "None" modulates none (frozen Hydra). Bold = Best per kernel.

Variant	Latency (ms) seq=4096	Memory (MiB) seq=4096	Total Params (M)	Additional Params (M)	Overhead seq=512	Overhead seq=4096
Frozen Hydra	267.43 ± 0.06	4446.58 $\pm$ 0.06	111.68	0.00	-	-
Input Injection	270.86 ± 0.03	4455.25 $\pm$ 0.06	113.96	2.28	+3.6%	+1.3%
AdaLN Zero	276.84 ± 0.02	4474.52 $\pm$ 0.60	118.51	6.84	+7.6%	+3.5%
Hypernet	304.75 ± 0.03	4470.62 $\pm$ 0.06	117.95	6.27	+65.6%	+14.0%
Chebyshev	309.74 ± 0.10	4488.61 $\pm$ 0.06	122.65	10.97	+104.0%	+15.8%
DCT	307.28 ± 0.05	4473.93 $\pm$ 0.06	118.75	7.07	+88.7%	+14.9%

Table 3 : Inference performance and relative latency overhead over Frozen Hydra at batch size 8 on an NVIDIA A100-SXM4-80GB with the same frozen 23-layer backbone. Each variant: 15 seeds, 100 forward passes per seed, means

\pm

95% CI (t-distribution, 14df). Bold = MaRK (Ours)

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$t,\tau$	diffusion timestep and internal continuous time
$\mathbf{c}_{t}$	context/timestep embedding used as the LPV scheduling variable
$Q,Q^{\prime}$	base and modulated SSM parameters, $Q\in\{A,B,C,D,\Delta\}$
$A(\mathbf{c}_{t}),B(\mathbf{c}_{t})$	LPV-SSM matrix-valued functions for recurrence and read-in parameters
$C(\mathbf{c}_{t}),D(\mathbf{c}_{t})$	LPV-SSM matrix-valued functions for read-out and skip parameters
$x(\tau),u(\tau),y(\tau)$	LPV-SSM vector-valued functions for state, input and output

Appendix

Table 4 : Compact symbol guide for the main text.

Source	Ratio (%)
common-pile StackExchange	10.00
common-pile LibreTexts	1.67
common-pile YouTube	22.50
common-pile PubMed	7.50
common-pile CCCC	10.00
common-pile Project Gutenberg	5.00

Appendix

Table 5: Source mixture used to construct the Hydra train/validation corpus. Percentages are the relative shard-pool weights of the reported training runs and normalize at sampling.

Setting	Value
Trainable parameters	MaRK adapters and timestep embeddings only
Backbone	Frozen 23-layer Hydra-BERT SSM
Sequence length	4096 tokens
Epochs	3
Optimizer	Fused AdamW, $\beta=(0.9,0.999)$ , weight decay $0.01$
MaRK learning rate	$6.0\times 10^{-4}$

Appendix

Table 6: Training settings shared by all three MaRK kernels. Values are taken from the training configurations and the corresponding optimization implementation.

Kernel	Batch size	Warmup / total steps	Kernel-specific parameters
Hypernet	112	720 / 14,400	–
Chebyshev	112	720 / 14,400	Polynomial degree $=5$
DCT	112	720 / 14,400	Timepoints $=256$ , frequencies $=8$

Appendix

Table 7: Kernel-specific training settings. Shared kernel dimensions are rank

r=2

, adapter MLP dimension 256, and timestep embedding dimension 128.

Parameter	Value
$A$ -shift bound $\beta_{A}$	$\leq 0.5$
Scale and shift initial magnitude $\alpha,\beta$	$0.2$ (learned)
CART $p$ / scale	0.45 / 1.0

Appendix

Table 8: Modulation bounds and CART settings, identical across the three kernels.

Explore similar work

Jun 29, 2026cs.LG

MuonSSM: Orthogonalizing State Space Models for Sequence Modeling

State space models (SSMs) have emerged as efficient linear-time alternatives to attention for long-sequence modeling. However, existing SSMs often suffer from instability and memory degradation over extended horizons due to poorly conditioned first-order updates and unbalanced update geometry. We introduce MuonSSM, a general framework that stabilizes SSM training by explicitly conditioning the geometry of memory updates rather than the recurrent transition matrix. MuonSSM augments SSMs with a momentum-based pathway and a lightweight Newton Schulz transformation on low-rank input injections, yielding bounded and spectrally conditioned updates while preserving parallel scan complexity. Theory shows that MuonSSM improves gradient propagation, mitigates spectral amplification, and enriches memory representations over long horizons. Extensive experiments across language, vision, and time-series benchmarks show consistent gains in accuracy, robustness, and long-context performance when integrated into diverse SSM backbones. These results establish geometric conditioning of updates as a principled pathway to stable, scalable sequence modeling.

Thai-Khanh Nguyen, Ngoc-Bich-Uyen Vo, Thieu N. Vo +2

Faculty of Information Technology, Dainam University, Hanoi University of Science and Technology, Hanoi 10000, Vietnam · Faculty of Artificial Intelligence, Posts and Telecommunications Institute of Technology, Hanoi 10000, Vietnam · Department of Computer Science, University of Bath, Bath BA2 7AY, UK +1

State Space Models Sequence Modeling Momentum

Ask PDF

Dec 22, 2025cs.LG

Lag Operator SSMs: A Geometric Framework for Structured State Space Modeling

Structured State Space Models (SSMs), which are at the heart of the recently popular Mamba architecture, are powerful tools for sequence modeling. However, their theoretical foundation relies on a complex, multistage process of continuous-time modeling and subsequent discretization, which can obscure intuition. We introduce a direct, first-principles framework for constructing discrete-time SSMs that is both flexible and modular. Our approach is based on a novel lag operator, which geometrically derives the discrete-time recurrence by measuring how the system's basis functions undergo what we call a domain expansion from one timestep to the next. The resulting state matrices are computed via a single inner product involving this operator, enabling a modular design space for creating novel SSMs by flexibly combining different basis functions and time-warping schemes. To validate our framework, we demonstrate that a specific instance exactly recovers the recurrence of the influential HiPPO model. Numerical simulations confirm our derivation, providing new theoretical tools for designing flexible and robust sequence models.

Sutashu Tomonaga, Kenji Doya, Noboru Murata

Okinawa Institute of Science and Technology · Waseda University

State Space Models State Space Sequence Modeling Basis Functions First-Principles Calculation

Ask PDF

Jun 24, 2026cs.LG

SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning

While parameter-efficient fine-tuning (PEFT) typically targets attention projectors, its efficacy for tasks requiring sequential state accumulation remains under-explored. We examine if PEFT for such tasks can benefit from state space model (SSMs) adapters, and if MLP blocks are better injection sites. We introduce Hankel Reduced order Model (HRM) adapter, an SSM-based residual module initialized via Balanced Truncation of empirical Hankel Grammians. By leveraging the time-invariance of the system matrix

\bar{A}

, HRM enables an exact FFT-based parallel scan, achieving computational parity with LoRA across all context lengths. In iso-parametric evaluations on Mistral-7B (8.4M trainable parameters), HRM outperforms LoRA variants on LongBench tasks, including QuALITY (+34.8% relative accuracy) and QMSum (+71.6% relative ROUGE-1). HRM further demonstrates consistent superiority across 18 configurations of synthetic state-tracking (DFA, Parity) and character-level language modeling (enwik8). Gate analysis reveals that HRM adapters effectively learn to modulate recurrence, providing a robust architectural alternative to low-rank adaptation for long-context sequence modeling.

Omanshu Thapliyal

Hitachi America Ltd., Santa Clara, CA, USA.

Efficient Long-Context Inference Model Fine-Tuning Sequence Modeling State Space Models Recurrent State Reduced-Order Models

Ask PDF