State Space Models

Also known as SSM

Follow this topic Ask about this topic

Momentum

28 papers in the last four weeks, up 250% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 181

Apr 23, 2026cs.IT

MambaCSP: Hybrid-Attention State Space Models for Hardware-Efficient Channel State Prediction

Recent works have demonstrated that attention-based transformer and large language model (LLM) architectures can achieve strong channel state prediction (CSP) performance by capturing long-range temporal dependencies across channel state information (CSI) sequences. However, these models suffer from quadratic scaling in sequence length, leading to substantial computational cost, memory consumption, and inference latency, which limits their applicability in real-time and resource-constrained wireless deployments. In this paper, we investigate whether selective state space models (SSMs) can serve as a hardware-efficient alternative for CSI prediction. We propose MambaCSP, a hybrid-attention SSM architecture that replaces LLM-based prediction backbones with a linear-time Mamba model. To overcome the local-only dependencies of pure SSMs, we introduce lightweight patch-mixer attention layers that periodically inject cross-token attentions, helping with long-context CSI prediction. Extensive MISO-OFDM simulations show that MambaCSP improves prediction accuracy over LLM-based approaches by 9-12%, while delivering up to 3.0x higher throughput, 2.6x lower VRAM usage, and 2.9x faster inference. Our results demonstrate that hybrid state space architectures provide a promising direction for scalable and hardware-efficient AI-native CSI prediction in future wireless networks.

Aladin Djuhera, Haris Gacanin, Holger Boche

Technical University Munich, Germany · RWTH Aachen University, Germany

Efficient Neural Network Inference Selective SSMs Wireless Communications State Space Models

Apr 22, 2026cs.CV

Beyond ZOH: Advanced Discretization Strategies for Vision Mamba

Vision Mamba, as a state space model (SSM), employs a zero-order hold (ZOH) discretization, which assumes that input signals remain constant between sampling instants. This assumption degrades temporal fidelity in dynamic visual environments and constrains the attainable accuracy of modern SSM-based vision models. In this paper, we present a systematic and controlled comparison of six discretization schemes instantiated within the Vision Mamba framework: ZOH, first-order hold (FOH), bilinear/Tustin transform (BIL), polynomial interpolation (POL), higher-order hold (HOH), and the fourth-order Runge-Kutta method (RK4). We evaluate each method on standard visual benchmarks to quantify its influence in image classification, semantic segmentation, and object detection. Our results demonstrate that POL and HOH yield the largest gains in accuracy at the cost of higher training-time computation. In contrast, the BIL provides consistent improvements over ZOH with modest additional overhead, offering the most favorable trade-off between precision and efficiency. These findings elucidate the pivotal role of discretization in SSM-based vision architectures and furnish empirically grounded justification for adopting BIL as the default discretization baseline for state-of-the-art SSM models.

Fady Ibrahim, Guangjun Liu, Guanghui Wang

Toronto Metropolitan University, 350 Victoria St, Toronto, ON M5B 2K3

Visual SSMs State Space Models

Apr 22, 2026cs.NE

An explicit operator explains end-to-end computation in the modern neural networks used for sequence and language modeling

We establish a mathematical correspondence between state space models, a state-of-the-art architecture for capturing long-range dependencies in data, and an exactly solvable nonlinear oscillator network. As a specific example of this general correspondence, we analyze the diagonal linear time-invariant implementation of the Structured State Space Sequence model (S4). The correspondence embeds S4D, a specific implementation of S4, into a ring network topology, in which recent inputs are encoded, as waves of activity traveling over the one-dimensional spatial layout of the network. We then derive an exact operator expression for the full forward pass of S4D, yielding an analytical characterization of its complete input-output map. This expression reveals that the nonlinear decoder in the system induces interactions between these information-carrying waves that enable classifying real-world sequences. These results generalize across modern SSM architectures, and show that they admit an exact mathematical description with a clear physical interpretation. These insights enable a new level of interpretability for these systems in terms of nonlinear oscillator networks.

Anif N. Shikder, Ramit Dey, Sayantan Auddy +6

Department of Mathematics, Western University, London ON, Canada · Fields Lab for Network Computation, Fields Institute, Toronto ON, Canada · King’s University College at Western University, London ON, Canada +1

Dynamical Systems Neural Network Interpretability Sequence Modeling State Space Models

Apr 22, 2026cs.LG

Data-Driven Open-Loop Simulation for Digital-Twin Operator Decision Support in Wastewater Treatment

Wastewater treatment plants (WWTPs) need digital-twin-style decision support tools that can simulate plant response under prescribed control plans, tolerate irregular and missing sensing, and remain informative over 12-36 h planning horizons. Meeting these requirements with full-scale plant data remains an open engineering-AI challenge. We present CCSS-RS, a controlled continuous-time state-space model that separates historical state inference from future control and exogenous rollout. The model combines typed context encoding, gain-weighted forcing of prescribed and forecast drivers, semigroup-consistent rollouts, and Student-t plus hurdle outputs for heavy-tailed and zero-inflated WWTP sensor data. On the public Avedøre full-scale benchmark, with 906,815 timesteps, 43% missingness, and 1-20 min irregular sampling, CCSS-RS achieves RMSE 0.696 and CRPS 0.349 at H=1000 across 10,000 test windows. This reduces RMSE by 40-46% relative to Neural CDE baselines and by 31-35% relative to simplified internal variants. Four case studies using a frozen checkpoint on test data demonstrate operational value: oxygen-setpoint perturbations shift predicted ammonium by -2.3 to +1.4 over horizons 300-1000; a smoothed setpoint plan ranks first in multi-criterion screening; context-only sensor outages raise monitored-variable RMSE by at most 10%; and ammonium, nitrate, and oxygen remain more accurate than persistence throughout the rollout. These results establish CCSS-RS as a practical learned simulator for offline scenario screening in industrial wastewater treatment, complementary to mechanistic models.

Gary Simethy, Daniel Ortiz Arroyo, Petar Durdevic

Aalborg University, Department of Energy, Niels Bohrs Vej 8, Esbjerg, DK-6700, Denmark

Neural Surrogate Modeling Digital Twins State Space Models

Apr 20, 2026cs.LG

Sessa: Selective State Space Attention

Modern sequence modeling is dominated by two families: Transformers, whose self-attention can access arbitrary elements of the visible sequence, and structured state-space models, which propagate information through an explicit recurrent state. These mechanisms face different limitations on long contexts: when attention is diffuse, the influence of individual tokens is diluted across the effective support, while recurrent state propagation can lose long-range sensitivity unless information is actively preserved. As a result, both mechanisms face challenges in preserving and selectively retrieving information over long contexts. We propose Sessa, a decoder that places attention inside a recurrent feedback path. This creates many attention-based paths through which past tokens can influence future states, rather than relying on a single attention read or a single recurrent chain. We prove that, under explicit assumptions and matched regimes, Sessa admits power-law memory tails

O(\ell^{-β})

for

0 < β< 1

, with slower decay than in the corresponding Transformer and Mamba-style baselines. We further give an explicit construction that achieves this power-law rate. Under the same assumptions, Sessa is the only model class among those considered that realizes flexible selective retrieval, including profiles whose influence does not decay with distance. Consistent with this theoretical advantage, across matched experiments, Sessa achieves the strongest performance on long-context benchmarks while remaining competitive with Transformer and Mamba-style baselines on short-context language modeling.

Liubomyr Horbatko

Self-Attention Long-Context Language Modeling Persistent Memory for Language Models Recurrent Transformers Attention Mechanisms State Space Models

Apr 20, 2026cs.CL

Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models

Transformer-based embedding models suffer from quadratic computational and linear memory complexity, limiting their utility for long sequences. We propose recurrent architectures as an efficient alternative, introducing a vertically chunked inference strategy that enables fast embedding generation with memory usage that becomes constant in the input length once it exceeds the vertical chunk size. By fine-tuning Mamba2 models, we demonstrate their viability as general-purpose text embedders, achieving competitive performance across a range of benchmarks while maintaining a substantially smaller memory footprint compared to transformer-based counterparts. We empirically validate the applicability of our inference strategy to Mamba2, RWKV, and xLSTM models, confirming consistent runtime-memory trade-offs across architectures and establishing recurrent models as a compelling alternative to transformers for efficient embedding generation.

Tobias Grantner, Emanuel Sallinger, Martin Flechl

Dynatrace Research · TU Wien · University of Oxford

Memory-Efficient Inference Text Embedding Models State Space Models

Apr 19, 2026eess.SY

Bilinear Input Modulation for Mamba: Koopman Bilinear Forms for Memory Retention and Multiplicative Computation

Selective State Space Models (SSMs), notably Mamba, employ diagonal state transitions that limit both memory retention and bilinear computational capacity. We propose a factorized bilinear input modulation that augments the SSM with a state-input product, interpretable as a finite-dimensional Koopman bilinear form. After introducing a shared state across channels (Coupled SSM), the modulation admits three implementations. Coupled Bilinear Input Modulation (seq-BIM) retains the full bilinear product on the input side at the cost of sequential computation, Coupled Gated Modulation (GM) linearizes it into a gate modulation that is compatible with the parallel scan, and Parallel Bilinear Input Modulation (p-BIM) places the same bilinear product on the state transition while remaining parallel-scannable. Experiments on a multiple input-delay pendulum (memory retention) and NARMA-10 (bilinear computation) reveal a clear dissociation. GM substantially improves memory retention but not bilinear computation, while both seq-BIM and p-BIM improve both. A pathway ablation confirms that the two downstream routes of the bilinear signal serve complementary roles. The improvement is statistically robust, with the bilinear variants consistently outperforming the other variants on bilinear computation. Furthermore, only the bilinear variants benefit from increasing the SSM state dimension, while coupling or gate modulation alone show no improvement, establishing the bilinear mechanism as uniquely capable of exploiting larger state spaces.

Hiroki Fujii, Masaki Yamakita

Department of Systems and Control Engineering, Institute of Science Tokyo, Tokyo, Japan

Selective SSMs State Space Models

Apr 18, 2026eess.AS

A state-space representation of the boundary integral equation for room acoustic modelling

We introduce a new framework for room acoustics modelling based on a state-space model of the boundary integral equation representing the sound field in a room. Whereas state-space models of linear time-invariant systems are traditionally constructed by means of a state vector and a 4-tuple of system matrices, the state-space representation introduced in this work consists of a state function representing the pressure distribution at the room boundary, and a 4-tuple of integral operators. We refer to this representation as a boundary integral operator state-space (BIOSS) model and provide a physical interpretation for each of the integral operators. As many mathematical operations on vectors and matrices translate to functions and operators, the BIOSS representation can be manipulated to obtain two transfer function representations, having either a feedback or a parallel feedforward structure. Consequently, various equivalent representations for room acoustics are obtained in the BIOSS framework, in the time or frequency domain, and in continuous or discrete space. We discuss two future directions for how the proposed framework can be fertile for research on room acoustics modelling. Firstly, we identify equivalences between the BIOSS framework and various existing room acoustics models (boundary element models, delay networks, geometric models), which may be used to establish relations between existing models and to develop novel room acoustics models. Secondly, we postulate on how concepts from state-space theory, such as observability, controllability, and state realization, can be used for developing new inference and control methods for room acoustics.

Randall Ali, Thomas Dietzen, Matteo Scerbo +2

Institute of Sound Recording, University of Surrey, Guildford, UK · STADIUS, Dept. of Electrical Engineering, KU Leuven, Leuven, Belgium

Room Acoustics State Space Models

Apr 17, 2026cs.CV

PA-TCNet: Pathology-Aware Temporal Calibration with Physiology-Guided Target Refinement for Cross-Subject Motor Imagery EEG Decoding in Stroke Patients

Stroke patient cross-subject electroencephalography (EEG) decoding of motor imagery (MI) brain-computer interface (BCI) is essential for motor rehabilitation, yet lesion-related abnormal temporal dynamics and pronounced inter-patient heterogeneity often undermine generalization. Existing adaptation methods are easily misled by pathological slow-wave activity and unstable target-domain pseudo-labels. To address this challenge, we propose PA-TCNet, a pathology-aware temporal calibration framework with physiology-guided target refinement for stroke motor imagery decoding. PA-TCNet integrates two coordinated components. The Pathology-aware Rhythmic State Mamba (PRSM) module decomposes EEG spatiotemporal features into slowly varying rhythmic context and fast transient perturbations, injecting the fused pathological context into selective state propagation to more effectively capture abnormal temporal dynamics. The Physiology-Guided Target Calibration (PGTC) module constructs source-domain sensorimotor region-of-interest templates, imposing physiological consistency constraints and dynamically refining target-domain pseudo-labels, thereby improving adaptation reliability. Leave-one-subject-out experiments on two independent stroke EEG datasets, XW-Stroke and 2019-Stroke, yielded mean accuracies of 66.56% and 72.75%, respectively, outperforming state-of-the-art baselines. These results indicate that jointly modeling pathological temporal dynamics and physiology-constrained pseudo-supervision can provide more robust cross-subject initialization for personalized post-stroke MI-BCI rehabilitation. The implemented code is available at https://github.com/wxk1224/PA-TCNet.

Xiangkai Wang, Yun Zhao, Dongyi He +5

School of Artificial Intelligence, Chongqing University of Technology, Chongqing 401135, China; Chongqing Key Laboratory of Embodied Intelligence Perception and Autonomous Learning for Humanoid Robots; Key Laboratory of Advanced Equipment Intelligence of the Chongqing Education Commission, Chongqing 401135, China · School of Smart Health, Chongqing Polytechnic University of Electronic Technology, Chongqing 401131, China · Department of Language Science and Technology, The Hong Kong Polytechnic University, Hung Hom, Hong Kong SAR, China +2

Cross-Subject EEG Decoding Brain-Computer Interfaces Domain Adaptation EEG Decoding State Space Models

Apr 17, 2026cs.CV

MambaBack: Bridging Local Features and Global Contexts in Whole Slide Image Analysis

Whole Slide Image (WSI) analysis is pivotal in computational pathology, enabling cancer diagnosis by integrating morphological and architectural cues across magnifications. Multiple Instance Learning (MIL) serves as the standard framework for WSI analysis. Recently, Mamba has become a promising backbone for MIL, overtaking Transformers due to its efficiency and global context modeling capabilities originating from Natural Language Processing (NLP). However, existing Mamba-based MIL approaches face three critical challenges: (1) disruption of 2D spatial locality during 1D sequence flattening; (2) sub-optimal modeling of fine-grained local cellular structures; and (3) high memory peaks during inference on resource-constrained edge devices. Studies like MambaOut reveal that Mamba's SSM component is redundant for local feature extraction, where Gated CNNs suffice. Recognizing that WSI analysis demands both fine-grained local feature extraction akin to natural images, and global context modeling akin to NLP, we propose MambaBack, a novel hybrid architecture that harmonizes the strengths of Mamba and MambaOut. First, we propose the Hilbert sampling strategy to preserve the 2D spatial locality of tiles within 1D sequences, enhancing the model's spatial perception. Second, we design a hierarchical structure comprising a 1D Gated CNN block based on MambaOut to capture local cellular features, and a BiMamba2 block to aggregate global context, jointly enhancing multi-scale representation. Finally, we implement an asymmetric chunking design, allowing parallel processing during training and chunking-streaming accumulation during inference, minimizing peak memory usage for deployment. Experimental results on five datasets demonstrate that MambaBack outperforms seven state-of-the-art methods. Source code and datasets are publicly available.

Sicheng Chen, Chad Wong, Tianyi Zhang +3

Nhu Department of Electrical Engineering and Computer Science, University of California, Irvine · Department of Electrical & Computer Engineering, National University of Singapore · PuzzleLogic Pte Ltd, Singapore 229594, Singapore

Efficient Neural Network Inference Multiple Instance Learning Whole-Slide Image Analysis State Space Models

Apr 17, 2026cs.CV

SSMamba: A Self-Supervised Hybrid State Space Model for Pathological Image Classification

Pathological diagnosis is highly reliant on image analysis, where Regions of Interest (ROIs) serve as the primary basis for diagnostic evidence, while whole-slide image (WSI)-level tasks primarily capture aggregated patterns. To extract these critical morphological features, ROI-level Foundation Models (FMs) based on Vision Transformers (ViTs) and large-scale self-supervised learning (SSL) have been widely adopted. However, three core limitations remain in their application to ROI analysis: (1) cross-magnification domain shift, as fixed-scale pretraining hinders adaptation to diverse clinical settings; (2) inadequate local-global relationship modeling, wherein the ViT backbone of FMs suffers from high computational overhead and imprecise local characterization; (3) insufficient fine-grained sensitivity, as traditional self-attention mechanisms tend to overlook subtle diagnostic cues. To address these challenges, we propose SSMamba, a hybrid SSL framework that enables effective fine-grained feature learning without relying on large external datasets. This framework incorporates three domain-adaptive components: Mamba Masked Image Modeling (MAMIM) for mitigating domain shift, a Directional Multi-scale (DMS) module for balanced local-global modeling, and a Local Perception Residual (LPR) module for enhanced fine-grained sensitivity. Employing a two-stage pipeline, SSL pretraining on target ROI datasets followed by supervised fine-tuning (SFT), SSMamba outperforms 11 state-of-the-art (SOTA) pathological FMs on 10 public ROI datasets and surpasses 8 SOTA methods on 6 public WSI datasets. These results validate the superiority of task-specific architectural designs for pathological image analysis.

Enhui Chai, Sicheng Chen, Tianyi Zhang +2

School of Computer Science, Northwest University · PuzzleLogic Pte Ltd, Singapore 229594, Singapore · Department of Electrical & Computer Engineering, National University of Singapore +1

Self-Supervised Learning Pathology Foundation Models Histopathology Image Classification Medical Image Representation Learning Whole-Slide Image Analysis State Space Models

Apr 16, 2026cs.CV

HAMSA: Scanning-Free Vision State Space Models via SpectralPulseNet

Vision State Space Models (SSMs) like Vim, VMamba, and SiMBA rely on complex scanning strategies to adapt sequential SSMs to process 2D images, introducing computational overhead and architectural complexity. We propose HAMSA, a scanning-free SSM operating directly in the spectral domain. HAMSA introduces three key innovations: (1) simplified kernel parameterization-a single Gaussian-initialized complex kernel replacing traditional (A, B, C) matrices, eliminating discretization instabilities; (2) SpectralPulseNet (SPN)-an input-dependent frequency gating mechanism enabling adaptive spectral modulation; and (3) Spectral Adaptive Gating Unit (SAGU)-magnitude-based gating for stable gradient flow in the frequency domain. By leveraging FFT-based convolution, HAMSA eliminates sequential scanning while achieving O(L log L) complexity with superior simplicity and efficiency. On ImageNet-1K, HAMSA reaches 85.7% top-1 accuracy (state-of-the-art among SSMs), with 2.2 X faster inference than transformers (4.2ms vs 9.2ms for DeiT-S) and 1.4-1.9X speedup over scanning-based SSMs, while using less memory (2.1GB vs 3.2-4.5GB) and energy (12.5J vs 18-25J). HAMSA demonstrates strong generalization across transfer learning and dense prediction tasks.

Badri N. Patro, Vijay S. Agneeswaran

Microsoft

Frequency-Domain Feature Learning Visual SSMs Image Classification State Space Models

Apr 16, 2026cs.LG

On the Expressive Power and Limitations of Multi-Layer SSMs

We study how depth, finite precision, state dimension, and chain-of-thought (CoT) affect the expressive power of multi-layer state-space models (SSMs). For the explicit-table

K

-function-composition problem, a canonical benchmark for sequential information propagation, we prove that any

L

-layer SSM solving

(L+3)

-function composition must satisfy

d^2p=Ω(N/L^3)

, where

d

is the state dimension and

p

is the per-scalar precision. Conversely,

K

-function composition is solved exactly by a

(K+1)

-layer generalized SSM with

d=1

and

p=Θ(\log N)

. This gives a worst-case depth hierarchy for this formal problem family. We then distinguish post-input reasoning, in which all thought tokens are generated after the input, from input-interleaved reasoning, in which thought tokens may be inserted while the input stream is being read. Post-input reasoning does not circumvent our communication-based lower-bound pipeline, whereas input-interleaved reasoning admits bidirectional simulations with general deterministic one-pass streaming algorithms at the granularity of persistent memory. Finally, width and precision are not interchangeable under exact step-preserving simulation in the base affine-state model, but become interchangeable through the streaming-memory characterization once input-interleaved reasoning is allowed.

Nikola Zubić, Qian Li, Yuyi Wang +1

Robotics and Perception Group University of Zurich · Shenzhen International Center For Industrial And Applied Mathematics Shenzhen Research Institute of Big Data · Tengen Intelligence Institute2026 CRRC Zhuzhou Institute

CoT Reasoning State Space Models Streaming Algorithms

Mar 24, 2026cs.CV

Mamba-driven MRI-to-CT Synthesis for MRI-only Radiotherapy Planning

Radiotherapy workflows for oncological patients increasingly rely on multi-modal medical imaging, commonly involving both Magnetic Resonance Imaging (MRI) and Computed Tomography (CT). MRI-only treatment planning has emerged as an attractive alternative, as it reduces patient exposure to ionizing radiation and avoids errors introduced by inter-modality registration. While nnU-Net-based frameworks are predominantly used for MRI-to-CT synthesis, we explore Mamba-based architectures for this task to investigate the applicability of state-space modeling for cross-modality medical image translation and assess its performance relative to established convolutional architectures. Specifically, we adapt the SegMamba architecture, originally proposed for segmentation, to perform image-to-image generation. Our 3D Mamba architecture effectively captures complex volumetric features and long-range dependencies, thus allowing accurate CT synthesis while maintaining a relatively low parameter count. Experiments were conducted on a subset of SynthRAD2025 dataset, comprising registered single-channel MRI-CT volume pairs across three anatomical regions. Quantitative evaluation is performed via a combination of image similarity metrics computed in Hounsfield Units (HU) and segmentation-based metrics obtained from TotalSegmentator to ensure geometric consistency is preserved. The findings pave the way for the integration of state-space models into radiotherapy workflows.

Konstantinos Barmpounakis, Theodoros P. Vagenas, Maria Vakalopoulou +1

School of Electrical and Computer Engineering, National Technical University of Athens · Archimedes, Athena Research Center · CentraleSupélec, University of Paris-Saclay

Medical Image Synthesis State Space Models

Mar 19, 2026stat.ML

Extending SSMs with the Exponentially Weighted Signature

We introduce the exponentially weighted signature (EWS), a continuous-time model that computes iterated integrals of a path, where each increment is weighted by the matrix exponential of a learnable generator over elapsed clock time. We prove that it solves a linear controlled differential equation, keeps the group-like structure and the universality of the signature, and satisfies a modified Chen identity, enabling a parallel scan. At depth one the EWS is a state-space model (SSM), and we map linear time-invariant SSMs, Mamba channels and Mamba-

2

heads to it in closed form. The EWS extends SSMs through an arbitrary matrix generator, a clock that generalises the step size to causal functionals of the input, and higher truncation depths that are non-linear in the path within a single layer. Empirically, the EWS achieves the highest average accuracy and rank on six long time-series classification datasets, where depth generally helps. Learned clocks prove necessary for state tracking on formal language tasks, and at depth one, the EWS matches or exceeds competing SSMs on regression and forecasting with far fewer parameters.

Alexandre Bloch, Benjamin Walker, Joël Mouterde +3

Mathematical Institute, University of Oxford · SKF · Department of Mathematics, Imperial College London

Dynamical Systems Time Series Classification Time Series Forecasting State Space Models

Mar 13, 2026math.OC

State-space models through the lens of ensemble control

State-space models (SSMs) are effective architectures for sequential modeling, but a rigorous theoretical understanding of their training dynamics is still lacking. We formulate continuous-time SSM parameter-path optimization as an ensemble optimal control problem: each input sequence generates a corresponding state trajectory in the ensemble, while the trainable parameter path forms a common control shared by all trajectories. Within this formulation, model evaluation is represented by the forward state equation and backward sensitivity propagation by the corresponding adjoint equation. We show that the Hamiltonian gradient with respect to the control variable represents the negative first-variation density of the reduced ensemble objective. Using this representation, we analyze a Bregman mirror-descent scheme for the continuous-time ensemble objective; in Euclidean geometry this reduces to functional projected gradient descent. State-adjoint and Hamiltonian-gradient stability yield two-sided relative curvature of the reduced objective and relative convexity under sufficient regularization. We further show that in the relatively convex regime the optimal-control problem admits a minimizer, which is unique under relative strong convexity. For SSMs, the explicit stability-generated defect

κ_{\mathrm{stab}}^{\mathrm{SSM}}

gives an

O(1/k)

objective rate at

τ=κ_{\mathrm{stab}}^{\mathrm{SSM}}

and geometric convergence to the unique minimizer for

τ>κ_{\mathrm{stab}}^{\mathrm{SSM}}

, under the stated stepsize condition.

Ye Feng, Jianfeng Lu

Mirror Descent Optimal Control State Space Models

Mar 3, 2026cs.AI

Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures

Transformers excel at in-context retrieval but suffer from quadratic complexity with sequence length, while State Space Models (SSMs) offer efficient linear-time processing but have limited retrieval capabilities. We investigate whether hybrid architectures combining Transformers and SSMs can achieve the best of both worlds on two synthetic in-context retrieval tasks. The first task, n-gram retrieval, requires the model to reproduce an n-gram that succeeds the query within the input sequence. The second task, position retrieval, presents the model with a query token and requires it to perform a two-hop lookup: first locating the corresponding element in the sequence, and then outputting its positional index. Under controlled conditions, we assess data efficiency, length generalization, robustness to out of domain training examples, and learned representations across Transformers, SSMs, and hybrid architectures. We find that hybrid models outperform SSMs and match or exceed Transformers in terms of data efficiency and extrapolation for tasks that require precise information retrieval from the input context. However, Transformers maintain superiority in position retrieval tasks. Through representation analysis, we discover that SSM-based models develop locality-aware embeddings where tokens representing adjacent positions become neighbors in embedding space, forming interpretable structures. This property is absent in Transformers as causal attention is sufficient for acquiring positional associations, and the introduction of positional encoding amplifies this behavior, leading to improvement in data efficiency. SSMs on the other hand update their internal representations incrementally and without positional encodings, are required to learn these associations. Our findings reveal fundamental differences in how Transformers and SSMs, and hybrid models learn positional associations.

Georgios Pantazopoulos, Malvina Nikandrou, Ioannis Konstas +1

University of Edinburgh · Heriot-Watt University

In-Context Retrieval Transformer State Space Models

Jan 30, 2026cs.LG

Learning-to-Defer in Non-Stationary Time Series via Switching State-Space Models

Learning-to-defer (L2D) lets a predictor decide, at each round, whether to issue its own forecast or pay for an expert's. In non-stationary time series this decision must keep adapting, although deployment reveals only the consulted expert's forecast while a historical archive records every expert with the target. L2D-SLDS learns from this archive a switching state-space model of the target and all expert forecasts, whose shared and expert-specific states describe how experts move together and apart. Its predictive law supplies the internal forecast and the expected cost of every consultation, which a greedy router minimizes, and one consultation also updates the beliefs about unconsulted and unavailable experts. We prove sublinear regret against a changing conditional-risk oracle without exploration, when the candidate models are accurate and either the archive separates them or live feedback reveals cost differences. On three real datasets, L2D-SLDS has the lowest cost among eight bandit routers and adapts its consultation rate to the fee.

Yannis Montreuil, Letian Yu, Axel Carlier +2

School of Computing National University of Singapore · Université de Toulouse Fédération ENAC ISAE-SUPAERO ONERA · Institute for Infocomm Research A*STAR, Singapore

Cost-Sensitive Learning Adaptive Model Routing Learning to Defer Decision-Focused Learning Non-Stationary Time Series Forecasting State Space Models

Jan 4, 2026stat.ML

Modeling Information Blackouts in Missing Not-At-Random Time Series Data

Traffic forecasting systems rely on fixed sensor networks that frequently exhibit contiguous blackouts. Such outages are usually treated as ignorable missingness, although dropout can depend on unobserved traffic conditions. We study this possibility with an MNAR-aware latent state-space model that combines linear traffic dynamics with a Bernoulli missingness channel whose probability depends on the latent state. Inference uses an Extended Kalman Filter (EKF) followed by Rauch-Tung-Striebel (RTS) smoothing, and parameters are learned by approximate EM. We evaluate Seattle using a leakage-free, month-balanced set of 300 unique all-horizon-aligned blackout windows. On this benchmark, MAR-LDS attains 4.264 mph pooled imputation RMSE and MNAR-LDS improves it to 4.177 (difference -0.086); the detector-cluster bootstrap 95% interval is [-0.182,-0.002]. A causal one-step predicted latent representation raises missingness ROC-AUC from 0.685 using observed-only features to 0.784. We further test whether this compact probabilistic model remains competitive with substantially larger neural time-series architectures under the identical masked-imputation protocol. MNAR-LDS ranks second in pooled RMSE and outperforms 8 of 9 evaluated neural architectures; it is within 1.22% of the best neural result, with no statistically resolved difference under detector-cluster bootstrap, while achieving lower P95 error, lower long-blackout RMSE, and orders of magnitude fewer stored scalar entries. MNAR roughly doubles end-to-end training time relative to MAR and increases EKF+RTS inference time by 41%, making the accuracy-complexity-cost tradeoff explicit. Controlled state-dependent blackouts further show larger gains when dropout is genuinely informative, including a 6.34% reduction in 30-minute forecast RMSE relative to MAR.

Aman Sunesh, Allan Ma, Siddarth Nilol

New York University

Time Series Imputation Incomplete Data Imputation Traffic Flow Estimation State Space Models

Dec 22, 2025cs.LG

Lag Operator SSMs: A Geometric Framework for Structured State Space Modeling

Structured State Space Models (SSMs), which are at the heart of the recently popular Mamba architecture, are powerful tools for sequence modeling. However, their theoretical foundation relies on a complex, multistage process of continuous-time modeling and subsequent discretization, which can obscure intuition. We introduce a direct, first-principles framework for constructing discrete-time SSMs that is both flexible and modular. Our approach is based on a novel lag operator, which geometrically derives the discrete-time recurrence by measuring how the system's basis functions undergo what we call a domain expansion from one timestep to the next. The resulting state matrices are computed via a single inner product involving this operator, enabling a modular design space for creating novel SSMs by flexibly combining different basis functions and time-warping schemes. To validate our framework, we demonstrate that a specific instance exactly recovers the recurrence of the influential HiPPO model. Numerical simulations confirm our derivation, providing new theoretical tools for designing flexible and robust sequence models.

Sutashu Tomonaga, Kenji Doya, Noboru Murata

Okinawa Institute of Science and Technology · Waseda University

Sequence Modeling State Space Models

Nov 19, 2025cs.LG

Recursive Entropic Variational Inference for Nonlinear State-Space Models

We present a class of algorithms for state estimation in nonlinear, non-Gaussian state-space models. Our approach is based on a variational Lagrangian formulation that casts Bayesian inference as a sequence of entropic trust-region updates subject to dynamic consistency constraints. This framework gives rise to a family of forward-backward algorithms whose structure is determined by the chosen factorization of the variational posterior. By focusing on Gauss--Markov approximations, we derive recursive schemes with favorable computational complexity. For general nonlinear, non-Gaussian models, we close the recursions using generalized statistical linear regression and Fourier--Hermite moment matching.

Hany Abdulsamad, Ángel F. García-Fernández, Simo Särkkä

Amsterdam Machine Learning Lab University of Amsterdam, Netherlands · Information Processing and Telecommunications Center Universidad Politécnica de Madrid, Spain · ELLIS Institute Finland Department of Electrical Engineering and Automation Aalto University, Finland

Bayesian Filtering Variational Inference State Space Models

Oct 29, 2025eess.SP

PyDPF: A Python Package for Differentiable Particle Filtering

State-space models (SSMs) are a widely used tool in time series analysis. In the complex systems that arise from real-world data, it is common to employ particle filtering (PF), an efficient Monte Carlo method for estimating the hidden state corresponding to a sequence of observations. Applying particle filtering requires specifying both the parametric form and the parameters of the system, which are often unknown and must be estimated. Gradient-based optimisation techniques cannot be applied directly to standard particle filters, as the filters themselves are not differentiable. However, several recently proposed methods modify the resampling step to make particle filtering differentiable. In this paper, we present an implementation of several such differentiable particle filters (DPFs) with a unified API built on the popular PyTorch framework. Our implementation makes these algorithms easily accessible to a broader research community and facilitates straightforward comparison between them. We validate our framework by reproducing experiments from several existing studies and demonstrate how DPFs can be applied to address several common challenges with state space modelling.

John-Joseph Brady, Benjamin Cox, Yunpeng Li +1

King’s College London · Max-Planck Institut für Physik · University of Edinburgh

Particle Filtering Bayesian Filtering Differentiable Programming State Space Models

Oct 26, 2025cs.CL

Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models

In-context learning is governed by both temporal and semantic relationships, shaping how Large Language Models (LLMs) retrieve contextual information. Analogous to human episodic memory, where the retrieval of specific events is enabled by separating events that happened at different times, this work probes the ability of various pretrained LLMs, including transformer and state-space models, to differentiate and retrieve temporally separated events. Specifically, we prompted models with sequences containing multiple presentations of the same token, which reappears at the sequence end. By fixing the positions of these repeated tokens and permuting all others, we removed semantic confounds and isolated temporal effects on next-token prediction. Across diverse sequences, models consistently placed the highest probabilities on tokens following a repeated token, but with a notable bias for those nearest the beginning or end of the input. An ablation experiment linked this phenomenon in transformers to induction heads. Extending the analysis to unique semantic contexts with partial overlap further demonstrated that memories embedded in the middle of a prompt are retrieved less reliably. Despite architectural differences, state-space and transformer models showed comparable temporal biases. Our findings deepen the understanding of temporal biases in in-context learning and offer an illustration of how these biases can enable temporal separation and episodic retrieval.

Anooshka Bajaj, Deven Mahesh Mistry, Sahaj Singh Maini +2

Department of Computer Science Indiana University Bloomington

In-Context Learning Induction Heads State Space Models

Oct 22, 2025cs.LG

Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge

State-space models (SSMs) have emerged as promising alternatives to Transformers for sequence modeling. However, training competitive SSMs from scratch remains computationally intensive, and the ecosystem around them is far less mature than that of Transformers. Moreover, the architectural differences between SSMs and Transformers make it challenging to efficiently transfer knowledge from pretrained Transformers. In this work, we propose Cross-architecture distillation via Attention Bridge (CAB), a distillation framework that transfers attention-related representations from Transformer teachers to state-space student models. Unlike conventional knowledge distillation that supervises only final predictions, CAB enables token-level intermediate supervision through a lightweight bridge and flexible layer-wise alignment. By aligning Transformer attention-related representations with Mamba's token-dependent state projections, CAB facilitates efficient cross-architecture knowledge transfer without inference-time overhead. Experiments on image classification and language modeling demonstrate that CAB improves Transformer-to-SSM distillation, particularly under limited-supervision settings. Overall, CAB provides an efficient pathway for transferring Transformer representations to SSM-based models, helping bridge the gap between mature Transformer ecosystems and emerging SSM ecosystems.

Penghao Wang, Yuhao Zhou, Mengxuan Wu +3

National University of Singapore · The University of Texas at Austin

Cross-Architecture Knowledge Distillation Knowledge Distillation State Space Models

Oct 17, 2025cs.CV

Cortical-SSM: A Deep State Space Model for Motor Imagery Decoding from EEG Signals

Classification of electroencephalogram (EEG) signals obtained during motor imagery (MI) has substantial application potential, including communication assistance and rehabilitation support for patients with motor impairments. These signals remain inherently susceptible to physiological artifacts (e.g., eye blinking and swallowing), which pose persistent challenges. Although Transformer-based approaches for classifying EEG signals have been widely adopted, they often struggle to capture fine-grained dependencies within them. To overcome these limitations, we propose Cortical-SSM, a novel architecture that extends deep state space models to capture integrated dependencies of EEG signals across temporal, spatial, and frequency domains. We validated our method across two large-scale public MI EEG datasets containing more than 50 subjects. Our method outperformed baseline methods on both benchmarks. Furthermore, visual explanations derived from our model indicate that it effectively captures neurophysiologically relevant regions of EEG signals. These results indicate that Cortical-SSM provides a robust and interpretable alternative to attention-based architectures for MI EEG decoding. By enabling physiologically grounded feature learning, our method advances the reliability of subject-independent EEG classification and supports the development of practical and clinically deployable brain-computer interface systems.

Shuntaro Suzuki, Shunya Nagashima, Komei Sugiura

Keio University

MI Classification Brain-Computer Interfaces EEG Decoding State Space Models

Sep 25, 2025cs.LG

Aligning Inductive Bias for Data-Efficient Generalization in State Space Models

The remarkable success of modern AI has been closely tied to scaling laws, yet the finite supply of high-quality data makes data efficiency--learning more from less--an increasingly important frontier. A model's inductive bias is a critical lever for data efficiency, but foundational sequence models such as State Space Models (SSMs) often rely on fixed, task-agnostic biases. When this fixed prior is misaligned with the underlying structure of a task, the model may require additional samples to overcome its own bias before learning the relevant signal. In this work, we introduce a principled framework for understanding and aligning the inductive bias of linear time-invariant SSMs. We first formalize this bias through an SSM-induced kernel and show theoretically and empirically that its spectrum is governed by the model's frequency response. This characterization motivates Task-Dependent Initialization (TDI), a fast power-spectrum matching method that aligns the initial SSM bias with the task's spectral characteristics before downstream training. Across controlled synthetic experiments, trainable one-layer SSMs, and deep SSMs on diverse real-world benchmarks, TDI can improve data-efficient generalization primarily when task-relevant spectral structure is present and the default SSM bias is spectrally mismatched. Our results provide both a theoretical lens and a practical tool for task-adaptive inductive bias, suggesting a path toward more data-efficient sequence modeling.

Qiyu Chen, Guozhang Chen

School of Computer Science, National Key Laboratory for Multimedia Information Processing, Peking University, China · School of Physics, Peking University, China

Sequence Modeling Data-Efficient Learning State Space Models

Aug 13, 2025cs.LG

EEGDM: Label-Efficient EEG Representation Learning with Generative Diffusion Model

Electroencephalography (EEG) is a critical tool for monitoring brain activity and diagnosing neurological disorders such as epilepsy. However, learning meaningful representations from raw EEG signals remains challenging due to limited annotations, substantial inter-subject variability, and complex temporal dynamics. Recent EEG foundation models (FMs) have demonstrated promising performance through transformer-based architectures and large-scale self-supervised pretraining, yet they often incur substantial computational costs and exhibit diminishing returns with increasing model and dataset scale, limiting their practicality in clinical settings. To address these challenges, we propose EEGDM, a diffusion-based EEG representation learning framework. Specifically, EEGDM introduces a Structured State-Space Model for Diffusion Pretraining (SSMDP) that effectively captures long-range temporal dependencies through generative diffusion training on unlabeled EEG data. The representations are subsequently leveraged for downstream tasks via our Latent Fusion Module (LFM), which integrates multi-layer latent features of SSMDP. We evaluate EEGDM on three EEG benchmarks spanning EEG event classification (TUEV), seizure detection (CHB-MIT), and seizure classification (IIIC). Compared with existing state-of-the-art methods, including EEG FMs, EEGDM achieves competitive performance across diverse tasks and datasets, exceeding existing methods in most cases while requiring substantially fewer samples for both pretraining and downstream adaptation. These results demonstrate that diffusion-based learning can unlock more discriminative EEG representations, with direct implications for epilepsy diagnosis and management. Our source code and pretrained checkpoints are publicly available at: https://github.com/jhpuah/EEGDM.

Jia Hong Puah, Sim Kuan Goh, Ziwei Zhang +6

School of Artificial Intelligence and Robotics, Xiamen University Malaysia · School of Energy and Chemical Engineering, Xiamen University Malaysia · Department of Biomedical Engineering, Universiti Malaya +3

Electroencephalography EEG Seizure Detection Representation Learning Self-Supervised Time Series Representation Learning Diffusion Models Data-Efficient Learning+1

Mar 25, 2025cs.CV

Keyframe-Centric State-Space Modeling for Burst Image Super-Resolution

Burst image super-resolution (BISR) reconstructs a high-resolution keyframe by aggregating complementary sub-pixel evidence from a short burst of low-resolution frames. Existing methods often process all burst frames with heavy backbones or maintain deep cross-frame interaction throughout the network, leading to redundant computation on non-key frames and limiting scalability with burst length. In this work we propose BurstMamba, a BISR architecture built on a simple principle: allocate most compute to reconstructing the keyframe, and process the burst primarily to extract sub-pixel priors. To this end, BurstMamba decouples BISR into a high-capacity keyframe super-resolution stream and a lightweight burst stream that interacts with it only through stage-wise residual injection. To improve burst-to-keyframe transfer, we introduce Gather -> Aggregate -> Scatter (GAS), which uses correspondence only for cross-frame message passing while preserving native-view features through a residual connection, and a wavelet-conditioned state update that biases selective routing toward high-frequency regions. Across SyntheticSR, RealBSR-RAW, and RealBSR-RGB, BurstMamba achieves state-of-the-art results. Extensive ablations show that the proposed compute separation, GAS, and wavelet-conditioned routing each contribute to the model's accuracy and scaling.

Ozan Unal, Steven Marty, Dengxin Dai

Computer Vision Lab, Huawei Research Center Zurich, Switzerland

Image Super-Resolution State Space Models

Jul 8, 2024cs.CV

Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learning

Few-shot class-incremental learning (FSCIL) aims to incrementally learn novel classes from limited examples while preserving knowledge of previously learned classes. Existing methods face a critical dilemma: static architectures rely on a constant parameter space to learn from data that arrive sequentially, making them prone to overfitting to the current session, while dynamic architectures continually expand the parameter space, leading to increased complexity. In this study, we explore the potential of Selective State Space Models (SSMs) for FSCIL. Mamba leverages its input-dependent parameters to dynamically adjust its processing patterns and generate content-aware scan patterns without session-wise projector expansion. This enables it to configure distinct processing for base and novel classes, helping preserve existing knowledge while adapting to new ones. To leverage Mamba's potential for FSCIL, we design two key modules: First, we propose a dual selective SSM projector that generates input-conditioned state-space parameters from intermediate features for dynamic adaptation. The dual design structurally decouples base and novel-class processing, employing a frozen base branch to maintain stable base-class features and a dynamic incremental branch that adaptively learns distinctive feature shifts for novel classes. Second, we develop a class-sensitive selective scan mechanism to guide dynamic adaptation of the incremental branch. It reduces the disruption to base-class representations caused by training on novel data, and meanwhile, encourages the selective scan to perform in distinct patterns between base and novel classes. Extensive experiments on miniImageNet, CIFAR-100, and CUB-200 demonstrate that Mamba-FSCIL achieves state-of-the-art performance.

Xiaojie Li, Yibo Yang, Jianlong Wu +4

School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · King Abdullah University of Science and Technology, Jeddah, Saudi Arabia +1

Class-Incremental Learning Incremental Learning Few-Shot Image Classification State Space Models

Mar 12, 2024cs.CV

SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spaces

Given the remarkable achievements in image generation through diffusion models, the research community has shown increasing interest in extending these models to video generation. Recent diffusion models for video generation have predominantly utilized attention layers to extract temporal features. However, attention layers are limited by their computational costs, which increase quadratically with the sequence length. This limitation presents significant challenges when generating longer video sequences using diffusion models. To overcome this challenge, we propose leveraging state-space models (SSMs) as temporal feature extractors. SSMs (e.g., Mamba) have recently gained attention as promising alternatives due to their linear-time memory consumption relative to sequence length. In line with previous research suggesting that using bidirectional SSMs is effective for understanding spatial features in image generation, we found that bidirectionality is also beneficial for capturing temporal features in video data, rather than relying on traditional unidirectional SSMs. We conducted comprehensive evaluations on multiple long-term video datasets, such as MineRL Navigate, across various model sizes. For sequences up to 256 frames, SSM-based models require less memory to achieve the same FVD as attention-based models. Moreover, SSM-based models often deliver better performance with comparable GPU memory usage. Our codes are available at https://github.com/shim0114/SSM-Meets-Video-Diffusion-Models.

Yuta Oshima, Shohei Taniguchi, Masahiro Suzuki +1

The University of Tokyo

Video Diffusion Models Long-Horizon Video Generation Diffusion Model Acceleration State Space Models