State Space Models

Also known as SSM

Follow this topic Ask about this topic

State space models describe how hidden states evolve over time and generate observations, supporting both probabilistic system identification and neural sequence learning. Approaches range from Kalman filtering, smoothing, and particle or variational inference to structured S4/HiPPO models, selective Mamba scans, and hybrids that combine recurrence with attention. They are applied to long-context language and video, time-series forecasting, and signals such as EEG, with evaluation spanning retrieval and sequence benchmarks as well as real-world prediction tasks. Their recurrent structure can offer linear-time processing and bounded memory, but efficient computation does not guarantee reliable long-range retrieval or state tracking, and practical speed, robustness, and accuracy remain important trade-offs.

Latest papers

299 papers30 papers in the last four weeks, down 12% on the four weeks before. 0.2% of all new papers.

Mar 17, 2026

RASLF: Representation-Aware State Space Model for Light Field Super-Resolution

Current SSM-based light field super-resolution (LFSR) methods often fail to fully leverage the complementarity among various LF representations, leading to the loss of fine textures and geometric misalignments across views. To address these issues, we propose RASLF, a representation-aware state-space framework that explicitly models structural correlations across multiple LF representations. Specifically, a Progressive Geometric Refinement (PGR) block is created that uses a panoramic epipolar representation to explicitly encode multi-view parallax differences, thereby enabling integration across different LF representations. Furthermore, we introduce a Representation Aware Asymmetric Scanning (RAAS) mechanism that dynamically adjusts scanning paths based on the physical properties of different representation spaces, optimizing the balance between performance and efficiency through path pruning. Additionally, a Dual-Anchor Aggregation (DAA) module improves hierarchical feature flow, reducing redundant deeplayer features and prioritizing important reconstruction information. Experiments on various public benchmarks show that RASLF achieves the highest reconstruction accuracy while remaining highly computationally efficient.

Zeqiang Wei, Kai Jin, Kuan Song +3

Capital Normal University Information Engineering College, Beijing 100048, China · Bigo Technology Pte. Ltd., Beijing 100020, China · Explorer Global (Suzhou) Artificial Intelligence Technology Co., Ltd., Suzhou, Jiangsu 215123, China +1

Image Super-Resolution State Space Models

Mar 17, 2026

NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling

Spiking neural networks promise brain-inspired neural dynamics, yet native large-scale spiking language-model pretraining remains largely unexplored relative to conversion, distillation, and backbone-adaptation approaches. We diagnose a central obstacle: standard spiking neurons apply the same firing sensitivity to every token, creating dense- or silent-firing regimes that are difficult to resolve with globally tuned parameters alone. This problem is particularly acute in language modeling, where a single sequence contains tokens of vastly different informational roles, making a uniform firing regime poorly matched to the data. We address this with NeuronSpark, a self-attention-free language model built around input-dependent spiking membrane dynamics, where each neuron's decay rate, gain, and firing threshold adapt to the current token content. Together with gated associative memory and adaptive temporal aggregation, this selectivity enables billion-scale pretraining from random initialization without backbone transfer, conversion, or distillation. We relate the neuron to fixed-parameter spiking models and the memory to linear attention. At 1.1B parameters and 2.1B tokens, NeuronSpark reaches a 40.0 eight-task zero-shot average, exceeds comparable-budget Pythia/Mamba checkpoints, retains 95.2%/93.2% of their fully trained scores, and spontaneously develops a fast--slow hierarchy. At 2,048 tokens, parameter-matched VHDL points reduce operation-energy proxy by 90.1%/67.0% and cycles by 94.1%/80.3% versus Transformer/Mamba; a physical FPGA path reaches 24.000s TTFT and 0.082440 TPS.

Zhengzheng Tang

Boston University

Spiking Language Models Spiking Neural Networks Language Model Pretraining Language Modeling Efficient Language Model Inference State Space Models

Mar 16, 2026

Mamba-3: Improved Sequence Modeling using State Space Principles

Scaling inference-time compute has emerged as an important driver of LLM performance, making inference efficiency a central focus of model design alongside model quality. While the current Transformer-based models deliver strong model quality, their quadratic compute and linear memory make inference expensive. This has spurred the development of sub-quadratic models with reduced linear compute and constant memory requirements. However, many recent linear models trade off model quality and capability for algorithmic efficiency, failing on tasks such as state tracking. Moreover, their theoretically linear inference remains hardware-inefficient in practice. Guided by an inference-first perspective, we introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models. We combine: (1) a more expressive recurrence derived from SSM discretization, (2) a complex-valued state update rule that enables richer state tracking, and (3) a multi-input, multi-output (MIMO) formulation for better model performance without increasing decode latency. Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with Mamba-3's MIMO variant further improving accuracy by another 1.2 points for a total 1.8 point gain. Across state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half of its predecessor's state size. Our evaluations demonstrate Mamba-3's ability to advance the performance-efficiency Pareto frontier.

Aakash Lahoti, Kevin Y. Li, Berlin Chen +5

Carnegie Mellon University · Princeton University · Together AI +1

State Space Language Models Sequence Modeling Efficient Language Model Inference State Space Models State Tracking

Mar 14, 2026

Enhancing Eye Feature Estimation from Event Data Streams through Adaptive Inference State Space Modeling

Eye feature extraction from event-based data streams can be performed efficiently and with low energy consumption, offering great utility to real-world eye tracking pipelines. However, few eye feature extractors are designed to handle sudden changes in event density caused by the changes between gaze behaviors that vary in their kinematics, leading to degraded prediction performance. In this work, we address this problem by introducing the adaptive inference state space model (AISSM), a novel architecture for feature extraction that is capable of dynamically adjusting the relative weight placed on current versus recent information. This relative weighting is determined via estimates of the signal-to-noise ratio and event density produced by a complementary dynamic confidence network. Lastly, we craft and evaluate a novel learning technique that improves training efficiency. Experimental results demonstrate that the AISSM system outperforms state-of-the-art models for event-based eye feature extraction.

Viet Dung Nguyen, Mobina Ghorbaninejad, Chengyi Ma +5

Rochester Institute of Technology · Meta Reality Labs

Eye Tracking Event-Based Vision Adaptive Inference State Space Models

Mar 14, 2026

Deep Learning for Virtual Reality User Identification: A Benchmark

Virtual Reality (VR) applications require robust user identification systems to ensure secure access to equipment and protect worker identities. Motion tracking data from VR headsets and controllers has emerged as a powerful behavioral biometric, with recent studies demonstrating identification accuracies exceeding 94% across a large user base. However, the application of modern deep learning architectures, particularly State Space Models (SSM), to VR scenarios remains largely unexplored. In this work, we benchmark user identification performance across the large-scale Who is Alyx VR dataset, gathering data from 71 users playing the popular Half-Life:Alyx game. We evaluate both established architectures (Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), Convolutional Neural Network (CNN), Temporal Convolutional Network (TCN), Transformer) and the emerging SSMs on time series motion data. Our results provide the first comprehensive benchmark of state-of-the-art and novel architectures for VR user identification, establishing baseline performance metrics for future privacy preserving authentication systems in manufacturing environments.

Davide Frizzo, Fabrizio Genilotti, David Petrovic +6

University of Padova, Italy

Biometric Identification Virtual Reality State Space Models

Mar 13, 2026

State-space models through the lens of ensemble control

State-space models (SSMs) are effective architectures for sequential modeling, but a rigorous theoretical understanding of their training dynamics is still lacking. We formulate continuous-time SSM parameter-path optimization as an ensemble optimal control problem: each input sequence generates a corresponding state trajectory in the ensemble, while the trainable parameter path forms a common control shared by all trajectories. Within this formulation, model evaluation is represented by the forward state equation and backward sensitivity propagation by the corresponding adjoint equation. We show that the Hamiltonian gradient with respect to the control variable represents the negative first-variation density of the reduced ensemble objective. Using this representation, we analyze a Bregman mirror-descent scheme for the continuous-time ensemble objective; in Euclidean geometry this reduces to functional projected gradient descent. State-adjoint and Hamiltonian-gradient stability yield two-sided relative curvature of the reduced objective and relative convexity under sufficient regularization. We further show that in the relatively convex regime the optimal-control problem admits a minimizer, which is unique under relative strong convexity. For SSMs, the explicit stability-generated defect

κ_{\mathrm{stab}}^{\mathrm{SSM}}

gives an

O(1/k)

objective rate at

τ=κ_{\mathrm{stab}}^{\mathrm{SSM}}

and geometric convergence to the unique minimizer for

τ>κ_{\mathrm{stab}}^{\mathrm{SSM}}

, under the stated stepsize condition.

Ye Feng, Jianfeng Lu

Mirror Descent Optimal Control State Space Models

Mar 13, 2026

Robust Automatic Differentiation of Square-Root Kalman Filters via Gramian Differentials

Square-root Kalman filters propagate state covariances in Cholesky-factor form for numerical stability, and are a natural target for gradient-based parameter learning in state-space models. Their core operation, triangularization of a matrix

M \in \mathbb{R}^{n \times m}

, is computed via a QR decomposition in practice, but naively differentiating through it causes two problems: the semi-orthogonal factor is non-unique when

m > n

, yielding undefined gradients; and the standard Jacobian formula involves inverses, which diverges when

M

is rank-deficient. Both are resolved by the observation that all filter outputs relevant to learning depend on the input matrix only through the Gramian

MM^\top

, so the composite loss is smooth in

M

even where the triangularization is not. We derive a closed-form chain-rule directly from the differential of this Gramian identity, prove it exact for the Kalman log-marginal likelihood and filtered moments, and extend it to rank-deficient inputs via a two-component decomposition: a column-space term based on the Moore--Penrose pseudoinverse, and a null-space correction for perturbations outside the column space of

M

.

Adrien Corenflos

Department of Statistics, University of Warwick, Coventry, UK

Automatic Differentiation Kalman Filtering State Space Models

Mar 13, 2026

From Gradients to Riccati Geometry: Kalman World Models for Single-Pass Learning

Backpropagation dominates modern machine learning, yet it is not the only principled method for optimizing dynamical systems. We propose Kalman World Models (KWM), a class of learned state-space models trained via recursive Bayesian filtering rather than reverse-mode automatic differentiation. Instead of gradient descent updates, we replace parameter learning with Kalman-style gain adaptation. Training becomes online filtering; error signals become innovations. We further extend this framework to transformer-based large language models (LLMs), where internal activations are treated as latent dynamical states corrected via innovation terms. This yields a gradient-free training and adaptation paradigm grounded in control theory. We derive stability conditions, analyze computational complexity, and provide empirical results on sequence modeling tasks demonstrating competitive performance with improved robustness and continual adaptation properties.

Andrew Kiruluta

UC Berkeley School of Information

Gradient-Free Optimization State Space Models World Model Learning Kalman Filtering Continual Learning for LLMs

Mar 12, 2026

SpectralGuard: Detecting Memory Collapse Attacks in State Space Models

State Space Models (SSMs) such as Mamba achieve linear-time sequence processing through input-dependent recurrence, but this mechanism introduces a critical safety vulnerability. We show that the spectral radius rho(A-bar) of the discretized transition operator governs effective memory horizon: when an adversary drives rho toward zero through gradient-based Hidden State Poisoning, memory collapses from millions of tokens to mere dozens, silently destroying reasoning capacity without triggering output-level alarms. We prove an Evasion Existence Theorem showing that for any output-only defense, adversarial inputs exist that simultaneously induce spectral collapse and evade detection, then introduce SpectralGuard, a real-time monitor that tracks spectral stability across all model layers. SpectralGuard achieves F1=0.961 against non-adaptive attackers and retains F1=0.842 under the strongest adaptive setting, with sub-15ms per-token latency. Causal interventions and cross-architecture transfer to hybrid SSM-Attention systems confirm that spectral monitoring provides a principled, deployable safety layer for recurrent foundation models.

Davi Bonetto

Independent Researcher

LLM Security State Space Models

Mar 12, 2026

CLASP: Defending Hybrid Large Language Models Against Hidden State Poisoning Attacks

State space models (SSMs) like Mamba have gained significant traction as efficient alternatives to Transformers, achieving linear complexity while maintaining competitive performance. However, Hidden State Poisoning Attacks (HiSPAs), a recently discovered vulnerability that corrupts SSM memory through adversarial strings, pose a critical threat to these architectures and their hybrid variants. Framing the HiSPA mitigation task as a binary classification problem at the token level, we introduce the CLASP model (Classifier Against State Poisoning) to defend against this threat. CLASP exploits distinct patterns in Mamba's block output embeddings (BOEs) and uses an XGBoost classifier to identify malicious tokens with minimal computational overhead. We consider a realistic scenario in which both SSMs and HiSPAs are likely to be used: an LLM screening résumés to identify the best candidates for a role. Evaluated on a corpus of 2,483 résumés totaling 9.5M tokens with controlled injections, CLASP achieves 95.9% token-level F1 score and 99.3% document-level F1 score on malicious tokens detection. Crucially, the model generalizes to unseen attack patterns: under leave-one-out cross-validation, performance remains high (96.9% document-level F1), while under clustered cross-validation with structurally novel triggers, it maintains useful detection capability (91.6% average document-level F1). Operating independently of any downstream model, CLASP processes 1,032 tokens per second with under 4GB VRAM consumption, potentially making it suitable for real-world deployment as a lightweight front-line defense for SSM-based and hybrid architectures. All code and detailed results are available at https://anonymous.4open.science/r/hispikes-91C0.

Alexandre Le Mercier, Thomas Demeester, Chris Develder

IDLab–T2K, Ghent University–imec

LLM Security Language Model Robustness State Space Models

Mar 12, 2026

SEMamba++: A General Speech Restoration Framework Leveraging Global, Local, and Periodic Spectral Patterns

General speech restoration demands techniques that can interpret complex speech structures under various distortions. While State-Space Models like SEMamba have advanced the state-of-the-art in speech denoising, they are not inherently optimized for critical speech characteristics, such as spectral periodicity or multi-resolution frequency analysis. In this work, we introduce an architecture tailored to incorporate speech-specific features as inductive biases. In particular, we propose the Global, Local, and Periodic (GLP) module, a frequency feature extraction block that effectively and efficiently leverages the properties of frequency bins. Then, we design a multi-resolution parallel time-frequency dual-processing block to capture diverse spectral patterns, and a learnable mapping to further enhance model performance. With all our ideas combined, the proposed SEMamba++ achieves the best performance among multiple baseline models while remaining computationally efficient.

Yongjoon Lee, Jung-Woo Choi

Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea

Speech Enhancement Frequency-Domain Feature Learning State Space Models

Mar 11, 2026

Single molecule localization microscopy challenge: a biologically inspired benchmark for long-sequence modeling

State space models (SSMs) have recently achieved strong performance on long sequence modeling tasks while offering improved memory and computational efficiency compared to transformer based architectures. However, their evaluation has been largely limited to synthetic benchmarks and application domains such as language and audio, leaving their behavior on sparse and stochastic temporal processes in biological imaging unexplored. In this work, we introduce the Single Molecule Localization Microscopy Challenge (SMLM-C), a benchmark dataset consisting of ten SMLM simulations spanning dSTORM and DNA-PAINT modalities with varying hyperparameter designed to evaluate state space models on biologically realistic spatiotemporal point process data with known ground truth. Using a controlled subset of these simulations, we evaluate state space models and find that performance degrades substantially as temporal discontinuity increases, revealing fundamental challenges in modeling heavy-tailed blinking dynamics. These results highlight the need for sequence models better suited to sparse, irregular temporal processes encountered in real world scientific imaging data.

Fatemeh Valeh, Monika Farsang, Radu Grosu +1

Department of Biophysics, Technische Universität Wien, Vienna, Austria · Department of Informatics, Technische Universität Wien, Vienna, Austria

Sequence Modeling State Space Models Irregular Time-Series Modeling Microscopy Image Analysis

Mar 10, 2026

Compiler-First State Space Duality and Portable $O(1)$ Autoregressive Caching for Inference

High-throughput Mamba-2 inference is usually tied to fused CUDA and Triton kernels, limiting portability across accelerator backends. We show that the state space duality (SSD) recurrence has a compiler-friendly structure: diagonal per-head dynamics, fixed-size chunking, einsum-dominated compute, and static control flow. Expressing this structure in standard JAX primitives gives a single-source inference path with no custom kernels, a registered JAX PyTree cache, and a compiled on-device autoregressive loop. On a single Google Cloud TPU v6e, batch-1 prefill reaches approximately 140 TFLOPS, or 15% model FLOP utilisation (MFU), the roofline ceiling for this regime, and cached decode reaches up to 64% hardware bandwidth utilisation (HBU). At a 4096-token context, cached decode is 27x--36x faster than full-prefix recomputation across five Mamba-2 checkpoints from 130M to 2.7B parameters. The same source runs unmodified on NVIDIA L40S, where cached decode remains sequence-length independent across all model scales. WikiText-103 validation perplexity matches the Triton reference mamba_ssm v2.2.2 within +/-0.0005 points, and hidden states agree to float32 rounding tolerance. Code is available at https://github.com/CosmoNaught/mamba2-jax.

Cosmo Santoni, Anmol Thapar

Imperial College London

Efficient Inference LLM Inference Acceleration Mamba State Space Models

Mar 7, 2026

Kinematics-Aware Latent World Models for Data-Efficient Autonomous Driving

Data-efficient learning remains a central challenge in autonomous driving due to the high cost and safety risks of large-scale real-world interaction. Although world-model-based reinforcement learning enables policy optimization through latent imagination, existing approaches often lack explicit mechanisms to encode spatial and kinematic structure essential for driving tasks. In this work, we build upon the Recurrent State-Space Model (RSSM) and propose a kinematics-aware latent world model framework for autonomous driving. Vehicle kinematic information is incorporated into the observation encoder to ground latent transitions in physically meaningful motion dynamics, while geometry-aware supervision regularizes the RSSM latent state to capture task-relevant spatial structure beyond pixel reconstruction. The resulting structured latent dynamics improve long-horizon imagination fidelity and stabilize policy optimization. Experiments in a driving simulation benchmark demonstrate consistent gains over both model-free and pixel-based world-model baselines in terms of sample efficiency and driving performance. Ablation studies further verify that the proposed design enhances spatial representation quality within the latent space. These results suggest that integrating kinematic grounding into RSSM-based world models provides a scalable and physically grounded paradigm for autonomous driving policy learning.

Jiazhuo Li, Linjiang Cao, Qi Liu +1

Key Laboratory of Road and Traffic Engineering, Ministry of Education, Tongji University, Shanghai, China

World Models for Autonomous Driving World Model Learning Autonomous Driving Model-Based RL State Space Models Autonomous Driving Simulation

Mar 7, 2026

Permutation-Equivariant 2D State Space Models: Theory and Canonical Architecture for Multivariate Time Series

Multivariate time series (MTS) modeling often implicitly imposes an artificial ordering over variables, violating the inherent exchangeability found in many real-world systems where no canonical variable axis exists. We formalize this limitation as a violation of the permutation symmetry principle and require state-space dynamics to be permutation-equivariant along the variable axis. In this work, we theoretically characterize the complete canonical form of linear variable coupling under this symmetry constraint. We prove that any permutation-equivariant linear 2D state-space system naturally decomposes into local self-dynamics and a global pooled interaction, rendering ordered recurrence not only unnecessary but structurally suboptimal. Motivated by this theoretical foundation, we introduce the Variable-Invariant Two-Dimensional State Space Model (VI 2D SSM), which realizes the canonical equivariant form via permutation-invariant aggregation. This formulation eliminates sequential dependency chains along the variable axis, reducing the dependency depth from

\mathcal{O}(C)

to

\mathcal{O}(1)

and simplifying stability analysis to two scalar modes. Furthermore, we propose VI 2D Mamba, a unified architecture integrating multi-scale temporal dynamics and spectral representations. Extensive experiments on forecasting, classification, and anomaly detection benchmarks demonstrate that our model achieves state-of-the-art performance with superior structural scalability, validating the theoretical necessity of symmetry-preserving 2D modeling.

Seungwoo Jeong, Heung-Il Suk

Department of Artificial Intelligence, Korea University, Seoul 02841, Republic of Korea

State Space Models Multivariate Time Series Forecasting Multivariate Time Series Classification

Mar 6, 2026

Swimba: Switch Mamba Model Scales State Space Models

Mixture-of-experts (MoE) is a common approach for increasing parameter capacity, but applying MoE to state space model (SSM) token mixers can multiply the cost of the recurrent state update. We study how to introduce expert specialization into selective SSMs while preserving computational efficiency. We show that MoE--SSM can refer to two designs: (1) MoE over separated SSMs, which maintains multiple state trajectories and thus scales compute with the number of experts; and (2) MoE-parameterized SSM, which mixes experts in parameter space, maintains a single state trajectory, and evaluates the recurrence once. Our method, Switch Mamba (Swimba), follows the second design by routing over expert-produced SSM streams. Theoretically, we establish well-definedness and stability for MoE-parameterized SSMs and characterize the relationship between the two designs. Empirically, we evaluate Swimba on standard benchmark tasks and measure real-time throughput and latency. Under matched FLOPs, Swimba achieves slightly better average performance than the baseline, with a small slowdown in real-time latency and throughput. Overall, these results suggest that parameter-space MoE can increase SSM capacity while keeping the dominant recurrence cost fixed.

Zhixu Du, Krishna Teja Chitty-Venkata, Murali Emani +3

Duke University · Red Hat, Inc. · Argonne National Laboratory

State Space Models Selective SSMs Mixture-of-Experts Models Sparse Mixture-of-Experts

Mar 6, 2026

DLRMamba: Distilling Low-Rank Mamba for Edge Multispectral Fusion Object Detection

Multispectral fusion object detection is a critical task for edge-based maritime surveillance and remote sensing, demanding both high inference efficiency and robust feature representation for high-resolution inputs. However, current State Space Models (SSMs) like Mamba suffer from significant parameter redundancy in their standard 2D Selective Scan (SS2D) blocks, which hinders deployment on resource-constrained hardware and leads to the loss of fine-grained structural information during conventional compression. To address these challenges, we propose the Low-Rank Two-Dimensional Selective Structured State Space Model (Low-Rank SS2D), which reformulates state transitions via matrix factorization to exploit intrinsic feature sparsity. Furthermore, we introduce a Structure-Aware Distillation strategy that aligns the internal latent state dynamics of the student with a full-rank teacher model to compensate for potential representation degradation. This approach substantially reduces computational complexity and memory footprint while preserving the high-fidelity spatial modeling required for object recognition. Extensive experiments on five benchmark datasets and real-world edge platforms, such as Raspberry Pi 5, demonstrate that our method achieves a superior efficiency-accuracy trade-off, significantly outperforming existing lightweight architectures in practical deployment scenarios.

Qianqian Zhang, Leon Tabaro, Ahmed M. Abdelmoniem +1

National Space Science Center, Chinese Academy of Sciences, Beijing, 101499, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, 100049, China · School of Electronic Engineering and Computer Science, Queen Mary University of London, London, E1 4NS, UK +1

Efficient Object Detection State Space Models Multispectral Object Detection Edge Inference Low-Rank Compression Knowledge Distillation

Mar 5, 2026

Warm Starting State-Space Models with Automata Learning

We prove that Moore machines can be exactly realized as state-space models (SSMs), establishing a formal correspondence between symbolic automata and these continuous machine learning architectures. These Moore-SSMs preserve both the complete symbolic structure and input-output behavior of the original Moore machine, but operate in Euclidean space. With this correspondence, we compare the training of SSMs with both passive and active automata learning. In recovering automata from the SYNTCOMP benchmark, we show that SSMs require orders of magnitude more data than symbolic methods and fail to learn state structure. This suggests that symbolic structure provides a strong inductive bias for learning these systems. We leverage this insight to combine the strengths of both automata learning and SSMs in order to learn complex systems efficiently. We learn an adaptive arbitration policy on a suite of arbiters from SYNTCOMP and show that initializing SSMs with symbolically-learned approximations learn both faster and better. We see 2-5 times faster convergence compared to randomly initialized models and better overall model accuracies on test data. Our work lifts automata learning out of purely discrete spaces, enabling principled exploitation of symbolic structure in continuous domains for efficiently learning in complex settings.

William Fishell, Sam Nicholas Kouteili, Mark Santolucito

Columbia University · Yale University · Barnard College, Columbia University

Automata Learning State Space Models Neurosymbolic Learning

Mar 5, 2026

BLINK: Behavioral Latent Modeling of NK Cell Cytotoxicity

Machine learning models of cellular interaction dynamics hold promise for understanding cell behavior. Natural killer (NK) cell cytotoxicity is a prominent example of such interaction dynamics and is commonly studied using time-resolved multi-channel fluorescence microscopy. Although tumor cell death events can be annotated at single frames, NK cytotoxic outcome emerges over time from cellular interactions and cannot be reliably inferred from frame-wise classification alone. We introduce BLINK, a trajectory-based recurrent state-space model that serves as a cell world model for NK-tumor interactions. BLINK learns latent interaction dynamics from partially observed NK-tumor interaction sequences and predicts apoptosis increments that accumulate into cytotoxic outcomes. Experiments on long-term time-lapse NK-tumor recordings show improved cytotoxic outcome detection and enable forecasting of future outcomes, together with an interpretable latent representation that organizes NK trajectories into coherent behavioral modes and temporally structured interaction phases. BLINK provides a unified framework for quantitative evaluation and structured modeling of NK cytotoxic behavior at the single-cell level.

Iman Nematollahi, Jose Francisco Villena-Ossa, Alina Moter +6

Department of Computer Science, University of Freiburg, Freiburg, Germany · Institute for Transfusion Medicine and Gene Therapy, University Medical Center Freiburg, Freiburg, Germany · Goethe University, Department of Pediatrics, Experimental Immunology and Cell Therapy, Frankfurt am Main, Germany +2

Toxicity Detection Latent Dynamics Modeling State Space Models

Mar 5, 2026

DeformTrace: A Deformable State Space Model with Relay Tokens for Temporal Forgery Localization

Temporal Forgery Localization (TFL) aims to precisely identify manipulated segments in video and audio, offering strong interpretability for security and forensics. While recent State Space Models (SSMs) show promise in precise temporal reasoning, their use in TFL is hindered by ambiguous boundaries, sparse forgeries, and limited long-range modeling. We propose DeformTrace, which enhances SSMs with deformable dynamics and relay mechanisms to address these challenges. Specifically, Deformable Self-SSM (DS-SSM) introduces dynamic receptive fields into SSMs for precise temporal localization. To further enhance its capacity for temporal reasoning and mitigate long-range decay, a Relay Token Mechanism is integrated into DS-SSM. Besides, Deformable Cross-SSM (DC-SSM) partitions the global state space into query-specific subspaces, reducing non-forgery information accumulation and boosting sensitivity to sparse forgeries. These components are integrated into a hybrid architecture that combines the global modeling of Transformers with the efficiency of SSMs. Extensive experiments show that DeformTrace achieves state-of-the-art performance with fewer parameters, faster inference, and stronger robustness.

Xiaodong Zhu, Suting Wang, Yuanming Zheng +5

National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, China

Hybrid Transformer-State Space Model Architectures Temporal Video Forgery Localization Video Forensics Digital Forensics State Space Models Deepfake Detection

Mar 4, 2026

Mask-aware inference with State-Space Models

Many real-world computer vision tasks, such as depth completion, must handle inputs with arbitrarily shaped regions of missing or invalid data. For Convolutional Neural Networks (CNNs), Partial Convolutions solved this by a mask-aware re-normalization conditioned only on valid pixels. Recently, State Space Models (SSMs) like Mamba have emerged, offering high performance with linear complexity. However, these architectures lack an inherent mechanism for handling such arbitrarily shaped invalid data at inference time. To bridge this gap, we introduce Partial Vision Mamba (PVM), a novel architectural component that ports the principles of partial operations to the Mamba backbone. We also define a series of rules to design architectures using PVM. We show the efficacy and generalizability of our approach in the tasks of depth completion, image inpainting, and classification with invalid data.

Ignasi Mas, Ramon Morros, Javier-Ruiz Hidalgo +1

Visual SSMs State Space Models Learning with Missing Data

Mar 4, 2026

Architectural Proprioception in State Space Models: Thermodynamic Training Induces Anticipatory Halt Detection

We introduce the Probability Navigation Architecture (PNA) framework, which treats neural computation as navigation through a probability manifold governed by thermodynamic principles. We train State Space Models (SSMs) and Transformers with a novel thermodynamic loss function that penalizes computational waste alongside standard cross-entropy. Across 19 experimental phases, we discover that thermodynamically-trained SSMs develop architectural proprioception: a strong anticipatory coupling between recurrent state entropy and halt confidence (r = -0.836, p < 0.001) in which the halt signal leads state entropy collapse by exactly two tokens (tau = -2.0). This Universal Stopping Signature (USS) reproduces to four decimal places across random seeds and generalizes to a structurally distinct sorting task. Critically, Transformers trained identically show no such coupling (r = -0.07), demonstrating that the phenomenon is architecture-dependent. Cross-task transfer experiments confirm that SSM halt detection reflects genuine meta-cognition (zero-shot transfer F1: SSMs 64.2% vs. Transformers 69.3%; post-adaptation: SSMs 94.5% vs. Transformers 86.4%), while Transformer halt detection relies on syntactic pattern matching. A 2D hyperparameter sweep over energy penalty (alpha) and halt supervision (beta) reveals that the anticipatory coupling is continuously controllable through training, with thermodynamic pressure serving as the primary induction mechanism and explicit halt supervision as an amplifier. Our results establish that SSMs are thermodynamically native architectures whose fixed-size recurrent states naturally support the Markovian compression that enables computational self-awareness, with implications for cost-aware inference, dynamic token budgets, and confidence-based routing in production systems.

Jay Noon

State Space Models Metacognition in Language Models Adaptive Computation

Mar 3, 2026

MaBERT:A Padding Safe Interleaved Transformer Mamba Hybrid Encoder for Efficient Extended Context Masked Language Modeling

Self attention encoders such as Bidirectional Encoder Representations from Transformers(BERT) scale quadratically with sequence length, making long context modeling expensive. Linear time state space models, such as Mamba, are efficient; however, they show limitations in modeling global interactions and can suffer from padding induced state contamination. We propose MaBERT, a hybrid encoder that interleaves Transformer layers for global dependency modeling with Mamba layers for linear time state updates. This design alternates global contextual integration with fast state accumulation, enabling efficient training and inference on long inputs. To stabilize variable length batching, we introduce paddingsafe masking, which blocks state propagation through padded positions, and mask aware attention pooling, which aggregates information only from valid tokens. On GLUE, MaBERT achieves the best mean score on five of the eight tasks, with strong performance on the CoLA and sentence pair inference tasks. When extending the context from 512 to 4,096 tokens, MaBERT reduces training time and inference latency by 2.36x and 2.43x, respectively, relative to the average of encoder baselines, demonstrating a practical long context efficient encoder.

Jinwoong Kim, Sangjin Park

Graduate School of Industrial Data Engineering, Hanyang University, Seoul, Republic of Korea

Hybrid Transformer-State Space Model Architectures Long-Context Language Modeling Hybrid Mamba-Transformer Architectures Sequence Modeling Long-Context Language Model Inference Masked Language Modeling+1

Mar 3, 2026

Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures

Transformers excel at in-context retrieval but suffer from quadratic complexity with sequence length, while State Space Models (SSMs) offer efficient linear-time processing but have limited retrieval capabilities. We investigate whether hybrid architectures combining Transformers and SSMs can achieve the best of both worlds on two synthetic in-context retrieval tasks. The first task, n-gram retrieval, requires the model to reproduce an n-gram that succeeds the query within the input sequence. The second task, position retrieval, presents the model with a query token and requires it to perform a two-hop lookup: first locating the corresponding element in the sequence, and then outputting its positional index. Under controlled conditions, we assess data efficiency, length generalization, robustness to out of domain training examples, and learned representations across Transformers, SSMs, and hybrid architectures. We find that hybrid models outperform SSMs and match or exceed Transformers in terms of data efficiency and extrapolation for tasks that require precise information retrieval from the input context. However, Transformers maintain superiority in position retrieval tasks. Through representation analysis, we discover that SSM-based models develop locality-aware embeddings where tokens representing adjacent positions become neighbors in embedding space, forming interpretable structures. This property is absent in Transformers as causal attention is sufficient for acquiring positional associations, and the introduction of positional encoding amplifies this behavior, leading to improvement in data efficiency. SSMs on the other hand update their internal representations incrementally and without positional encodings, are required to learn these associations. Our findings reveal fundamental differences in how Transformers and SSMs, and hybrid models learn positional associations.

Georgios Pantazopoulos, Malvina Nikandrou, Ioannis Konstas +1

University of Edinburgh · Heriot-Watt University

In-Context Retrieval Transformer Hybrid Transformer-State Space Model Architectures State Space Models

Mar 2, 2026

The Expressive Limits of Diagonal SSMs for State-Tracking

State-Space Models (SSMs) have recently been shown to achieve strong empirical performance on a variety of long-range sequence modeling tasks while remaining efficient and highly-parallelizable. However, the theoretical understanding of their expressive power remains limited. In this work, we study the expressivity of input-Dependent Complex-valued Diagonal (DCD) SSMs on sequential state-tracking tasks. We show that single-layer DCD SSMs cannot express state-tracking of any non-Abelian group at finite precision. More generally, we show that

k

-layer DCD SSMs can express state-tracking of a group if and only if that group has a subnormal series of length

k

, with Abelian factors. That is, we identify the precise expressivity range of

k

-layer DCD SSMs within the solvable groups. Empirically, we find that multi-layer models often fail to learn state-tracking for non-Abelian groups, highlighting a gap between expressivity and learnability.

Mehran Shakerinava, Behnoush Khavari, Siamak Ravanbakhsh +1

School of Computer Science, McGill University · Mila – Quebec AI Institute · Chandar Research Lab +1

State Space Models State Tracking

Mar 2, 2026

TacMamba: A Tactile History Compression Adapter Bridging Fast Reflexes and Slow VLA Reasoning

In visually ambiguous manipulation such as detecting button click tactile feedback is often the sole source of ground truth. However, fusing tactile data poses a significant challenge due to a spatiotemporal mismatch: tactile perception requires high-frequency processing with long-horizon memory (System 1), whereas visual policies operate at low control frequencies (System 2). Existing architectures struggle to bridge this gap: Transformers are computationally prohibitive for high-frequency loops (>100Hz), while LSTMs suffer from forgetting over extended interaction histories. In this paper, we introduce TacMamba, a hierarchical architecture that aligns high-bandwidth tactile reflexes with low-frequency visual planning. Our approach comprises three core contributions: (1) a custom high-frequency tactile interface designed for flexible integration; (2) a Mamba-based Tactile History Compressor that encodes continuous force history into a compact state with O(1) inference latency (0.45 ms), enabling plug-and-play fusion with VLA models without joint pre-training and (3) a Tactile-Guided Dual-Stage Training strategy that leverages temporal discrimination for self-supervised representation learning and phase-uniform sampling to mitigate data sparsity. Experiments on discrete counting and implicit state switching demonstrate that TacMamba achieves 100% success rates, significantly outperforming the visual-only pi_0.5 baseline, while strictly satisfying hard real-time constraints.

Zhenan Wang, Yanzhe Wang, Meixuan Ren +8

Zhejiang University, Hangzhou, China · GigaAI, Beijing, China · Harbin Institute of Technology, Harbin, China

Tactile Sensing for Robotic Manipulation Visuo-Tactile Representation Learning VLMs for Robotics State Space Models

Mar 1, 2026

Efficient Extractive Summarization with MAMBA-Transformer Hybrids for Low-Resource Scenarios

Extractive summarization of long documents is bottlenecked by quadratic complexity, often forcing truncation and limiting deployment in resource-constrained settings. We introduce the first Mamba-Transformer hybrid for extractive summarization, combining the semantic strength of pre-trained transformers with the linear-time processing of state space models. Leveraging Mamba's ability to process full documents without truncation, our approach preserves context while maintaining strong summarization quality. The architecture includes: (1) a transformer encoder for sentence-level semantics, (2) a Mamba state space model to capture inter-sentence dependencies efficiently, and (3) a linear classifier for sentence relevance prediction. Across news, argumentative, and scientific domains under low-resource conditions, our method achieves: (1) large gains over BERTSUM and MATCHSUM, including +0.23 ROUGE-1 on ArXiv and statistically significant improvements on all datasets (p < 0.001); (2) consistent advantages across domains, strongest on the longest documents; (3) robust performance with limited training data; and (4) 24-27% faster inference on news summarization (CNN/DailyMail). We introduce the first hybrid Transformer-state space architecture for summarization, showing significant ROUGE improvements in low-resource scenarios.

Nisrine Ait Khayi

Independent Researcher · University of Memphis

Hybrid Transformer-State Space Model Architectures Text Summarization Hybrid Mamba-Transformer Architectures Long-Context Modeling Sequence Modeling State Space Models

Mar 1, 2026

Improved MambdaBDA Framework for Robust Building Damage Assessment Across Disaster Domains

Reliable post-disaster building damage assessment (BDA) from satellite imagery is hindered by severe class imbalance, background clutter, and domain shift across disaster types and geographies. In this work, we address these problems and explore ways to improve the MambaBDA, the BDA network of ChangeMamba architecture, one of the most successful BDA models. The approach enhances the MambaBDA with three modular components: (i) Focal Loss to mitigate class imbalance damage classification, (ii) lightweight Attention Gates to suppress irrelevant context, and (iii) a compact Alignment Module to spatially warp pre-event features toward post-event content before decoding. We experiment on multiple satellite imagery datasets, including xBD, Pakistan Flooding, Turkey Earthquake, and Ida Hurricane, and conduct in-domain and crossdataset tests. The proposed modular enhancements yield consistent improvements over the baseline model, with 0.8% to 5% performance gains in-domain, and up to 27% on unseen disasters. This indicates that the proposed enhancements are especially beneficial for the generalization capability of the system.

Alp Eren Gençoğlu, Hazım Kemal Ekenel

Department of Computer Engineering, Istanbul Technical University, Türkiye · Division of Engineering, New York University Abu Dhabi, UAE

Disaster Damage Assessment Building Damage Assessment Domain Generalization State Space Models

Feb 27, 2026

BiM-GeoAttn-Net: Linear-Time Depth Modeling with Geometry-Aware Attention for 3D Aortic Dissection CTA Segmentation

Accurate segmentation of aortic dissection (AD) lumens in CT angiography (CTA) is essential for quantitative morphological assessment and clinical decision-making. However, reliable 3D delineation remains challenging due to limited long-range context modeling, which compromises inter-slice coherence, and insufficient structural discrimination under low-contrast conditions. To address these limitations, we propose BiM-GeoAttn-Net, a lightweight framework that integrates linear-time depth-wise state-space modeling with geometry-aware vessel refinement. Our approach is featured by Bidirectional Depth Mamba (BiM) to efficiently capture cross-slice dependencies and Geometry-Aware Vessel Attention (GeoAttn) module that employs orientation-sensitive anisotropic filtering to refine tubular structures and sharpen ambiguous boundaries. Extensive experiments on a multi-source AD CTA dataset demonstrate that BiM-GeoAttn-Net achieves a Dice score of 93.35% and an HD95 of 12.36 mm, outperforming representative CNN-, Transformer-, and SSM-based baselines in overlap metrics while maintaining competitive boundary accuracy. These results suggest that coupling linear-time depth modeling with geometry-aware refinement provides an effective, computationally efficient solution for robust 3D AD segmentation.

Yuan Zhang, Lei Liu, Jialin Zhang +3

College of Computer Science Sichuan Normal University Chengdu, Sichuan 610068, China · Zhejiang University & Ant Group Hangzhou, Zhejiang, China · State Key Laboratory of Public Big Data Guizhou University Guiyang, Guizhou 550025, China

Medical Image Segmentation 3D Medical Image Segmentation State Space Models Computed Tomography

Feb 26, 2026

Latent Matters: Learning Deep State-Space Models

Deep state-space models (DSSMs) enable temporal predictions by learning the underlying dynamics of observed sequence data. They are often trained by maximising the evidence lower bound. However, as we show, this does not ensure the model actually learns the underlying dynamics. We therefore propose a constrained optimisation framework as a general approach for training DSSMs. Building upon this, we introduce the extended Kalman VAE (EKVAE), which combines amortised variational inference with classic Bayesian filtering/smoothing to model dynamics more accurately than RNN-based DSSMs. Our results show that the constrained optimisation framework significantly improves system identification and prediction accuracy on the example of established state-of-the-art DSSMs. The EKVAE outperforms previous models w.r.t. prediction accuracy, achieves remarkable results in identifying dynamical systems, and can furthermore successfully learn state-space representations where static and dynamic features are disentangled.

Alexej Klushyn, Richard Kurle, Maximilian Soelch +2

Technical University of Munich · AWS AI Labs · Machine Learning Research Lab, Volkswagen Group, Munich +1

State Space Models Latent Dynamics Modeling Variational Autoencoders System Identification State Representation Learning