Neural state-space identification learns dynamical models from measured inputs and outputs. SUBNET combines this task with an encoder that estimates initial states from short measurement histories. Its continuous-time version requires many successive neural-network evaluations during numerical integration, resulting in slow training speed which is a major limitation for practical use cases. We propose deep structured state space models (DS3M), which combine compact nonlinear dynamical components in parallel and in series. Custom GPU kernels implemented with Triton reduce the launch and memory overhead of their sequential evaluation. We explain this construction, describe a port-Hamiltonian variant, and present various benchmark comparisons of accuracy and training time.
Figures & tables
Figure 1: A comparison of relevant training parameters for the standard machine learning task of training ResNet50 He et al. (2016) on Imagenet Russakovsky et al. (2015) and SUBNET Beintema et al. (2023b) on the Wiener-Hammerstein benchmark from Schoukens et al. (2009) ; Schoukens et al. (2026) .
Figure 2: DS3M architecture. Panel (A) shows the overall model pipeline. Here the encoder supplies initial states for all NH components. Each block propagates H CT-SUBNETS in parallel, mixes their states, and adds the result to its input features. The N blocks are connected in series.
Figure 3: Architecture for MLP that are used to parametrize fθ(j) in the DS3M blocks.
T
Baseline
Base [ms]
Triton [ms]
Speedup
100
CPU eager
154.651
1.850
83.61×
100
CPU compiled
52.854
1.850
28.58×
100
CUDA eager
280.963
1.850
151.91×
100
CUDA compiled
79.145
1.850
42.79×
200
CPU eager
370.954
3.779
98.16×
200
CPU compiled
117.881
3.779
31.19×
Table 1: Runtime comparison of PyTorch baselines and the fused Triton kernel of a single CT-SUBNET forward with n=6,m=3,h=32 , batch size 128 , σ=SiLU and RK4 integrator. Timings include forward and backward passes. Here ’compliled’ means that the torch.compile functionality is used.
Dataset
N
n
m
h
σ
I
T
training iterations
Δt/τ
batch size
validation frequency
history length
Silverbox
2
2
2
32
SiLU
RK4
150
200,000
0.5
128
500
50
EMPS
1
3
2
64
tanh
Heun
2500
15,000
0.25
256
50
20
WH
2
4
4
32
SiLU
RK4
200
500,000
1
128
500
50
WH (port-Hamiltonian)
2
4
4
32
SiLU
RK4
200
200,000
0.3
128
500
50
ParWH
2
8
8
32
SiLU
RK4
250
800,000
0.5
128
500
50
Table 2: Parameters that change throughout the experiments.
Method
Training time
RMSE [mV]
SUBNET
>200h
0.24
PNLSS
n.a.
0.42
DS3M
1.5h
0.24
DS3M (port-Hamiltonian)
2.4h
0.28
Table 3: Wiener–Hammerstein benchmark.
Method
Training time
RMSE [mm]
CT-SUBNET
1 – 24h
4.6
dynoNet (original)
n.a.
2.6
PNARX
1 – 60min
2.6
DS3M
0.5h
1.8
Table 4: EMPS benchmark. The dynoNet record uses a different historical data split.
Recent advances in deep-learning-based nonlinear system identification have led to encoder-based estimation of neural state-space (ANN-SS) models that achieve state-of-the-art performance in offline settings by estimating initial model states from past input-output data. These methods are typically used in multiple-shooting-based offline identification, and online learning of these models remains largely unexplored. This paper presents a batch-wise learning pipeline and a direct recursive identification algorithm for subspace encoder-based ANN-SS models. We provide convergence analysis of the recursive formulation and validate its performance through extensive simulation studies. The results demonstrate that the proposed approach enables computationally efficient online adaptation with high model accuracy.
Bendegúz Györök, Tamás Péni, Maarten Schoukens +1
Systems and Control Laboratory, HUN-REN Institute for Computer Science and Control, 1111 Budapest, Hungary · Control Systems Group, Eindhoven University of Technology, 5612 Eindhoven, The Netherlands
Deep state-space models (DSSMs) enable temporal predictions by learning the underlying dynamics of observed sequence data. They are often trained by maximising the evidence lower bound. However, as we show, this does not ensure the model actually learns the underlying dynamics. We therefore propose a constrained optimisation framework as a general approach for training DSSMs. Building upon this, we introduce the extended Kalman VAE (EKVAE), which combines amortised variational inference with classic Bayesian filtering/smoothing to model dynamics more accurately than RNN-based DSSMs. Our results show that the constrained optimisation framework significantly improves system identification and prediction accuracy on the example of established state-of-the-art DSSMs. The EKVAE outperforms previous models w.r.t. prediction accuracy, achieves remarkable results in identifying dynamical systems, and can furthermore successfully learn state-space representations where static and dynamic features are disentangled.
Alexej Klushyn, Richard Kurle, Maximilian Soelch +2
Technical University of Munich · AWS AI Labs · Machine Learning Research Lab, Volkswagen Group, Munich +1
Latent state space systems are ubiquitous in statistical modelling, arising naturally when time series are observed through noisy measurements. However, training deep state space models (DSSMs) at scale remains difficult. Two largely distinct strategies have emerged for training DSSMs. The first, auto-encoding DSSMs, trains generative models by optimising a variational lower bound. The second backpropagates through the outputs of classical sequential Monte Carlo (SMC) algorithms. Such approaches can train DSSMs for both discriminative and generative tasks, but their inherently sequential forward passes scale poorly on modern hardware. We propose parallel variational Monte Carlo (PVMC), a new training method that bridges these paradigms and robustly trains DSSMs for both discriminative and generative tasks. Across a set of benchmark experiments, PVMC matches or exceeds state-of-the-art performance while training 10× faster than the fastest competing SMC-based approach.
John-Joseph Brady, Nikolas Nusken, Yunpeng Li
Centre for Oral, Clinical and Translational Sciences, King’s College London, London, United Kingdom · Department of Mathematics, King’s College London, London, United Kingdom