Generalizing Dynamics Modeling More Easily from Representation Perspective
Authors: Yiming Wang, Zhengnan Zhang, Genghe Zhang, Jiawen Dan, Changchun Li, Chenlong Hu, Chris Nugent, Jun Liu, Ximing Li, Bo Yang
Organizations: College of Computer Science and Technology, and Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China · College of Software and Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China · School of Computing, Belfast, Northern Ireland, UK · College of Computer Science, and Technology, Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China
Learning system dynamics from observations is a critical problem in many applications over various real-world complex systems, e.g., climate, ecology, and fluid systems. Recently, neural dynamics modeling method have become a prevalent solution that embeds the object's observations into a latent space before learning dynamics using neural methods such as neural Ordinary Differential Equations (ODE). Existing dynamics modeling methods induce a specific model for each observation of different complex systems, resulting in poor generalization across systems. Inspired by the great success of pre-trained models, we conduct a generalized Pre-trained Dynamics EncoDER (PDEDER) which can embed the original state observations into a latent space where the dynamics can be captured more easily. To conduct the generalized PDEDER, we pre-train any Pre-trained Language Model (PLM) by minimizing the Lyapunov exponent objective, which constrains the chaotic behavior of governing dynamics learned in the latent space. By penalizing the divergence of embedded observations, our PDEDER promotes locally stable and well-structured latent dynamics, thereby facilitating more effective dynamics modeling than in the original observation space. In addition, we incorporate reconstruction and forecasting objectives to mitigate the risk of obtaining an over-smoothed latent space. Specifically, we collect 152 sets of real-world and synthetic observations from 23 complex systems as pre-training corpora and employ them to pre-train PDEDER. Given any future dynamic observation, we can fine-tune PDEDER with any specific dynamics modeling method. We evaluate PDEDER on 12 dynamic systems by short/long-term forecasting under both in-domain and cross-domain settings, and the empirical results indicate the effectiveness and generalizability of PDEDER.
Figures & tables
Fig. 1: The overall pipeline of pre-training (top) and fine-tuning (bottom) PDE der . (1) Pre-training of PDE der on large-scale dynamics observations: The sequences are first tokenized and normalized, followed by projection with system-specific data projection layers to align the dimension. The projected sequences are then encoded into latent space and decoded for reconstruction and forecasting. To pre-train PDE der , we employ a maximal Lyapunov exponent objective on sequence embeddings along with reconstruction and forecasting objectives. (2) Fine-tuning PDE der to approximate specific dynamics: The first token of an observation sequence is taken as the initial state and encoded into the latent space via the pre-trained encoder of PDE der . Then we approximate governing dynamics by integrating using the embedded initial states within the latent space through a GNN-based ODE module. Finally, we decode the integrated solutions by the pre-trained decoder and fine-tuning PDE der by minimizing the forecasting objective against the ground-truth observation sequence.
System
Type
Domain
Ns,h,m
Ts,h,m
Hs
Ms,h
Vs
Charged [ 44 ]
Synthetic
Physics
{ 5,10,15,20 }
{ 400,600 }
8
5000
4
Springs [ 44 ]
Synthetic
Physics
{ 20,25,30,35,40 }
{ 200,300 }
10
3500
4
Mutualistic Interaction Dynamics [ 3 ]
Synthetic
Physics
{ 100,121,169,196,225 }
{ 300,350,400 }
15
1500
1
Heat Diffusion [ 45 ]
Synthetic
Physics
{ 225,256,289,324 }
{ 200,250,300 }
12
1500
1
1D Diffusion-Reaction [ 46 ]
Synthetic
Fluid
{ 256,368,464,512 }
{ 200,225,250,275 }
16
700
1
1D Compressible Navier-Stokes [ 47 ]
Synthetic
Fluid
{ 300,350,400 }
{ 600,625 }
6
300
3
TABLE I: Statistics of collected dynamics. Ns,h,m is the number of objects; Ts,h,m is the length of observation sequence; Vs is the feature dimension; Ms,h is the number of generated samples; Hs is the number of different parameter settings.
Method Type
Single-Agent Method
Multi-Agent Methods
Multivariate Time Series Forecasting Methods
Our Method
System
Forecast Length
LatentODE [ 57 ]
GNS [ 19 ]
NDCN [ 12 ]
TREAT [ 23 ]
STGODE [ 58 ]
MTGODE [ 59 ]
PDE der
PDE der -small
RMSE
MAE
MAPE
RMSE
MAE
MAPE
RMSE
MAE
MAPE
RMSE
MAE
MAPE
RMSE
MAE
MAPE
RMSE
MAE
MAPE
RMSE
MAE
MAPE
RMSE
MAE
MAPE
Synthetic Systems
Mutualistic
10%
2.706
2.541
32.56
1.383
0.869
13.90
0.913
0.775
10.50
4.195
4.013
51.62
-
-
-
-
-
-
0.921
0.663
10.14
1.266
0.685
11.86
20%
2.198
1.989
25.37
1.720
1.042
15.40
1.116
0.895
11.90
3.517
3.302
41.95
-
-
-
-
-
-
1.171
0.698
10.24
1.551
0.779
12.16
50%
1.695
1.457
18.46
1.997
1.191
16.40
1.167
0.898
11.70
2.375
1.876
23.62
-
-
-
-
-
-
1.404
0.758
10.49
1.594
0.778
11.16
70%
1.527
1.279
16.20
2.048
1.221
16.60
1.083
0.818
10.63
2.009
1.370
17.25
-
-
-
-
-
-
1.411
0.771
10.53
1.474
0.709
10.02
TABLE II: Full results of dynamics forecasting varying forecasting lengths. Each score is the averaged results of multiple sample with different initial values. The best scores are in boldface and the second best scores are in underline . “Forecast Length” denotes the forecasting length. For synthetic systems, we truncate the test sequence by ratios of {10%,20%,50%,70%,80%,100%} to form short/long-term forecasting. “Avg” denotes the averaged results of all forecasting lengths. MAPE is in %. “-” means that results are not available. “PDE der ” and “PDE der -small” denotes the versions using pre-trained T5 and T5-small . (Due to the model setting of STGODE and MTGODE that the input and output lengths should be identical for training and testing, we only compare with them on realistic systems.)
Fig. 2: The interacting graphs among objects of 2D Compressible Navier-Stokes (2D-CFD) and PEMS08.
System
Forecast Length
PDE der -small
PDE der -para
PDE der -sys
PDE der w/o MLE
PDE der w/o pre
PDE der frz
RMSE
MAE
MAPE
RMSE
MAE
MAPE
RMSE
MAE
MAPE
RMSE
MAE
MAPE
RMSE
MAE
MAPE
RMSE
MAE
MAPE
Synthetic Systems
Mutualistic
10%
1.266
0.685
11.86
1.461
0.734
13.92
2.573
2.293
30.19
1.443
1.067
15.14
1.892
1.377
20.39
4.438
4.339
57.54
20%
1.551
0.779
12.16
1.682
0.792
14.85
2.709
2.417
31.21
1.694
1.228
16.74
1.974
1.358
19.27
4.595
4.505
58.36
50%
1.594
0.778
11.16
1.634
0.801
14.46
2.688
2.367
30.03
1.830
1.313
17.29
1.807
1.180
16.08
4.648
4.563
57.9
70%
1.474
0.709
10.02
1.602
0.797
14.07
2.530
2.177
27.58
1.788
1.311
17.12
1.622
1.036
13.99
4.525
4.431
56.04
80%
1.404
0.668
9.40
1.593
0.763
13.46
2.421
2.051
25.99
1.756
1.315
17.09
1.536
0.970
13.06
4.401
4.282
54.15
TABLE III: Full results of cross-domain settings and pre-training variants on PDE der -small. Each score is the averaged results of multiple sample with different initial values. The best scores are in boldface and the second best scores are in underline . “Forecast Length” denotes the forecasting length. For synthetic systems, we truncate the test sequence by ratios of {10%,20%,50%,70%,80%,100%} to form short/long-term forecasting. “Avg” denotes the averaged results of all forecasting lengths. MAPE is in %. “PDE der -para” denotes the variant fine-tuning under the cross-domain setting of pre-training learning one parameter out; “PDE der -sys” denotes the variant fine-tuning under the cross-domain setting of pre-training leaving one system out; “PDE der w/o MLE” denotes the ablative version pre-training without maximal Lyapunov Exponent objective; “PDE der w/o pre” denotes the variant which fine-tunes without pre-training; “PDE der frz” denotes the variant which fine-tunes by freezing our pre-trained PLM module. “-” denotes there are no available system-specific parameters.
Fig. 3: Forecasting visualizations on Heat dynamics over the variants of PDE der . “PDE der w/o MLE” denotes the ablative version pre-training without M-Lyapunov exponent; “PDE der w/o pre” denotes fine-tuning PDE der without pre-training; “PDE der frz” denotes fine-tuning by freezing the PLM module; “PDE der -sys” denotes the variant fine-tuning under the cross-domain setting of pre-training leaving one system out; “PDE der -para” denotes the variant fine-tuning under the cross-domain setting of pre-training learning one parameter out. The horizontal axes denote object indices, while the vertical axis represents object state values.
Fig. 4: Traffic flow forecasting visualizations on PDE der and baselines of real-world systems (a) T-Drive and (b) PEMS08. The vertical coordinate denotes the traffic flow values.
Method
T-Drive
NYCTaxi
CHIBike
PEMS03
PEMS04
PEMS07
PEMS08
LatentODE
13.9
188.8
15.4
20.5
14.0
19.0
12.0
GNS
78.0
44.7
9.9
31.3
26.2
24.9
19.7
NDCN
120.1
59.4
14.6
35.6
27.1
22.4
22.5
TREAT
49.5
177.9
69.2
42.9
49.2
20.7
48.0
STGODE
38.3
35.6
40.6
17.6
13.9
11.0
10.9
MTGODE
253.9
76.5
102.7
14.4
100.1
9.3
10.8
TABLE IV: Averaged MAPE (in %) scores of the predicted Incidence Proportion on traffic flows. The best scores are in boldface and second best scores are in underline .
Dynamical systems studies how states evolve over time, including stability, feedback, chaos, and transitions in physical, biological, and engineered systems.
Foundation models for dynamical systems can forecast unseen partial differential equations (PDEs), ordinary differential equations (ODEs), and time series during inference. However, it remains an open question whether these foundation models truly learn generalizable in-context forecasting strategies. We show that looped transformers are capable of emulating propagators for dynamical systems in-context by first establishing a link between linear attention and (extended) dynamic mode decomposition. We then generalize this insight into the Test-Time Forecasting (TTF) framework in which a one-step forecaster leverages iterations of gradient descent as recurrent depth. We demonstrate that lightweight TTF models trained on ODEs and PDEs generalize to out-of-distribution system dynamics during autoregressive rollout after being trained for one-step prediction. Our results reinforce TTF as a theoretically motivated framework for designing recurrent depth models capable of forecasting unseen dynamical systems at test-time.
Jeffrey B. Lai, John Quinn, Anthony Bao +1
Oden Institute University of Texas at Austin · Department of Physics University of Texas at Austin
Joint-embedding predictive architectures learn abstract states by predicting target embeddings from context embeddings, but their transition models are typically opaque neural maps. We introduce SJEPA, a reconstruction-free JEPA framework that learns predictive representations whose induced dynamics admit compact symbolic descriptions. Its hybrid transition combines a symbolic law with a regularised neural correction for dynamics outside the selected grammar. The central principle is to learn the simplest adequate dynamics: representation constraints preserve informative, non-collapsed predictive coordinates, while operator compression favours low-complexity symbolic-neural transitions that remain predictively adequate. We formalise this principle through induced-dynamics complexity, analyse predictive-coordinate non-identifiability, and show that unconstrained operator compression creates a direct shortcut to representation collapse. The framework supports both alternating representation-equation learning and symbolic dynamics fitted to fixed representations. In controlled pendulum experiments, joint learning discovers substantially simpler symbolic dynamics with lower long-horizon rollout error and divergence than post-hoc fitting, while an unconstrained one-step diagnostic realises the predicted collapse shortcut. Under grammar misspecification, correction regularisation preserves the representable symbolic mechanism and directs the neural component towards residual dynamics. The results expose a controllable trade-off among predictive fidelity, representation quality, symbolic parsimony, and symbolic-neural allocation.
Deep state-space models (DSSMs) enable temporal predictions by learning the underlying dynamics of observed sequence data. They are often trained by maximising the evidence lower bound. However, as we show, this does not ensure the model actually learns the underlying dynamics. We therefore propose a constrained optimisation framework as a general approach for training DSSMs. Building upon this, we introduce the extended Kalman VAE (EKVAE), which combines amortised variational inference with classic Bayesian filtering/smoothing to model dynamics more accurately than RNN-based DSSMs. Our results show that the constrained optimisation framework significantly improves system identification and prediction accuracy on the example of established state-of-the-art DSSMs. The EKVAE outperforms previous models w.r.t. prediction accuracy, achieves remarkable results in identifying dynamical systems, and can furthermore successfully learn state-space representations where static and dynamic features are disentangled.
Alexej Klushyn, Richard Kurle, Maximilian Soelch +2
Technical University of Munich · AWS AI Labs · Machine Learning Research Lab, Volkswagen Group, Munich +1