Generalizing Dynamics Modeling More Easily from Representation Perspective

Yiming Wang; Zhengnan Zhang; Genghe Zhang; Jiawen Dan; Changchun Li; Chenlong Hu; Chris Nugent; Jun Liu

cs.LGMar 24, 2026

Generalizing Dynamics Modeling More Easily from Representation Perspective

Authors: Yiming Wang, Zhengnan Zhang, Genghe Zhang, Jiawen Dan, Changchun Li, Chenlong Hu, Chris Nugent, Jun Liu,

Organizations: College of Computer Science and Technology, and Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China · College of Software and Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China · School of Computing, Belfast, Northern Ireland, UK · College of Computer Science, and Technology, Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China

Abstract

Learning system dynamics from observations is a critical problem in many applications over various real-world complex systems, e.g., climate, ecology, and fluid systems. Recently, neural dynamics modeling method have become a prevalent solution that embeds the object's observations into a latent space before learning dynamics using neural methods such as neural Ordinary Differential Equations (ODE). Existing dynamics modeling methods induce a specific model for each observation of different complex systems, resulting in poor generalization across systems. Inspired by the great success of pre-trained models, we conduct a generalized Pre-trained Dynamics EncoDER (PDEDER) which can embed the original state observations into a latent space where the dynamics can be captured more easily. To conduct the generalized PDEDER, we pre-train any Pre-trained Language Model (PLM) by minimizing the Lyapunov exponent objective, which constrains the chaotic behavior of governing dynamics learned in the latent space. By penalizing the divergence of embedded observations, our PDEDER promotes locally stable and well-structured latent dynamics, thereby facilitating more effective dynamics modeling than in the original observation space. In addition, we incorporate reconstruction and forecasting objectives to mitigate the risk of obtaining an over-smoothed latent space. Specifically, we collect 152 sets of real-world and synthetic observations from 23 complex systems as pre-training corpora and employ them to pre-train PDEDER. Given any future dynamic observation, we can fine-tune PDEDER with any specific dynamics modeling method. We evaluate PDEDER on 12 dynamic systems by short/long-term forecasting under both in-domain and cross-domain settings, and the empirical results indicate the effectiveness and generalizability of PDEDER.

Figures & tables

Fig. 1: The overall pipeline of pre-training (top) and fine-tuning (bottom) PDE der . (1) Pre-training of PDE der on large-scale dynamics observations: The sequences are first tokenized and normalized, followed by projection with system-specific data projection layers to align the dimension. The projected sequences are then encoded into latent space and decoded for reconstruction and forecasting. To pre-train PDE der , we employ a maximal Lyapunov exponent objective on sequence embeddings along with reconstruction and forecasting objectives. (2) Fine-tuning PDE der to approximate specific dynamics: The first token of an observation sequence is taken as the initial state and encoded into the latent space via the pre-trained encoder of PDE der . Then we approximate governing dynamics by integrating using the embedded initial states within the latent space through a GNN-based ODE module. Finally, we decode the integrated solutions by the pre-trained decoder and fine-tuning PDE der by minimizing the forecasting objective against the ground-truth observation sequence.

System	Type	Domain	$N_{s,h,m}$	$T_{s,h,m}$	$H_{s}$	$M_{s,h}$	$V_{s}$
Charged [ 44 ]	Synthetic	Physics	$\{$ 5,10,15,20 $\}$	$\{$ 400,600 $\}$	8	5000	4
Springs [ 44 ]	Synthetic	Physics	$\{$ 20,25,30,35,40 $\}$	$\{$ 200,300 $\}$	10	3500	4
Mutualistic Interaction Dynamics [ 3 ]	Synthetic	Physics	$\{$ 100,121,169,196,225 $\}$	$\{$ 300,350,400 $\}$	15	1500	1
Heat Diffusion [ 45 ]	Synthetic	Physics	$\{$ 225,256,289,324 $\}$	$\{$ 200,250,300 $\}$	12	1500	1
1D Diffusion-Reaction [ 46 ]	Synthetic	Fluid	$\{$ 256,368,464,512 $\}$	$\{$ 200,225,250,275 $\}$	16	700	1
1D Compressible Navier-Stokes [ 47 ]	Synthetic	Fluid	$\{$ 300,350,400 $\}$	$\{$ 600,625 $\}$	6	300	3

TABLE I: Statistics of collected dynamics.

N_{s,h,m}

is the number of objects;

T_{s,h,m}

is the length of observation sequence;

V_{s}

is the feature dimension;

M_{s,h}

is the number of generated samples;

H_{s}

is the number of different parameter settings.

Method Type			Single-Agent Method			Multi-Agent Methods									Multivariate Time Series Forecasting Methods						Our Method
System		Forecast Length	LatentODE [ 57 ]			GNS [ 19 ]			NDCN [ 12 ]			TREAT [ 23 ]			STGODE [ 58 ]			MTGODE [ 59 ]			PDE der			PDE der -small
			RMSE	MAE	MAPE	RMSE	MAE	MAPE	RMSE	MAE	MAPE	RMSE	MAE	MAPE	RMSE	MAE	MAPE	RMSE	MAE	MAPE	RMSE	MAE	MAPE	RMSE	MAE	MAPE
Synthetic Systems	Mutualistic	10%	2.706	2.541	32.56	1.383	0.869	13.90	0.913	0.775	10.50	4.195	4.013	51.62	-	-	-	-	-	-	0.921	0.663	10.14	1.266	0.685	11.86
		20%	2.198	1.989	25.37	1.720	1.042	15.40	1.116	0.895	11.90	3.517	3.302	41.95	-	-	-	-	-	-	1.171	0.698	10.24	1.551	0.779	12.16
		50%	1.695	1.457	18.46	1.997	1.191	16.40	1.167	0.898	11.70	2.375	1.876	23.62	-	-	-	-	-	-	1.404	0.758	10.49	1.594	0.778	11.16
		70%	1.527	1.279	16.20	2.048	1.221	16.60	1.083	0.818	10.63	2.009	1.370	17.25	-	-	-	-	-	-	1.411	0.771	10.53	1.474	0.709	10.02

TABLE II: Full results of dynamics forecasting varying forecasting lengths. Each score is the averaged results of multiple sample with different initial values. The best scores are in boldface and the second best scores are in underline . “Forecast Length” denotes the forecasting length. For synthetic systems, we truncate the test sequence by ratios of

\{10\%,20\%,50\%,70\%,80\%,100\%\}

to form short/long-term forecasting. “Avg” denotes the averaged results of all forecasting lengths. MAPE is in %. “-” means that results are not available. “PDE der ” and “PDE der -small” denotes the versions using pre-trained T5 and T5-small . (Due to the model setting of STGODE and MTGODE that the input and output lengths should be identical for training and testing, we only compare with them on realistic systems.)

Fig. 2: The interacting graphs among objects of 2D Compressible Navier-Stokes (2D-CFD) and PEMS08.

System		Forecast Length	PDE der -small			PDE der -para			PDE der -sys			PDE der w/o MLE			PDE der w/o pre			PDE der frz
			RMSE	MAE	MAPE	RMSE	MAE	MAPE	RMSE	MAE	MAPE	RMSE	MAE	MAPE	RMSE	MAE	MAPE	RMSE	MAE	MAPE
Synthetic Systems	Mutualistic	10%	1.266	0.685	11.86	1.461	0.734	13.92	2.573	2.293	30.19	1.443	1.067	15.14	1.892	1.377	20.39	4.438	4.339	57.54
		20%	1.551	0.779	12.16	1.682	0.792	14.85	2.709	2.417	31.21	1.694	1.228	16.74	1.974	1.358	19.27	4.595	4.505	58.36
		50%	1.594	0.778	11.16	1.634	0.801	14.46	2.688	2.367	30.03	1.830	1.313	17.29	1.807	1.180	16.08	4.648	4.563	57.9
		70%	1.474	0.709	10.02	1.602	0.797	14.07	2.530	2.177	27.58	1.788	1.311	17.12	1.622	1.036	13.99	4.525	4.431	56.04
		80%	1.404	0.668	9.40	1.593	0.763	13.46	2.421	2.051	25.99	1.756	1.315	17.09	1.536	0.970	13.06	4.401	4.282	54.15

TABLE III: Full results of cross-domain settings and pre-training variants on PDE der -small. Each score is the averaged results of multiple sample with different initial values. The best scores are in boldface and the second best scores are in underline . “Forecast Length” denotes the forecasting length. For synthetic systems, we truncate the test sequence by ratios of

\{10\%,20\%,50\%,70\%,80\%,100\%\}

to form short/long-term forecasting. “Avg” denotes the averaged results of all forecasting lengths. MAPE is in %. “PDE der -para” denotes the variant fine-tuning under the cross-domain setting of pre-training learning one parameter out; “PDE der -sys” denotes the variant fine-tuning under the cross-domain setting of pre-training leaving one system out; “PDE der w/o MLE” denotes the ablative version pre-training without maximal Lyapunov Exponent objective; “PDE der w/o pre” denotes the variant which fine-tunes without pre-training; “PDE der frz” denotes the variant which fine-tunes by freezing our pre-trained PLM module. “-” denotes there are no available system-specific parameters.

Fig. 3: Forecasting visualizations on Heat dynamics over the variants of PDE der . “PDE der w/o MLE” denotes the ablative version pre-training without M-Lyapunov exponent; “PDE der w/o pre” denotes fine-tuning PDE der without pre-training; “PDE der frz” denotes fine-tuning by freezing the PLM module; “PDE der -sys” denotes the variant fine-tuning under the cross-domain setting of pre-training leaving one system out; “PDE der -para” denotes the variant fine-tuning under the cross-domain setting of pre-training learning one parameter out. The horizontal axes denote object indices, while the vertical axis represents object state values.

Panel of Figure 7 — Fig. 4: Traffic flow forecasting visualizations on PDE der and baselines of real-world systems (a) T-Drive and (b) PEMS08. The vertical coordinate denotes the traffic flow values.

Method	T-Drive	NYCTaxi	CHIBike	PEMS03	PEMS04	PEMS07	PEMS08
LatentODE	13.9	188.8	15.4	20.5	14.0	19.0	12.0
GNS	78.0	44.7	9.9	31.3	26.2	24.9	19.7
NDCN	120.1	59.4	14.6	35.6	27.1	22.4	22.5
TREAT	49.5	177.9	69.2	42.9	49.2	20.7	48.0
STGODE	38.3	35.6	40.6	17.6	13.9	11.0	10.9
MTGODE	253.9	76.5	102.7	14.4	100.1	9.3	10.8

TABLE IV: Averaged MAPE (in %) scores of the predicted Incidence Proportion on traffic flows. The best scores are in boldface and second best scores are in underline .

Dynamical Systems

Dynamical systems studies how states evolve over time, including stability, feedback, chaos, and transitions in physical, biological, and engineered systems.

See all 468 papers

Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction
Oct 14, 2026Jeffrey B. Lai, John Quinn, Anthony Bao +1Dynamical Systems In-Context Learning
Oden Institute University of Texas at Austin · Department of Physics University of Texas at Austin
SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictors
Aug 4, 2026Yongchao HuangLatent Dynamics Modeling Joint-Embedding Predictive Architecture
Latent Matters: Learning Deep State-Space Models
Feb 26, 2026Alexej Klushyn, Richard Kurle, Maximilian Soelch +2State Space Models Latent Dynamics Modeling
Technical University of Munich · AWS AI Labs · Machine Learning Research Lab, Volkswagen Group, Munich +1

Oct 14, 2026J/K move · Enter open · S save

Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction

Jeffrey B. Lai, John Quinn, Anthony Bao, William Gilpin

Oden Institute University of Texas at Austin · Department of Physics University of Texas at Austin

Dynamical Systems In-Context Learning Time-Series Foundation Models Recurrent Transformers PDE Operator Learning

Open paper AskPDF arXiv

Foundation models for dynamical systems can forecast unseen partial differential equations (PDEs), ordinary differential equations (ODEs), and time series during inference. However, it remains an open question whether these foundation models truly learn generalizable in-context forecasting strategies. We show that looped transformers are capable of emulating propagators for dynamical systems in-context by first establishing a link between linear attention and (extended) dynamic mode decomposition. We then generalize this insight into the

\textit{Test-Time Forecasting}

(TTF) framework in which a one-step forecaster leverages iterations of gradient descent as recurrent depth. We demonstrate that lightweight TTF models trained on ODEs and PDEs generalize to out-of-distribution system dynamics during autoregressive rollout after being trained for one-step prediction. Our results reinforce TTF as a theoretically motivated framework for designing recurrent depth models capable of forecasting unseen dynamical systems at test-time.

Abstract

Dynamical Systems

Explore similar work

Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction

SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictors

Latent Matters: Learning Deep State-Space Models