Physical dynamical systems can be viewed as natural information processors: their systems preserve, transform, and disperse input information. This perspective motivates learning not only from data generated by such systems, but also how to measure them in a way that extracts the most useful information for a given task. We propose a general computing framework for adaptive information extraction from dynamical systems, in which a trainable attention module learns both where to probe the system state and how to combine these measurements to optimize prediction performance. As a concrete instantiation, we implement this idea using a spatiotemporal field governed by a partial differential equation as the underlying dynamics, though the framework applies equally to any system whose state can be sampled. Our results show that adaptive spatial sensing significantly improves prediction accuracy on canonical chaotic benchmarks. This work provides a perspective on attention-enhanced reservoir computing as a special case of a broader paradigm: neural networks as trainable measurement devices for extracting information from physical dynamical systems.
Figures & tables
Figure 1: Flow chart of the Adaptive-Sensing-Attention–Enhanced-Reservoir-Computer (ASAERC). Inputs drive a fixed continuous reservoir. The reservoir is measured at fixed locations given via the measurement kernels ψt(j)(x) and the resulting Nfix states r~t are fed into the attention module. The trainable attention module outputs parameterized kernels ϕt+T(i)(x) to guide sampling and attention weights Watt,t+T to combine sampled features. The weighted combination produces yˉn , and gradient descent (dashed path) updates the attention module. No gradients pass through the PDE reservoir.
Figure 2: Illustration of ASAERC behavior. (a) Example predicted time series (orange-dashed line) compared with ground truth (blue-solid line) on the Lorenz attractor. (b) Snapshot of the PDE reservoir field u(x,t⋆) with fixed injection points (light-blue triangles), fixed measurement points (attention input, crosses) r~t(j)=∫Ωψt(j)(x)u(x,t)dx, and dynamic measurement locations (attention output) overlaid rt+T(j)=∫Ωϕt+T(i)(x)u(x,t+T)dx, . The adaptive sensors (blue circles) cluster around dynamically active regions, demonstrating the network’s ability to focus its attention on informative spatial locations. (c) A histogram of the attention output locations, giving an intricate pattern that is a nonlinear projection of the Lorenz attractor on the measurement process.
Figure 3: Prediction error as a function of number of measurement points. All methods use the same PDE reservoir. (a) Classical reservoir computing with linear readout; (b) AERC with fixed measurement positions; (c) ASAERC with adaptive sensing. Color indicates the number of measurement points at fixd locations.
Figure 4: Number of trainable parameters versus number of measurement points. The dotted line with triangles denotes the linear readout, the dashed line with squares denotes AERC, and the solid lines with dots denote ASAERC. For both AERC and ASAERC, we show three cases: one with 16, 64, and 256 adaptive queries. All three methods use the same PDE reservoir. ASAERC has the largest number of parameters, however only slightly more than AERC. Linear readout has far fewer parameters.
Figure 5: Correlation analysis of readout contributions. Distribution of pairwise Pearson correlation coefficients ρij between readout nodes i and j for (left column) raw PDE values, (middle) attention weights, and (right) their product. Rows correspond to (top row) linear readout, (middle row) AERC, and (bottom row) ASAERC ASAERC exhibits substantially lower correlation than linear readout and AERC, indicating that its measurements are less redundant and more complementary, which likely aids generalization and lowers prediction error.
Aspect
Classical RC
AERC
ASAERC
Sensing / readout
Static linear map yˉt=Woutrt .
State–conditional weights Watt,t=Fatt(rt) (small MLP) used in prediction, yˉt=Watt,trt .
Dual mechanism: (i) adaptive kernels {ϕt+T(i)} select where to measure ; (ii) attention weights Watt,t+T decide how to combine .
Trainable parameters
Only Wout trained by ridge regression or gradient descent.
Parameters of Fatt trained by gradient descent.
Parameters of Fθ and Fatt (often shared backbone) trained by gradient descent.
Table 1: Comparison of classical Reservoir Computing (RC), Attention–Enhanced RC (AERC), and Adaptive-Sensing AERC (ASAERC).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Delay-embedding MLP baseline. Full (all eight systems) and individual normalized MSE versus number of delay steps k and parameter count. Double Pendulum (blue), Duffing Oscillator (orange), Henon Map (green), Logistic Map (red), Lorenz System (purple), Mackey-Glass Equation (brown), Roessler System (pink), Van der Pool Oscillator (grey). Performance improves for small k however worsens beyond an optimal value, likely due to increasing input dimensionality and insufficient training or overfitting.
Figure 7: LSTM baseline. Full (all eight systems) and individual normalized MSE versus total parameter count and parameter count over hidden size. Performance improves monotonically for higher parameter counts, showing a more stable behaviour than the delay-embedding approach. The performance is on par with the best ASAERC models while also having roughly the same parameter count. Double Pendulum (blue), Duffing Oscillator (orange), Henon Map (green), Logistic Map (red), Lorenz System (purple), Mackey-Glass Equation (brown), Roessler System (pink), Van der Pool Oscillator (grey).
Figure 8: Fig. 3 MSE for the individual dynamical systems.
Machine learning approaches to spatiotemporal physical systems have primarily focused on next-frame prediction, with the goal of learning an accurate emulator for the system's evolution in time. However, these emulators are computationally expensive to train and are subject to performance pitfalls, such as compounding errors during autoregressive rollout. In this work, we take a different perspective and look at scientific tasks further downstream of predicting the next frame, such as estimation of a system's governing physical parameters. Accuracy on these tasks offers a uniquely quantifiable glimpse into the physical relevance of the representations of these models. We evaluate the effectiveness of general-purpose self-supervised methods in learning physics-grounded representations that are useful for downstream scientific tasks. Surprisingly, we find that not all methods designed for physical modeling outperform generic self-supervised learning methods on these tasks, and methods that learn in the latent space (e.g., joint embedding predictive architectures, or JEPAs) outperform those optimizing pixel-level prediction objectives. Code is available at https://github.com/helenqu/physical-representation-learning.
Helen Qu, Rudy Morel, Michael McCabe +4
Flatiron Institute · Princeton University · Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM +1
Accurately learning nonlinear dynamics from a finite-duration experiment requires the efficient collection of informative data. We address this challenge for stochastic controlled nonlinear dynamical systems whose state is observed along a single trajectory. Our goal is to reconstruct the unknown controlled state-increment map over a prescribed compact subset of state-input space. We construct a parametric estimator of the map using fixed nonlinear features, so that the model is nonlinear in the state and input, but linear in the unknown parameters. A Gaussian prior over the parameters yields recursive Bayesian posterior updates as data stream in, enabling online quantification of predictive uncertainty in the reconstructed dynamics over the target set. We formulate an optimal adaptive-design problem over an information state, using a prediction-oriented acquisition criterion based on the mean marginal mutual information between candidate future trajectories and the reconstructed dynamics over the target set. We then approximate the resulting adaptive-design problem by a non-myopic receding-horizon formulation, evaluate its remaining expectation using a scenario-based sample average, and solve the resulting deterministic program with the cross-entropy method, leveraging parallel candidate-scenario evaluations. Numerical experiments on a noisy multistable system demonstrate that the proposed adaptive information-seeking strategy reduces predictive uncertainty and reconstruction error more efficiently than common excitation baselines under comparable experimental constraints.
Juncal Arbelaiz, Anushri Arora, Jonathan W. Pillow
Princeton Neuroscience Institute and Center for Statistics and Machine Learning, Princeton University · Computer Science Department, Princeton University
We present an adaptive reservoir computing framework for the CTF-4-Science Lorenz benchmark, which evaluates machine learning models across twelve distinct tasks spanning five qualitatively different scenarios: baseline forecasting, noisy signal reconstruction, forecasting under noise, few-shot learning, and parametric generalization. Rather than applying a uniform inference strategy, we tailor the training and prediction procedure of Echo State Networks (ESNs) to the specific demands of each evaluation scenario. Our key contributions are fourfold: (1) exact reservoir state synchronization that eliminates warmup approximation error in short-time prediction; (2) histogram-guided candidate selection that directly optimizes the long-time ergodic evaluation metric; (3) multi-seed reservoir search for few-shot regimes with severely limited training data; and (4) sequential multi-sequence training that resolves state-distribution mismatch in parametric generalization tasks. The proposed framework achieves a score of 74.91 on the public benchmark leaderboard, demonstrating that carefully adapted reservoir computing constitutes a competitive and computationally efficient approach for diverse chaotic system modeling challenges.