cs.CVJul 3, 2025
SaveNeuralDMD: Interpretable Neural Representation of Dynamics from Sparse and Noisy Measurements
Organizations: University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · NVIDIA Corporation, USA
Abstract
Many challenges in scientific imaging involve solving ill-posed inverse problems, where the goal is to recover spatio-temporal fields from indirect, noisy, and highly sparse measurements - often without access to ground truth data or reliable simulators. To address this challenging scenario, we present NeuralDMD, an interpretable, untrained (per-instance) reconstruction framework that combines neural implicit representations with Dynamic Mode Decomposition (DMD) to reconstruct continuous spatio-temporal dynamics directly from measurements. NeuralDMD parameterizes DMD modes as continuous neural fields, and imposes a low-rank linear dynamics prior with spectral time evolution to enforce temporal continuity. This formulation enables both forecasting under sparsity, and yields interpretable modes and spectra. We find that NeuralDMD outperforms baselines on a wide variety of tasks: from weather data assimilation from sparse station observations to interferometric (Fourier domain) observations of Sagittarius A, the black hole at the center of our galaxy. Moreover, NeuralDMD remains stable when extrapolating into the future. While this framework is most naturally suited to linear dynamics, we show that it can be applied to nonlinear regimes, though with extrapolation performance that degrades with increasing nonlinearity. Together, these results show that NeuralDMD enables interpretable reconstruction and forecasting of spatio-temporal dynamics directly from sparse and indirect measurements without relying on numerical simulators or training data.
Figures & tables
Figure 1: NeuralDMD reconstructs spatio-temporal dynamics from sparse observations in both indirect and direct sensing regimes, including sparse Fourier (top) and image-domain (bottom) measurements. It models the field as a low-rank linear combination of spatial modes with simple spectral time evolution, yielding an interpretable decomposition that supports forecasting. We demonstrate this on black hole imaging (top) and atmospheric data assimilation (bottom).
Figure 2: (a) Block diagram of the NeuralDMD architecture. The model comprises three MLPs defining the spatio-temporal dynamics: , mapping positionally encoded spatial coordinates to mode values ; , producing complex spectral components (decay rate , frequency ) from a learnable latent vector; and , mapping a learnable latent vector to the initial state for temporal evolution. (b), (c) The learned modes and coefficients are combined to produce the predicted dynamics . For the black hole setting, predictions are Fourier transformed ( ) and sampled at measured baseline coordinates ( ). The loss is computed between the observed visibilities and the corresponding model prediction. For image-domain experiments (weather data assimilation) the loss is instead computed directly on the sparse measurements.
| Method | Samples per frame | RMSE | PSNR | SSIM | LPIPS |
|---|---|---|---|---|---|
| NeuralDMD (ours) | 2.3 0.1 | 19.1 0.8 | 0.53 0.03 | 0.59 0.02 | |
| 1.5 0.1 | 24.7 0.5 | 0.70 0.01 | 0.32 0.01 | ||
| 3D-Var | - | - | - | - | |
| 3.6 0.8 | 18 4 | 0.48 0.09 | 0.5 0.1 | ||
| Vanilla MLP | 2.7 0.2 | 17 1 | 0.49 0.04 | 0.65 0.03 | |
| 2.3 0.2 | 21.0 0.6 | 0.56 0.01 | 0.6 0.2 |
Table 1: Quantitative evaluation: Weather data assimilation (ERA5). Quantitative comparison (mean std over frames) for dense ( ) and sparse ( ) observations per frame. We report results at both sparsity levels to highlight performance in a highly undersampled, operationally realistic ( ) and in a denser regime ( ), where 3D-Var is applicable. NeuralDMD achieves the best overall performance at both sparsity levels.
Figure 3: Qualitative evaluation: Weather data assimilation (ERA5). Rows show hrs, hrs, and extrapolation, with samples per-frame. 3D-Var is accurate near initialization but degrades over time and cannot extrapolate. NeuralDMD better preserves storm structure and forecasts more reliably than INR baselines, which lack explicit dynamics.
| CORAL ID | NeuralDMD ID | CORAL OOD | NeuralDMD OOD | |
|---|---|---|---|---|
| RMSE | 0.02 0.01 | (3.6 0.8)e-3 | 0.05 0.02 | 0.014 0.004 |
| PSNR | 39 1 | 56 1.3 | 30 0.3 | 43 1 |
| SSIM | 0.89 0.0 | 0.9994 0.0002 | 0.69 0.05 | 0.997 0.001 |
| LPIPS | 0.012 0.001 | 0.0002 0.0001 | 0.020 0.001 | 0.002 0.001 |
Table 2: In distribution (ID) and out of distribution (OOD) quantitative performance of CORAL vs. NeuralDMD . Sparsity rates of and of the total image pixels are used for fitting for ID and OOD cases respectively.
Figure 4: Spatio-temporal reconstruction from sparse EHT 2017 visibilities. (a) Reconstructions of a GRMHD sequence from sparse observations during fitting and extrapolation (forecasting). Values denote image-space MSE (lower is better). NeuralDMD produces lower error and better temporal consistency than StarWarps , resolve, and the spatio-temporal INR (vanilla MLP); The bottom row shows the average optical flow, with denoting cosine similarity to the ground truth flow (higher is better). (b) Per-frame normalized image-space MSE. Gray region denotes the extrapolation window, where NeuralDMD remains stable in contrast to the vanilla MLP diverges. (c) Robustness under increased visibility noise and reduced Fourier sampling. Fractional noise is relative to the data amplitude. A Fourier grid is used; Fourier components correspond sparsity.
| Method | RMSE | PSNR | SSIM | LPIPS | |
|---|---|---|---|---|---|
| SIREN | 3 | 0.072 0.011 | 23.0 1.2 | 0.58 0.032 | 0.29 0.026 |
| 10 | 0.075 0.0058 | 23.0 0.68 | 0.56 0.019 | 0.32 0.011 | |
| MFN | 3 | 0.13 0.0095 | 18.0 0.68 | 0.54 0.0087 | 0.30 0.0036 |
| 10 | 0.15 0.025 | 17.0 1.6 | 0.53 0.020 | 0.32 0.022 | |
| FFM | 256 | 0.093 0.0037 | 21.0 0.33 | 0.42 0.0061 | 0.66 0.018 |
| 1024 | 0.094 0.0047 | 21.0 0.42 | 0.47 0.031 | 0.59 0.017 |
Table 3: EHT 2017 ablations. Reconstruction metrics for different coordinate-network backbones, positional-encoding bandwidths , and the direct optimization baseline for .
Figure 5: top: PSNR vs. normalized time for Navier-Stokes cylinder flow simulations at Re and Re . The white and gray regions denotes the observation and forecasting regions respectively. bottom: Ground truth and NeuralDMD reconstructions at . Error grows more rapidly for the highly non-linear case of Re than Re (the same fact is reflected in the PSNR plot).
Figure S1 : GRMHD modes and eigenvalues of NeuralDMD reconstruction from ngEHT+ coverage. The top panel shows both the real (red) and imaginary (blue) parts of the eigenvalues. The imaginary part of the eigenvalues, , indicates the oscillatory behavior and the real part, , indicates the decay rate. Note that the higher order modes have highly decaying behavior , indicating that the reconstruction quality and interpretability of the dynamical system does not depend on these modes. None of the modes depicted here are highly decaying.
Figure S2 : zeroth-order modes of the GRMHD experiment as compared to the average frames, for ngEHT and ngEHT+ coverages. Note that there is but small differences between and the average.
Figure S3 : NeuralDMD reconstruction error with varying number of modes. Note that the only high-RMSE case is which means that using only 2 modes is not enough for high-fidelity reconstruction.
Figure S4 : CORAL and NeuralDMD reconstructions of ID, OOD, and far OOD 2d wave equation test cases. NeuralDMD outperforms CORAL on all cases, most notably when the test case is far outside of the distribution CORAL is trained on. The left two frames for CORAL and NeuralDMD at times are reconstructed from 10 uniformly sampled pixels for ID case and samples for OOD and far OOD case. The right two frames are then extrapolated with no observational data.
Figure S5 : GRMHD reconstruction from sparse interferometric data with and without additional gain corruptions.
Figure S6 : NeuralDMD reconstruction results as compared to additional INR baselines along with NeuralDMD but with direct optimization on parameters. NeuralDMD outperforms all baselines.
| Metric | Vanilla MLP | SIREN | MFN | FFM | Direct | NeuralDMD |
|---|---|---|---|---|---|---|
| 0.92 | 0.94 | 0.86 | 0.7 | 1.24 | 1.09 |
Table S1 : Comparison of values across different model architectures for black hole EHT 2017 experiment.
| Method | Freq | RMSE | PSNR | SSIM | LPIPS |
|---|---|---|---|---|---|
| SIREN | 1 | 0.1 0.02 | 20 1 | 0.5 0.02 | 0.4 0.03 |
| 3 | 0.07 0.01 | 20 1 | 0.6 0.03 | 0.3 0.03 | |
| 10 | 0.08 0.006 | 20 0.7 | 0.6 0.02 | 0.3 0.01 | |
| MFN | 1 | 0.1 0.007 | 20 0.5 | 0.5 0.01 | 0.3 0.006 |
| 3 | 0.1 0.01 | 20 0.7 | 0.5 0.009 | 0.3 0.004 | |
| 10 | 0.2 0.03 | 20 2 | 0.5 0.02 | 0.3 0.02 |
Table S2 : Ablation study across architectures and frequency choices.
Figure S7 : Top 11 Zernicke polynomials used for pretraining NeuralDMD . The “disk” initialization for neural representation is equivalent to the first Zernicke polynomial used for mode 1 of NeuralDMD .
Figure S8 : This figure demonstrates that the dynamics observed by the EHT, when blurred with a as beam, are well approximated by a linear dynamical system. The inset displays the frame at hours after the initial time: without blurring (blur column), highly nonlinear structures are present and not captured by NeuralDMD , while under blurring, these features are smoothed out. The nonlinear structure is outlined and shown in the zoomed-in panel. The curves show the per-frame normalized error between the NeuralDMD reconstructions and the ground truth, with error decreasing as blurring increases and the nonlinear features are progressively suppressed.
Figure S9 : Reconstruction of a GRMHD sequence from ngEHT - synthetic interferometric data. Rows (top to bottom): ground truth, NeuralDMD , neural representation, and StarWarps . The left two frames are data - constrained reconstructions (“data-fitting”), the right two are model extrapolations. Numbers at each frame’s top - left give the normalized L2 error versus ground truth (lower is better). NeuralDMD shows the lowest errors and best visual match across both observed and predicted frames. Since StarWarps directly fits pixels to the observation data, it cannot extrapolate The final column presents mean optical flow; is the cosine similarity with the ground - truth flow (higher is better).
Figure S10 : Reconstruction results on a GRMHD with interferometric observations synthesized for the ngEHT+ array. From left to right, the columns show the ground truth, NeuralDMD , neural representation, StarWarps , NeuralDMD reconstruction from ngEHT array. In each row, the first two frames are reconstructed from the observations, i.e., constrained by data for fitting the model. The second two frames are extrapolation results obtained by advancing the recovered dynamical (or neural) model forward in time to unobserved frames. The value at the top left corner of each frame shows the normalized L2-error of that reconstruction as compared to the ground truth (lower is better).
Figure S11 : Reconstruction error for NeuralDMD (blue) and the neural representation (orange) for ngEHT coverage. Solid curves give total error, dashed curves the dynamics - only error (mean image removed), and reconstructed frames are shown at selected points. Data-fitting relies on observations from a hour window; the grey band marks the extrapolation time window. The neural representation error exhibits rapidly increasing error beyond the observational window, whereas NeuralDMD maintains stable reconstructions – preserving both the average structure and dynamic behavior – over extended time horizons.
Figure S12 : EHT2017 coverage cylinder plots. The top row panels show the cylinder plots along the ring at the radius of peak emission. The second row shows the average frames where the selected ring used for the cylinder plot is depicted. The position angle (PA) is measured clockwise from the direction of East ( ) and NorthWest ( ), is highlighted in both for orientation. The white dashed line shows an analytical counter-clockwise Keplerian angular velocity, while the magenta denotes a sub-Keplerian (slower) angular velocity extracted from an image feature. Note that we do not show these lines for neural representation, and StarWarps as the poor reconstruction quality prohibits precise angular velocity measurements.
Figure S13 : ngEHT coverage cylinder plots. The top row panels show the cylinder plots along the ring at the radius of peak emission. The second row shows the average frames where the selected ring used for the cylinder plot is depicted. The position angle (PA) is measured clockwise from the direction of East ( ) and NorthWest ( ), is highlighted in both for orientation. The white dashed line shows an analytical counter-clockwise Keplerian angular velocity, while the magenta denotes a sub-Keplerian (slower) angular velocity extracted from an image feature. Note that we do not show these lines for neural representation, and StarWarps as the poor reconstruction quality prohibits precise angular velocity measurements.
Figure S14 : ngEHT+ coverage cylinder plots. First row shows the cylinder plots at the radius of the maximum emission. The second row shows the average frames where the selected ring is depicted. The white dashed line shows Keplerian angular velocity, while the magenta denotes a sub-Keplerian angular velocity of the point of maximum emission for the ground truth due to its smaller slope. We do not show these lines for StarWarps as the reconstructions are not fine enough to find the anuglar velocity of these features. The reconstructions are from ngEHT+ coverage unless otherwise stated.
Figure S15 : Mean optical flows of ground truth, and ngEHT+ coverage reconstruction results of NeuralDMD , neural representation, StarWarps, and ngEHT coverage reconstruction result of NeuralDMD . The value shows the cosine similarity (higher is better) of the reconstructions with the ground truth. Note that does not capture the noise reconstruction; hence, even though for neural representation is close to NeuralDMD , it hallucinates dynamics, resulting in a poor reconstruction quality.
Figure S16 : NeuralDMD and neural representation (Neural Rep.) imaging performance under different amounts of noise. NeuralDMD consistently outperforms neural representation. The percentage shows the amount of fractional noise added to the data. The two right-most columns are extrapolation after the data-fitting window.
Figure S17 : NeuralDMD and optDMD imaging performance under different sparsity settings. NeuralDMD outperforms TV-regularized optDMD (while there are some numerical issues for some of the first frames). The number in the format pts is the number of Fourier frequencies used in data fitting for each frame. The two right-most columns are extrapolation after the data-fitting window.
Figure S18 : A comparison of NeuralDMD with the data assimilation baseline 3D-Var . The results are shown on sparsely sampled ERA5 datasets of wind velocity magnitudes. [ First row ]: Results for hours from the initial frame [ Second row ] hours from the initial frame with of data given as sparse observations, [ Third Row ] hours from the initial frame with of the data given. [ Fourth row ] demonstrates NeuralDMD ’s ability to extrapolate by forecasting 5 hours beyond the observation window. From left to right: sparsely sampled observations, 3D-Var , and NeuralDMD reconstructions, and ground truth.
Figure S19 : NeuralDMD and neural representation (Neural Rep.) imaging performance on an edge-on GRMHD simulation. This test is more challenging than a face-on GRMHD since the rotation dynamics of the accretion moves in and out of the image plane as the line of observation is almost parallel to the image plane. Whereas, in the face-on case, the majority of the dynamics is well-captured within the image plane.
Figure S20 : Orbiting hot spot modes (real part) and eigenvalues. The top panel shows both the real (red) and imaginary (blue) parts of the eigenvalues. The imaginary part of the eigenvalues, , indicates the oscillatory behavior and the real part, , indicates the decay rate. The gray-shaded region highlights higher-order modes with fast decay ( ) which do not impact the reconstruction. The bottom panel showcases four of the recovered NeuralDMD modes, illustrating the multi-scale nature of the modal decomposition.
Figure S21 : Frequency coverage of EHT2017 , ngEHT , and the enhanced ngEHT+ . The scatter plots overlay the observed frequencies collected during the entire time window of 3.2 hours, color-coded by time. Note that the uv sampling smoothly changes as the Earth rotates throughout the acquisition period.
| Station | x (m) | y (m) | z (m) |
|---|---|---|---|
| PV | 5,088,967.900 | -301,681.600 | 3,825,015.800 |
| SMT | -1,828,796.200 | -5,054,406.800 | 3,427,865.200 |
| SMA | -5,464,523.400 | -2,493,147.080 | 2,150,611.750 |
| LMT | -768,713.964 | -5,988,541.798 | 2,063,275.947 |
| ALMA | 2,225,061.164 | -5,440,057.370 | -2,481,681.150 |
| APEX | 2,225,039.530 | -5,441,197.630 | -2,479,303.360 |
Table S3 : Selected station coordinates for EHT2017 , ngEHT , and ngEHT+ .
More in this topic
Spatiotemporal Forecasting
Spatiotemporal forecasting predicts how quantities change across locations and time, from traffic and weather to disease and environmental systems.
See all 410 papers