LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observations

Ziyu Tian; Kaichen Shen; Wenbo Hao; Phillip Si; Peng Chen; Wei Zhu

cs.LGOct 2, 2026

LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observations

Authors: Ziyu Tian, Kaichen Shen, Wenbo Hao, Phillip Si, Peng Chen, Wei Zhu

Organizations: School of Mathematics, Georgia Institute of Technology · School of Computational Science and Engineering, Georgia Institute of Technology

Abstract

Data assimilation combines model forecasts with noisy, incomplete observations to estimate the evolving state of a dynamical system. Existing methods face two compounding challenges: high-dimensional nonlinear dynamics make repeated forward simulation computationally expensive, while sparse observations provide limited direct information about the full state. To address these challenges, we propose the Latent-Dynamics Ensemble Flow Filter (LD-EnFF), a sequential Bayesian filtering framework that performs both forecast propagation and filtering updates in a compact latent space. LD-EnFF combines a latent dynamics surrogate for ensemble propagation with a variational autoencoder (VAE)-based observation model that evaluates a state-dependent observation likelihood in latent space. At each assimilation step, an ensemble filtering update based on flow matching uses the forecast ensemble and this likelihood to generate posterior samples, jointly updating latent states and uncertain parameters. This design avoids repeated full-state simulation during forecasting and full-field reconstruction during likelihood evaluation. LD-EnFF substantially outperforms a broad range of data assimilation algorithms on benchmarks spanning Kolmogorov flow, tsunami propagation, and atmospheric modeling, all featuring complex dynamics and sparse, noisy observations.

Figures & tables

Figure 1: Pipeline of LD-EnFF. Training phase 1: LDNet learns parameter-conditioned latent dynamics and a reconstruction map from

\mathbf{s}_{j}

to the physical state. Training phase 2: A variational encoder maps the sparse noisy observation

\mathbf{y}_{j}

to a latent observation representation, while the

y^{\star}

operator

H_{\alpha}

maps the augmented latent state

\bm{\kappa}_{j}=(\mathbf{s}_{j},\bm{\varepsilon}_{j})

to its predicted noiseless observation. Online assimilation: With all learned components frozen, LDNet advances the latent states to form the forecast ensemble. EnFF then assimilates each new observation using this ensemble and the state-dependent local Gaussian likelihood in Eq. 16 , jointly updating the latent states and uncertain parameters. The reconstruction network is invoked only when a physical-state estimate is needed.

Figure 2: Mean full-state RelRMSE over 20 held-out test trajectories and three random seeds per trajectory for Kolmogorov Flow (left), Tsunami Modeling (center), and Atmospheric Modeling (right). Full-space EnFF methods are not shown for Tsunami Modeling and Atmospheric Modeling because the corresponding runs exhibited severe error growth under the sparse-observation setting.

Method	Kolmogorov Flow	Tsunami Modeling	Atmospheric Modeling
LD-EnFF-OT (ours)	$\bm{5.74\pm 0.26}$	$\bm{6.53\pm 0.25}$	$\bm{4.42\pm 0.21}$
LD-EnFF-F2P (ours)	$5.99\pm 0.29$	$7.48\pm 0.26$	$4.46\pm 0.23$
LD-IEnKF	$14.23\pm 0.47$	$8.79\pm 0.45$	$5.89\pm 0.42$
LDNet+LSTM EnSF	$14.27\pm 0.49$	$10.35\pm 0.42$	$5.26\pm 0.31$
LDNet+LSTM EnFF-OT	$13.09\pm 0.38$	$9.87\pm 0.32$	$6.18\pm 0.27$
LDNet+LSTM EnFF-F2P	$14.40\pm 0.42$	$14.48\pm 0.39$	$5.52\pm 0.29$

Table 1: Full-state RelRMSE (%) at the final assimilation step for the latent-space methods, reported as mean

\pm

standard deviation over 20 held-out test trajectories after averaging over three random seeds per trajectory. Best and second-best results are shown in bold and underlined, respectively.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Kolmogorov Flow	Shallow Water	PlanetSWE
Downsampled time steps	$40$	$50$	$63$
$\Delta t$	$0.04$	$0.036$	$0.04$
Latent-state dimension	$9$	$10$	$50$
Dynamics network hidden layers	$9$	$8$	$8$
Dynamics network hidden dimension	$200$	$50$	$200$
Reconstruction network hidden layers	$14$	$10$	$15$

Appendix

Table 2: LDNet training hyperparameters used for the three benchmarks.

Configuration	Kolmogorov Flow	Shallow Water	PlanetSWE
Augmented latent input dimension	$10$	$12$	$52$
Output shape	$2\times 10\times 10$	$3\times 10\times 10$	$3\times 16\times 8$
MLP hidden dimensions	$(128,256,512,1024)$	$(256,512,1024,2048)$	$(256,512,1024,2048)$
Batch size	$100$	$100$	$100$

Appendix

Table 3: Learned

y^{\star}

operator configurations for the three benchmarks.

Method	State propagation	Observation construction	Analysis
LD-EnFF-OT (ours)	LDNet	VAE encoder + $y^{\star}$ operator	EnFF-OT
LD-EnFF-F2P (ours)	LDNet	VAE encoder + $y^{\star}$ operator	EnFF-F2P
LD-IEnKF	LDNet	VAE encoder + $y^{\star}$ operator	IEnKF
LDNet+LSTM EnSF	LDNet	LSTM encoder	EnSF
LDNet+LSTM EnFF-OT	LDNet	LSTM encoder	EnFF-OT
LDNet+LSTM EnFF-F2P	LDNet	LSTM encoder	EnFF-F2P

Appendix

Table 4: Summary of the methods compared in the experiments.

Figure 3: Mean uncertain-parameter relative error over 20 held-out test trajectories and three random seeds per trajectory for all assimilating latent-space methods on Kolmogorov Flow (left), Tsunami Modeling (center), and Atmospheric Modeling (right).

Figure 4: Speed-magnitude fields and pointwise errors at the final assimilation time for Kolmogorov Flow. The ground truth is shown on the left, followed by the estimates and corresponding errors for the compared methods. Each pointwise error is defined as the prediction minus the ground truth.

Figure 5: Free-surface elevation fields and pointwise errors at the final assimilation time for Tsunami Modeling. The ground truth is shown on the left. For each compared method, the prediction and corresponding pointwise error are shown side by side. Each pointwise error is defined as the prediction minus the ground truth.

Figure 6: Zonal wind velocity fields and pointwise errors at the final assimilation time for Atmospheric Modeling. The ground truth is shown on the left. For each compared method, the estimate and corresponding pointwise error (estimate minus ground truth) are shown side by side.

Data Assimilation

Data assimilation combines observations with dynamical models to estimate hidden system states and their uncertainty over time.

See all 58 papers

Aug 18, 2025

Flow Matching for Efficient and Scalable Data Assimilation

Data assimilation (DA) estimates a dynamical system's state from noisy observations. Recent generative models like the ensemble score filter (EnSF) improve DA in high-dimensional nonlinear settings but are computationally expensive. We introduce the ensemble flow filter (EnFF), a training-free, flow matching (FM)-based framework that accelerates sampling and offers flexibility in flow design. EnFF uses Monte Carlo estimators for the marginal flow field, localized guidance for observation assimilation, and utilizes a novel flow path that exploits the Bayesian DA formulation. It generalizes classical filters such as the bootstrap particle filter and ensemble Kalman filter. Experiments on high-dimensional benchmarks demonstrate EnFF's improved cost-accuracy tradeoffs and scalability, highlighting FM's potential for efficient, scalable DA. Code is available at https://github.com/Utah-Math-Data-Science/Data-Assimilation-Flow-Matching.

Taos Transue, Bohan Chen, So Takao +1

Department of Mathematics and Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT, 84112 · The Computing and Mathematical Sciences Department, California Institute of Technology, 1200 E California Blvd, Pasadena, CA 91125

Flow Matching Kalman Filtering Bayesian Filtering Data Assimilation Ensemble Kalman Filtering

Ask PDF

Sep 23, 2026

Improving Ensemble Filters with Flow Matching

Data assimilation estimates a dynamical state from partial and noisy observations. Classical ensemble filters are efficient but restrict analysis updates through finite sample covariance and affine Gaussian distribution. We introduce the Flow Ensemble Filter (FlowEF), which uses conditional flow matching to transport the forecast ensemble from a classical baseline filter to an analysis ensemble. FlowEF uses a localized Gaussian source during training, transports forecast ensemble members from a baseline filter at deployment, and conditions its velocity field on ensembles from that baseline filter and the observation. The proposed model therefore learns a nonlinear update while mapping each baseline ensemble independently. For sparsely observed dynamical systems, FlowEF improves both deterministic and probabilistic metrics over all four classical ensemble filters. It also achieves the best performance among the state-of-the-art generative data assimilation models.

Haoyuan Chen, Alexandre Thiéry

Department of Statistics and Data Science National University of Singapore 6 Science Drive 2, Singapore 117546

Data Assimilation Ensemble Kalman Filtering

Ask PDF

Feb 23, 2026

LEVDA: Latent Ensemble Variational Data Assimilation via Differentiable Dynamics

Long-range geophysical forecasts are fundamentally limited by chaotic dynamics and numerical errors. While data assimilation can mitigate these issues, classical variational smoothers require computationally expensive tangent-linear and adjoint models. Conversely, recent efficient latent filtering methods often enforce weak trajectory-level constraints and assume fixed observation grids. To bridge this gap, we propose Latent Ensemble Variational Data Assimilation (LEVDA), an ensemble-space variational smoother that operates in the low-dimensional latent space of a pretrained differentiable neural dynamics surrogate. By performing four-dimensional ensemble-variational (4DEnVar) optimization within an ensemble subspace, LEVDA jointly assimilates states and unknown parameters without the need for adjoint code or auxiliary observation-to-latent encoders. Leveraging the fully differentiable, continuous-in-time-and-space nature of the surrogate, LEVDA naturally accommodates highly irregular sampling at arbitrary spatiotemporal locations. Across three challenging geophysical benchmarks, LEVDA matches or outperforms state-of-the-art latent filtering baselines under severe observational sparsity while providing more reliable uncertainty quantification. Simultaneously, it achieves substantially improved assimilation accuracy and computational efficiency compared to full-state 4DEnVar.

Phillip Si, Peng Chen

School of Computational Sciences and Engineering, Georgia Institute of Technology

Data Assimilation Latent Space Optimization

Ask PDF

Abstract

Data Assimilation

Explore similar work

Flow Matching for Efficient and Scalable Data Assimilation

Improving Ensemble Filters with Flow Matching

LEVDA: Latent Ensemble Variational Data Assimilation via Differentiable Dynamics