LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observations

Ziyu Tian; Kaichen Shen; Wenbo Hao; Phillip Si; Peng Chen; Wei Zhu

cs.LGOct 2, 2026

LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observations

Authors: Ziyu Tian, Kaichen Shen, Wenbo Hao, Phillip Si, Peng Chen, Wei Zhu

Organizations: School of Mathematics, Georgia Institute of Technology · School of Computational Science and Engineering, Georgia Institute of Technology

Abstract

Data assimilation combines model forecasts with noisy, incomplete observations to estimate the evolving state of a dynamical system. Existing methods face two compounding challenges: high-dimensional nonlinear dynamics make repeated forward simulation computationally expensive, while sparse observations provide limited direct information about the full state. To address these challenges, we propose the Latent-Dynamics Ensemble Flow Filter (LD-EnFF), a sequential Bayesian filtering framework that performs both forecast propagation and filtering updates in a compact latent space. LD-EnFF combines a latent dynamics surrogate for ensemble propagation with a variational autoencoder (VAE)-based observation model that evaluates a state-dependent observation likelihood in latent space. At each assimilation step, an ensemble filtering update based on flow matching uses the forecast ensemble and this likelihood to generate posterior samples, jointly updating latent states and uncertain parameters. This design avoids repeated full-state simulation during forecasting and full-field reconstruction during likelihood evaluation. LD-EnFF substantially outperforms a broad range of data assimilation algorithms on benchmarks spanning Kolmogorov flow, tsunami propagation, and atmospheric modeling, all featuring complex dynamics and sparse, noisy observations.

Figures & tables

Figure 1: Pipeline of LD-EnFF. Training phase 1: LDNet learns parameter-conditioned latent dynamics and a reconstruction map from

\mathbf{s}_{j}

to the physical state. Training phase 2: A variational encoder maps the sparse noisy observation

\mathbf{y}_{j}

to a latent observation representation, while the

y^{\star}

operator

H_{\alpha}

maps the augmented latent state

\bm{\kappa}_{j}=(\mathbf{s}_{j},\bm{\varepsilon}_{j})

to its predicted noiseless observation. Online assimilation: With all learned components frozen, LDNet advances the latent states to form the forecast ensemble. EnFF then assimilates each new observation using this ensemble and the state-dependent local Gaussian likelihood in Eq. 16 , jointly updating the latent states and uncertain parameters. The reconstruction network is invoked only when a physical-state estimate is needed.

Figure 2: Mean full-state RelRMSE over 20 held-out test trajectories and three random seeds per trajectory for Kolmogorov Flow (left), Tsunami Modeling (center), and Atmospheric Modeling (right). Full-space EnFF methods are not shown for Tsunami Modeling and Atmospheric Modeling because the corresponding runs exhibited severe error growth under the sparse-observation setting.

Method	Kolmogorov Flow	Tsunami Modeling	Atmospheric Modeling
LD-EnFF-OT (ours)	$\bm{5.74\pm 0.26}$	$\bm{6.53\pm 0.25}$	$\bm{4.42\pm 0.21}$
LD-EnFF-F2P (ours)	$5.99\pm 0.29$	$7.48\pm 0.26$	$4.46\pm 0.23$
LD-IEnKF	$14.23\pm 0.47$	$8.79\pm 0.45$	$5.89\pm 0.42$
LDNet+LSTM EnSF	$14.27\pm 0.49$	$10.35\pm 0.42$	$5.26\pm 0.31$
LDNet+LSTM EnFF-OT	$13.09\pm 0.38$	$9.87\pm 0.32$	$6.18\pm 0.27$
LDNet+LSTM EnFF-F2P	$14.40\pm 0.42$	$14.48\pm 0.39$	$5.52\pm 0.29$

Table 1: Full-state RelRMSE (%) at the final assimilation step for the latent-space methods, reported as mean

\pm

standard deviation over 20 held-out test trajectories after averaging over three random seeds per trajectory. Best and second-best results are shown in bold and underlined, respectively.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Kolmogorov Flow	Shallow Water	PlanetSWE
Downsampled time steps	$40$	$50$	$63$
$\Delta t$	$0.04$	$0.036$	$0.04$
Latent-state dimension	$9$	$10$	$50$
Dynamics network hidden layers	$9$	$8$	$8$
Dynamics network hidden dimension	$200$	$50$	$200$
Reconstruction network hidden layers	$14$	$10$	$15$

Appendix

Table 2: LDNet training hyperparameters used for the three benchmarks.

Configuration	Kolmogorov Flow	Shallow Water	PlanetSWE
Augmented latent input dimension	$10$	$12$	$52$
Output shape	$2\times 10\times 10$	$3\times 10\times 10$	$3\times 16\times 8$
MLP hidden dimensions	$(128,256,512,1024)$	$(256,512,1024,2048)$	$(256,512,1024,2048)$
Batch size	$100$	$100$	$100$

Appendix

Table 3: Learned

y^{\star}

operator configurations for the three benchmarks.

Method	State propagation	Observation construction	Analysis
LD-EnFF-OT (ours)	LDNet	VAE encoder + $y^{\star}$ operator	EnFF-OT
LD-EnFF-F2P (ours)	LDNet	VAE encoder + $y^{\star}$ operator	EnFF-F2P
LD-IEnKF	LDNet	VAE encoder + $y^{\star}$ operator	IEnKF
LDNet+LSTM EnSF	LDNet	LSTM encoder	EnSF
LDNet+LSTM EnFF-OT	LDNet	LSTM encoder	EnFF-OT
LDNet+LSTM EnFF-F2P	LDNet	LSTM encoder	EnFF-F2P

Appendix

Table 4: Summary of the methods compared in the experiments.

Figure 3: Mean uncertain-parameter relative error over 20 held-out test trajectories and three random seeds per trajectory for all assimilating latent-space methods on Kolmogorov Flow (left), Tsunami Modeling (center), and Atmospheric Modeling (right).

Figure 4: Speed-magnitude fields and pointwise errors at the final assimilation time for Kolmogorov Flow. The ground truth is shown on the left, followed by the estimates and corresponding errors for the compared methods. Each pointwise error is defined as the prediction minus the ground truth.

Figure 5: Free-surface elevation fields and pointwise errors at the final assimilation time for Tsunami Modeling. The ground truth is shown on the left. For each compared method, the prediction and corresponding pointwise error are shown side by side. Each pointwise error is defined as the prediction minus the ground truth.

Figure 6: Zonal wind velocity fields and pointwise errors at the final assimilation time for Atmospheric Modeling. The ground truth is shown on the left. For each compared method, the estimate and corresponding pointwise error (estimate minus ground truth) are shown side by side.

Data Assimilation

Data assimilation combines observations with dynamical models to estimate hidden system states and their uncertainty over time.

See all 58 papers

Flow Matching for Efficient and Scalable Data Assimilation
Aug 18, 2025Taos Transue, Bohan Chen, So Takao +1Flow Matching Kalman Filtering
Department of Mathematics and Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT, 84112 · The Computing and Mathematical Sciences Department, California Institute of Technology, 1200 E California Blvd, Pasadena, CA 91125
Improving Ensemble Filters with Flow Matching
Sep 23, 2026Haoyuan Chen, Alexandre ThiéryData Assimilation Ensemble Kalman Filtering
Department of Statistics and Data Science National University of Singapore 6 Science Drive 2, Singapore 117546
LEVDA: Latent Ensemble Variational Data Assimilation via Differentiable Dynamics
Feb 23, 2026Phillip Si, Peng ChenData Assimilation Latent Space Optimization
School of Computational Sciences and Engineering, Georgia Institute of Technology

Aug 18, 2025J/K move · Enter open · S save

Flow Matching for Efficient and Scalable Data Assimilation

Taos Transue, Bohan Chen, So Takao, Bao Wang

Department of Mathematics and Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT, 84112 · The Computing and Mathematical Sciences Department, California Institute of Technology, 1200 E California Blvd, Pasadena, CA 91125

Flow Matching Kalman Filtering Bayesian Filtering Data Assimilation Ensemble Kalman Filtering

Open paper AskPDF arXiv

Data assimilation (DA) estimates a dynamical system's state from noisy observations. Recent generative models like the ensemble score filter (EnSF) improve DA in high-dimensional nonlinear settings but are computationally expensive. We introduce the ensemble flow filter (EnFF), a training-free, flow matching (FM)-based framework that accelerates sampling and offers flexibility in flow design. EnFF uses Monte Carlo estimators for the marginal flow field, localized guidance for observation assimilation, and utilizes a novel flow path that exploits the Bayesian DA formulation. It generalizes classical filters such as the bootstrap particle filter and ensemble Kalman filter. Experiments on high-dimensional benchmarks demonstrate EnFF's improved cost-accuracy tradeoffs and scalability, highlighting FM's potential for efficient, scalable DA. Code is available at https://github.com/Utah-Math-Data-Science/Data-Assimilation-Flow-Matching.

Abstract

Data Assimilation

Explore similar work

Flow Matching for Efficient and Scalable Data Assimilation

Improving Ensemble Filters with Flow Matching

LEVDA: Latent Ensemble Variational Data Assimilation via Differentiable Dynamics