Data Assimilation

Ask about this topic

Overview

Data assimilation estimates the evolving state of a system by combining imperfect observations with forecasts from a physical or learned model. Kalman and ensemble filters, particle filters, and variational methods are established approaches; newer work also uses latent-space models, neural operators, and generative methods such as diffusion and flow matching. These methods support applications including weather prediction, fluid reconstruction, ocean modeling, and subsurface characterization, often when observations are sparse or irregular. A central challenge is balancing computational cost and scalability with accurate uncertainty estimates and physically plausible results, especially for nonlinear systems and non-Gaussian posteriors.

Latest papers

CardsList

58 papers9 papers in the last four weeks, up 29% on the four weeks before. 0.1% of all new papers.

May 18, 2026

SURGE: Approximation and Training Free Particle Filter for Diffusion Surrogate

Data assimilation (DA) addresses the problem of sequentially estimating the state of a dynamical system from noisy and incomplete observations. In this work, we employ a diffusion model as a world model to simulate and predict the system's dynamics. Recently, score-based diffusion models have learned global diffusion priors that effectively model (stochastic) dynamics, revealing strong potential for data assimilation. In this paper, we investigate how information from noisy observations can be incorporated to enable continuous correction and refinement of the predicted system state when using a diffusion prior. Motivated by particle filtering methods, we represent the posterior distribution using a set of particles. After receiving noisy observations, the diffusion model is guided using the observation likelihood to steer the generation process toward observation-consistent states. Nevertheless, such guidance does not guarantee sampling from the true posterior. We therefore employ a Sequential Monte Carlo approach over the diffusion trajectory, viewed as a path measure, to reweight and resample particles, thereby correcting the generation process and ensuring convergence toward the desired posterior distribution. This leads to an unbiased particle filtering method that rigorously fuses observational data with diffusion model simulations.

Lifu Wei, Yinuo Ren, Naichen Shi +1

Department of Mechanical Engineering, Northwestern University, Evanston, IL, United States · Institute for Computational & Mathematical Engineering, Stanford University, Stanford, CA, United States · Department of Industrial Engineering & Management Sciences, Northwestern University, Evanston, IL, United States

Particle FilteringBayesian FilteringData AssimilationDiffusion Models
AskPDF
May 14, 2026

ForcingDAS: Unified and Robust Data Assimilation via Diffusion Forcing

Data assimilation (DA) estimates the state of an evolving dynamical system from noisy, partial observations, and is widely used in scientific simulation as well as weather and climate science. In practice, filtering methods rely on frame-to-frame transition models. However, these models are fragile when observations are non-Markovian (when they form only a partial slice of a higher-dimensional latent state as in real-world weather data): they tend to accumulate errors over long horizons. At the same time, learned DA methods typically commit to a single regime, either filtering (nowcasting, real-time forecasting) or smoothing (retrospective reanalysis), which splits what should be a shared prior across application-specific pipelines. To address both issues, we introduce ForcingDAS, a unified and robust DA framework. Built on Diffusion Forcing with an independent noise level assigned to each frame, ForcingDAS learns a joint-trajectory prior instead of frame-to-frame transitions. This allows it to capture long-horizon temporal dependencies and reduce error accumulation. In addition, the same trained model spans the full filtering to smoothing spectrum at inference time. Specifically, nowcasting, fixed-lag smoothing, and batch reanalysis are selected through the inference schedule alone, without retraining. We evaluate ForcingDAS on 2D Navier-Stokes vorticity, precipitation nowcasting, and global atmospheric state estimation. Across all settings, a single model is competitive with or outperforms both learned and classical baselines that are specialized for individual regimes, with the largest gains observed on real-world weather benchmarks.

Yixuan Jia, Siyi Chen, Yida Pan +9

University of Michigan, Ann Arbor · Michigan State University · Los Alamos National Laboratory

Data Assimilation
AskPDF
May 10, 2026

WindINR: Latent-State INR for Fast Local Wind Query and Correction in Complex Terrain

Many decisions in complex terrain require fast wind estimates at specific locations and heights rather than dense forecasts on fixed grids. We present WindINR, a latent-state implicit neural representation framework for continuous high-resolution local wind queries and correction using sparse observations. A latent-conditioned decoder maps terrain descriptors, a low-resolution background field, and continuous query coordinates to local wind estimates. WindINR separates reusable representation learning from sample-specific latent correction. During training, a privileged encoder infers reference latent states from high-resolution supervision, while a deployable predictor estimates initial states from inference-time inputs. Their discrepancies define a dataset-adaptive Gaussian prior over latent corrections. At inference time, network weights remain fixed, and only the latent state is updated through a regularized objective incorporating sparse observations and their uncertainty. We train WindINR on the large-scale FuXi-CFD dataset and evaluate it using controlled observing system simulation experiments, simulated helicopter trajectories, and real UAV measurements over Tromsø. Across these settings, sparse-observation assimilation consistently improves local wind predictions, with WindINR outperforming several baselines, including WindSeer. Updating only a low-dimensional latent state yields a 14-fold speedup over full-network fine-tuning while preserving continuous queries at arbitrary spatial coordinates. WindINR thus provides an efficient interface between kilometer-scale background products, sparse local observations, and local wind queries in complex terrain.

Yi Xiao, Qilong Jia, Hang Fan +4

Tsinghua University · MBZUAI · Columbia University +2

Implicit Neural RepresentationsInference-Time OptimizationData Assimilation
AskPDF
May 4, 2026

Online Generalised Predictive Coding

This paper introduces an extension of generalised filtering for online applications. Generalised filtering refers to data assimilation schemes that jointly infer latent states, learn unknown model parameters, and estimate uncertainty in an integrated framework -- e.g., estimate state and observation noise -- at the same time (i.e., triple estimation). This framework appears across disciplines under different names, including variational Kalman-Bucy filtering in engineering, generalised predictive coding in neuroscience, and Dynamic Expectation Maximisation (DEM) in time-series analysis. Here, we specialise DEM for ``online'' data assimilation, through a separation of temporal scales. We describe the variational principles and procedures that allow one to assimilate data in a way that allows for a slow updating of parameters and precisions, which contextualise fast Bayesian belief updating about the dynamic hidden states. Using numerical studies, we demonstrate the validity of online DEM (ODEM) using a non-linear -- and potentially chaotic -- generative model, to show that the ODEM scheme can track the latent states of the generative process, even when its functional form differs fundamentally from the dynamics of the generative model. Framed from a neuro-mimetic predictive coding perspective, ODEM offers a biologically inspired solution to online inference, learning, and uncertainty estimation in dynamic environments.

Mehran H. Z. Bazargani, Szymon Urbas, Adeel Razi +2

Insight Research Ireland Centre, UCD, Dublin, Ireland. · UCD School of Mathematics and Statistics, UCD, Belfield, Dublin, Ireland. · Turner Institute for Brain and Mental Health, School of Psychological Sciences, Monash University, Melbourne, Victoria, Australia. +5

Parameter EstimationLatent Dynamics ModelingPredictive CodingData AssimilationExpectation-Maximization
AskPDF
May 1, 2026

Learning Discriminators for Resampling in the Ensemble Gaussian Mixture Filter through a Normalizing Flow Approach

The ensemble Gaussian mixture filter (EnGMF) is a powerful, convergent particle filter capable of medium-to-high dimensional non-linear filtering. The EnGMF relies on a resampling step that can generate physically unrealistic posterior samples, that would subsequently produce physically meaningless forecasts. This work introduces the discriminator-informed resampling procedure, that augments the posterior resampling step with a discriminator that accepts or rejects candidate particles based on their physical plausibility. In this work these discriminators are learned through a normalizing flow approach. Numerical experiments on both the Ikeda map and the Lorenz '63 system show that discriminator informed resampling procedure consistently reduces error relative to the standard EnGMF in low-ensemble regimes.

Zain Jabbar, Andrey A. Popov

Particle FilteringBayesian FilteringData Assimilation
AskPDF
Apr 17, 2026

Global Attention with Linear Complexity for Exascale Generative Data Assimilation in Earth System Prediction

Accurate Earth system prediction requires state inference from incomplete observations, but conventional two-stage data assimilation (DA) is computationally prohibitive because repeated PDE-based ensemble forecasts, observation updates, and intermediate data movement limit ensemble size at high resolution. We introduce STORM, a one-stage generative AI framework that reformulates DA as diffusion-based Bayesian posterior sampling, replacing online PDE ensemble forecasts with scalable AI inference. It further combines a spatiotemporal transformer with a global-attention algorithm that reduces complexity from quadratic to linear through scalable gradient propagation, enabling high-resolution, long-context Earth modeling. STORM scales to 74,400 GPUs on Frontier with 96--99% strong-scaling efficiency and up to 6 ExaFLOPs sustained BF16 throughput, while enabling 32,768-member ensembles for uncertainty quantification in 34 seconds on 4,096 GPUs. It scales to 20 billion spatiotemporal tokens and 177,000 temporal frames. Hurricane tracking and long-term climate reanalysis demonstrate improved accuracy, including benefits from longer temporal context and recovery of temperature extremes missed by forecast-only predictions.

Xiao Wang, Zezhong Zhang, Isaac Lyngaas +10

Oak Ridge National Laboratory, Oak Ridge, TN, USA · Auburn University, Auburn, AL, USA · Florida State University, Tallahassee, FL, USA +2

Climate ScienceHigh-Performance ComputingData AssimilationClimate ModelingConditional Generative Modeling
AskPDF
Apr 8, 2026

Amortized Filtering and Smoothing with Conditional Normalizing Flows

Bayesian filtering and smoothing are central to data assimilation in nonlinear dynamical systems. Recent advances in deep generative models provide flexible approximations of the associated non-Gaussian posterior distributions. However, several existing approaches require the score to be re-estimated or a transport map to be constructed at each assimilation step. We propose an amortized framework for filtering and smoothing that reuses trained conditional models across observation sequences and assimilation times. The framework jointly learns a shared recurrent summary network and two conditional normalizing flows from simulated state and observation trajectories. The recurrent network represents each observation history by a fixed-dimensional summary that conditions the filtering approximation and, together with the next state, the backward transition approximation. An information-theoretic analysis shows that, under the Markov assumption, a summary sufficient for filtering is also sufficient for the backward kernel. Combining the terminal filtering approximation with the backward kernels yields an approximate joint smoothing distribution. Numerical experiments on advection-diffusion, Burgers, and Lorenz systems demonstrate the accuracy of the proposed filtering and smoothing approximations and characterize the evolution of their errors beyond the training horizon.

Tiangang Cui, Xiaodong Feng, Chenlong Pei +2

School of Mathematics and Statistics, The University of Sydney, Camperdown, NSW 2006, Australia · Institute of Advanced Study, Beijing Normal-Hong Kong Baptist University, Zhuhai 519087, China · SKLMS & Institute of Computational Mathematics and Scientific/Engineering Computing, Academy of Mathematics and Systems Science, Chinese Academy of Sciences +1

Bayesian FilteringData AssimilationBayesian InferenceAmortized Inference
AskPDF
Apr 3, 2026

Rethinking Forward Processes for Score-Based Nonlinear Data Assimilation in High Dimensions

Data assimilation is the process of estimating the state of a dynamical system over time by combining model predictions with measurements. This task becomes challenging when the system is nonlinear and high-dimensional. To address this, score-based Bayesian filters have recently emerged. However, these methods still show unsatisfactory performance in certain cases, particularly under spatially sparse measurements. Such degradation stems from heuristic approximations of the likelihood score, whose errors can accumulate over time. This limitation arises because the methods simply adopt a classical forward process for generative modeling that transforms a data distribution toward a Gaussian distribution, which is independent of the measurement equation. Here, we propose a forward process tailored for filtering that transforms the system state toward the measurement space, enabling a theoretically sound formulation of the likelihood score. Based on this, we develop the Measurement-Aware Score-based Filter (MASF). We evaluate MASF on Kolmogorov flow, a high-dimensional fluid benchmark with up to O(105)\mathcal{O}(10^5) dimensions, under diverse measurement operators, including nonlinear cases with a dimensional mismatch between the state and the measurements. MASF shows improved performance over existing score-based filters and ensemble-type Kalman filters. Notably, MASF achieves up to a 28.2×28.2\times wall-clock speedup compared with the baselines when using amortized pretraining. Our implementation is available at https://github.com/tcnllab-oss/masf.

Eunbi Yoon, Won Chang, Donghan Kim +1

KAIST · Seoul National University

Data AssimilationBayesian FilteringScore-Based Generative Modeling
AskPDF
Mar 22, 2026

Closed-form conditional diffusion models for data assimilation

We propose closed-form conditional diffusion models for data assimilation. Diffusion models use data to learn the score function (defined as the gradient of the log-probability density of a data distribution), allowing them to generate new samples from the data distribution by reversing a noise injection process. While it is common to train neural networks to approximate the score function, we leverage the analytical tractability of the score function to assimilate the states of a system with measurements. To enable the efficient evaluation of the score function, we use kernel density estimation to model the joint distribution of the states and their corresponding measurements. The proposed approach also inherits the capability of conditional diffusion models of operating in black-box settings, i.e., the proposed data assimilation approach can accommodate systems and measurement processes without their explicit knowledge. The ability to accommodate black-box systems combined with the superior capabilities of diffusion models in approximating complex, non-Gaussian probability distributions means that the proposed approach offers advantages over many widely used filtering methods. We evaluate the proposed method on nonlinear data assimilation problems based on the Lorenz-63 and Lorenz-96 systems of moderate dimensionality and nonlinear measurement models. Results show the proposed approach outperforms the widely used ensemble Kalman and particle filters when small to moderate ensemble sizes are used.

Brianna Binder, Agnimitra Dasgupta, Assad Oberai

Department of Aerospace & Mechanical Engineering, University of Southern California, 3650 McClintock Ave, Los Angeles, 90089, California, USA. · Optimization & Uncertainty Quantification, Sandia National Laboratories, 1450 Innovation Pkwy SE, Albuquerque, 87123, New Mexico, USA.

Data AssimilationConditional Diffusion ModelsConditional Distribution Estimation
AskPDF
Mar 21, 2026

Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithms

Data assimilation algorithms estimate the state of a dynamical system from partial observations, where the successful performance of these algorithms hinges on costly parameter tuning and on employing an accurate model for the dynamics. This paper introduces a framework for jointly learning the state, dynamics, and parameters of filtering algorithms in data assimilation through a process we refer to as auto-differentiable filtering. The framework leverages a theoretically motivated loss function that enables learning from partial, noisy observations via gradient-based optimization using auto-differentiation. We further demonstrate how several well-known data assimilation methods can be learned or tuned within this framework. To underscore the versatility of auto-differentiable filtering, we perform experiments on dynamical systems spanning multiple scientific domains, such as the Clohessy-Wiltshire equations from aerospace engineering, the Lorenz-96 system from atmospheric science, and the generalized Lotka-Volterra equations from systems biology. Finally, we provide guidelines for practitioners to customize our framework according to their observation model, accuracy requirements, and computational budget.

Melissa Adrian, Daniel Sanz-Alonso, Rebecca Willett

School of Mechanical Engineering, Purdue University, West Lafayette, IN 47907 · Department of Statistics, The University of Chicago, Chicago, IL 60637 · Department of Computer Science, The University of Chicago, Chicago, IL 60637 +1

Data AssimilationState Space ModelsSystem Identification
AskPDF
Mar 20, 2026

Operator Learning for Smoothing and Forecasting

Machine learning has opened new frontiers in purely data-driven algorithms for data assimilation in, and for forecasting of, dynamical systems; the resulting methods are showing some promise. However, in contrast to model-driven algorithms, analysis of these data-driven methods is poorly developed. In this paper we address this issue, developing a theory to underpin data-driven methods to solve smoothing problems arising in data assimilation and forecasting problems. The theoretical framework relies on two key components: (i) establishing the existence of the mapping to be learned; (ii) the properties of the operator learning architecture used to approximate this mapping. By studying these two components in conjunction, we establish novel universal approximation theorems for purely data driven algorithms for both smoothing and forecasting of dynamical systems. We work in the continuous time setting, hence deploying neural operator architectures. The theoretical results are illustrated with experiments studying the Lorenz 63, Lorenz 96 and Kuramoto-Sivashinsky dynamical systems.

Edoardo Calvello, Elizabeth Carlson, Nikola Kovachki +2

California Institute of Technology, Pasadena, CA · Oregon State University, Corvallis, OR · NVIDIA Corporation, Santa Clara, CA +1

Data AssimilationOperator LearningDynamical SystemsNeural Operators
AskPDF
Mar 20, 2026

Sparsely-Supervised Data Assimilation via Physics-Informed Schrödinger Bridge

Data assimilation (DA) for systems governed by partial differential equations (PDE) aims to reconstruct full spatiotemporal fields from sparse high-fidelity (HF) observations while respecting physical constraints. While full-grid low-fidelity (LF) simulations provide informative priors in multi-fidelity settings, recovering an HF field consistent with both sparse observations and the governing PDE typically requires per-instance test-time optimization, which becomes a major bottleneck in time-critical applications. To alleviate this, amortized reconstruction using generative models has recently been proposed; however, such approaches rely on full-field HF supervision during training, which is often impractical in real-world settings. From a more realistic perspective, we propose the Physics-Informed Conditional Schrödinger Bridge (PICSB), which transports an informative LF prior toward an observation-conditioned HF posterior without any additional inference-time guidance. To enable learning without HF endpoints, PICSB employs an iterative surrogate-endpoint refresh scheme, and directly incorporates PDE residuals into the training objective while enforcing observations via hard conditioning throughout sampling. Experiments on fluid PDE benchmarks demonstrate that PICSB enables extremely fast spatiotemporal field reconstruction while maintaining competitive accuracy under sparse HF supervision.

Dohyun Bu, Chanho Kim, Seokun Choi +1

Department of Industrial & Systems Engineering, KAIST, Daejeon, Repulic of Korea · Department of Mathematical Sciences, KAIST, Daejeon, Repulic of Korea

Data AssimilationPhysics-Informed Generative ModelingAmortized Inference
AskPDF
Mar 19, 2026

Enhancing the Parameterization of Reservoir Properties for Data Assimilation Using Deep VAE-GAN

Currently, the methods called Iterative Ensemble Smoothers, especially the method called Ensemble Smoother with Multiple Data Assimilation (ESMDA) can be considered state-of-the-art for history matching in petroleum reservoir simulation. However, this approach has two important limitations: the use of an ensemble with finite size to represent the distributions and the Gaussian assumption in parameter and data uncertainties. This latter is particularly important because many reservoir properties have non-Gaussian distributions. Parameterization involves mapping non-Gaussian parameters to a Gaussian field before the update and then mapping them back to the original domain to forward the ensemble through the reservoir simulator. A promising approach to perform parameterization is through deep learning models. Recent studies have shown that Generative Adversarial Networks (GAN) performed poorly concerning data assimilation, but generated more geologically plausible realizations of the reservoir, while the Variational Autoencoder (VAE) performed better than the GAN in data assimilation, but generated less geologically realistic models. This work is innovative in combining the strengths of both to implement a deep learning model called Variational Autoencoder Generative Adversarial Network (VAE-GAN) integrated with ESMDA. The methodology was applied in two case studies, one case being categorical and the other with continuous values of permeability. Our findings demonstrate that by applying the VAE-GAN model we can obtain high quality reservoir descriptions (just like GANs) and a good history matching on the production curves (just like VAEs) simultaneously.

M. A. Sampaio, P. H. Ranazzi, M. J. Blunt

Departamento de Engenharia de Minas e de Petróleo, Escola Politécnica, Universidade de São Paulo, 05508-030, SP, Brasil. · Department of Earth Science and Engineering, Imperial College London, South Kensington, London, UK

Data AssimilationParameter Estimation
AskPDF
Mar 17, 2026

Implementation of tangent linear and adjoint models for neural networks based on a compiler library tool

This paper presents TorchNWP, a compilation library tool for the efficient coupling of artificial intelligence components and traditional numerical models. It aims to address the issues of poor cross-language compatibility, insufficient coupling flexibility, and low data transfer efficiency between operational numerical models developed in Fortran and Python-based deep learning frameworks. Based on LibTorch, it optimizes and designs a unified application-layer calling interface, converts deep learning models under the PyTorch framework into a static binary format, and provides C/C++ interfaces. Then, using hybrid Fortran/C/C++ programming, it enables the deployment of deep learning models within numerical models. Integrating TorchNWP into a numerical model only requires compiling it into a callable link library and linking it during the compilation and linking phase to generate the executable. On this basis, tangent linear and adjoint model based on neural networks are implemented at the C/C++ level, which can shield the internal structure of neural network models and simplify the construction process of four-dimensional variational data assimilation systems. Meanwhile, it supports deployment on heterogeneous platforms, is compatible with mainstream neural network models, and enables mapping of different parallel granularities and efficient parallel execution. Using this tool requires minimal code modifications to the original numerical model, thus reducing coupling costs. It can be efficiently integrated into numerical weather prediction models such as CMA-GFS and MCV, and has been applied to the coupling of deep learning-based physical parameterization schemes (e.g., radiation, non-orographic gravity wave drag) and the development of their tangent linear and adjoint models, significantly improving the accuracy and efficiency of numerical weather prediction.

Sa Xiao, Hao Jing, Honglu Sun +1

State Key Laboratory of Severe Weather Meteorological Science and Technology (LaSW), Beijing, China · CMA Earth System Modeling and Prediction Centre (CEMC), Beijing, China

Weather ForecastingData AssimilationEfficient Neural Network Inference
AskPDF
Mar 16, 2026

FuXiWeather2: Learning accurate atmospheric state estimation for operational global weather forecasting

Numerical weather prediction has long been constrained by the computational bottlenecks inherent in data assimilation and numerical modeling. While machine learning has accelerated forecasting, existing models largely serve as "emulators of reanalysis products," thereby retaining their systematic biases and operational latencies. Here, we present FuXiWeather2, a unified end-to-end neural framework for assimilation and forecasting. We align training objectives directly with a combination of real-world observations and reanalysis data, enabling the framework to effectively rectify inherent errors within reanalysis products. To address the distribution shift between NWP-derived background inputs during training and self-generated backgrounds during deployment, we introduce a recursive unrolling training method to enhance the precision and stability of analysis generation. Furthermore, our model is trained on a hybrid dataset of raw and simulated observations to mitigate the impact of observational distribution inconsistency. FuXiWeather2 generates high-resolution (0.25∘0.25^{\circ}) global analysis fields and 10-day forecasts within minutes. The analysis fields surpass the NCEP-GFS across most variables and demonstrate superior accuracy over both ERA5 and the ECMWF-HRES system in lower-tropospheric and surface variables. These high-quality analysis fields drive deterministic forecasts that exceed the skill of the HRES system in 91% of evaluated metrics. Additionally, its outstanding performance in typhoon track prediction underscores its practical value for rapid response to extreme weather events. The FuXiWeather2 analysis dataset is available at https://doi.org/10.5281/zenodo.18872728.

Xiaoze Xu, Xiuyu Sun, Songling Zhu +5

Shanghai Academy of Artificial Intelligence for Science, Shanghai, & 200232, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai & 200433, China. · FuXi Intelligent Computing Technology Co., Ltd., Shanghai & 200233, China.

Weather ForecastingData Assimilation
AskPDF
Mar 16, 2026

A Score Filter Enhanced Data Assimilation Framework for Data-Driven Dynamical Systems

We introduce a score-filter-enhanced data assimilation framework designed to reduce predictive uncertainty in machine learning (ML) models for data-driven dynamical system forecasting. Machine learning serves as an efficient numerical model for predicting dynamical systems. However, even with sufficient data, model uncertainty remains and accumulates over time, causing the long-term performance of ML models to deteriorate. To overcome this difficulty, we integrate data assimilation techniques into the training process to iteratively refine the model predictions by incorporating observational information. Specifically, we apply the Ensemble Score Filter (EnSF), a generative AI-based training-free diffusion model approach, for solving the data assimilation problem in high-dimensional nonlinear complex systems. This leads to a hybrid data assimilation-training framework that combines ML with EnSF to improve long-term predictive performance. We shall demonstrate that EnSF-enhanced ML can effectively reduce predictive uncertainty in ML-based Lorenz-96 system prediction and the Korteweg-De Vries (KdV) equation prediction.

Jingqiao Tang, Ryan Bausback, Feng Bao +2

Data AssimilationScientific MLChaotic Time-Series Prediction
AskPDF
Mar 13, 2026

Adaptive Diffusion Posterior Sampling for Data and Model Fusion of Complex Nonlinear Dynamical Systems

High-fidelity numerical simulations of chaotic, high dimensional nonlinear dynamical systems are computationally expensive, necessitating the development of efficient surrogate models. Most surrogate models for such systems are deterministic, for example when neural operators are involved. However, deterministic models often fail to capture the intrinsic distributional uncertainty of chaotic systems. This work presents a surrogate modeling formulation that leverages generative machine learning, where a deep learning diffusion model is used to probabilistically forecast turbulent flows over long horizons. We introduce a multi-step autoregressive diffusion objective that significantly enhances long-rollout stability compared to standard single-step training. To handle complex, unstructured geometries, we utilize a multi-scale graph transformer architecture incorporating diffusion preconditioning and voxel-grid pooling. More importantly, our modeling framework provides a unified platform that also predicts spatiotemporally important locations for sensor placement, either via uncertainty estimates or through an error-estimation module. Finally, the observations of the ground truth state at these dynamically varying sensor locations are assimilated using diffusion posterior sampling requiring no retraining of the surrogate model. We present our methodology on two-dimensional homogeneous and isotropic turbulence and for a flow over a backwards-facing step, demonstrating its utility in forecasting, adaptive sensor placement, and data assimilation for high dimensional chaotic systems.

Dibyajyoti Chakraborty, Hojin Kim, Romit Maulik

College of Information Sciences and Technology, Pennsylvania State University, University Park, 16802, PA, USA · School of Mechanical Engineering, Purdue University, West Lafayette, 47907, IN, USA

PDE Surrogate ModelingChaotic Time-Series PredictionAutoregressive DiffusionActive SensingData AssimilationDiffusion Posterior Sampling
AskPDF
Mar 6, 2026

Latent Autoencoder Ensemble Kalman Filter for Nonlinear Data assimilation

The ensemble Kalman filter (EnKF) is widely used for data assimilation in high-dimensional systems, but its performance often deteriorates for strongly nonlinear dynamics due to the structural mismatch between the Kalman update and the underlying system behavior. In this work, we propose a latent autoencoder ensemble Kalman filter (LAE-EnKF) that addresses this limitation by reformulating the assimilation problem in a learned latent space with linear and stable dynamics. The proposed method learns a nonlinear encoder--decoder together with a stable linear latent evolution operator and a consistent latent observation mapping, yielding a closed linear state-space model in the latent coordinates. This construction restores compatibility with the Kalman filtering framework and allows both forecast and analysis steps to be carried out entirely in the latent space. Compared with existing autoencoder-based and latent assimilation approaches that rely on unconstrained nonlinear latent dynamics, the proposed formulation emphasizes structural consistency, stability, and interpretability. We provide a theoretical analysis of learning linear dynamics on low-dimensional manifolds and establish generalization error bounds for the proposed latent model. Numerical experiments on representative nonlinear and chaotic systems demonstrate that the LAE-EnKF yields more accurate and stable assimilation than the standard EnKF and related latent-space methods, while maintaining comparable computational cost and data-driven.

Xin T. Tong, Yanyan Wang, Liang Yan

Department of Mathematics, National University of Singapore, 119076, Singapore · School of Mathematics, Southeast University, Nanjing 210096, China

Data AssimilationEnsemble Kalman FilteringLatent Dynamics Modeling
AskPDF
Mar 4, 2026

Accurate and Efficient Hybrid-Ensemble Atmospheric Data Assimilation in Latent Space with Uncertainty Quantification

Data assimilation (DA) combines model forecasts and observations to estimate the optimal state of the atmosphere with its uncertainty, providing initial conditions for weather prediction and reanalyses for climate research. Yet, existing traditional and machine-learning DA methods struggle to achieve accuracy, efficiency and uncertainty quantification simultaneously. Here, we propose HLOBA (Hybrid-Ensemble Latent Observation-Background Assimilation), a three-dimensional hybrid-ensemble DA method that operates in an atmospheric latent space learned via an autoencoder (AE). HLOBA maps both model forecasts and observations into a shared latent space via the AE encoder and an end-to-end Observation-to-Latent-space mapping network (O2Lnet), respectively, and fuses them through a Bayesian update with weights inferred from time-lagged ensemble forecasts. Both idealized and real-observation experiments demonstrate that HLOBA matches dynamically constrained four-dimensional DA methods in both analysis and forecast skill, while achieving end-to-end inference-level efficiency and theoretical flexibility applies to any forecasting model. Moreover, by exploiting the error decorrelation property of latent variables, HLOBA enables element-wise uncertainty estimates for its latent analysis and propagates them to model space via the decoder. Idealized experiments show that this uncertainty highlights large-error regions and captures their seasonal variability.

Hang Fan, Juan Nathaniel, Yi Xiao +5

Department of Earth and Environmental Engineering, School of Engineering and Applied Sciences, Climate School, Columbia University, New York, NY, USA · Learning the Earth with Artificial Intelligence and Physics (LEAP) Center, Columbia University, New York, NY, USA · Shanghai Artificial Intelligence Laboratory, Shanghai, China +2

Data AssimilationUncertainty Quantification
AskPDF
Feb 27, 2026

BLISSNet: Deep Operator Learning for Fast and Accurate Flow Reconstruction from Sparse Sensor Measurements

Reconstructing fluid flows from sparse sensor measurements is a fundamental challenge in science and engineering. Widely separated measurements and complex, multiscale dynamics make accurate recovery of fine-scale structures difficult. In addition, existing methods face a persistent tradeoff: high-accuracy models are often computationally expensive, whereas faster approaches typically compromise fidelity. In this work, we introduce BLISSNet, a model that strikes a strong balance between reconstruction accuracy and computational efficiency for both flow reconstruction and nudging-based data assimilation. The model follows a DeepONet-like architecture, enabling zero-shot inference on domains of arbitrary size. After the first model call on a given domain, certain network components can be precomputed, leading to low inference cost for subsequent evaluations on large domains. Consequently, the model can achieve faster inference than classical interpolation methods such as radial basis function or bicubic interpolation. This combination of high accuracy, low cost, and zero-shot generalization makes BLISSNet well-suited for large-scale real-time flow reconstruction and data assimilation tasks.

Maksym Veremchuk, K. Andrea Scott, Zhao Pan

Department of Mechanical and Mechatronics Engineering, University of Waterloo

Operator LearningComputational Fluid DynamicsData AssimilationEfficient Neural Network Inference
AskPDF
Feb 26, 2026

Neural ensemble Kalman filter: Data assimilation for compressible flows with shocks

Data assimilation (DA) for compressible flows with shocks is challenging because many classical DA methods generate spurious oscillations and nonphysical features near uncertain shocks. We focus here on the ensemble Kalman filter (EnKF). We show that the poor performance of the EnKF may be attributed to the bimodal forecast distribution that can arise in the vicinity of an uncertain shock location; this violates the assumptions underpinning the EnKF, which assume a forecast which is close to Gaussian. To address this issue we introduce the new neural EnKF. The basic idea is to systematically embed neural function approximations within ensemble DA by mapping the forecast ensemble of shocked flows to the parameter space (weights and biases) of a deep neural network (NN) and to subsequently perform DA in that space. The nonlinear mapping encodes sharp and smooth flow features in an ensemble of NN parameters. Neural EnKF updates are therefore well-behaved only if the NN parameters vary smoothly within the neural representation of the forecast ensemble. We show that such a smooth variation of network parameters can be enforced via physics-informed transfer learning, and demonstrate that in so-doing the neural EnKF avoids the spurious oscillations and nonphysical features that plague the EnKF. The applicability of the neural EnKF is demonstrated through a series of systematic numerical experiments with the inviscid Burgers' equation, the Sod shock tube, and a two-dimensional blast wave.

Xu-Hui Zhou, Lorenzo Beronilla, Michael K. Sleeman +4

Scripps Institution of Oceanography, University of California San Diego, La Jolla, 92093, CA, USA · Computing + Mathematical Sciences, California Institute of Technology, Pasadena, 91125, CA, USA · Department of Mechanical Engineering, Johns Hopkins University, Baltimore, 21218, MD, USA

Data AssimilationEnsemble Kalman FilteringComputational Fluid Dynamics
AskPDF
Feb 25, 2026

D-Flow SGLD: Source-Space Posterior Sampling for Scientific Inverse Problems with Flow Matching

Data assimilation and scientific inverse problems require reconstructing high-dimensional physical states from sparse and noisy observations, ideally with uncertainty-aware posterior samples that remain faithful to learned priors and governing physics. While training-free conditional generation is well developed for diffusion models, corresponding conditioning and posterior sampling strategies for Flow Matching (FM) priors remain comparatively under-explored, especially on scientific benchmarks where fidelity must be assessed beyond measurement misfit. In this work, we study training-free conditional generation for scientific inverse problems under FM priors and organize existing inference-time strategies by where measurement information is injected: (i) guided transport dynamics that perturb sampling trajectories using likelihood information, and (ii) source-distribution inference that performs posterior inference over the source variable while keeping the learned transport fixed. Building on the latter, we propose D-Flow SGLD, a source-space posterior sampling method that augments differentiable source inference with preconditioned stochastic gradient Langevin dynamics, enabling scalable exploration of the source posterior induced by new measurement operators without retraining the prior or modifying the learned FM dynamics. We benchmark representative methods from both families on a hierarchy of problems: 2D toy posteriors, chaotic Kuramoto-Sivashinsky trajectories, and wall-bounded turbulence reconstruction. Across these settings, we quantify trade-offs among measurement assimilation, posterior diversity, and physics/statistics fidelity, and establish D-Flow SGLD as a practical FM-compatible posterior sampler for scientific inverse problems.

Meet Hemant Parikh, Yaqin Chen, Jian-Xun Wang

Sibley School of Mechanical and Aerospace Engineering, Cornell University, Ithaca, NY, USA · Department of Aerospace and Mechanical Engineering, University of Notre Dame, Notre Dame, IN, USA

Posterior SamplingBayesian Inverse ProblemsData AssimilationFlow MatchingStochastic Gradient Langevin Dynamics
AskPDF
Feb 23, 2026

LEVDA: Latent Ensemble Variational Data Assimilation via Differentiable Dynamics

Long-range geophysical forecasts are fundamentally limited by chaotic dynamics and numerical errors. While data assimilation can mitigate these issues, classical variational smoothers require computationally expensive tangent-linear and adjoint models. Conversely, recent efficient latent filtering methods often enforce weak trajectory-level constraints and assume fixed observation grids. To bridge this gap, we propose Latent Ensemble Variational Data Assimilation (LEVDA), an ensemble-space variational smoother that operates in the low-dimensional latent space of a pretrained differentiable neural dynamics surrogate. By performing four-dimensional ensemble-variational (4DEnVar) optimization within an ensemble subspace, LEVDA jointly assimilates states and unknown parameters without the need for adjoint code or auxiliary observation-to-latent encoders. Leveraging the fully differentiable, continuous-in-time-and-space nature of the surrogate, LEVDA naturally accommodates highly irregular sampling at arbitrary spatiotemporal locations. Across three challenging geophysical benchmarks, LEVDA matches or outperforms state-of-the-art latent filtering baselines under severe observational sparsity while providing more reliable uncertainty quantification. Simultaneously, it achieves substantially improved assimilation accuracy and computational efficiency compared to full-state 4DEnVar.

Phillip Si, Peng Chen

School of Computational Sciences and Engineering, Georgia Institute of Technology

Data AssimilationLatent Space Optimization
AskPDF
Jan 27, 2026

Machine learning enhanced data assimilation framework for multiscale carbonate rock characterization

Carbonate reservoirs offer significant capacity for subsurface carbon storage, oil production, and underground hydrogen storage. X-ray computed tomography (X-ray CT) coupled with numerical simulations is commonly used to investigate the multiphase flow behaviors in carbonate rocks. Carbonates exhibit pore size distribution across scales, hindering the comprehensive investigation with conventional X-ray CT images. Imaging samples at both macro and micro-scales (multi-scale imaging) proved to be a viable option in this context. However, multi-scale imaging faces two key limitations: the trade-off between field of view and voxel size necessitates resource-intensive imaging, while multi-scale multi-physics numerical simulations on resulting digital models incur prohibitive computational costs. To address these challenges, we propose a machine learning-enhanced data assimilation framework that leverages experimental drainage relative permeability measurements to achieve efficient characterization of micro-scale structures, delivering a data-driven solution toward a high-fidelity multiscale digital rock modeling. We train a dense neural network (DNN) as a proxy to a multi-scale pore network simulator and couple it with an ensemble smoother with multiple data assimilation (ESMDA) algorithm. DNN-ESMDA framework simultaneously infers the CO2-brine drainage relative permeability of microporosity phases with associated uncertainty estimation, revealing the relative importance of each rock phase and guiding future characterization. Our DNN-ESMDA framework achieves a computational speedup, reducing inference time from thousands of hours to seconds compared with the usage of conventional multiscale numerical simulation. Given this computational efficiency and applicability, the machine learning-enhanced ESMDA framework presents a generalizable approach for characterizing multiscale carbonate rocks.

Zhenkai Bo, Ahmed H. Elsheikh, Hannah P. Menke +5

Institute of GeoEnergy Engineering, Heriot-Watt University, Edinburgh, EH14 4AS, United Kingdom · Department of Geoscience & Engineering, Technical University of Delft, Delft, The Netherlands · PETRONAS Research Sdn Bhd, Bangi, Selangor, Malaysia

Data Assimilation
AskPDF
Jan 16, 2026

GenDA: Generative Data Assimilation on Complex Urban Areas via Classifier-Free Diffusion Guidance

Urban wind flow reconstruction is essential for assessing air quality, heat dispersion, and pedestrian comfort, yet remains challenging when only sparse sensor data are available. We propose GenDA, a generative data assimilation framework that reconstructs high-resolution wind fields on unstructured meshes from limited observations. The model employs a multiscale graph-based diffusion architecture trained on computational fluid dynamics (CFD) simulations and interprets classifier-free guidance as a learned posterior reconstruction mechanism: the unconditional branch learns a geometry-aware flow prior, while the sensor-conditioned branch injects observational constraints during sampling. This formulation enables obstacle-aware reconstruction and generalization to held-out mesh geometries, wind directions, and sensor configurations within the studied urban-flow setting, without retraining. We consider both sparse fixed sensors and trajectory-based observations using the same reconstruction procedure. When evaluated against supervised graph neural network (GNN) baselines and classical reduced-order data assimilation methods, GenDA reduces the relative root-mean-square error (RRMSE) by 25-57% and increases the structural similarity index (SSIM) by 23-33% across the tested meshes. Experiments are conducted on Reynolds-averaged Navier-Stokes (RANS) simulations of a real urban neighborhood in Bristol, United Kingdom, at a characteristic Reynolds number of Re≈2×107\mathrm{Re}\approx2\times10^{7}, featuring complex building geometry and irregular terrain. The proposed framework provides a scalable path toward generative, geometry-aware data assimilation for environmental monitoring in complex domains.

Francisco Giral, Álvaro Manzano, Ignacio Gómez +2

Applied Mathematics Department, ETSIAE-School of Aeronautics, Universidad Politécnica de Madrid, Spain · Department of Aerospace Engineering, University of Michigan, Ann Arbor, MI 48109, USA.

Diffusion Model GuidanceData AssimilationDiffusion ModelsComputational Fluid Dynamics
AskPDF
Aug 25, 2025

Huracan: A skillful end-to-end data-driven system for ensemble data assimilation and weather prediction

Over the past few years, machine learning-based data-driven weather prediction has been transforming operational weather forecasting by providing more accurate forecasts while using a mere fraction of computing power compared to traditional numerical weather prediction (NWP). However, those models still rely on initial conditions from NWP, putting an upper limit on their forecast abilities. A few end-to-end systems have since been proposed, but they have yet to match the forecast skill of state-of-the-art NWP competitors. In this work, we propose Huracan, an observation-driven weather forecasting system which combines an ensemble data assimilation model with a forecast model to produce highly accurate forecasts relying only on observations as inputs. Huracan is not only the first to provide ensemble initial conditions and end-to-end ensemble weather forecasts, but also the first end-to-end system to achieve an accuracy comparable with that of ECMWF ENS, the state-of-the-art NWP competitor, despite using a smaller amount of available observation data. Notably, Huracan matches or exceeds the continuous ranked probability score of ECMWF ENS on 80.2% of the variable and lead time combinations. Our work is a major step forward in end-to-end data-driven weather prediction and opens up opportunities for further improving and revolutionizing operational weather forecasting.

Zekun Ni, Jonathan Weyn, Hang Zhang +7

Microsoft Corporation. · Department of Earth System Science, Ministry of Education Key Laboratory for Earth System Modeling, Institute for Global Change Studies, Tsinghua University, Beijing, China.

Ensemble LearningData AssimilationProbabilistic ForecastingWeather Forecasting
AskPDF
Aug 18, 2025

Flow Matching for Efficient and Scalable Data Assimilation

Data assimilation (DA) estimates a dynamical system's state from noisy observations. Recent generative models like the ensemble score filter (EnSF) improve DA in high-dimensional nonlinear settings but are computationally expensive. We introduce the ensemble flow filter (EnFF), a training-free, flow matching (FM)-based framework that accelerates sampling and offers flexibility in flow design. EnFF uses Monte Carlo estimators for the marginal flow field, localized guidance for observation assimilation, and utilizes a novel flow path that exploits the Bayesian DA formulation. It generalizes classical filters such as the bootstrap particle filter and ensemble Kalman filter. Experiments on high-dimensional benchmarks demonstrate EnFF's improved cost-accuracy tradeoffs and scalability, highlighting FM's potential for efficient, scalable DA. Code is available at https://github.com/Utah-Math-Data-Science/Data-Assimilation-Flow-Matching.

Taos Transue, Bohan Chen, So Takao +1

Department of Mathematics and Scientific Computing and Imaging Institute, University of Utah, Salt Lake City, UT, 84112 · The Computing and Mathematical Sciences Department, California Institute of Technology, 1200 E California Blvd, Pasadena, CA 91125

Flow MatchingKalman FilteringBayesian FilteringData AssimilationEnsemble Kalman Filtering
AskPDF
Jul 1, 2025

Guided Unconditional and Conditional Generative Models for Super-Resolution and Inference of Quasi-Geostrophic Turbulence

Typically, numerical simulations of Earth systems are coarse, and Earth observations are sparse and gappy. We apply four generative diffusion modeling approaches to super-resolution and inference of forced two-dimensional quasi-geostrophic turbulence on the beta-plane from coarse, sparse, and gappy observations. Two guided approaches minimally adapt a pre-trained unconditional model: SDEdit modifies the initial condition, and Diffusion Posterior Sampling (DPS) modifies the reverse diffusion process score. Two conditional approaches, a vanilla variant and classifier-free guidance, require training with paired high-resolution and observation data. We consider multiple test cases spanning: two regimes, eddy and anisotropic-jet turbulence; two Reynolds numbers, 10^3 and 10^4; and two observation types, 4x coarse-resolution fields and coarse, sparse and gappy observations. Our comprehensive skill metrics include norms of the reconstructed vorticity fields, turbulence statistical quantities, and quantifications of the super-resolved probabilistic ensembles and their errors. We also study the sensitivity to tuning parameters such as guidance strength. Results show that the generated super-resolution fields of SDEdit are unphysical, while those of DPS are reasonable but with smoothed fine-scale features; however, neither of these lower-cost models propagates observational information effectively to unobserved regions. The two conditional models require re-training, but reconstruct missing fine-scale features, are cycle-consistent with observations, and predict correct turbulence statistics, including the tails. Further, their mean errors are highly correlated with and predictable from their ensemble standard deviations. Results highlight the tradeoffs between ease of implementation, fidelity (sharpness), and cycle-consistency of the diffusion models, and offer practical guidance for deployment.

Anantha Narayanan Suresh Babu, Akhil Sadam, Pierre F. J. Lermusiaux

Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, USA · Center for Computational Science and Engineering, Massachusetts Institute of Technology, Cambridge, USA

Diffusion Model GuidanceData AssimilationConditional Diffusion ModelsDiffusion Posterior SamplingImage Super-ResolutionComputational Fluid Dynamics
AskPDF