Accurate and Efficient Hybrid-Ensemble Atmospheric Data Assimilation in Latent Space with Uncertainty Quantification

Hang Fan; Juan Nathaniel; Yi Xiao; Ce Bian; Fenghua Ling; Ben Fei; Lei Bai; Pierre Gentine

cs.LGMar 4, 2026

Accurate and Efficient Hybrid-Ensemble Atmospheric Data Assimilation in Latent Space with Uncertainty Quantification

Authors: Hang Fan, Juan Nathaniel, Yi Xiao, Ce Bian, Fenghua Ling, Ben Fei, Lei Bai, Pierre Gentine

Organizations: Department of Earth and Environmental Engineering, School of Engineering and Applied Sciences, Climate School, Columbia University, New York, NY, USA · Learning the Earth with Artificial Intelligence and Physics (LEAP) Center, Columbia University, New York, NY, USA · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China ·

Abstract

Data assimilation (DA) combines model forecasts and observations to estimate the optimal state of the atmosphere with its uncertainty, providing initial conditions for weather prediction and reanalyses for climate research. Yet, existing traditional and machine-learning DA methods struggle to achieve accuracy, efficiency and uncertainty quantification simultaneously. Here, we propose HLOBA (Hybrid-Ensemble Latent Observation-Background Assimilation), a three-dimensional hybrid-ensemble DA method that operates in an atmospheric latent space learned via an autoencoder (AE). HLOBA maps both model forecasts and observations into a shared latent space via the AE encoder and an end-to-end Observation-to-Latent-space mapping network (O2Lnet), respectively, and fuses them through a Bayesian update with weights inferred from time-lagged ensemble forecasts. Both idealized and real-observation experiments demonstrate that HLOBA matches dynamically constrained four-dimensional DA methods in both analysis and forecast skill, while achieving end-to-end inference-level efficiency and theoretical flexibility applies to any forecasting model. Moreover, by exploiting the error decorrelation property of latent variables, HLOBA enables element-wise uncertainty estimates for its latent analysis and propagates them to model space via the decoder. Idealized experiments show that this uncertainty highlights large-error regions and captures their seasonal variability.

Figures & tables

Figure 1: Overview of the HLOBA method. a , HLOBA pipeline. The background field and observations are mapped into a shared latent space using the Encoder and O2Lnet, respectively, and fused in a Bayesian manner based on their uncertainty estimates derived from time-lagged ensembles. The resulting latent analysis is then decoded to obtain the final analysis in model space. b , Generation of time-lagged ensembles. Analyses produced at different assimilation cycles are propagated forward to form a flow-dependent ensemble that represents background uncertainty without requiring parallel ensemble forecasts. c , Comparison of HLOBA with other hybrid DA methods when assimilating real observations on an NVIDIA A100. The x-axis shows the assimilation time per observation, and the y-axis shows forecast improvement relative to 3DVar. Marker size indicates GPU memory usage. d , Climatological estimates of error correlations for latent-space background $\boldsymbol{z}_{b}$ and observation $\boldsymbol{z}_{o}$ (see Methods for details). Inter-variable correlations are computed across latent dimensions at each grid point and averaged in absolute value; spatial correlations are computed across grid points within each latent dimension and averaged in absolute value.

Figure 2: Performance comparison of HLOBA and other hybrid DA methods in cycling DA experiments for 2017. a , Analysis errors from idealized cycling DA experiments, evaluated against ERA5. Observations were sampled from ERA5 using radiosonde and surface observation locations from GDAS at 0000 UTC on 1 January. Four observation times were assimilated in each cycle at 6-hour intervals. b , Annual-mean errors for 5-day forecasts initialized from the analyses in a after each assimilation cycle in a , evaluated against ERA5. c , Analysis errors from cycling experiments with real observations, evaluated using a withheld 10% subset of observations not assimilated. Four observation times were assimilated in each cycle at 12-hour intervals. Errors of ERA5 evaluated at the same observation locations are also shown for comparison. For clarity, H3DVar and HL3DVar are not shown. d , Annual-mean 5-day forecast errors initialized from the analyses in c , evaluated using all available observations.

Figure 3: Uncertainty estimation of HLOBA analyses in idealized experiments. a , Examples of analysis RMSE and corresponding estimated standard deviation (Std.) for Z500 at a single time step, daily mean, and monthly mean. Shown are results for 0000 UTC on 1 February 2017, the daily mean on 1 February, and the monthly mean for February 2017. b , Pearson correlation coefficients (PCC) between the estimated standard deviation and the true RMSE for single-step, daily-mean, and monthly-mean analyses over 2017. c , Seasonal variations in analysis RMSE are reflected in the estimated uncertainty. Results are shown for Q500 and T2m.

Figure 4: Impact of time-lagged ensembles on hybrid DA methods in idealized experiments. a , Impact of introducing time-lagged ensembles on different hybrid DA methods, relative to their non-ensemble counterparts. b , Improvement from using hybrid-ensemble (Hybrid) latent observation error covariance $\mathbf{R}_{z}$ in HLOBA, relative to a climatological configuration (clim.) without ensemble information. c , Same as b , but using hybrid-ensemble latent background error covariance $\mathbf{B}_{z}$ . d , Additional improvement from introducing a hybrid $\mathbf{B}_{z}$ in HLOBA when $\mathbf{R}_{z}$ is fully ensemble-based (Ens.). Values show mean improvements averaged over the data assimilation (DA) and forecast (FC) stages; colors indicate improvements in the forecast stage only.

Figure 5: Advantages of introducing O2Lnet , demonstrated under idealized settings. a , Illustration of observation-only analysis (OOA) based on O2Lnet, obtained by decoding the latent observation variable $\boldsymbol{z}_{o}$ produced by O2Lnet. Observations are sampled from ERA5, with observation locations indicated by black dots. Shown are Z500 and T2m at 0000 UTC on 1 February 2017. b , Accuracy of OOA and its robustness to observation noise, compared with HL3DVar and HL4DVar. Solid lines show results with observation noise drawn from a zero-mean Gaussian distribution with variance equal to 0.03 times the climatological variance, while dashed lines correspond to a variance of 0.1 times the climatological variance. c , Example illustrating the consistency between ensemble-estimated uncertainties of the background and OOA terms and their realized mean squared errors (MSE). Shown is T500 at 0000 UTC on 1 February 2017, with all fields normalized to the range [0, 1]. d , Pearson correlation coefficients between estimated error variances and realized MSEs for the background and observation terms, $\boldsymbol{z}_{b}$ and $\boldsymbol{z}_{o}$ . Results based on climatological variance estimates are also shown for comparison.

Figure S1: Extended analysis performance comparison of HLOBA and other hybrid DA methods in idealized cycling experiments for 2017. This figure extends Fig. 2 a by reporting WRMSE for variables at 300 hPa, 500 hPa, 850 hPa, and the surface, evaluated against ERA5.

Figure S2: Extended forecast performance comparison of HLOBA and other hybrid DA methods in idealized cycling experiments for 2017. This figure extends Fig. 2 b by reporting WRMSE for variables at 300 hPa, 500 hPa, 850 hPa, and the surface, evaluated against ERA5.

Figure S3: Extended analysis performance comparison of HLOBA and other hybrid DA methods with real observations in cycling experiments for 2017. This figure extends Fig. 2 c by reporting RMSE for variables at 300 hPa, 500 hPa, 850 hPa, and the surfacee, evaluated against observations withheld from assimilation.

Figure S4: Extended forecast performance comparison of HLOBA and other hybrid DA methods with real observations in cycling experiments for 2017. This figure extends Fig. 2 d by reporting RMSE for variables at 300 hPa, 500 hPa, 850 hPa, and the surfacee, evaluated against all real observations.

Figure S5: The O2Lnet-derived observation-only analysis (OOA) at 0000 UTC on 1 February 2017, using simulated observations sampled from ERA5. Shown are 2-m temperature (T2m), 850-hPa specific humidity (Q850), 500-hPa geopotential height (Z500), and 200-hPa zonal wind (U200).The observation locations are indicated by black dots.

Figure S6: Same as Figure S5 , but for the case at 0000 UTC on 1 October 2017.

Figure S7: Extended accuracy of OOA and its robustness to observation noise in idealized experiments, compared with HL3DVar and HL4DVar This figure extends Fig. 5 b by reporting WRMSE for variables at 300 hPa, 500 hPa, 850 hPa, and the surface, evaluated against ERA5.

Data Assimilation

Data assimilation combines observations with dynamical models to estimate hidden system states and their uncertainty over time.

See all 58 papers

Multimodal Spatiotemporal Atmospheric Data Assimilation with Latent Video Flow-matching
Aug 5, 2026Dibyajyoti Chakraborty, Romit MaulikData Assimilation Spatiotemporal Forecasting
Information Sciences and Technology The Pennsylvania State University University Park, Pennsylvania · School of Mechanical Engineering Purdue University West Lafayette, Indiana
LEVDA: Latent Ensemble Variational Data Assimilation via Differentiable Dynamics
Feb 23, 2026Phillip Si, Peng ChenData Assimilation Latent Space Optimization
School of Computational Sciences and Engineering, Georgia Institute of Technology
LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observations
Oct 2, 2026Ziyu Tian, Kaichen Shen, Wenbo Hao +3Data Assimilation Bayesian Filtering
School of Mathematics, Georgia Institute of Technology · School of Computational Science and Engineering, Georgia Institute of Technology

Aug 5, 2026J/K move · Enter open · S save

Multimodal Spatiotemporal Atmospheric Data Assimilation with Latent Video Flow-matching

Dibyajyoti Chakraborty, Romit Maulik

Information Sciences and Technology The Pennsylvania State University University Park, Pennsylvania · School of Mechanical Engineering Purdue University West Lafayette, Indiana

Data Assimilation Spatiotemporal Forecasting

Open paper AskPDF arXiv

Data assimilation (DA) uses Bayesian inference to update the state of a numerical forecast model with observed data. In this study, we propose a fundamentally different, unified approach to atmospheric data assimilation. We use latent video flow-matching to sample temporally consistent trajectories from a prior trained using ERA5 reanalysis (69 variables over an 8-day window). We also use posterior sampling to assimilate real observation sources, such as those from the NOAA Integrated Global Radiosonde Archive and the Integrated Surface Database. Because the prior generates a continuous trajectory, it naturally propagates information between observed and unobserved frames. Therefore, we can perform various DA tasks, such as filtering and smoothing, simply by changing the observed frames. Moreover, we generate full-state ensemble forecasts directly from sparse observations, achieving performance competitive with state-of-the-art observation-to-forecast models.

Abstract

Data Assimilation

Explore similar work

Multimodal Spatiotemporal Atmospheric Data Assimilation with Latent Video Flow-matching

LEVDA: Latent Ensemble Variational Data Assimilation via Differentiable Dynamics

LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observations