Accurate and Efficient Hybrid-Ensemble Atmospheric Data Assimilation in Latent Space with Uncertainty Quantification
Authors: Hang Fan, Juan Nathaniel, Yi Xiao, Ce Bian, Fenghua Ling, Ben Fei, Lei Bai, Pierre Gentine
Organizations: Department of Earth and Environmental Engineering, School of Engineering and Applied Sciences, Climate School, Columbia University, New York, NY, USA · Learning the Earth with Artificial Intelligence and Physics (LEAP) Center, Columbia University, New York, NY, USA · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China · The Chinese University of Hong Kong, Hong Kong, China
Data assimilation (DA) combines model forecasts and observations to estimate the optimal state of the atmosphere with its uncertainty, providing initial conditions for weather prediction and reanalyses for climate research. Yet, existing traditional and machine-learning DA methods struggle to achieve accuracy, efficiency and uncertainty quantification simultaneously. Here, we propose HLOBA (Hybrid-Ensemble Latent Observation-Background Assimilation), a three-dimensional hybrid-ensemble DA method that operates in an atmospheric latent space learned via an autoencoder (AE). HLOBA maps both model forecasts and observations into a shared latent space via the AE encoder and an end-to-end Observation-to-Latent-space mapping network (O2Lnet), respectively, and fuses them through a Bayesian update with weights inferred from time-lagged ensemble forecasts. Both idealized and real-observation experiments demonstrate that HLOBA matches dynamically constrained four-dimensional DA methods in both analysis and forecast skill, while achieving end-to-end inference-level efficiency and theoretical flexibility applies to any forecasting model. Moreover, by exploiting the error decorrelation property of latent variables, HLOBA enables element-wise uncertainty estimates for its latent analysis and propagates them to model space via the decoder. Idealized experiments show that this uncertainty highlights large-error regions and captures their seasonal variability.
Figures & tables
Figure 1: Overview of the HLOBA method. a , HLOBA pipeline. The background field and observations are mapped into a shared latent space using the Encoder and O2Lnet, respectively, and fused in a Bayesian manner based on their uncertainty estimates derived from time-lagged ensembles. The resulting latent analysis is then decoded to obtain the final analysis in model space. b , Generation of time-lagged ensembles. Analyses produced at different assimilation cycles are propagated forward to form a flow-dependent ensemble that represents background uncertainty without requiring parallel ensemble forecasts. c , Comparison of HLOBA with other hybrid DA methods when assimilating real observations on an NVIDIA A100. The x-axis shows the assimilation time per observation, and the y-axis shows forecast improvement relative to 3DVar. Marker size indicates GPU memory usage. d , Climatological estimates of error correlations for latent-space background zb and observation zo (see Methods for details). Inter-variable correlations are computed across latent dimensions at each grid point and averaged in absolute value; spatial correlations are computed across grid points within each latent dimension and averaged in absolute value.
Figure 2: Performance comparison of HLOBA and other hybrid DA methods in cycling DA experiments for 2017. a , Analysis errors from idealized cycling DA experiments, evaluated against ERA5. Observations were sampled from ERA5 using radiosonde and surface observation locations from GDAS at 0000 UTC on 1 January. Four observation times were assimilated in each cycle at 6-hour intervals. b , Annual-mean errors for 5-day forecasts initialized from the analyses in a after each assimilation cycle in a , evaluated against ERA5. c , Analysis errors from cycling experiments with real observations, evaluated using a withheld 10% subset of observations not assimilated. Four observation times were assimilated in each cycle at 12-hour intervals. Errors of ERA5 evaluated at the same observation locations are also shown for comparison. For clarity, H3DVar and HL3DVar are not shown. d , Annual-mean 5-day forecast errors initialized from the analyses in c , evaluated using all available observations.
Figure 3: Uncertainty estimation of HLOBA analyses in idealized experiments. a , Examples of analysis RMSE and corresponding estimated standard deviation (Std.) for Z500 at a single time step, daily mean, and monthly mean. Shown are results for 0000 UTC on 1 February 2017, the daily mean on 1 February, and the monthly mean for February 2017. b , Pearson correlation coefficients (PCC) between the estimated standard deviation and the true RMSE for single-step, daily-mean, and monthly-mean analyses over 2017. c , Seasonal variations in analysis RMSE are reflected in the estimated uncertainty. Results are shown for Q500 and T2m.
Figure 4: Impact of time-lagged ensembles on hybrid DA methods in idealized experiments. a , Impact of introducing time-lagged ensembles on different hybrid DA methods, relative to their non-ensemble counterparts. b , Improvement from using hybrid-ensemble (Hybrid) latent observation error covariance Rz in HLOBA, relative to a climatological configuration (clim.) without ensemble information. c , Same as b , but using hybrid-ensemble latent background error covariance Bz . d , Additional improvement from introducing a hybrid Bz in HLOBA when Rz is fully ensemble-based (Ens.). Values show mean improvements averaged over the data assimilation (DA) and forecast (FC) stages; colors indicate improvements in the forecast stage only.
Figure 5: Advantages of introducing O2Lnet , demonstrated under idealized settings. a , Illustration of observation-only analysis (OOA) based on O2Lnet, obtained by decoding the latent observation variable zo produced by O2Lnet. Observations are sampled from ERA5, with observation locations indicated by black dots. Shown are Z500 and T2m at 0000 UTC on 1 February 2017. b , Accuracy of OOA and its robustness to observation noise, compared with HL3DVar and HL4DVar. Solid lines show results with observation noise drawn from a zero-mean Gaussian distribution with variance equal to 0.03 times the climatological variance, while dashed lines correspond to a variance of 0.1 times the climatological variance. c , Example illustrating the consistency between ensemble-estimated uncertainties of the background and OOA terms and their realized mean squared errors (MSE). Shown is T500 at 0000 UTC on 1 February 2017, with all fields normalized to the range [0, 1]. d , Pearson correlation coefficients between estimated error variances and realized MSEs for the background and observation terms, zb and zo . Results based on climatological variance estimates are also shown for comparison.
Figure S1: Extended analysis performance comparison of HLOBA and other hybrid DA methods in idealized cycling experiments for 2017. This figure extends Fig. 2 a by reporting WRMSE for variables at 300 hPa, 500 hPa, 850 hPa, and the surface, evaluated against ERA5.
Figure S2: Extended forecast performance comparison of HLOBA and other hybrid DA methods in idealized cycling experiments for 2017. This figure extends Fig. 2 b by reporting WRMSE for variables at 300 hPa, 500 hPa, 850 hPa, and the surface, evaluated against ERA5.
Figure S3: Extended analysis performance comparison of HLOBA and other hybrid DA methods with real observations in cycling experiments for 2017. This figure extends Fig. 2 c by reporting RMSE for variables at 300 hPa, 500 hPa, 850 hPa, and the surfacee, evaluated against observations withheld from assimilation.
Figure S4: Extended forecast performance comparison of HLOBA and other hybrid DA methods with real observations in cycling experiments for 2017. This figure extends Fig. 2 d by reporting RMSE for variables at 300 hPa, 500 hPa, 850 hPa, and the surfacee, evaluated against all real observations.
Figure S5: The O2Lnet-derived observation-only analysis (OOA) at 0000 UTC on 1 February 2017, using simulated observations sampled from ERA5. Shown are 2-m temperature (T2m), 850-hPa specific humidity (Q850), 500-hPa geopotential height (Z500), and 200-hPa zonal wind (U200).The observation locations are indicated by black dots.
Figure S6: Same as Figure S5 , but for the case at 0000 UTC on 1 October 2017.
Figure S7: Extended accuracy of OOA and its robustness to observation noise in idealized experiments, compared with HL3DVar and HL4DVar This figure extends Fig. 5 b by reporting WRMSE for variables at 300 hPa, 500 hPa, 850 hPa, and the surface, evaluated against ERA5.
Data assimilation (DA) uses Bayesian inference to update the state of a numerical forecast model with observed data. In this study, we propose a fundamentally different, unified approach to atmospheric data assimilation. We use latent video flow-matching to sample temporally consistent trajectories from a prior trained using ERA5 reanalysis (69 variables over an 8-day window). We also use posterior sampling to assimilate real observation sources, such as those from the NOAA Integrated Global Radiosonde Archive and the Integrated Surface Database. Because the prior generates a continuous trajectory, it naturally propagates information between observed and unobserved frames. Therefore, we can perform various DA tasks, such as filtering and smoothing, simply by changing the observed frames. Moreover, we generate full-state ensemble forecasts directly from sparse observations, achieving performance competitive with state-of-the-art observation-to-forecast models.
Dibyajyoti Chakraborty, Romit Maulik
Information Sciences and Technology The Pennsylvania State University University Park, Pennsylvania · School of Mechanical Engineering Purdue University West Lafayette, Indiana
Long-range geophysical forecasts are fundamentally limited by chaotic dynamics and numerical errors. While data assimilation can mitigate these issues, classical variational smoothers require computationally expensive tangent-linear and adjoint models. Conversely, recent efficient latent filtering methods often enforce weak trajectory-level constraints and assume fixed observation grids. To bridge this gap, we propose Latent Ensemble Variational Data Assimilation (LEVDA), an ensemble-space variational smoother that operates in the low-dimensional latent space of a pretrained differentiable neural dynamics surrogate. By performing four-dimensional ensemble-variational (4DEnVar) optimization within an ensemble subspace, LEVDA jointly assimilates states and unknown parameters without the need for adjoint code or auxiliary observation-to-latent encoders. Leveraging the fully differentiable, continuous-in-time-and-space nature of the surrogate, LEVDA naturally accommodates highly irregular sampling at arbitrary spatiotemporal locations. Across three challenging geophysical benchmarks, LEVDA matches or outperforms state-of-the-art latent filtering baselines under severe observational sparsity while providing more reliable uncertainty quantification. Simultaneously, it achieves substantially improved assimilation accuracy and computational efficiency compared to full-state 4DEnVar.
Phillip Si, Peng Chen
School of Computational Sciences and Engineering, Georgia Institute of Technology
Data assimilation combines model forecasts with noisy, incomplete observations to estimate the evolving state of a dynamical system. Existing methods face two compounding challenges: high-dimensional nonlinear dynamics make repeated forward simulation computationally expensive, while sparse observations provide limited direct information about the full state. To address these challenges, we propose the Latent-Dynamics Ensemble Flow Filter (LD-EnFF), a sequential Bayesian filtering framework that performs both forecast propagation and filtering updates in a compact latent space. LD-EnFF combines a latent dynamics surrogate for ensemble propagation with a variational autoencoder (VAE)-based observation model that evaluates a state-dependent observation likelihood in latent space. At each assimilation step, an ensemble filtering update based on flow matching uses the forecast ensemble and this likelihood to generate posterior samples, jointly updating latent states and uncertain parameters. This design avoids repeated full-state simulation during forecasting and full-field reconstruction during likelihood evaluation. LD-EnFF substantially outperforms a broad range of data assimilation algorithms on benchmarks spanning Kolmogorov flow, tsunami propagation, and atmospheric modeling, all featuring complex dynamics and sparse, noisy observations.
Ziyu Tian, Kaichen Shen, Wenbo Hao +3
School of Mathematics, Georgia Institute of Technology · School of Computational Science and Engineering, Georgia Institute of Technology