Bridging the Simulation-to-Reality Gap in Electron Microscope Calibration via VAE-EM Estimation

Jilles S. van Hulst; W. P. M. H. Heemels; Duarte J. Antunes

cs.CVMar 17, 2026

Bridging the Simulation-to-Reality Gap in Electron Microscope Calibration via VAE-EM Estimation

Authors: Jilles S. van Hulst, W. P. M. H. Heemels, Duarte J. Antunes

Organizations: The authors are with the Control Systems Technology Section, Department of Mechanical Engineering, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands.

Abstract

Electron microscopy has enabled many scientific breakthroughs across multiple fields. A key challenge is the tuning of microscope parameters based on images to overcome optical aberrations that deteriorate image quality. This calibration problem is challenging due to the high-dimensional and noisy nature of the diagnostic images, and the fact that optimal parameters cannot be identified from a single image. We tackle the calibration problem for Scanning Transmission Electron Microscopes (STEM) by employing variational autoencoders (VAEs), trained on simulated data, to learn low-dimensional representations of images, whereas most existing methods extract only scalar values. We then simultaneously estimate the model that maps calibration parameters to encoded representations and the optimal calibration parameters using an expectation maximization (EM) approach. This joint estimation explicitly addresses the simulation-to-reality gap inherent in data-driven methods that train on simulated data from a digital twin. We leverage the known symmetry property of the optical system to establish global identifiability of the joint estimation problem, ensuring that a unique optimum exists. We demonstrate that our approach is substantially faster and more consistent than existing methods on a real STEM, achieving a 2x reduction in estimation error while requiring fewer observations. This represents a notable advance in automated STEM calibration and demonstrates the potential of VAEs for information compression in images. Beyond microscopy, the VAE-EM framework applies to inverse problems where simulated training data introduces a reality gap and where non-injective mappings would otherwise prevent unique solutions.

Figures & tables

Fig. 1: Grid of Ronchigram images (left) and their Fourier power spectra (right) for different aberration values. The horizontal axis varies defocus ( $C1$ ) from $-200$ to $+200$ nm, while the vertical axis varies two-fold astigmatism ( $A1x$ ) over the same range. The center images correspond to well-tuned aberrations ( $C1=0$ , $A1x=0$ ). Colored rectangles group images in specific aberration regions. Note that the Fourier power spectra exhibit symmetry about the origin, with images at $(x_{1},x_{2})$ and $(-x_{1},-x_{2})$ appearing similar (e.g., purple and orange, cyan and red).

Fig. 2: Calibration pipeline overview. High-dimensional Ronchigram images are preprocessed and then encoded to a low-dimensional latent space using a VAE. The EM algorithm then iteratively refines both the aberration state estimate and the mapping from aberrations to latent representations. In the E-step, the distribution over aberration states is estimated by evaluating the likelihood of observed latent space values for different candidate states. These candidates ( $\bullet$ $\bullet$ $\bullet$ ) are highlighted in the aberration space, where the color indicates their estimated likelihood (low $\bullet$ to high $\bullet$ ). The $\boldsymbol{\times}$ indicates the ground truth aberration state. Note that this state and its symmetrical neighbor have an increased density of candidates due to the adaptive candidate refinement strategy. The most likely candidates are also clustered around the ground truth state. The M-step then updates the Gaussian process model of the aberration to latent space mapping. The current model belief based on 5 data points ( $\circ$ ) is illustrated as a function of the aberrations. Note that this model belief is symmetric due to the symmetry constraint in the model structure. The symmetry point is not necessarily at the origin (in the true aberration coordinates) due to the fact that the state estimate is not yet perfect. The M- and E-steps are iterated until convergence.

Fig. 3: Variational Autoencoder architecture. The encoder maps preprocessed Ronchigram power spectra to a low-dimensional latent distribution, from which samples are decoded to reconstruct the input. The weights and biases of the encoder and decoder networks are optimized to maximize the ELBO objective in ( 7 ), which balances reconstruction accuracy and latent space regularization. The example reconstruction to the right of the decoder resembles the input and shows the denoising effect of the VAE.

Fig. 4: Learned VAE latent space for simulated data that contains two varying aberration parameters: defocus (

C1

), and two-fold astigmatism in the x-direction (

A1x

). The color of the data points is determined by the aberration region, with the assignment shown in the top right of the figure and in Fig. 1 . We can observe that similar aberrations are grouped together. Additionally, different aberration values that produce similar Ronchigrams are still mapped to the same latent space values (example: purple and orange), demonstrating the non-injectivity of the observation model.

Fig. 5: Latent space distribution for Ronchigram datasets gathered on real STEM on different days. Left: dataset 1, middle: dataset 2, right: dataset 3. The latent space data points are colored according to their aberration values, with the color-coding scheme taken from Fig. 1 . We can observe consistency across datasets, as well as with respect to the simulated data from Fig. 4 , indicating that the VAE has learned a meaningful latent representation that reflects the underlying aberration states.

Fig. 6: Fitted prior mean function

\mu_{f}(x)

mapping aberration parameters to VAE latent space. Left: first latent dimension

z_{1}

as a function of defocus (

C1

) and two-fold astigmatism (

A1x

). Right: second latent dimension

z_{2}

. The prior mean is generated by fitting piecewise linear bases to a large set of simulated Ronchigrams. Note the symmetry with respect to the origin, reflecting the physical property

f(-x)=f(x)

Fig. 7: Estimation performance on real STEM data over 100 Monte Carlo runs with different random initializations with 3 aberration parameters (

C1,A1x,A1y

). We view the aberration estimation error 2-norm as a function of the number of Ronchigram observations used for calibration. The proposed VAE-EM method consistently outperforms existing automated calibration approaches in terms of speed and accuracy.

Fig. 8: Estimation error performance on real STEM data after 10 Ronchigram observations (left) and 30 observations (right). The violin plots summarize the distribution of estimation error 2-norms with 3 aberration parameters (

C1,A1x,A1y

) across 100 Monte Carlo runs with different random initializations.

Electron Microscopy

See all 36 papers

Jul 24, 2026

Frequency-Aware Dual-Stream Learning for Balanced Realism and Fidelity in Electron Microscopy Imaging

Electron microscopy enables nanoscale cellular visualization but faces a trade-off between imaging resolution and acquisition speed. Existing learning-based methods rely on single-stream architectures that struggle to balance perceptual realism and quantitative fidelity, either over-smoothing details or generating unrealistic hallucinations. This work introduces a frequency-adaptive dual-stream architecture to resolve this conflict. Using discrete wavelet transform, we decompose images into low-frequency structures and high-frequency details, then employ a conditional diffusion model for realistic global synthesis and a transformer network for precise detail recovery. Experiments on the EMDiffuse dataset show the method achieves superior LPIPS and resolution ratio, substantially outperforming existing approaches. The method also shows strong generalization across diverse biological samples, supporting fast and reliable electron microscopy imaging for structural biology and nanotechnology applications. The source code and associated dataset are publicly available to facilitate further research.

Longmi Gao, Zhengkai Zhao, Pan Gao +1

College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China. · School of computing, Mathematics, and Engineering, Charles Sturt University, NSW, Australia.

Diffusion Models for Image Restoration Microscopy Image Analysis Image Super-Resolution Electron Microscopy

Ask PDF

Jun 28, 2026

STEMGym: Benchmarking Sequential Decision-Making under Dose Budgets in Autonomous Electron Microscopy

A central premise of autonomous scientific imaging is that smarter navigation, whether Bayesian, RL-based, or otherwise adaptive, is the principal lever for sample-efficient acquisition. We present evidence to the contrary in scanning transmission electron microscopy (STEM), an atomic-resolution imaging modality whose every measurement deposits damaging electron dose. We introduce STEMGym, an open-source Gymnasium benchmark of 15 physics-simulated STEM worlds spanning five materials, three difficulty levels, and four characterisation tasks, scored by the Dose-Efficiency Curve area (DEC-AUC), a single scalar capturing the information-vs-dose Pareto frontier. Across 33 agent configurations under realistic dose budgets, the dominant determinant of dose efficiency is the analyst (perception) pipeline, not the navigator: pairing a trained CNN analyst with naïve raster scanning raises DEC-AUC by 5.5x over a CNN-free raster baseline (0.287 vs.\ 0.052), while substituting Bayesian or adaptive finite-state-machine navigation for raster yields no statistically significant further gain. Production-tier vision-language models further underperform task-specific CNNs by {\sim}13x on crystallographic defect analysis. By decoupling perception, navigation, and planning under a unified dose budget, STEMGym reframes where ML effort should be invested in autonomous electron microscopy and provides the measurement infrastructure to test it.

Can Polat, Erchin Serpedin, Mustafa Kurban +1

Dept. of Electrical & Computer Eng. Texas A&M Univ. College Station, TX, USA · Dept. of Prosthetics & Orthotics College of Science & Eng. Ankara Univ., Ankara, Turkey · Hamad Bin Khalifa Univ. and Texas A&M Univ. at Qatar Doha, Qatar

Scanning Transmission Electron Microscopy Autonomous Experimentation Microscopy Image Analysis Electron Microscopy

Ask PDF

Apr 27, 2026

Contrastive Image-Metadata Pre-Training for Materials Transmission Electron Microscopy

The transmission electron microscope facilitates the highest-resolution imaging of any instrument ever created, and its limiting factor is no longer spatial resolution but dose efficiency. Low electron doses avoid sample damage but produce noisy images for which, unlike in classical computer vision, there is no ground truth. Autonomous materials experimentation poses a related problem, since closed-loop instruments need representations grounded in the microscope state at acquisition. Both demand representations grounded in how an image was acquired. We release 7,330 paired high-angle annular dark-field scanning-TEM (HAADF-STEM) images and their seven-dimensional acquisition metadata, and propose Contrastive Image-Metadata Pre-training (CIMP), a CLIP-style encoder that aligns the two modalities and reaches 84.4% Top-1 cross-modal retrieval on a held-out split. All seven parameters are individually recoverable from the frozen visual embedding through a linear probe, and we use the embedding to condition a metadata-conditioned style-transfer model that re-renders experimental images under different acquisition parameters. Virtually scaling dwell time and beam current of low-dose images turns this model into a physics-informed denoiser; in a blind user study, experimental microscopists prefer it over the current state-of-the-art denoiser for STEM imagery on 70.2% of trials.

Georgia Channing, Debora Keller, Marta D. Rossell +3

Hugging Face · University of Oxford · Empa Dübendorf +1

Cross-Modal Alignment Multimodal Contrastive Learning Scanning Transmission Electron Microscopy Microscopy Image Analysis Image Denoising Electron Microscopy

Ask PDF

Abstract

Electron Microscopy

Explore similar work

Frequency-Aware Dual-Stream Learning for Balanced Realism and Fidelity in Electron Microscopy Imaging

STEMGym: Benchmarking Sequential Decision-Making under Dose Budgets in Autonomous Electron Microscopy

Contrastive Image-Metadata Pre-Training for Materials Transmission Electron Microscopy