Bridging the Simulation-to-Reality Gap in Electron Microscope Calibration via VAE-EM Estimation
Authors: Jilles S. van Hulst, W. P. M. H. Heemels, Duarte J. Antunes
Organizations: The authors are with the Control Systems Technology Section, Department of Mechanical Engineering, Eindhoven University of Technology, 5600 MB Eindhoven, The Netherlands.
Electron microscopy has enabled many scientific breakthroughs across multiple fields. A key challenge is the tuning of microscope parameters based on images to overcome optical aberrations that deteriorate image quality. This calibration problem is challenging due to the high-dimensional and noisy nature of the diagnostic images, and the fact that optimal parameters cannot be identified from a single image. We tackle the calibration problem for Scanning Transmission Electron Microscopes (STEM) by employing variational autoencoders (VAEs), trained on simulated data, to learn low-dimensional representations of images, whereas most existing methods extract only scalar values. We then simultaneously estimate the model that maps calibration parameters to encoded representations and the optimal calibration parameters using an expectation maximization (EM) approach. This joint estimation explicitly addresses the simulation-to-reality gap inherent in data-driven methods that train on simulated data from a digital twin. We leverage the known symmetry property of the optical system to establish global identifiability of the joint estimation problem, ensuring that a unique optimum exists. We demonstrate that our approach is substantially faster and more consistent than existing methods on a real STEM, achieving a 2x reduction in estimation error while requiring fewer observations. This represents a notable advance in automated STEM calibration and demonstrates the potential of VAEs for information compression in images. Beyond microscopy, the VAE-EM framework applies to inverse problems where simulated training data introduces a reality gap and where non-injective mappings would otherwise prevent unique solutions.
Figures & tables
Fig. 1: Grid of Ronchigram images (left) and their Fourier power spectra (right) for different aberration values. The horizontal axis varies defocus ( C1 ) from −200 to +200 nm, while the vertical axis varies two-fold astigmatism ( A1x ) over the same range. The center images correspond to well-tuned aberrations ( C1=0 , A1x=0 ). Colored rectangles group images in specific aberration regions. Note that the Fourier power spectra exhibit symmetry about the origin, with images at (x1,x2) and (−x1,−x2) appearing similar (e.g., purple and orange, cyan and red).
Fig. 2: Calibration pipeline overview. High-dimensional Ronchigram images are preprocessed and then encoded to a low-dimensional latent space using a VAE. The EM algorithm then iteratively refines both the aberration state estimate and the mapping from aberrations to latent representations. In the E-step, the distribution over aberration states is estimated by evaluating the likelihood of observed latent space values for different candidate states. These candidates ( ∙∙∙ ) are highlighted in the aberration space, where the color indicates their estimated likelihood (low ∙ to high ∙ ). The × indicates the ground truth aberration state. Note that this state and its symmetrical neighbor have an increased density of candidates due to the adaptive candidate refinement strategy. The most likely candidates are also clustered around the ground truth state. The M-step then updates the Gaussian process model of the aberration to latent space mapping. The current model belief based on 5 data points ( ∘ ) is illustrated as a function of the aberrations. Note that this model belief is symmetric due to the symmetry constraint in the model structure. The symmetry point is not necessarily at the origin (in the true aberration coordinates) due to the fact that the state estimate is not yet perfect. The M- and E-steps are iterated until convergence.
Fig. 3: Variational Autoencoder architecture. The encoder maps preprocessed Ronchigram power spectra to a low-dimensional latent distribution, from which samples are decoded to reconstruct the input. The weights and biases of the encoder and decoder networks are optimized to maximize the ELBO objective in ( 7 ), which balances reconstruction accuracy and latent space regularization. The example reconstruction to the right of the decoder resembles the input and shows the denoising effect of the VAE.
Fig. 4: Learned VAE latent space for simulated data that contains two varying aberration parameters: defocus ( C1 ), and two-fold astigmatism in the x-direction ( A1x ). The color of the data points is determined by the aberration region, with the assignment shown in the top right of the figure and in Fig. 1 . We can observe that similar aberrations are grouped together. Additionally, different aberration values that produce similar Ronchigrams are still mapped to the same latent space values (example: purple and orange), demonstrating the non-injectivity of the observation model.
Fig. 5: Latent space distribution for Ronchigram datasets gathered on real STEM on different days. Left: dataset 1, middle: dataset 2, right: dataset 3. The latent space data points are colored according to their aberration values, with the color-coding scheme taken from Fig. 1 . We can observe consistency across datasets, as well as with respect to the simulated data from Fig. 4 , indicating that the VAE has learned a meaningful latent representation that reflects the underlying aberration states.
Fig. 6: Fitted prior mean function μf(x) mapping aberration parameters to VAE latent space. Left: first latent dimension z1 as a function of defocus ( C1 ) and two-fold astigmatism ( A1x ). Right: second latent dimension z2 . The prior mean is generated by fitting piecewise linear bases to a large set of simulated Ronchigrams. Note the symmetry with respect to the origin, reflecting the physical property f(−x)=f(x) .
Fig. 7: Estimation performance on real STEM data over 100 Monte Carlo runs with different random initializations with 3 aberration parameters ( C1,A1x,A1y ). We view the aberration estimation error 2-norm as a function of the number of Ronchigram observations used for calibration. The proposed VAE-EM method consistently outperforms existing automated calibration approaches in terms of speed and accuracy.
Fig. 8: Estimation error performance on real STEM data after 10 Ronchigram observations (left) and 30 observations (right). The violin plots summarize the distribution of estimation error 2-norms with 3 aberration parameters ( C1,A1x,A1y ) across 100 Monte Carlo runs with different random initializations.
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China. · School of computing, Mathematics, and Engineering, Charles Sturt University, NSW, Australia.
Dept. of Electrical & Computer Eng. Texas A&M Univ. College Station, TX, USA · Dept. of Prosthetics & Orthotics College of Science & Eng. Ankara Univ., Ankara, Turkey · Hamad Bin Khalifa Univ. and Texas A&M Univ. at Qatar Doha, Qatar