A Mixture of Experts Foundation Model for Scanning Electron Microscopy Image Analysis
Authors: Sk Miraj Ahmed, Yuewei Lin, Chuntian Cao, Shinjae Yoo, Xinpei Wu, Won-Il Lee, Nikhil Tiwale, Dan N. Le, Thi Thu Huong Chu, Jiyoung Kim, Kevin G. Yager, Chang-Yong Nam
Organizations: Computing and Data Sciences, Brookhaven National Laboratory, Upton, 11973, New York, USA. · Center for Functional Nanomaterials, Brookhaven National Laboratory, Upton, 11973, New York, USA. · Department of Material Sciences and Engineering, The University of Texas at Dallas, Richardson, 75080, Texas, USA.
Scanning Electron Microscopy (SEM) is indispensable in modern materials science, enabling high-resolution imaging across a wide range of structural, chemical, and functional investigations. However, SEM imaging remains constrained by task-specific models and labor-intensive acquisition processes that limit its scalability across diverse applications. Here, we introduce the first foundation model for SEM images, pretrained on a large corpus of multi-instrument, multi-condition scientific micrographs, enabling generalization across diverse material systems and imaging conditions. Leveraging a self-supervised transformer architecture, our model learns rich and transferable representations that can be fine-tuned or adapted to a wide range of downstream tasks. As a compelling demonstration, we focus on defocus-to-focus image translation-an essential yet underexplored challenge in automated microscopy pipelines. Our method not only restores focused detail from defocused inputs without paired supervision but also outperforms state-of-the-art techniques across multiple evaluation metrics. This work lays the groundwork for a new class of adaptable SEM models, accelerating materials discovery by bridging foundational representation learning with real-world imaging needs.
Figures & tables
Figure 1: Overview of the proposed SEM foundation model pretraining framework. The framework consists of four stages: (1) large-scale self-supervised pretraining of a ViT-MAE teacher on diverse unlabeled SEM images; (2) knowledge distillation into an MAE+MoE student with multiple experts and top- k gating to enable conditional computation and specialization; (3) frequency-aware masked reconstruction to further adapt the foundation model to SEM-specific spectral statistics by emphasizing high-frequency/PSD-consistent detail recovery during masked prediction.
Figure 2: Downstream task adaptation. The pretrained SEM foundation encoder can be paired with any task-specific decoder head (e.g., restoration, segmentation, or measurement prediction) depending on the application. In this work, we instantiate the downstream task as SEM defocus-to-focus refocusing and compare multiple decoder choices; empirically, reusing the pretrained masked-reconstruction decoder yields the best performance for refocusing, providing the most accurate and stable metrology on real SEM data. For adaptation, we compare the predicted focused image to the ground-truth focused reference under different loss configurations and fine-tune by updating only the decoder while keeping the encoder fixed.
Model
Silhouette (cosine) ↑
Davies– Bouldin ↓
Calinski– Harabasz ↑
ViT-MAE Large
0.0643
2.9303
1462.40
ViT-MAE Large + MoE (8E)
0.1014
2.5721
2027.61
Relative change vs. MAE
+57.7%
−12.2%
+38.7%
Table 1: Clustering performance comparing ViT-MAE Large vs. ViT-MAE + MoE (8 experts). Higher is better for Silhouette and Calinski–Harabasz; lower is better for Davies–Bouldin.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
NIQE ↓
Defocused Input
18.3
0.31
0.57
11.8
Richardson–Lucy
15.9
0.51
0.76
10.9
Wiener Filter
14.5
0.30
0.83
19.7
BM3D
18.7
0.44
0.51
14.4
Noise2Void (N2V)
18.9
0.55
0.56
9.4
Noise2Noise (N2N)
18.8
0.55
0.60
9.6
Table 2: Synthetic-to-synthetic (S → S) evaluation using optics-simulated defocus.Both training and evaluation are performed on synthetically blurred SEM images. Metrics reflect visual fidelity and frequency preservation under fully controlled conditions.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
NIQE ↓
Fast-scan Input (Defocused)
17.7
0.62
0.35
7.9
Richardson–Lucy
15.8
0.53
0.37
11.2
Wiener Filter
11.9
0.23
0.54
11.3
BM3D
17.8
0.63
0.34
8.2
Noise2Void (N2V)
17.4
0.47
0.34
10.0
Noise2Noise (N2N)
17.2
0.46
0.34
11.1
Table 3: Image-quality evaluation for defocus-to-focus SEM restoration using a single real slow-scan image as the in-focus reference. All models are trained exclusively on 10 synthetically generated defocus–focus image pairs and evaluated on real fast-scan defocused SEM images. Higher is better for PSNR and SSIM, while lower is better for LPIPS and NIQE.
Method
CD
CD Std
LWR
LER
PSD
CD (MAE)
Avg (MAE)
I1
I2
I1
I2
I1
I2
I1
I2
I1
I2
Focused Input
16.3
9.7
0.4
0.6
4.2
2.2
3.3
1.6
4.2
2.2
0.00
0.00
Defocused Input
16.9
10.4
1.4
0.7
6.4
2.5
4.6
1.9
6.5
2.5
0.65
0.91
Richardson–Lucy
17.4
10.7
1.4
0.8
4.3
1.4
3.0
1.1
4.4
1.4
1.05
0.60
Wiener Filter
18.0
✗
0.4
✗
9.3
✗
7.2
✗
9.4
✗
✗
✗
BM3D
17.1
10.2
1.6
0.7
4.8
2.3
3.4
1.7
5.1
2.3
0.65
0.50
Table 4: Metrology accuracy for defocus-to-focus SEM restoration under a synthetic-to-real (S → R) protocol.Models are trained on synthetic defocus and evaluated on real defocused SEM images. Values are reported for two real images (Img1 / Img2). CD(MAE) is averaged over Img1 and Img2, and Avg(MAE) is computed by averaging the absolute error over all 10 entries (5 metrics × 2 images) with respect to the focused reference. Entries marked with ✗ indicate cases where the metrology software (SMILE) failed to reliably detect edges due to poor image quality.
Figure 3: Qualitative comparison for defocus-to-focus SEM restoration under the S → R protocol. The focused slow-scan image is used as reference (GT), and the fast-scan defocused image is the input. All remaining panels are restored outputs from classical baselines, denoising baselines, and learning-based methods.
Figure 4: SMILE-based line detection for metrology. Visualization of the line/edge detections produced by the SMILE pipeline for the focused reference (top-left), the defocused input, representative baselines, and our restored outputs (bottom row, last three columns). While several methods yield visually sharp reconstructions, they can still induce mis-localized or inconsistent SMILE line fits that lead to biased CD/roughness measurements. In contrast, our outputs produce line detections that are most consistent with the focused reference, aligning with the improved metrology accuracy reported in Table 4 .
Figure 5: LWR power spectral density (PSD) from SMILE. LWR PSD curves directly exported from the SMILE metrology pipeline for the focused reference (F), defocused input (DF), Wiener filtering (WF), MRN, Noise2Void (N2V), and our method (subset shown for readability). Our reconstruction produces a PSD that is most coherent with the focused reference across the relevant frequency band, while other methods show larger spectral distortions indicative of biased roughness statistics. Residual high-frequency attenuation in our outputs can be adjusted by tuning the relative weights of the loss components during fine-tuning.