A Mixture of Experts Foundation Model for Scanning Electron Microscopy Image Analysis
Authors: Sk Miraj Ahmed, Yuewei Lin, Chuntian Cao, Shinjae Yoo, Xinpei Wu, Won-Il Lee, Nikhil Tiwale, Dan N. Le, Thi Thu Huong Chu, Jiyoung Kim, Kevin G. Yager, Chang-Yong Nam
Organizations: Computing and Data Sciences, Brookhaven National Laboratory, Upton, 11973, New York, USA. · Center for Functional Nanomaterials, Brookhaven National Laboratory, Upton, 11973, New York, USA. · Department of Material Sciences and Engineering, The University of Texas at Dallas, Richardson, 75080, Texas, USA.
Scanning Electron Microscopy (SEM) is indispensable in modern materials science, enabling high-resolution imaging across a wide range of structural, chemical, and functional investigations. However, SEM imaging remains constrained by task-specific models and labor-intensive acquisition processes that limit its scalability across diverse applications. Here, we introduce the first foundation model for SEM images, pretrained on a large corpus of multi-instrument, multi-condition scientific micrographs, enabling generalization across diverse material systems and imaging conditions. Leveraging a self-supervised transformer architecture, our model learns rich and transferable representations that can be fine-tuned or adapted to a wide range of downstream tasks. As a compelling demonstration, we focus on defocus-to-focus image translation-an essential yet underexplored challenge in automated microscopy pipelines. Our method not only restores focused detail from defocused inputs without paired supervision but also outperforms state-of-the-art techniques across multiple evaluation metrics. This work lays the groundwork for a new class of adaptable SEM models, accelerating materials discovery by bridging foundational representation learning with real-world imaging needs.
Figures & tables
Figure 1: Overview of the proposed SEM foundation model pretraining framework. The framework consists of four stages: (1) large-scale self-supervised pretraining of a ViT-MAE teacher on diverse unlabeled SEM images; (2) knowledge distillation into an MAE+MoE student with multiple experts and top- k gating to enable conditional computation and specialization; (3) frequency-aware masked reconstruction to further adapt the foundation model to SEM-specific spectral statistics by emphasizing high-frequency/PSD-consistent detail recovery during masked prediction.
Figure 2: Downstream task adaptation. The pretrained SEM foundation encoder can be paired with any task-specific decoder head (e.g., restoration, segmentation, or measurement prediction) depending on the application. In this work, we instantiate the downstream task as SEM defocus-to-focus refocusing and compare multiple decoder choices; empirically, reusing the pretrained masked-reconstruction decoder yields the best performance for refocusing, providing the most accurate and stable metrology on real SEM data. For adaptation, we compare the predicted focused image to the ground-truth focused reference under different loss configurations and fine-tune by updating only the decoder while keeping the encoder fixed.
Model
Silhouette (cosine) ↑
Davies– Bouldin ↓
Calinski– Harabasz ↑
ViT-MAE Large
0.0643
2.9303
1462.40
ViT-MAE Large + MoE (8E)
0.1014
2.5721
2027.61
Relative change vs. MAE
+57.7%
−12.2%
+38.7%
Table 1: Clustering performance comparing ViT-MAE Large vs. ViT-MAE + MoE (8 experts). Higher is better for Silhouette and Calinski–Harabasz; lower is better for Davies–Bouldin.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
NIQE ↓
Defocused Input
18.3
0.31
0.57
11.8
Richardson–Lucy
15.9
0.51
0.76
10.9
Wiener Filter
14.5
0.30
0.83
19.7
BM3D
18.7
0.44
0.51
14.4
Noise2Void (N2V)
18.9
0.55
0.56
9.4
Noise2Noise (N2N)
18.8
0.55
0.60
9.6
Table 2: Synthetic-to-synthetic (S → S) evaluation using optics-simulated defocus.Both training and evaluation are performed on synthetically blurred SEM images. Metrics reflect visual fidelity and frequency preservation under fully controlled conditions.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
NIQE ↓
Fast-scan Input (Defocused)
17.7
0.62
0.35
7.9
Richardson–Lucy
15.8
0.53
0.37
11.2
Wiener Filter
11.9
0.23
0.54
11.3
BM3D
17.8
0.63
0.34
8.2
Noise2Void (N2V)
17.4
0.47
0.34
10.0
Noise2Noise (N2N)
17.2
0.46
0.34
11.1
Table 3: Image-quality evaluation for defocus-to-focus SEM restoration using a single real slow-scan image as the in-focus reference. All models are trained exclusively on 10 synthetically generated defocus–focus image pairs and evaluated on real fast-scan defocused SEM images. Higher is better for PSNR and SSIM, while lower is better for LPIPS and NIQE.
Method
CD
CD Std
LWR
LER
PSD
CD (MAE)
Avg (MAE)
I1
I2
I1
I2
I1
I2
I1
I2
I1
I2
Focused Input
16.3
9.7
0.4
0.6
4.2
2.2
3.3
1.6
4.2
2.2
0.00
0.00
Defocused Input
16.9
10.4
1.4
0.7
6.4
2.5
4.6
1.9
6.5
2.5
0.65
0.91
Richardson–Lucy
17.4
10.7
1.4
0.8
4.3
1.4
3.0
1.1
4.4
1.4
1.05
0.60
Wiener Filter
18.0
✗
0.4
✗
9.3
✗
7.2
✗
9.4
✗
✗
✗
BM3D
17.1
10.2
1.6
0.7
4.8
2.3
3.4
1.7
5.1
2.3
0.65
0.50
Table 4: Metrology accuracy for defocus-to-focus SEM restoration under a synthetic-to-real (S → R) protocol.Models are trained on synthetic defocus and evaluated on real defocused SEM images. Values are reported for two real images (Img1 / Img2). CD(MAE) is averaged over Img1 and Img2, and Avg(MAE) is computed by averaging the absolute error over all 10 entries (5 metrics × 2 images) with respect to the focused reference. Entries marked with ✗ indicate cases where the metrology software (SMILE) failed to reliably detect edges due to poor image quality.
Figure 3: Qualitative comparison for defocus-to-focus SEM restoration under the S → R protocol. The focused slow-scan image is used as reference (GT), and the fast-scan defocused image is the input. All remaining panels are restored outputs from classical baselines, denoising baselines, and learning-based methods.
Figure 4: SMILE-based line detection for metrology. Visualization of the line/edge detections produced by the SMILE pipeline for the focused reference (top-left), the defocused input, representative baselines, and our restored outputs (bottom row, last three columns). While several methods yield visually sharp reconstructions, they can still induce mis-localized or inconsistent SMILE line fits that lead to biased CD/roughness measurements. In contrast, our outputs produce line detections that are most consistent with the focused reference, aligning with the improved metrology accuracy reported in Table 4 .
Figure 5: LWR power spectral density (PSD) from SMILE. LWR PSD curves directly exported from the SMILE metrology pipeline for the focused reference (F), defocused input (DF), Wiener filtering (WF), MRN, Noise2Void (N2V), and our method (subset shown for readability). Our reconstruction produces a PSD that is most coherent with the focused reference across the relevant frequency band, while other methods show larger spectral distortions indicative of biased roughness statistics. Residual high-frequency attenuation in our outputs can be adjusted by tuning the relative weights of the loss components during fine-tuning.
Vision foundation models have substantially advanced computer vision, enabling state-of-the-art performance in zero- and few-shot settings. They have been successfully applied to biomedical imaging tasks ranging from organ segmentation in computed tomography to cell segmentation in light microscopy. Electron microscopy (EM) is a central modality for analyzing cellular ultrastructure due to its nanometer-scale resolution. However, the application of foundation models in EM has so far been limited to specific organelles, such as mitochondria, largely due to the diversity of segmentation tasks and the scarcity of comprehensively annotated data. As a result, EM segmentation still predominantly relies on supervised learning, requiring extensive manual annotation and limiting ultrastructural analysis. To address this gap, we propose μMatch, a framework for semi-supervised learning and domain adaptation that leverages foundation models. We implement state-of-the-art student-teacher-based methods and evaluate multiple foundation models (SAM, SAM2, μSAM, DINOv2/v3) on challenging EM tasks, including mitochondrion, nucleus, and neurite segmentation. Our results demonstrate consistent improvements over strong baselines and highlight a path toward substantially reducing the annotation effort in EM.
The transmission electron microscope facilitates the highest-resolution imaging of any instrument ever created, and its limiting factor is no longer spatial resolution but dose efficiency. Low electron doses avoid sample damage but produce noisy images for which, unlike in classical computer vision, there is no ground truth. Autonomous materials experimentation poses a related problem, since closed-loop instruments need representations grounded in the microscope state at acquisition. Both demand representations grounded in how an image was acquired. We release 7,330 paired high-angle annular dark-field scanning-TEM (HAADF-STEM) images and their seven-dimensional acquisition metadata, and propose Contrastive Image-Metadata Pre-training (CIMP), a CLIP-style encoder that aligns the two modalities and reaches 84.4% Top-1 cross-modal retrieval on a held-out split. All seven parameters are individually recoverable from the frozen visual embedding through a linear probe, and we use the embedding to condition a metadata-conditioned style-transfer model that re-renders experimental images under different acquisition parameters. Virtually scaling dwell time and beam current of low-dose images turns this model into a physics-informed denoiser; in a blind user study, experimental microscopists prefer it over the current state-of-the-art denoiser for STEM imagery on 70.2% of trials.
Georgia Channing, Debora Keller, Marta D. Rossell +3
Hugging Face · University of Oxford · Empa Dübendorf +1
Quantitative microstructural characterization is fundamental to materials science, and electron micrographs (EMs) provide indispensable high-resolution insights. However, progress in deep learning-based analysis of EMs has been hampered by the scarcity of large-scale, expert-annotated public datasets. To address this issue, we introduce EM3M, a large-scale and multimodal dataset for instance-level understanding of EMs. EM3M comprises 5,091 high-quality EMs, approximately 3 million instance segmentation annotations, and image-level textual descriptions with disentangled attributes. The dataset is constructed through a rigorous multi-stage curation and validation pipeline, with comprehensive statistical analyses to ensure reliability and reproducibility. Building upon these curated image-text pairs, we further provide a text-to-image diffusion model that serves as a controllable data augmentation engine, demonstrating that synthetic augmentation consistently improves downstream segmentation performance. To establish a systematic benchmark, we evaluate representative instance segmentation methods on EM3M. Our results reveal that conventional detection-based and query-based methods struggle with the extreme instance densities and textural complexities inherent in EMs. We additionally provide an optimized flow-based baseline to facilitate fair comparison and future research. EM3M {Dataset: https://huggingface.co/datasets/UniParser/EM3M}, the generative engine {Generation: https://huggingface.co/UniParser/EM3M-Gen}, and an online demo {Segmentation demo: https://www.bohrium.com/apps/uni-aims} are publicly available to support future research in automated materials analysis.