Organizations: Global Institute of Future Technology, Shanghai Jiao Tong University, Shanghai 200240, China · Global College, Shanghai Jiao Tong University, Shanghai 200240, China · Contemporary Amperex Technology Co., Ltd., Ningde, P.R. China · Materials Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
Four-dimensional scanning transmission electron microscopy (4D-STEM) records a two-dimensional diffraction pattern at each electron-probe position, yielding spatially resolved reciprocal-space information but large, heterogeneous data volumes. Here we describe 4D-ImageNet, a collection of 174,000 diffraction patterns comprising 145,000 experimental patterns selected from 29 acquisitions and 29,000 multislice simulations. The experimental data cover acquisition-level labels for Ag, Au, mixed Au-Ag, CoO, Pd and ZnO specimens across multiple fields of view, scan dimensions, camera lengths and exposure times. Each acquisition contributes 5,000 quality-ranked patterns with source scan coordinates and acquisition metadata. A set-prediction detector provides model-derived pseudo-labels for the direct-beam position and Bragg-disk centres, with a confidence score for each disk. The simulation data cover 13 crystal structures and include Euler rotations, reciprocal-space sampling and approximate low-index beam directions. A grouped mixed-domain masked-reconstruction benchmark is provided to assess leakage-resistant loading and evaluation across experimental and simulated data. The dataset is intended for representation learning, disk detection, diffraction-pattern retrieval, orientation analysis and simulation-to-experiment studies.
Figures & tables
Component
Material groups
Files or acquisitions
Patterns
Pattern shape
Stored size
Experimental
6
29
145,000
192×192
21.39 GB
Simulation
13
13
29,000
256×256
7.60 GB
Total
17
42
174,000
–
28.99 GB
Table 1: Composition of 4D-ImageNet. File sizes are decimal gigabytes measured for the PKL files.
Figure 1: Experimental and simulated 4D-STEM examples. (a) Experimental virtual bright-field image obtained by integrating a 12-pixel-radius region around the direct beam of an Ag acquisition; coloured markers identify four probe positions. Display used 0.5th–99.5th percentile clipping and linear normalisation. (b) Experimental diffraction patterns recorded at those positions, displayed after non-negative clipping, log(1+I) transformation, 15th–99.5th percentile clipping and a display gamma of 1.35. (c) Four distinct Ag multislice patterns selected by maximum cosine similarity of central-beam-excluded reciprocal-space features. The material phase and approximate [hkl] beam direction are shown within each pattern; display used log(1+I) , 15th–99.8th percentile clipping and a gamma of 0.92. Within each column, the border colour links the probe position, experimental frame and similarity-selected simulation. The simulations are visual references rather than paired ground truth, and all transformations are display-only.
Scan shape
Number of acquisitions
Source positions per acquisition
Retained patterns per acquisition
128×128
1
16,384
5,000
130×130
1
16,900
5,000
185×185
2
34,225
5,000
200×200
1
40,000
5,000
256×256
13
65,536
5,000
260×260
11
67,600
5,000
Table 2: Experimental source-scan dimensions represented in the curated dataset.
Material label
Acquisitions
Patterns
Ag
4
20,000
Au
3
15,000
Au–Ag
3
15,000
CoO
9
45,000
Pd
5
25,000
ZnO
5
25,000
Table 3: Experimental material groups and numbers of acquisition-level files. Every acquisition contributes 5,000 selected patterns.
Figure 2: Experimental diffraction patterns and corresponding model-derived pseudo-labels across the six material groups. Panels (a–f) show Ag, Au, Au–Ag, CoO, Pd and ZnO, respectively. The upper row contains the original experimental patterns and the lower row shows the identical detector pixels with r4 predictions read directly from the corresponding acquisition-level label JSON files at a confidence threshold of 0.5. Cyan crosses indicate predicted Bragg-disk centres, and the yellow plus sign indicates the output of the dedicated direct-beam query. For display, each pattern was independently transformed by non-negative clipping, log(1+I) and 1st–99.8th percentile normalisation. The overlays are point annotations rather than disk radii or segmentation masks; no diffraction feature was inserted, removed or redrawn.
Figure 3: Schematic diagram of the disk-detection annotation architecture. (a) A real experimental Ag diffraction pattern is used as input. (b) A ConvNeXt V2 Tiny backbone produces four feature levels of 64×64 , 32×32 , 16×16 and 8×8 , each projected to 192 channels. (c) Four multi-scale deformable-attention encoder layers, each using eight heads and four sampling points per feature level, form a 5,440×192 memory. (d) Six decoder layers refine one dedicated beam query and 384 learned disk queries through self-attention and deformable cross-attention. (e) Predictions at a confidence threshold of 0.5 are overlaid on the unchanged input: cyan crosses denote predicted Bragg-disk centres and the yellow plus sign denotes the predicted direct-beam position. The feature panels are stored activations from the frozen r4 checkpoint, and the markers are read from the corresponding acquisition-level label JSON file.
Record
Contents
Experimental PKL
One acquisition per file; 5,000 192×192 patterns in diffraction_patterns , with material , source_name , scan_positions , scan_shape , fov , camera_length and selection scores .
Disk-annotation JSON
One acquisition per file, linked to its PKL file by source_file . The predictions array contains one object per pattern; center_beam , disk_count and disks provide direct-beam and disk-centre coordinates. The top-level model object records the detector version, weight identifier, preprocessing, input size and confidence threshold.
Label manifest JSON
One dataset-level file listing the source and annotation path for every acquisition together with pattern- and disk-count summaries.
Simulation PKL
One structure per file; 256×256 patterns with material , source_cif , center_list , pixel_size_list , orientation_list , hkl_list and simulation parameters.
Table 4: Core data and annotation records in 4D-ImageNet.
Figure 4: Simulation coverage and experiment–simulation domain gap. (a) Marginal distributions of the three Euler coordinates for all 29,000 simulations after normalisation by their respective sampling ranges; the dashed line denotes the expectation for a coordinate-uniform distribution. (b) Approximate [h,k,l] coverage grouped by the maximum absolute index and the number of non-zero components. These low-index directions are simulation metadata and are not crystallographically equivalent phase labels across materials. (c) Radial profiles for 522 experimental and 520 simulated patterns processed with the same intensity transform and normalised detector radius. At each radius, the line is the median across patterns of the per-pattern annular mean of normalised log intensity, and the band spans the interquartile range. (d) Joint principal-component projection of frozen bottleneck features from the best mixed-domain U-Net checkpoint. Both domains were forwarded as unmasked [x,0] inputs; shaded ellipses show 90% covariance contours. The centroid separation of 3.43 pooled within-domain spread units documents residual domain structure and is not an accuracy score.
Metric
Experimental test
Simulation test
Number of patterns
1,035
769
Masked mean-squared error
0.001851
0.002020
Masked mean-absolute error
0.03180
0.02273
Intensity-weighted masked mean-squared error
0.001717
0.003831
Peak signal-to-noise ratio
27.33 dB
26.95 dB
Table 5: Held-out masked-reconstruction metrics for the mixed-domain usability benchmark. Metrics are calculated on masked pixels after the stated preprocessing.
Figure 5: Mixed-domain masked-reconstruction U-Net and experimental test example. (a) A held-out Au target, four-region binary mask M and masked pattern x(1−M) form the two-channel model input [x(1−M),M] . (b) The implemented Small U-Net is shown in a stepped encoder–decoder layout. Pairs of feature slabs represent the two 3×3 Conv–BN–ReLU layers in each encoder or bottleneck block. The feature maps progress from 1922 pixels and 16 channels to 962 pixels and 32 channels, and then to the 482 -pixel, 64-channel bottleneck. Each decoder group shows the 2×2 transposed convolution, concatenated encoder feature and two decoder convolutions; purple routes denote the two skip concatenations. A 1×1 convolution and sigmoid produce the one-channel prediction. (c) The prediction is inserted into the masked regions to form the reconstruction, and the masked absolute error is displayed on the same target-intensity scale. Pattern 4,708 was drawn from a held-out experimental acquisition and evaluated using the mixed-domain epoch-48 checkpoint.
Mapping crystalline phases at heterogeneous interfaces is essential for understanding material performance and degradation. However, structural heterogeneity, phase overlap, and local disorder complicate diffraction interpretation, while growing data volumes make manual analysis increasingly impractical. We introduce 4DMulti, a physics-guided learning framework for automated multicomponent identification from large-scale four-dimensional scanning transmission electron microscopy (4D-STEM) data. The supporting diffraction data resource comprises over 6 million high-quality experimental patterns and labeled patterns generated by Sim2real. A retrieval-conditioned latent diffusion transformer (Sim2real) translates simulated patterns into experimental-style examples under constraints designed to preserve Bragg geometry, while a rotation-invariant coordinate convolutional network identifies phases across in-plane rotations. 4DMulti achieves 98.82% classification accuracy on a five-phase experimental nanoparticle benchmark, with ablation studies supporting the complementary benefits of domain adaptation and rotation-invariant classification. We define diffraction-inferred structural complexity (DISC), a normalized predictive entropy score that quantifies phase-assignment ambiguity within a specified candidate phase library. We apply 4DMulti to generate structural maps of superconducting heterostructures, corroded alloy surfaces, and degraded solid-state battery interfaces down to single-nanometer spatial resolution. 4DMulti connects simulation-derived crystallographic knowledge to automated experimental interpretation, establishing a foundation for scalable analysis of complex interfaces and data-driven discovery of interfacial design principles.
Haoran Zhang, Zian Mao, Shufen Chu +6
Global Institute of Future Technology, Shanghai Jiao Tong University, 200240, Shanghai, China. · Global College, Shanghai Jiao Tong University, 200240, Shanghai, China. · School of Materials Science and Engineering, Shanghai Jiao Tong University, 200240, Shanghai, China.
Understanding and inferring dislocation microstructures from diffraction patterns remains an open challenge in materials characterization, as diffraction measurements provide only indirect information about the underlying dislocation structure. In this work, a cross-modal learning framework is developed to enable the prediction of 3D dislocation structures directly from diffraction data. Dislocation density fields generated from discrete dislocation dynamics simulations are paired with corresponding virtual X-ray diffraction patterns and embedded into a shared 2D latent space using contrastive learning. The alignment between structural and diffraction representations of dislocation structures is evaluated directly in the learned latent space using correlations between corresponding latent features. To estimate the role of dataset size for this approach, farthest point sampling is employed to construct representative and diverse training subsets of varying sizes. The results show strong cross-modal alignment and that model performance improves rapidly with increasing dataset size. Near-saturation is achieved with approximately 500 representative observations from a dataset of 10,000 observations, enabling accurate prediction of dislocation density fields from previously unseen diffraction data of the same distribution. Qualitative comparisons confirm that the predicted structures capture the dominant spatial features of the underlying dislocation microstructures. These findings demonstrate an efficient approach for learning structure-diffraction relationships and highlight the potential for inferring structural characteristics of dislocation networks directly from diffraction patterns, providing a pathway toward diffraction-based structural analysis and future extension to experimental data.
Benjamin Udofia, Nicolas Bertin, Markus Stricker
Interdisciplinary Centre for Advanced Materials Simulation Ruhr University Bochum Bochum, Germany · Lawrence Livermore National Laboratory Livermore Carlifornia, United States of America
Modern transmission electron microscopes are versatile instruments which have become indispensable tools for understanding structure and chemical composition at the nano- and atomic scale. In the physical sciences these instruments are still largely manually controlled, requiring significant operator expertise, limiting throughput, and precluding statistical analysis of large datasets. Recent technical advances in both hardware and in control software now allow for the interaction with almost every functionality of the microscope through a programming interface. This enables better experimental design and data collection automation while also reducing operator collection bias and required expertise. In this study, we present an automated data collection routine with machine-driven decision-making to enable the collection of hundreds of 4D-STEM nanobeam diffraction and ptychography data from a large distribution of size-selectively deposited Pt nanoparticles. We present a semi-automated data analysis workflow to extract pertinent information from the large volumes of collected data. For the nanobeam diffraction data, reducing each dataset to its azimuthal variance profile and combining automated crystal orientation mapping with per-particle morphology descriptors reveals the orientation, shape and phase distributions across the ensemble, including a weak {110} texture. For the ptychographic data, an automated screening pipeline identifies on-zone-axis particles and enables atomic-resolution phase imaging and lattice-strain mapping of individual grains. Together these demonstrate how automation turns instrument throughput into statistically meaningful, atomic-scale microstructural information.