cs.CVSep 18, 2026
SaveA benchmark dataset and baseline methods for four-dimensional STEM diffraction patterns
Organizations: Global Institute of Future Technology, Shanghai Jiao Tong University, Shanghai 200240, China · Global College, Shanghai Jiao Tong University, Shanghai 200240, China · Contemporary Amperex Technology Co., Ltd., Ningde, P.R. China · Materials Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
Abstract
Four-dimensional scanning transmission electron microscopy (4D-STEM) records a two-dimensional diffraction pattern at each electron-probe position, yielding spatially resolved reciprocal-space information but large, heterogeneous data volumes. Here we describe 4D-ImageNet, a collection of 174,000 diffraction patterns comprising 145,000 experimental patterns selected from 29 acquisitions and 29,000 multislice simulations. The experimental data cover acquisition-level labels for Ag, Au, mixed Au-Ag, CoO, Pd and ZnO specimens across multiple fields of view, scan dimensions, camera lengths and exposure times. Each acquisition contributes 5,000 quality-ranked patterns with source scan coordinates and acquisition metadata. A set-prediction detector provides model-derived pseudo-labels for the direct-beam position and Bragg-disk centres, with a confidence score for each disk. The simulation data cover 13 crystal structures and include Euler rotations, reciprocal-space sampling and approximate low-index beam directions. A grouped mixed-domain masked-reconstruction benchmark is provided to assess leakage-resistant loading and evaluation across experimental and simulated data. The dataset is intended for representation learning, disk detection, diffraction-pattern retrieval, orientation analysis and simulation-to-experiment studies.
Figures & tables
| Component | Material groups | Files or acquisitions | Patterns | Pattern shape | Stored size |
| Experimental | 6 | 29 | 145,000 | 21.39 GB | |
| Simulation | 13 | 13 | 29,000 | 7.60 GB | |
| Total | 17 | 42 | 174,000 | – | 28.99 GB |
Table 1: Composition of 4D-ImageNet. File sizes are decimal gigabytes measured for the PKL files.
Figure 1: Experimental and simulated 4D-STEM examples. (a) Experimental virtual bright-field image obtained by integrating a 12-pixel-radius region around the direct beam of an Ag acquisition; coloured markers identify four probe positions. Display used 0.5th–99.5th percentile clipping and linear normalisation. (b) Experimental diffraction patterns recorded at those positions, displayed after non-negative clipping, transformation, 15th–99.5th percentile clipping and a display gamma of 1.35. (c) Four distinct Ag multislice patterns selected by maximum cosine similarity of central-beam-excluded reciprocal-space features. The material phase and approximate beam direction are shown within each pattern; display used , 15th–99.8th percentile clipping and a gamma of 0.92. Within each column, the border colour links the probe position, experimental frame and similarity-selected simulation. The simulations are visual references rather than paired ground truth, and all transformations are display-only.
| Scan shape | Number of acquisitions | Source positions per acquisition | Retained patterns per acquisition |
|---|---|---|---|
| 1 | 16,384 | 5,000 | |
| 1 | 16,900 | 5,000 | |
| 2 | 34,225 | 5,000 | |
| 1 | 40,000 | 5,000 | |
| 13 | 65,536 | 5,000 | |
| 11 | 67,600 | 5,000 |
Table 2: Experimental source-scan dimensions represented in the curated dataset.
| Material label | Acquisitions | Patterns |
|---|---|---|
| Ag | 4 | 20,000 |
| Au | 3 | 15,000 |
| Au–Ag | 3 | 15,000 |
| CoO | 9 | 45,000 |
| Pd | 5 | 25,000 |
| ZnO | 5 | 25,000 |
Table 3: Experimental material groups and numbers of acquisition-level files. Every acquisition contributes 5,000 selected patterns.
Figure 2: Experimental diffraction patterns and corresponding model-derived pseudo-labels across the six material groups. Panels (a–f) show Ag, Au, Au–Ag, CoO, Pd and ZnO, respectively. The upper row contains the original experimental patterns and the lower row shows the identical detector pixels with r4 predictions read directly from the corresponding acquisition-level label JSON files at a confidence threshold of 0.5. Cyan crosses indicate predicted Bragg-disk centres, and the yellow plus sign indicates the output of the dedicated direct-beam query. For display, each pattern was independently transformed by non-negative clipping, and 1st–99.8th percentile normalisation. The overlays are point annotations rather than disk radii or segmentation masks; no diffraction feature was inserted, removed or redrawn.
Figure 3: Schematic diagram of the disk-detection annotation architecture. (a) A real experimental Ag diffraction pattern is used as input. (b) A ConvNeXt V2 Tiny backbone produces four feature levels of , , and , each projected to 192 channels. (c) Four multi-scale deformable-attention encoder layers, each using eight heads and four sampling points per feature level, form a memory. (d) Six decoder layers refine one dedicated beam query and 384 learned disk queries through self-attention and deformable cross-attention. (e) Predictions at a confidence threshold of 0.5 are overlaid on the unchanged input: cyan crosses denote predicted Bragg-disk centres and the yellow plus sign denotes the predicted direct-beam position. The feature panels are stored activations from the frozen r4 checkpoint, and the markers are read from the corresponding acquisition-level label JSON file.
| Record | Contents |
|---|---|
| Experimental PKL | One acquisition per file; 5,000 patterns in diffraction_patterns , with material , source_name , scan_positions , scan_shape , fov , camera_length and selection scores . |
| Disk-annotation JSON | One acquisition per file, linked to its PKL file by source_file . The predictions array contains one object per pattern; center_beam , disk_count and disks provide direct-beam and disk-centre coordinates. The top-level model object records the detector version, weight identifier, preprocessing, input size and confidence threshold. |
| Label manifest JSON | One dataset-level file listing the source and annotation path for every acquisition together with pattern- and disk-count summaries. |
| Simulation PKL | One structure per file; patterns with material , source_cif , center_list , pixel_size_list , orientation_list , hkl_list and simulation parameters. |
Table 4: Core data and annotation records in 4D-ImageNet.
Figure 4: Simulation coverage and experiment–simulation domain gap. (a) Marginal distributions of the three Euler coordinates for all 29,000 simulations after normalisation by their respective sampling ranges; the dashed line denotes the expectation for a coordinate-uniform distribution. (b) Approximate coverage grouped by the maximum absolute index and the number of non-zero components. These low-index directions are simulation metadata and are not crystallographically equivalent phase labels across materials. (c) Radial profiles for 522 experimental and 520 simulated patterns processed with the same intensity transform and normalised detector radius. At each radius, the line is the median across patterns of the per-pattern annular mean of normalised log intensity, and the band spans the interquartile range. (d) Joint principal-component projection of frozen bottleneck features from the best mixed-domain U-Net checkpoint. Both domains were forwarded as unmasked inputs; shaded ellipses show 90% covariance contours. The centroid separation of 3.43 pooled within-domain spread units documents residual domain structure and is not an accuracy score.
| Metric | Experimental test | Simulation test |
|---|---|---|
| Number of patterns | 1,035 | 769 |
| Masked mean-squared error | 0.001851 | 0.002020 |
| Masked mean-absolute error | 0.03180 | 0.02273 |
| Intensity-weighted masked mean-squared error | 0.001717 | 0.003831 |
| Peak signal-to-noise ratio | 27.33 dB | 26.95 dB |
Table 5: Held-out masked-reconstruction metrics for the mixed-domain usability benchmark. Metrics are calculated on masked pixels after the stated preprocessing.
Figure 5: Mixed-domain masked-reconstruction U-Net and experimental test example. (a) A held-out Au target, four-region binary mask and masked pattern form the two-channel model input . (b) The implemented Small U-Net is shown in a stepped encoder–decoder layout. Pairs of feature slabs represent the two Conv–BN–ReLU layers in each encoder or bottleneck block. The feature maps progress from pixels and 16 channels to pixels and 32 channels, and then to the -pixel, 64-channel bottleneck. Each decoder group shows the transposed convolution, concatenated encoder feature and two decoder convolutions; purple routes denote the two skip concatenations. A convolution and sigmoid produce the one-channel prediction. (c) The prediction is inserted into the masked regions to form the reconstruction, and the masked absolute error is displayed on the same target-intensity scale. Pattern 4,708 was drawn from a held-out experimental acquisition and evaluated using the mixed-domain epoch-48 checkpoint.