InformedXRD: Reproducible Benchmarks and Physics-Informed Evaluation for Powder Diffraction Symmetry Classification
Authors: Edward G Friedman, Elizabeth J. Baggett, Abhishek Shetty, Derrick Chan-Sew, Vanellsa Acha, Harshitha Dwaracherla, P. A. Kienzle, William Ratcliff
Organizations: School of Computer Science, Carnegie Mellon University · NIST Center for Neutron Research · Dept. of Physics, Boston College · School of Information, UC Berkeley · Dept. of Materials Science and Engineering, University of Maryland · Dept. of Physics, University of Maryland
Machine learning for powder diffraction (PXRD) symmetry classification is being trained at scale, yet the field lacks a common evaluation standard: papers report results on different subsets of experimental archives, with different target taxonomies, preprocessing, and metrics, and published models are rarely released in a form that permits rescoring. We present InformedXRD, an evaluation framework with three parts: (i) a machine-readable mapping from 230 space groups to 99 extinction groups, the finest classes powder diffraction can identify from systematic absences alone; (ii) a benchmark-construction method whose inclusion rules are specified as code and reproduce from a frozen snapshot, instantiated as RRUFF-473, RRUFF-325, and, as a small example, opXRD-98; (iii) a reporting protocol with prior-only frequency baselines, a label-permutation test, a hierarchy-aware error metric on the condensed translationengleiche subgroup graph, and stratified evaluation. We build on the mapping, RRUFF benchmarks, subgroup graph, and four ViT checkpoints released in our earlier work (Baggett et al., 2026). Using these checkpoints, a 1D residual CNN, and six architectures trained under the XRDBench recipe (Cao et al., 2025), we show that evaluation design changes what conclusions are justified: paired tests point in different directions at Top-1 and Top-5; paired Top-k tests do not separate the three leading XRDBench models, while hierarchy-aware analysis describes error distance and direction across all six; an input-blind frequency predictor matches the models on aggregate Top-k, yet a label-permutation test still detects pattern-specific evidence; and stratified analysis exposes a label-entropy confound that aggregate accuracy hides. All mapping tables, benchmark code, scripts, and checkpoints are released as shared infrastructure for PXRD-ML evaluation.
Figures & tables
Figure 1: The construction method of § 4.1 instantiated on two archives. Left: RRUFF, yielding the frozen RRUFF-473 and RRUFF-325 benchmarks with their nuisance-fit strata. Right: opXRD, a small example yielding opXRD-98. Rows are the steps of the method aligned across the two archives, with dashed boxes marking omitted steps and blue boxes marking the released evaluation sets. Per-step detail—thresholds, diagnostics, and the full gate cascade—is in Appendix E (RRUFF) and Appendix F (opXRD).
Model
Top-1 (%)
≤ 2 hops (%)
Mean dist. ( ↓ )
Branch (%, ↓ )
Top-1 rank
Dist. rank
ViT: no preferred orientation
9.54
38.4
2.72
25.8
3
3
ViT: preferred-orientation only
13.54
53.0
2.51
34.1
1
2
ViT: final mixed-data calibrated
10.46
51.55
2.46
32.6
2
1
1D residual CNN
6.77
28.05
2.92
53.8
4
4
ViT: legacy uncalibrated
4.92
17.8
3.50
56.4
5
5
Table 1: Five-model comparison on RRUFF-325: Top-1, ≤2 -hop locality, mean DAG distance, and branch-jump fraction, with the implied ranks under flat Top-1 versus mean DAG distance. The two ranking columns disagree on the preferred-orientation-only and final mixed-data calibrated ViT checkpoints; the 1D CNN baseline is set apart from the ViT family by its 53.8% branch-jump rate. Hierarchy columns are over wrong Top-1 predictions; matched nulls for the preferred-orientation-only and final mixed-data calibrated ViTs and the CNN are given in Table 9 . Boldface marks the best value in each column and does not denote statistical significance.
RRUFF-473
RRUFF-325
Model
Top-1 / 5 (%)
≤ 2 hops obs./null
Mean dist. obs./null
Top-1 / 5 (%)
≤ 2 hops obs./null
Mean dist. obs./null
CNN10
12.47 / 45.67
37.68 / 31.43
2.797 / 2.910
9.85 / 39.08
35.15 / 31.42
2.877 / 2.901
PatchTST
12.47 / 45.45
40.10 / 30.01
2.688 / 2.969
9.23 / 40.62
37.29 / 30.84
2.783 / 2.927
BiLSTM
8.67 / 30.87
34.49 / 31.26
2.894 / 2.929
4.92 / 24.00
34.95 / —
2.903 / —
CNN11
11.63 / 45.88
38.52 / 31.09
2.749 / 2.919
8.31 / 40.31
36.24 / —
2.842 / —
BiGRU
10.57 / 33.62
32.86 / 31.75
2.877 / 2.901
6.77 / 25.85
31.68 / —
2.921 / —
Table 2: Six XRDBench architectures rescored in extinction-group space. Models were trained on the unchanged public SimXRD train_1 contract ( Cao et al., 2025 ) and evaluated on RRUFF patterns exactly decimated onto that grid (contract specification in Appendix K ). Hierarchy columns are over wrong Top-1 predictions and give observed / model-matched null (§ 5.2 ); on RRUFF-325 matched nulls were computed for the two strongest models. No pair among CNN10, PatchTST, and CNN11 is separable by exact McNemar test at Top-1, Top-3, or Top-5 on either benchmark (smallest raw p=0.124 ). Boldface marks the best value per column, not statistical significance.
Figure 2: Decomposition of the catastrophic paradox on RRUFF-473. Left: the severity strata differ strongly in label-space structure, with the catastrophic stratum spanning far fewer unique extinction groups than the recoverable stratum. Right: classical profile-fit quality and neural classification difficulty do not align monotonically, and extinction-group reweighting removes about 45% of the recoverable-versus-catastrophic gap.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
SG model, SG space
Collapsed SG, EG space
Direct EG
Direct-EG advantage
Synthetic validation, full scale
15.17 / 27.81 / 35.25
18.46 / 33.27 / 42.35
19.32 / 34.28 / 43.38
+0.86 / +1.01 / +1.03
Synthetic, \nicefrac12 scale (Top-1)
—
—
—
+0.59
Synthetic, \nicefrac14 scale (Top-1)
—
—
—
+1.45
Synthetic, \nicefrac18 scale (Top-1)
—
—
—
+2.77
RRUFF-473
—
3.38 / 5.92 / 8.25
3.59 / 5.71 / 8.46
+0.21 [ −1.27 , 1.69], p=1.00
RRUFF-325
—
1.54 / 4.31 / 7.08
2.77 / 5.23 / 8.31
+1.23 [ −0.31 , 2.77], p=0.219
Appendix
Table 3: Matched comparison of a 230-way space-group (SG) model and a 99-way extinction-group (EG) model. “Collapsed SG” scores the SG model after its probabilities are summed into EG space. All accuracy cells are Top-1 / Top-3 / Top-5 (%). Advantage is direct EG minus collapsed SG in percentage points; RRUFF advantages are Top-1 with paired-bootstrap 95% CI and exact McNemar p (Top-3 and Top-5 differences on both benchmarks likewise have p≥0.42 and intervals containing zero).
Schopmans et al.
SimXRD
RRUFF-473
RRUFF-325
Patterns Count
942
∼ 3000
473
325
Classification target
145 SGs
230 SGs
99 EGs
99 EGs
Curation method
Hand-filtered
Partially documented
Algorithmic
Algorithmic
Unique minerals
—
—
227
168
Unique extinction groups
—
—
46
45
Difficult cases retained?
No (85 SGs excluded)
Unknown
Yes
No a
Appendix
Table 4: Benchmark audit against prior PXRD-ML evaluation sets. The main point is not raw score comparison but the mismatch in target taxonomy, filtering, difficult-case retention, and reproducibility.
Variant
Selected
Intersection
Selected-only
Reference-only
Baseline
471
471
0
0
Anchor full threshold 0.86
473
471
2
0
Anchor cell threshold 0.0020
472
471
1
0
Partial tie-break margin 0.04
471
469
2
2
Lattice threshold 0.0006
471
470
1
1
Appendix
Table 5: Local one-at-a-time sensitivity of the family-coherence curation defaults. Reported variants are the only tested perturbations that changed benchmark composition beyond the baseline 471/473 core match.
Benchmark / stratum
Top-1
Top-5
Top-10
RRUFF-473 prior
12.9
51.4
68.7
RRUFF-325 prior
12.6
43.7
62.8
Recoverable prior
12.2
41.3
—
Usable-or-better prior
15.7
48.4
—
Poor prior
16.2
66.2
—
Catastrophic prior
28.4
75.7
—
Appendix
Table 6: Benchmark-matched prior-only controls. “RRUFF prior” denotes the empirical extinction-group distribution of the named benchmark itself rather than the aggregate synthetic-training prior.
Gate
Records remaining
Initial archive
92,552
Declares a phase
2,179
Space group assignable to each phase (316 records with all phase labels stored explicitly; remainder derived from structure with spglib [ Togo et al., 2024 ] )
1,228
Single phase with lattice and derivable space group (891 from one source)
908
Cu-K α gate: explicit K α1 /K α2 pair or the conventional 2:1 weighted average
229
RRUFF angular support at native resolution, 10–80 ∘
98 (28 space groups, 23 extinction groups)
Appendix
Table 7: Filtering gates applied to opXRD to obtain opXRD-98, a one-pattern, one-label Cu-K α set comparable with the RRUFF benchmarks.
Model
EG Top-1 / 3 / 5 (%)
≤ 2 hops (%)
Mean DAG dist.
PatchTST
10.20 / 20.41 / 34.69
32.95
3.034
CNN11
6.12 / 20.41 / 26.53
31.52
3.174
CNN10
5.10 / 19.39 / 38.78
29.03
3.258
BiLSTM
5.10 / 15.31 / 22.45
23.66
3.441
BiGRU
4.08 / 17.35 / 24.49
20.21
3.394
Transformer
4.08 / 22.45 / 31.63
17.02
3.745
Appendix
Table 8: Six XRDBench architectures on opXRD-98 (98 Cu-K α records), rescored in extinction-group space. Hierarchy columns are over wrong Top-1 predictions.
Reference or model
Mean DAG distance
Within 2 hops (%)
Descendant / ancestor / branch (%)
Uniform null, RRUFF-473
2.997
29.76
8.65 / 35.61 / 55.69
Uniform null, RRUFF-325
2.944
30.65
10.04 / 29.53 / 60.38
Frequency-matched null, RRUFF-473
2.692
42.15
—
Frequency-matched null, RRUFF-325
2.656
43.35
—
Most-frequent-group predictor, RRUFF-325
2.116
63.4
—
Final mixed-data ViT, RRUFF-325
2.46
51.55
56.7 / 10.7 / 32.6
Appendix
Table 9: Reference distributions for the hierarchy-aware metric and three RRUFF-325 case-study models. Direction is the descendant / ancestor / branch-jump split over wrong predictions.
Figure 3: Flat accuracy and hierarchy-aware severity rank the same five models differently. Each line is one model; its left endpoint is its Top-1 rank, and its right endpoint is its mean-DAG-distance rank on the condensed t -subgroup graph. Lower mean DAG distance is better. The preferred-orientation-only ViT checkpoint wins under flat Top-1, while the final mixed-data calibrated ViT checkpoint has the best raw mean-DAG-distance rank. The 1D CNN baseline remains worse on both metrics and, more importantly for this paper, demonstrates that the hierarchy-aware metric distinguishes a separate branch-jump-heavy failure regime that flat Top-1 alone would compress into a single score.
Benchmark
Top-1 (%)
Top-5 (%)
RRUFF-473 ( n=473 )
9.94 [7.61, 12.68]
50.74 [46.29, 55.18]
RRUFF-325 ( n=325 )
10.46 [7.38, 13.85]
43.69 [38.46, 48.92]
Appendix
Table 10: Bootstrap 95% confidence intervals for the final mixed-data calibrated ViT checkpoint (point estimate [CI]).
Checkpoint
Val. loss
Val. Top-1/3/5 (%)
RRUFF-473 acc.
RRUFF-473 ≤ 2 hops / mean dist.
RRUFF-325 acc.
RRUFF-325 ≤ 2 hops / mean dist.
Submitted epoch 6
1.6631
50.25 / 75.47 / 84.85
9.09 / 19.66 / 28.54
28.84% / 2.956
6.77 / 18.77 / 25.23
28.05% / 2.921
Validation-selected epoch 46
1.3948
58.00 / 81.30 / 89.03
3.59 / 8.46 / 15.64
17.98% / 3.342
2.15 / 4.92 / 12.92
20.13% / 3.255
Appendix
Table 11: Convergence test for the 1D residual CNN control. Validation loss and Top-1/3/5 are on held-out synthetic data; benchmark accuracy columns are Top-1 / Top-3 / Top-5 (%); hierarchy columns are the within-2-hop fraction and mean DAG distance over wrong predictions. Epoch 46 is the validation-loss minimum after repeated learning-rate reductions; the formal early-stopping rule (patience 8) had not fired.
Figure 4: Extinction-group imbalance in the ICSD-derived source distribution underlying the aggregate synthetic training prior used for calibration. Horizontal axis: all 99 extinction groups, ordered by training frequency; vertical axis: prior probability; the ten most frequent groups are numbered on the bars and named in the inset. Heavy class imbalance means a naive prior-frequency baseline can attain non-trivial Top- k accuracy with no reference to the input.
Determining crystal structures from powder X-ray diffraction (PXRD) would enable routine characterization of newly synthesized materials, including those absent from reference databases. Generative models offer a promising approach, but existing evaluations remain fragmented and largely assume single-phase samples with known composition. To fill this gap, we introduce PXRD-Bench, a unified benchmark that extends evaluation to unknown per-element atom counts and recovery of both structures in two-phase mixtures. It provides approximately 4.1 million training structures with simulated PXRD, tests generalization to unseen compositions and structural prototypes, and evaluates recovery from experimental PXRD after excluding structurally matching training entries. To provide a strong reference baseline, we also introduce Bragglite, a PXRD-to-structure model developed through a systematic study of PXRD modeling choices. It achieves a 44.2% Top-1 match rate on experimental RRUFF patterns with known composition, 24.6% above the strongest prior baseline. Further analysis yields several key findings: some models fail to benefit from experimental PXRD, while generalization to unseen structural prototypes and recovery from two-phase mixtures remain challenging. Together, our benchmark, systematic studies, and Bragglite clarify the capabilities and limitations of current PXRD-to-structure models.
Dongyeop Woo, Nofit Segal, Nayoung Kim +3
Korea Advanced Institute of Science & Technology (kaist.edu) · Massachusetts Institute of Technology (mit.edu)
Miller-index identification from powder XRD patterns requires capabilities untested by existing multimodal benchmarks: the model must read a narrow peak location from a rendered scientific curve and then connect that observation to multi-step crystallographic reasoning. We introduce CrystalXRD-Bench, a 250-sample benchmark built from 10 public crystallographic databases for a single task: recover the full set of HKLs contributing to the highest-intensity peak in an XRD pattern. Each sample pairs the rendered XRD image with the source CIF text and chemical formula, so visual extraction errors and reasoning errors can be examined side by side. We evaluate seven vision-language models. The best Jaccard score is 0.5888 (GPT-5.4) with an exact-match rate of 37.6%, yet six of seven models remain below Jaccard 0.50; the task is far from solved. Error patterns vary systematically: double-peak cases are especially brittle, recall-heavy models gain coverage by over-predicting HKLs, and access to CIF text does not close the gap in crystallographic calculation. Alongside model rankings, the benchmark identifies the conditions under which current VLMs fail on quantitative scientific figures. All data and evaluation code will be publicly available.
Determining the crystal structure of a material from its powder X-ray diffraction (PXRD) pattern is a central challenge in materials science. PXRD is an accessible and widely used characterization technique, yet recovering the atomic structure from diffraction data requires solving an underdetermined inverse problem due to the loss of phase information. Generative modeling can provide a prior over atomic structure and learn the mapping from PXRD patterns to crystal structures via simulated structure-spectrum pairs. We present XRDiff, a diffusion model that recovers crystal structures from PXRD given either the stoichiometry or, in a more challenging setting, the elemental constituents and total number of atoms in the unit cell. We evaluate on datasets where each stoichiometry has multiple polymorphs and all polymorphs of a given composition are held out together, ensuring that high performance reflects genuine use of the diffraction signal. XRDiff achieves strong structure recovery rates on simulated benchmarks, indicating that the model learns a spectrum-to-structure mapping precise enough to differentiate between polymorphs. To address generalization to experimental data, we compare a full-spectrum encoding against an encoding based on peak descriptors. The peak-based encoding generalizes substantially better, outperforming even a model trained on full spectra with augmentations fitted to the experimental noise distribution. These results demonstrate that representations robust to the noise and artifacts present in real-world PXRD offer a practical and scalable path toward closing the simulation-to-experiment gap, enabling zero-shot crystal structure solution from experimental PXRD with full or partial chemical composition input.
Nofit Segal, Mingda Li, Benjamin Kurt Miller +1
Department of Materials Science and Engineering, MIT, Cambridge, MA, USA. · Department of Nuclear Science and Engineering, MIT, Cambridge, MA, USA. · FAIR, Meta, San Francisco, CA, USA.