InformedXRD: Reproducible Benchmarks and Physics-Informed Evaluation for Powder Diffraction Symmetry Classification

Edward G Friedman; Elizabeth J. Baggett; Abhishek Shetty; Derrick Chan-Sew; Vanellsa Acha; Harshitha Dwaracherla; P. A. Kienzle; William Ratcliff

cs.LGNov 1, 2026

InformedXRD: Reproducible Benchmarks and Physics-Informed Evaluation for Powder Diffraction Symmetry Classification

Authors: Edward G Friedman, Elizabeth J. Baggett, Abhishek Shetty, Derrick Chan-Sew, Vanellsa Acha, Harshitha Dwaracherla, P. A. Kienzle, William Ratcliff

Organizations: School of Computer Science, Carnegie Mellon University · NIST Center for Neutron Research · Dept. of Physics, Boston College · School of Information, UC Berkeley ·

Abstract

Machine learning for powder diffraction (PXRD) symmetry classification is being trained at scale, yet the field lacks a common evaluation standard: papers report results on different subsets of experimental archives, with different target taxonomies, preprocessing, and metrics, and published models are rarely released in a form that permits rescoring. We present InformedXRD, an evaluation framework with three parts: (i) a machine-readable mapping from 230 space groups to 99 extinction groups, the finest classes powder diffraction can identify from systematic absences alone; (ii) a benchmark-construction method whose inclusion rules are specified as code and reproduce from a frozen snapshot, instantiated as RRUFF-473, RRUFF-325, and, as a small example, opXRD-98; (iii) a reporting protocol with prior-only frequency baselines, a label-permutation test, a hierarchy-aware error metric on the condensed translationengleiche subgroup graph, and stratified evaluation. We build on the mapping, RRUFF benchmarks, subgroup graph, and four ViT checkpoints released in our earlier work (Baggett et al., 2026). Using these checkpoints, a 1D residual CNN, and six architectures trained under the XRDBench recipe (Cao et al., 2025), we show that evaluation design changes what conclusions are justified: paired tests point in different directions at Top-1 and Top-5; paired Top-k tests do not separate the three leading XRDBench models, while hierarchy-aware analysis describes error distance and direction across all six; an input-blind frequency predictor matches the models on aggregate Top-k, yet a label-permutation test still detects pattern-specific evidence; and stratified analysis exposes a label-entropy confound that aggregate accuracy hides. All mapping tables, benchmark code, scripts, and checkpoints are released as shared infrastructure for PXRD-ML evaluation.

Figures & tables

Figure 1: The construction method of § 4.1 instantiated on two archives. Left: RRUFF, yielding the frozen RRUFF-473 and RRUFF-325 benchmarks with their nuisance-fit strata. Right: opXRD, a small example yielding opXRD-98. Rows are the steps of the method aligned across the two archives, with dashed boxes marking omitted steps and blue boxes marking the released evaluation sets. Per-step detail—thresholds, diagnostics, and the full gate cascade—is in Appendix E (RRUFF) and Appendix F (opXRD).

Model	Top-1 (%)	$\leq$ 2 hops (%)	Mean dist. ( $\downarrow$ )	Branch (%, $\downarrow$ )	Top-1 rank	Dist. rank
ViT: no preferred orientation	9.54	38.4	2.72	25.8	3	3
ViT: preferred-orientation only	13.54	53.0	2.51	34.1	1	2
ViT: final mixed-data calibrated	10.46	51.55	2.46	32.6	2	1
1D residual CNN	6.77	28.05	2.92	53.8	4	4
ViT: legacy uncalibrated	4.92	17.8	3.50	56.4	5	5

Table 1: Five-model comparison on RRUFF-325: Top-1,

\leq 2

-hop locality, mean DAG distance, and branch-jump fraction, with the implied ranks under flat Top-1 versus mean DAG distance. The two ranking columns disagree on the preferred-orientation-only and final mixed-data calibrated ViT checkpoints; the 1D CNN baseline is set apart from the ViT family by its 53.8% branch-jump rate. Hierarchy columns are over wrong Top-1 predictions; matched nulls for the preferred-orientation-only and final mixed-data calibrated ViTs and the CNN are given in Table 9 . Boldface marks the best value in each column and does not denote statistical significance.

	RRUFF-473			RRUFF-325
Model	Top-1 / 5 (%)	$\leq$ 2 hops obs./null	Mean dist. obs./null	Top-1 / 5 (%)	$\leq$ 2 hops obs./null	Mean dist. obs./null
CNN10	12.47 / 45.67	37.68 / 31.43	2.797 / 2.910	9.85 / 39.08	35.15 / 31.42	2.877 / 2.901
PatchTST	12.47 / 45.45	40.10 / 30.01	2.688 / 2.969	9.23 / 40.62	37.29 / 30.84	2.783 / 2.927
BiLSTM	8.67 / 30.87	34.49 / 31.26	2.894 / 2.929	4.92 / 24.00	34.95 / —	2.903 / —
CNN11	11.63 / 45.88	38.52 / 31.09	2.749 / 2.919	8.31 / 40.31	36.24 / —	2.842 / —
BiGRU	10.57 / 33.62	32.86 / 31.75	2.877 / 2.901	6.77 / 25.85	31.68 / —	2.921 / —

Table 2: Six XRDBench architectures rescored in extinction-group space. Models were trained on the unchanged public SimXRD train_1 contract ( Cao et al., 2025 ) and evaluated on RRUFF patterns exactly decimated onto that grid (contract specification in Appendix K ). Hierarchy columns are over wrong Top-1 predictions and give observed / model-matched null (§ 5.2 ); on RRUFF-325 matched nulls were computed for the two strongest models. No pair among CNN10, PatchTST, and CNN11 is separable by exact McNemar test at Top-1, Top-3, or Top-5 on either benchmark (smallest raw

p=0.124

). Boldface marks the best value per column, not statistical significance.

Figure 2: Decomposition of the catastrophic paradox on RRUFF-473. Left: the severity strata differ strongly in label-space structure, with the catastrophic stratum spanning far fewer unique extinction groups than the recoverable stratum. Right: classical profile-fit quality and neural classification difficulty do not align monotonically, and extinction-group reweighting removes about 45% of the recoverable-versus-catastrophic gap.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	SG model, SG space	Collapsed SG, EG space	Direct EG	Direct-EG advantage
Synthetic validation, full scale	15.17 / 27.81 / 35.25	18.46 / 33.27 / 42.35	19.32 / 34.28 / 43.38	+0.86 / +1.01 / +1.03
Synthetic, $\nicefrac{{1}}{{2}}$ scale (Top-1)	—	—	—	+0.59
Synthetic, $\nicefrac{{1}}{{4}}$ scale (Top-1)	—	—	—	+1.45
Synthetic, $\nicefrac{{1}}{{8}}$ scale (Top-1)	—	—	—	+2.77
RRUFF-473	—	3.38 / 5.92 / 8.25	3.59 / 5.71 / 8.46	+0.21 [ $-1.27$ , 1.69], $p=1.00$
RRUFF-325	—	1.54 / 4.31 / 7.08	2.77 / 5.23 / 8.31	+1.23 [ $-0.31$ , 2.77], $p=0.219$

Appendix

Table 3: Matched comparison of a 230-way space-group (SG) model and a 99-way extinction-group (EG) model. “Collapsed SG” scores the SG model after its probabilities are summed into EG space. All accuracy cells are Top-1 / Top-3 / Top-5 (%). Advantage is direct EG minus collapsed SG in percentage points; RRUFF advantages are Top-1 with paired-bootstrap 95% CI and exact McNemar

p

(Top-3 and Top-5 differences on both benchmarks likewise have

p\geq 0.42

and intervals containing zero).

	Schopmans et al.	SimXRD	RRUFF-473	RRUFF-325
Patterns Count	942	$\sim$ 3000	473	325
Classification target	145 SGs	230 SGs	99 EGs	99 EGs
Curation method	Hand-filtered	Partially documented	Algorithmic	Algorithmic
Unique minerals	—	—	227	168
Unique extinction groups	—	—	46	45
Difficult cases retained?	No (85 SGs excluded)	Unknown	Yes	No a

Appendix

Table 4: Benchmark audit against prior PXRD-ML evaluation sets. The main point is not raw score comparison but the mismatch in target taxonomy, filtering, difficult-case retention, and reproducibility.

Variant	Selected	Intersection	Selected-only	Reference-only
Baseline	471	471	0	0
Anchor full threshold $0.86$	473	471	2	0
Anchor cell threshold $0.0020$	472	471	1	0
Partial tie-break margin $0.04$	471	469	2	2
Lattice threshold $0.0006$	471	470	1	1

Appendix

Table 5: Local one-at-a-time sensitivity of the family-coherence curation defaults. Reported variants are the only tested perturbations that changed benchmark composition beyond the baseline 471/473 core match.

Benchmark / stratum	Top-1	Top-5	Top-10
RRUFF-473 prior	12.9	51.4	68.7
RRUFF-325 prior	12.6	43.7	62.8
Recoverable prior	12.2	41.3	—
Usable-or-better prior	15.7	48.4	—
Poor prior	16.2	66.2	—
Catastrophic prior	28.4	75.7	—

Appendix

Table 6: Benchmark-matched prior-only controls. “RRUFF prior” denotes the empirical extinction-group distribution of the named benchmark itself rather than the aggregate synthetic-training prior.

Gate	Records remaining
Initial archive	92,552
Declares a phase	2,179
Space group assignable to each phase (316 records with all phase labels stored explicitly; remainder derived from structure with spglib [ Togo et al., 2024 ] )	1,228
Single phase with lattice and derivable space group (891 from one source)	908
Cu-K $\alpha$ gate: explicit K $\alpha_{1}$ /K $\alpha_{2}$ pair or the conventional 2:1 weighted average	229
RRUFF angular support at native resolution, 10–80 ∘	98 (28 space groups, 23 extinction groups)

Appendix

Table 7: Filtering gates applied to opXRD to obtain opXRD-98, a one-pattern, one-label Cu-K

\alpha

set comparable with the RRUFF benchmarks.

Model	EG Top-1 / 3 / 5 (%)	$\leq$ 2 hops (%)	Mean DAG dist.
PatchTST	10.20 / 20.41 / 34.69	32.95	3.034
CNN11	6.12 / 20.41 / 26.53	31.52	3.174
CNN10	5.10 / 19.39 / 38.78	29.03	3.258
BiLSTM	5.10 / 15.31 / 22.45	23.66	3.441
BiGRU	4.08 / 17.35 / 24.49	20.21	3.394
Transformer	4.08 / 22.45 / 31.63	17.02	3.745

Appendix

Table 8: Six XRDBench architectures on opXRD-98 (98 Cu-K

\alpha

records), rescored in extinction-group space. Hierarchy columns are over wrong Top-1 predictions.

Reference or model	Mean DAG distance	Within 2 hops (%)	Descendant / ancestor / branch (%)
Uniform null, RRUFF-473	2.997	29.76	8.65 / 35.61 / 55.69
Uniform null, RRUFF-325	2.944	30.65	10.04 / 29.53 / 60.38
Frequency-matched null, RRUFF-473	2.692	42.15	—
Frequency-matched null, RRUFF-325	2.656	43.35	—
Most-frequent-group predictor, RRUFF-325	2.116	63.4	—
Final mixed-data ViT, RRUFF-325	2.46	51.55	56.7 / 10.7 / 32.6

Appendix

Table 9: Reference distributions for the hierarchy-aware metric and three RRUFF-325 case-study models. Direction is the descendant / ancestor / branch-jump split over wrong predictions.

Figure 3: Flat accuracy and hierarchy-aware severity rank the same five models differently. Each line is one model; its left endpoint is its Top-1 rank, and its right endpoint is its mean-DAG-distance rank on the condensed $t$ -subgroup graph. Lower mean DAG distance is better. The preferred-orientation-only ViT checkpoint wins under flat Top-1, while the final mixed-data calibrated ViT checkpoint has the best raw mean-DAG-distance rank. The 1D CNN baseline remains worse on both metrics and, more importantly for this paper, demonstrates that the hierarchy-aware metric distinguishes a separate branch-jump-heavy failure regime that flat Top-1 alone would compress into a single score.

Benchmark	Top-1 (%)	Top-5 (%)
RRUFF-473 ( $n=473$ )	9.94 [7.61, 12.68]	50.74 [46.29, 55.18]
RRUFF-325 ( $n=325$ )	10.46 [7.38, 13.85]	43.69 [38.46, 48.92]

Appendix

Table 10: Bootstrap 95% confidence intervals for the final mixed-data calibrated ViT checkpoint (point estimate [CI]).

Checkpoint	Val. loss	Val. Top-1/3/5 (%)	RRUFF-473 acc.	RRUFF-473 $\leq$ 2 hops / mean dist.	RRUFF-325 acc.	RRUFF-325 $\leq$ 2 hops / mean dist.
Submitted epoch 6	1.6631	50.25 / 75.47 / 84.85	9.09 / 19.66 / 28.54	28.84% / 2.956	6.77 / 18.77 / 25.23	28.05% / 2.921
Validation-selected epoch 46	1.3948	58.00 / 81.30 / 89.03	3.59 / 8.46 / 15.64	17.98% / 3.342	2.15 / 4.92 / 12.92	20.13% / 3.255

Appendix

Table 11: Convergence test for the 1D residual CNN control. Validation loss and Top-1/3/5 are on held-out synthetic data; benchmark accuracy columns are Top-1 / Top-3 / Top-5 (%); hierarchy columns are the within-2-hop fraction and mean DAG distance over wrong predictions. Epoch 46 is the validation-loss minimum after repeated learning-rate reductions; the formal early-stopping rule (patience 8) had not fired.

Figure 4: Extinction-group imbalance in the ICSD-derived source distribution underlying the aggregate synthetic training prior used for calibration. Horizontal axis: all 99 extinction groups, ordered by training frequency; vertical axis: prior probability; the ten most frequent groups are numbered on the bars and named in the inset. Heavy class imbalance means a naive prior-frequency baseline can attain non-trivial Top-

k

accuracy with no reference to the input.

Benchmark Reproducibility

See all 27 papers

PXRD-Bench: How Close Are We to Determining Crystal Structures from Powder Diffraction?
Oct 12, 2026Dongyeop Woo, Nofit Segal, Nayoung Kim +3Crystal Structure Prediction Materials Science
Korea Advanced Institute of Science & Technology (kaist.edu) · Massachusetts Institute of Technology (mit.edu)
CrystalXRD-Bench: Benchmarking Vision-Language Models for XRD Peak Indexing Across Diverse Crystalline Materials
May 28, 2026Chengliang Xu, Xiaogang Li, Peiyao Xiao +3VLM Evaluation Materials Science
Alibaba Group, Beijing, China
XRDiff: Crystal Structure Prediction from Powder X-Ray Diffraction Data Using Diffusion Models
Jun 12, 2026Nofit Segal, Mingda Li, Benjamin Kurt Miller +1Crystal Structure Prediction Diffusion Models
Department of Materials Science and Engineering, MIT, Cambridge, MA, USA. · Department of Nuclear Science and Engineering, MIT, Cambridge, MA, USA. · FAIR, Meta, San Francisco, CA, USA.

Oct 12, 2026J/K move · Enter open · S save

PXRD-Bench: How Close Are We to Determining Crystal Structures from Powder Diffraction?

Dongyeop Woo, Nofit Segal, Nayoung Kim, Minkyu Kim+2

Korea Advanced Institute of Science & Technology (kaist.edu) · Massachusetts Institute of Technology (mit.edu)

Crystal Structure Prediction Materials Science Benchmark Construction

Open paper AskPDF arXiv

Determining crystal structures from powder X-ray diffraction (PXRD) would enable routine characterization of newly synthesized materials, including those absent from reference databases. Generative models offer a promising approach, but existing evaluations remain fragmented and largely assume single-phase samples with known composition. To fill this gap, we introduce PXRD-Bench, a unified benchmark that extends evaluation to unknown per-element atom counts and recovery of both structures in two-phase mixtures. It provides approximately 4.1 million training structures with simulated PXRD, tests generalization to unseen compositions and structural prototypes, and evaluates recovery from experimental PXRD after excluding structurally matching training entries. To provide a strong reference baseline, we also introduce Bragglite, a PXRD-to-structure model developed through a systematic study of PXRD modeling choices. It achieves a 44.2% Top-1 match rate on experimental RRUFF patterns with known composition, 24.6% above the strongest prior baseline. Further analysis yields several key findings: some models fail to benefit from experimental PXRD, while generalization to unseen structural prototypes and recovery from two-phase mixtures remain challenging. Together, our benchmark, systematic studies, and Bragglite clarify the capabilities and limitations of current PXRD-to-structure models.

Abstract

Benchmark Reproducibility

Explore similar work

PXRD-Bench: How Close Are We to Determining Crystal Structures from Powder Diffraction?

CrystalXRD-Bench: Benchmarking Vision-Language Models for XRD Peak Indexing Across Diverse Crystalline Materials

XRDiff: Crystal Structure Prediction from Powder X-Ray Diffraction Data Using Diffusion Models