Verifiable pXRD: Hill-climbing pXRD structure determination with LLM agents
Organizations: Stanford University, Stanford, CA, USA · University of Toronto, Toronto, ON, Canada · Vector Institute for Artificial Intelligence, Toronto, ON, Canada · Acceleration Consortium, University of Toronto, Toronto, ON, Canada ·
Abstract
Autonomous crystal-structure determination from powder X-ray diffraction (pXRD) remains a key bottleneck for self-driving materials laboratories. We formulate the task of pXRD structure determination as a verifiable agentic search problem: candidate structures can be assessed during inference through forward-simulated profile agreement and during evaluation through structural matching to a known reference. To benchmark progress, we introduce pXRD-Bench, a suite of 200 simulated single-phase problems spanning in-distribution structure determination, challenging and polymorphic structures, and controlled compositional and topological distribution shifts. Agent adaptation raises selected structural recovery from 27% to 69%, with further gains from repeated sampling and verifier-guided inference-time scaling. We extend this approach to experimental multiphase pXRD through a reflexive evolution framework that learns from self-generated synthetic problems. A replaceable laboratory specification defines the sample family and measurement conditions. An evolver agent uses the known generating structures and measurement parameters to diagnose solver failures and develop persistent guidance and executable tools, which are then provided to agents on measured patterns. This supplies verifiable training experience without requiring a collection of solved experimental measurements. On 15 measured patterns from solid-state synthesis, the evolved package improves average profile agreement and transfers an analogue-substitution procedure that recovers a phase whose structure is absent from the supplied reference library. These results demonstrate how learning from verifiable synthetic experience can improve agentic characterization of experimental samples whose structures are not known in advance.
Figures & tables
| Corpus | Category | Construction and intended test | |
| pXRD-100 | Representative | 30 | Distribution-matched sample of eligible MP-20 structures, balancing crystal system, primitive-cell size, chemical complexity, stability, density, and elemental prevalence. |
| pXRD-100 | Challenge | 30 | Six structures selected along each of five predefined difficulty axes: rare symmetry, site and chemical complexity, peak congestion, scattering ambiguity, and large primitive cells. |
| pXRD-100 | Polymorph | 40 | Twenty structurally distinct pairs with identical reduced compositions, stratified by diffraction profile similarity |
| pXRD-ood | Substitution | 25 | Composition-novel structures obtained by full-orbit isovalent substitution of an exposed structural parent. |
| pXRD-ood | Ordered decoration | 25 | Composition-novel structures obtained by ordered decoration of symmetry orbits in an exposed structural parent. |
| pXRD-ood | Wyckoff enumeration | 25 | Topology-novel structures with exposed compositions generated by symmetry-constrained Wyckoff enumeration. |
| pXRD-100 categories | pXRD-ood categories | Aggregate | ||||||||||||||||||
| Method | Rep. | Chall. | Poly. | Sub. | Ord. | Wyck. | CSS | pXRD-100 | pXRD-ood | Overall | ||||||||||
| Crystalyze | 5 | 9 | 2 | 4 | 4 | 6 | 2 | 4 | 0 | 1 | 0 | 0 | 0 | 1 | 11 | 19 | 2 | 6 | 6.5% | 12.5% |
| DiffractGPT | 5 | – | 0 | – | 4 | – | 5 | – | 1 | – | 0 | – | 0 | – | 9 | – | 6 | – | 7.5% | – |
| Ab-PXRD-Solver | 21 | – | 5 | – | 16 | – | 9 | – | 9 | – | 1 | – | 1 | – | 42 | – | 20 | – | 31.0% | – |
| deCIFer † | 9 | 12 | 3 | 9 | 6 | 11 | 4 | 5 | 3 | 5 | 0 | 0 | 2 | 2 | 18 | 32 | 9 | 12 | 13.5% | 22.0% |
| PXRDGen ‡ | 21 | 30 | 11 | 22 | 15 | 32 | 10 | 12 | 7 | 13 | 0 | 0 | 0 | 1 | 47 | 84 | 17 | 26 | 32.0% | 55.0% |
| pXRD-100 categories | pXRD-ood categories | Aggregate | ||||||||||||||||||
| Condition | Rep. | Chall. | Poly. | Sub. | Ord. | Wyck. | CSS | pXRD-100 | pXRD-ood | Overall | ||||||||||
| GPT-5.6 Sol (high) | 15 | 16 | 6 | 6 | 9 | 11 | 11 | 12 | 6 | 6 | 2 | 2 | 5 | 5 | 30 | 33 | 24 | 25 | 27.0% | 29.0% |
| matsci-codex | 25 | 26 | 14 | 14 | 21 | 21 | 15 | 15 | 12 | 12 | 7 | 7 | 10 | 10 | 60 | 61 | 44 | 44 | 52.0% | 52.5% |
| pxrd-codex | 22 | 23 | 12 | 12 | 24 | 24 | 20 | 20 | 16 | 18 | 8 | 8 | 12 | 12 | 58 | 59 | 56 | 58 | 57.0% | 58.5% |
| adapted-codex | 27 | 27 | 22 | 22 | 28 | 28 | 19 | 19 | 21 | 21 | 10 | 12 | 11 | 12 | 77 | 77 | 61 | 64 | 69.0% | 70.5% |
| Condition | Mean (%) | Median (%) | Mean best-of-3 (%) | Within-scan SD (pp) | Mean time (min) |
|---|---|---|---|---|---|
| Deposited PG | 12.59 | 10.22 | — | — | — |
| Baseline | 7.90 | 7.20 | 6.92 | 1.04 | 27.0 |
| Evolved v6 | 6.71 | 5.86 | 6.04 | 0.80 | 50.2 |
| Case | Deposited PG | Baseline | Evolved v6 |
|---|---|---|---|
| 003 (Ca) | CaCO 3 : 13.5 wt% | CaCO 3 : 3/3 27.7–33.8 wt% | CaCO 3 : 3/3 28.4–30.2 wt% |
| 006 (Cu) | 7 phase models; CuWO 4 : 68.1 wt% | 2 phase models in 3/3 CuWO 4 : 97.7–97.9 wt% | 2 phase models in 3/3 CuWO 4 : 96.8–97.5 wt% |
| 007 (Mg) | MgO: absent | MgO: 3/3 7.0–7.6 wt% | MgO: 3/3 7.1–7.4 wt% |
| 008 (Mn) | WO 3 : absent | WO 3 : 3/3 2.0–2.1 wt% | WO 3 : 3/3 2.0–3.3 wt% |
| 013 (Pb) | Pb 3 WO 6 : 15.3 wt% | Pb 3 WO 6 : 0/3 | Pb 3 WO 6 : 0/3 |
| 014 (Sr) | Sr 2 WO 5 : 24.8 wt% | Sr 2 WO 5 : 0/3 | Sr 2 WO 5 : 2/3 24.9–25.9 wt% |
| Condition | Mean (%) | Median (%) | Mean best-of-3 (%) | Mean time (min) |
|---|---|---|---|---|
| Baseline | 7.90 | 7.20 | 6.92 | 27.0 |
| Evolved v6 | 6.71 | 5.86 | 6.04 | 50.2 |
| Framework access | 5.98 | 5.49 | 5.47 | 66.1 |
| Framework + v6 | 6.49 | 6.36 | 5.68 | 67.0 |
| Evolved framework v5 † | 6.44 | 6.15 | 5.65 | 43.8 |
| Initial attempts | Added attempts | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Condition | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Total |
| Baseline | 12.89 | 16.48 | 16.72 | 13.99 | 13.53 | 12.54 | 18.68 | 16.71 | 15.27 | 0/9 |
| Evolved v6 | 14.66 | 7.12 | 7.88 | 12.97 | 12.90 | 14.67 | 7.32 | 15.77 | 15.34 | 3/9 |
| Framework access | 14.65 | 8.98 | 11.73 | 11.13 † | 11.14 | 11.36 | 13.28 | 10.21 | 12.90 | 0/9 |
| Framework + v6 | 15.85 | 13.04 | 7.53 | 13.44 | 7.19 | 6.82 | 6.93 | 13.40 | 13.08 | 5/9 |
| Evolved framework v5 | 6.64 | 15.23 | 15.59 | 15.22 | 14.38 | 12.97 | 13.07 | 13.18 | 12.93 | 1/9 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Native diffraction input | Structural side information | Candidates | Generation and reporting protocol |
|---|---|---|---|---|
| Crystalyze | Cu K Caglioti–Gaussian encoder profile | Element set; stoichiometric counts not enforced | 8 | Released diffusion sampler with allowed-element constraints; samples are reported in generation order. |
| DiffractGPT | Released 180-bin nearest-bin vector | Reduced composition | 1 | One greedy decode with a 1,024-token generation limit; no stochastic best-of-eight policy is introduced. |
| Ab-PXRD-Solver | Released noisy GSAS-II profile | Reduced composition; space group inferred | 1 | Deterministic search with Halton quasi-random sampling and a four-hour CPU limit per case; a candidate emitted before timeout remains eligible. |
| deCIFer | Clean 1,000-bin -space pseudo-Voigt profile | Full conventional-cell composition (oracle ) | 8 | Stochastic autoregressive sampling at temperature 1 and top- ; raw samples are reported in generation order. |
| PXRDGen | Frozen GSAS-II Cu K profile | Exact primitive-cell composition and atom count (oracle ) | 8 | 200 diffusion steps with step size 5.0; the reported structures are raw generated samples without the paper’s subsequent Rietveld stage. |
| RealPXRD-Solver | Variable-length native sparse peak list | Exact primitive-cell composition and atom count (oracle ); no lattice | 20 (first 8 reported) | 200 diffusion steps with step size 5.0; candidate order is released sampler order because the public GSAS-II ranking/refinement stage is unavailable. |
| Method | Candidate slots | Valid structures | Valid and composition-matched |
|---|---|---|---|
| Crystalyze | 1,600 | 1,600 | 351 |
| DiffractGPT | 200 | 146 | 138 |
| Ab-PXRD-Solver | 200 | 160 | 160 |
| deCIFer | 1,600 | 1,500 | 1,406 |
| PXRDGen | 1,600 | 1,592 | 1,592 |
| RealPXRD-Solver | 1,600 | 1,587 | 1,587 |
| Method | Corpus | Pass@1 | Pass@8 |
|---|---|---|---|
| Crystalyze / StructSnap | pXRD-100 | ||
| pXRD-ood | |||
| PXRDGen | pXRD-100 | ||
| pXRD-ood | |||
| Ab-PXRD-Solver | pXRD-100 | — | |
| pXRD-ood | — |
| All candidate slots | Exact first candidates | |||
|---|---|---|---|---|
| Method | Exported/ available | Failures | Refined | Retained original |
| Crystalyze / StructSnap | 1,600/1,600 | 0 | 14 | 0 |
| PXRDGen | 1,225/1,600 | 375 | 27 | 11 |
| Ab-PXRD-Solver | 159/160 | 1 | 64 | 0 |
| RealPXRD-Solver adapter | 3,975/4,000 | 25 | 47 | 0 |
| Method | Corpus | Mean (%) | Median (%) | |
|---|---|---|---|---|
| Crystalyze / StructSnap | pXRD-100 | 100 | ||
| pXRD-ood | 100 | |||
| PXRDGen | pXRD-100 | 76 | ||
| pXRD-ood | 71 | |||
| Ab-PXRD-Solver | pXRD-100 | 76 | ||
| pXRD-ood | 83 |
| Condition | Interface | Scientific environment | Persistent pXRD capability | Per-case limit |
|---|---|---|---|---|
| GPT-5.6 Sol (high) | Single Responses API call; no workspace or tools | None | None | 65,536 output tokens |
| matsci-codex | Stateful Codex coding agent | Scientific Python, including NumPy, SciPy, pymatgen, ASE, spglib, and PyXtal; read-only local COD | None | 2 vCPU, 5 GiB, 1 hour |
| pxrd-codex | Stateful Codex coding agent | matsci-codex environment plus GSAS-II revision 5837 and bare DICVOL14 | None | 2 vCPU, 5 GiB, 1 hour |
| adapted-codex | Stateful Codex coding agent | Same pXRD environment as pxrd-codex | Frozen 139-word skill and neutral DICVOL14 adapter | 2 vCPU, 5 GiB, 1 hour |
| Condition | Attempts | Timeouts / other incomplete | Mean / median wall (min) | Total wall-hours | Usage records | Mean input / cached / output tokens |
|---|---|---|---|---|---|---|
| GPT-5.6 Sol (high) | 200 | 0 / 0 | 6.94 / 6.95 | 23.1 | 200 | 0.140M / 0 / 0.021M |
| matsci-codex | 200 | 10 / 3 infrastructure | 25.28 / 21.32 | 84.3 | 187 | 6.488M / 6.294M / 0.041M |
| pxrd-codex | 200 | 1 / 7 nonzero exits | 16.30 / 12.09 | 54.3 | 192 | 7.112M / 6.976M / 0.039M |
| adapted-codex | 200 | 3 / 0 | 21.21 / 16.16 | 70.7 | 197 | 4.203M / 4.045M / 0.036M |
| Condition | Usage records | Input tokens | Cached-input subset | Output tokens | Wall hours | Timeouts |
|---|---|---|---|---|---|---|
| pxrd-codex , Sol (high) | 156/160 | 1,110.30M | 1,089.35M | 6.28M | 49.4 | 2 |
| pxrd-codex , Luna (max) | 112/160 | 1,827.73M | 1,792.60M | 12.02M | 79.5 | 29 |
| adapted-codex , Sol (high) | 156/160 | 621.02M | 605.17M | 5.56M | 44.9 | 3 |
| Stage | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| E | E | M | E | M | M | E | E | E | E | M | E | M | M | E | E | E | E | M | E | |
| E | E | E | – | A | M | – | – | – | – | M | – | E | M | A | – | – | M | E | E | |
| – | – | – | – | – | – | – | – | |||||||||||||
| E | E | E | – | – | M | – | – | – | – | E | – | – | M | – | – | – | M | M | E | |
| – | – | – | – | – | – | – | – | – | – | – |
| Case | Generating phase inventory | Intended difficulty |
|---|---|---|
| c01 | Single-phase control with an available parent reference. | |
| c02 | , , | Low counts; a 7.3 wt% residual precursor. |
| c03 | , , | All entries withheld; carbonate-derived mixture. |
| c04 | , , | All entries withheld; weak residual oxide. |
| c05 | , , | Withheld 10.7 wt% intermediate and overlapping reflections. |
| c06 | , , , | Four phases; weak PbO and specimen broadening. |
| Component | Setting |
|---|---|
| Radiation and geometry | Cu K : , intensity ratio ; Bragg–Brentano reflection, radius . |
| Angular grid | – in , step . |
| Count level and noise | Curator-selected signal maximum of 5,000, 15,000, or 40,000 counts, before background; independent Poisson noise on signal plus background. |
| Instrument profile | , , ; . , polarization . |
| Angular corrections | Zero offset ; specimen displacement . |
| Size and thermal motion | Crystallite size ; per-phase . |
| Case | Run | PG | Sol high | Baseline Sol-max | Evolved v6 | Framework access | Framework + v6 | Evolved framework v5 | Opus |
|---|---|---|---|---|---|---|---|---|---|
| 001 (BaCO 3 ) | 1 | 8.00 | 8.73 | 7.22 | 7.15 | 5.85 | 6.87 | 6.67 | 7.30 |
| 2 | 8.73 | 7.22 | 7.23 | 6.08 | 6.65 | 5.83 | |||
| 3 | 8.82 | 7.15 | 7.58 | 5.92 | 6.79 | 5.96 | |||
| 002 (BaO 2 ) | 1 | 7.20 | 8.87 | 7.59 | 8.73 | 8.51 | 7.06 | 7.12 | 8.30 |
| 2 | 7.68 | 12.67 | 7.65 | 7.24 | 5.78 | 7.21 | |||
| 3 | 7.80 | 7.12 | 7.10 | 7.21 | 8.29 | 7.60 |
| Version | Mean change | Required mean gain | Worst case change |
|---|---|---|---|
| v1 | -11.81 | 10.13 | -39.66 |
| v2 | -3.55 | 13.94 | -32.62 |
| v3 | +4.16 | 11.95 | -21.83 |
| v4 | -6.68 | 8.46 | -38.34 |
| v5 | +2.96 | 10.06 | -37.62 |
| v6 | -6.04 | 8.43 | -35.93 |
| Condition | Background terms | Nondefault | Phases |
|---|---|---|---|
| Baseline | 15 (6–20) | 12/45 | 2.47 |
| Evolved v6 | 8 (6–12) | 17/45 | 2.91 |
| Framework access | 16 (10–100) | 44/45 | 2.64 |
| Framework + v6 | 8 (3–33) | 45/45 | 2.80 |
| Evolved framework v5 | 12 (8–30) | 45/45 | 2.51 |
Crystal Structure Prediction
Crystal structure prediction seeks stable atomic arrangements from composition, molecular identity, or experimental measurements.
See all 37 papers