Verifiable pXRD: Hill-climbing pXRD structure determination with LLM agents
Authors: Samuel Chen, Jiaru Bai, Austin Henry Cheng, Varinia Bernales, Alán Aspuru-Guzik
Organizations: Stanford University, Stanford, CA, USA · University of Toronto, Toronto, ON, Canada · Vector Institute for Artificial Intelligence, Toronto, ON, Canada · Acceleration Consortium, University of Toronto, Toronto, ON, Canada · Canadian Institute for Advanced Research (CIFAR), Toronto, ON, Canada · NVIDIA, Toronto, ON, Canada
Autonomous crystal-structure determination from powder X-ray diffraction (pXRD) remains a key bottleneck for self-driving materials laboratories. We formulate the task of pXRD structure determination as a verifiable agentic search problem: candidate structures can be assessed during inference through forward-simulated profile agreement and during evaluation through structural matching to a known reference. To benchmark progress, we introduce pXRD-Bench, a suite of 200 simulated single-phase problems spanning in-distribution structure determination, challenging and polymorphic structures, and controlled compositional and topological distribution shifts. Agent adaptation raises selected structural recovery from 27% to 69%, with further gains from repeated sampling and verifier-guided inference-time scaling. We extend this approach to experimental multiphase pXRD through a reflexive evolution framework that learns from self-generated synthetic problems. A replaceable laboratory specification defines the sample family and measurement conditions. An evolver agent uses the known generating structures and measurement parameters to diagnose solver failures and develop persistent guidance and executable tools, which are then provided to agents on measured patterns. This supplies verifiable training experience without requiring a collection of solved experimental measurements. On 15 measured patterns from solid-state synthesis, the evolved package improves average profile agreement and transfers an analogue-substitution procedure that recovers a phase whose structure is absent from the supplied reference library. These results demonstrate how learning from verifiable synthetic experience can improve agentic characterization of experimental samples whose structures are not known in advance.
Figures & tables
Figure 1: pXRD structure determination as a verifiable inverse problem. Ground truth: pXRD-Bench case pxrd100-rep-010 . Proposed: pxrd-codex selected.cif . Structures rendered with Crystal Toolkit ( Horton et al., 2023 ) .
Corpus
Category
n
Construction and intended test
pXRD-100
Representative
30
Distribution-matched sample of eligible MP-20 structures, balancing crystal system, primitive-cell size, chemical complexity, stability, density, and elemental prevalence.
pXRD-100
Challenge
30
Six structures selected along each of five predefined difficulty axes: rare symmetry, site and chemical complexity, peak congestion, scattering ambiguity, and large primitive cells.
pXRD-100
Polymorph
40
Twenty structurally distinct pairs with identical reduced compositions, stratified by diffraction profile similarity
pXRD-ood
Substitution
25
Composition-novel structures obtained by full-orbit isovalent substitution of an exposed structural parent.
pXRD-ood
Ordered decoration
25
Composition-novel structures obtained by ordered decoration of symmetry orbits in an exposed structural parent.
pXRD-ood
Wyckoff enumeration
25
Topology-novel structures with exposed compositions generated by symmetry-constrained Wyckoff enumeration.
Table 1: Composition of pXRD-Bench. pXRD-100 measures recovery of structures represented in the training distributions of existing generative methods, whereas pXRD-ood separates compositional extrapolation from structural-topology extrapolation.
pXRD-100 categories
pXRD-ood categories
Aggregate
Method
Rep.
Chall.
Poly.
Sub.
Ord.
Wyck.
CSS
pXRD-100
pXRD-ood
Overall
Crystalyze
5 ∣
9
2 ∣
4
4 ∣
6
2 ∣
4
0 ∣
1
0 ∣
0
0 ∣
1
11 ∣
19
2 ∣
6
6.5% ∣
12.5%
DiffractGPT
5 ∣
–
0 ∣
–
4 ∣
–
5 ∣
–
1 ∣
–
0 ∣
–
0 ∣
–
9 ∣
–
6 ∣
–
7.5% ∣
–
Ab-PXRD-Solver
21 ∣
–
5 ∣
–
16 ∣
–
9 ∣
–
9 ∣
–
1 ∣
–
1 ∣
–
42 ∣
–
20 ∣
–
31.0% ∣
–
deCIFer †
9 ∣
12
3 ∣
9
6 ∣
11
4 ∣
5
3 ∣
5
0 ∣
0
2 ∣
2
18 ∣
32
9 ∣
12
13.5% ∣
22.0%
PXRDGen ‡
21 ∣
30
11 ∣
22
15 ∣
32
10 ∣
12
7 ∣
13
0 ∣
0
0 ∣
1
47 ∣
84
17 ∣
26
32.0% ∣
55.0%
Table 2: Exact structural recovery of existing pXRD structure-determination methods on pXRD-Bench. Category and corpus entries report pass@1 ∣ pass@8 as the number of recovered structures.
pXRD-100 categories
pXRD-ood categories
Aggregate
Condition
Rep.
Chall.
Poly.
Sub.
Ord.
Wyck.
CSS
pXRD-100
pXRD-ood
Overall
GPT-5.6 Sol (high)
15 ∣
16
6 ∣
6
9 ∣
11
11 ∣
12
6 ∣
6
2 ∣
2
5 ∣
5
30 ∣
33
24 ∣
25
27.0% ∣
29.0%
matsci-codex
25 ∣
26
14 ∣
14
21 ∣
21
15 ∣
15
12 ∣
12
7 ∣
7
10 ∣
10
60 ∣
61
44 ∣
44
52.0% ∣
52.5%
pxrd-codex
22 ∣
23
12 ∣
12
24 ∣
24
20 ∣
20
16 ∣
18
8 ∣
8
12 ∣
12
58 ∣
59
56 ∣
58
57.0% ∣
58.5%
adapted-codex
27 ∣
27
22 ∣
22
28 ∣
28
19 ∣
19
21 ∣
21
10 ∣
12
11 ∣
12
77 ∣
77
61 ∣
64
69.0% ∣
70.5%
Table 3: Exact structural recovery across the agent-adaptation ladder on pXRD-Bench. Entries report selected exact ∣ any exact , where any exact permits a match from either selected.cif or alternative-<n>.cif .
Figure 2: Repeated-sampling results on pXRD-Bench-20. Panels (a)–(c) show candidate-set exact StructureMatcher recovery for pxrd-codex Sol, pxrd-codex Luna, and adapted-codex Sol, respectively. A colored square denotes a candidate-structure match; the rightmost Exact column counts these matches. Panel (d) shows order-invariant expected oracle pass@ k , computed by averaging cumulative candidate-set coverage over all (k8) subsets of the observed rollouts.
Figure 3: Inference-time verifier accuracy and token-cost-scaled recovery. (a) Decisions on 200 adapted-codex pXRD-Bench trajectories. False stops are incorrect structures judged complete; false continues are exact structures allocated another solve. The Rwp thresholds were selected retrospectively to match the two LLM policies’ false-stop counts. (b) Verifier-gated continuation framework plotted against the 8 independent adapted-codex Sol reruns. The eight independent reruns points are placed at expected token cost for k=1,…,8 rollouts. The verifier-gated curve shows cumulative recovery after cumulative token cost following the initial blind pass, a coverage-directed continuation, and two verifier-authorized fresh solves.
Figure 4: Synthetic experience for experimental characterization. A laboratory specification defines the scientific setting and permitted synthetic variation. Fresh solver workspaces receive only public measurements and metadata. A persistent evolver can inspect trajectories and use synthetic reference information to diagnose failures, revise a capability package, and test it on further solves. The frozen package is subsequently applied to measured patterns. Ground-truth access is confined to synthetic research and evaluation; experimental fits retain the solving agent’s own model and calibration.
Figure 5: Evolution improved the initial curriculum but was neither monotonic nor uniformly reliable. (a) Case-level scores for the baseline, single-attempt revision screens, and the retained v6 package. Gray cells were not evaluated under that revision. The v6 mean includes its initial screen and two additional attempts per case; it is a repeated development evaluation, not an independent held-out test. (b) Individual baseline and v6 scores, with larger dots denoting case means. (c) Initial-panel means and prospective comparisons on two subsequent pairs of new synthetic cases, each with three attempts per condition and no package edits before that comparison. Scores are the composite Q .
Figure 6: Experimental transfer of the v6 package. (a) Agent points show the mean verified Rwp for each measurement; horizontal lines span the observed attempts, not a confidence interval. Each PG point scores the saved deposited profile. The horizontal axis is logarithmic. Labels give the case and second precursor; every reaction also contains WO3 . (b) Change in the measurement-level mean from baseline to v6; negative values indicate improved profile agreement. Only the initial three attempts per measurement are included.
Condition
Mean Rwp (%)
Median (%)
Mean best-of-3 (%)
Within-scan SD (pp)
Mean time (min)
Deposited PG
12.59
10.22
—
—
—
Baseline
7.90
7.20
6.92
1.04
27.0
Evolved v6
6.71
5.86
6.04
0.80
50.2
Table 4: Transfer from synthetic experience to measured patterns. Mean and median summarize the 15 measurement-level mean residuals. Best-of-three averages the lowest Rwp among three attempts on each measurement. Within-scan SD is the mean sample standard deviation across measurements, in percentage points (pp). Time is mean wall time per attempt. Lower residuals indicate better profile agreement.
Figure 7: Same phase inventory, improved profile agreement: the Zn measurement. Baseline and evolved attempt 3 on scan 015, shown as an illustrative trajectory comparison rather than a best-of-three selection. All curves are taken from saved independent project replays. Observations are black points, calculations are colored lines, and differences are gray with a vertical offset. The right panels resolve the WO3 -rich region. Both fits contain ZnWO4 , WO3 , and ZnO ; the evolved attempt changes the source structure and profile treatment. Intensities share the observed maximum as a display normalization, while reported Rwp uses raw counts over the full evaluation window.
Table 5: Selected differences in phase assignment and quantity. Agent entries give the number of initial attempts containing the indicated phase and the range of its reported crystalline weight fraction where present. Rwp averages all three attempts, including those without the phase. The Cu row instead gives the total number of phase models and the dominant-phase fraction. Sr and Pb presence requires a structural match to the deposited reference structure; other rows use the submitted phase formula. The table highlights substantial disagreements rather than listing every trace phase.
Condition
Mean Rwp (%)
Median (%)
Mean best-of-3 (%)
Mean time (min)
Baseline
7.90
7.20
6.92
27.0
Evolved v6
6.71
5.86
6.04
50.2
Framework access
5.98
5.49
5.47
66.1
Framework + v6
6.49
6.36
5.68
67.0
Evolved framework v5 †
6.44
6.15
5.65
43.8
Table 6: Scientific context and evolved guidance. Summary conventions follow Table 4 . Best-of-three averages the lowest Rwp among three attempts on each of the 15 measurements; the later Pb/Sr repeats are excluded.
Figure 8: The effect of framework access varies across measurements. (a) Points show the mean verified Rwp ; horizontal lines span the observed attempts. Labels give the second precursor and firing temperature; every reaction also contains WO3 . (b) Differences from the baseline’s measurement-level mean. The framework-access condition improves on baseline on every scan, but is not the best condition on every scan. Only the initial three attempts per measurement are included.
Figure 9: The Sr measurement exposes a difference in the hypotheses explored. The illustrated baseline r1 and framework-access r2 are their lowest- Rwp attempts in the original three-repeat panels; evolved r2 is the successful prototype-transfer example discussed in the text. Framework access fits several weak features using a larger phase set but does not recover Sr2WO5 . The evolved attempt constructs Sr2WO5 from a Ba2ReO5 prototype and fits a three-phase mixture. Dashed lines mark diagnostic reflection positions identified in that trajectory, not a separate reference-assisted fit. All plotted profiles are frozen project replays; display and residual conventions follow Figure 7 . The repeat frequencies in Table 7 provide context for these selected examples.
Initial attempts
Added attempts
Condition
1
2
3
4
5
6
7
8
9
Total
Baseline
12.89
16.48
16.72
13.99
13.53
12.54
18.68
16.71
15.27
0/9
Evolved v6
14.66
7.12
7.88
12.97
12.90
14.67
7.32
15.77
15.34
3/9
Framework access
14.65
8.98
11.73
11.13 †
11.14
11.36
13.28
10.21
12.90
0/9
Framework + v6
15.85
13.04
7.53
13.44
7.19
6.82
6.93
13.40
13.08
5/9
Evolved framework v5
6.64
15.23
15.59
15.22
14.38
12.97
13.07
13.18
12.93
1/9
Table 7: Profile agreement and phase recovery on the Sr measurement (014). Entries are verified Rwp values (%). Bold entries indicate a submitted structure matching the deposited Sr2WO5 phase; Total counts matches among the nine completed solves. Initial attempts belong to the full-panel experiment; added attempts were collected after inspecting those results. All repeats concern the same specimen.
Figure 10: Two framework-access evolution campaigns retained their starting agent. (a) Early selective screening: baseline cells average three solves; revision cells show single solves. Gray cells were not scored. Passing cases were set aside; combining different revisions does not validate a single package. (b) Later 3-repeat evaluation: each revision has three solves on six cases. Changes are relative to the baseline on the same cases. Mean gains must reach the plotted threshold and every case must remain above the dashed regression limit. Vertical separators mark panel changes; the boxed v5 was also tested experimentally. Appendix A.8 gives the protocols and scores; further details can be found in the verifiable-pxrd repo.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Native diffraction input
Structural side information
Candidates
Generation and reporting protocol
Crystalyze
Cu K α Caglioti–Gaussian encoder profile
Element set; stoichiometric counts not enforced
8
Released diffusion sampler with allowed-element constraints; samples are reported in generation order.
DiffractGPT
Released 180-bin nearest-bin vector
Reduced composition
1
One greedy decode with a 1,024-token generation limit; no stochastic best-of-eight policy is introduced.
Ab-PXRD-Solver
Released noisy GSAS-II profile
Reduced composition; space group inferred
1
Deterministic search with Halton quasi-random sampling and a four-hour CPU limit per case; a candidate emitted before timeout remains eligible.
deCIFer
Clean 1,000-bin Q -space pseudo-Voigt profile
Full conventional-cell composition (oracle Z )
8
Stochastic autoregressive sampling at temperature 1 and top- k=50 ; raw samples are reported in generation order.
PXRDGen
Frozen GSAS-II Cu K α1/α2 profile
Exact primitive-cell composition and atom count (oracle Z )
8
200 diffusion steps with step size 5.0; the reported structures are raw generated samples without the paper’s subsequent Rietveld stage.
RealPXRD-Solver
Variable-length native sparse peak list
Exact primitive-cell composition and atom count (oracle Z ); no lattice
20 (first 8 reported)
200 diffusion steps with step size 5.0; candidate order is released sampler order because the public GSAS-II ranking/refinement stage is unavailable.
Appendix
Table 8: Reproduction protocol for the specialist pXRD baselines. Oracle inputs are derived from the hidden reference structure and therefore provide more information than the reduced-composition benchmark prompt.
Method
Candidate slots
Valid structures
Valid and composition-matched
Crystalyze
1,600
1,600
351
DiffractGPT
200
146
138
Ab-PXRD-Solver
200
160
160
deCIFer
1,600
1,500
1,406
PXRDGen
1,600
1,592
1,592
RealPXRD-Solver
1,600
1,587
1,587
Appendix
Table 9: Output completeness and validity before additional refinement, pooled across all 200 problems. Counts refer to candidate slots, not recovered problems. We use one slot per problem for DiffractGPT and Ab-PXRD-Solver, and the first eight for the other methods, including RealPXRD-Solver. Missing outputs remain in the slot denominator.
Method
Corpus
Pass@1
Pass@8
Crystalyze / StructSnap
pXRD-100
11→12
19→20
pXRD-ood
2→2
6→6
PXRDGen
pXRD-100
47→32
84→67
pXRD-ood
17→6
26→17
Ab-PXRD-Solver
pXRD-100
42→41
—
pXRD-ood
20→23
—
Appendix
Table 10: Exact recovery before → after method-specific refinement, out of 100 problems per corpus. Execution/export failures retain the original candidate. Crystalyze results use the first StructSnap stage.
All candidate slots
Exact first candidates
Method
Exported/ available
Failures
Refined
Retained original
Crystalyze / StructSnap
1,600/1,600
0
14
0
PXRDGen
1,225/1,600
375
27
11
Ab-PXRD-Solver
159/160
1
64
0
RealPXRD-Solver adapter
3,975/4,000
25
47
0
Appendix
Table 11: Execution and fallback accounting across both corpora. Export/failure counts cover all available candidate slots. The final two columns count exact first candidates out of 200 problems and sum to post-refinement pass@1. Export completion does not establish structural correctness or optimizer convergence.
Method
Corpus
n
Mean Rwp (%)
Median Rwp (%)
Crystalyze / StructSnap
pXRD-100
100
93.35→80.93
95.66→86.23
pXRD-ood
100
94.57→87.68
95.40→92.66
PXRDGen
pXRD-100
76
89.47→51.10
93.79→51.60
pXRD-ood
71
93.10→82.10
96.17→92.09
Ab-PXRD-Solver
pXRD-100
76
41.47→30.50
36.54→17.70
pXRD-ood
83
54.66→46.12
53.08→30.67
Appendix
Table 12: Profile agreement before → after refinement. Here n counts successfully exported first candidates with both residuals available; the same n candidates contribute on each side. Failed refinements and missing originals are excluded from this paired table, but remain in recovery denominators.
Condition
Interface
Scientific environment
Persistent pXRD capability
Per-case limit
GPT-5.6 Sol (high)
Single Responses API call; no workspace or tools
None
None
65,536 output tokens
matsci-codex
Stateful Codex coding agent
Scientific Python, including NumPy, SciPy, pymatgen, ASE, spglib, and PyXtal; read-only local COD
None
2 vCPU, 5 GiB, 1 hour
pxrd-codex
Stateful Codex coding agent
matsci-codex environment plus GSAS-II revision 5837 and bare DICVOL14
None
2 vCPU, 5 GiB, 1 hour
adapted-codex
Stateful Codex coding agent
Same pXRD environment as pxrd-codex
Frozen 139-word skill and neutral DICVOL14 adapter
2 vCPU, 5 GiB, 1 hour
Appendix
Table 13: Interfaces and scientific environments in the agent-adaptation ladder. The capability column describes persistent task-specific material supplied before a rollout; scripts written by an agent during its own rollout are not persistent across cases.
Condition
Attempts
Timeouts / other incomplete
Mean / median wall (min)
Total wall-hours
Usage records
Mean input / cached / output tokens
GPT-5.6 Sol (high)
200
0 / 0
6.94 / 6.95
23.1
200
0.140M / 0 / 0.021M
matsci-codex
200
10 / 3 infrastructure
25.28 / 21.32
84.3
187
6.488M / 6.294M / 0.041M
pxrd-codex
200
1 / 7 nonzero exits
16.30 / 12.09
54.3
192
7.112M / 6.976M / 0.039M
adapted-codex
200
3 / 0
21.21 / 16.16
70.7
197
4.203M / 4.045M / 0.036M
Appendix
Table 14: Compute and token accounting for the 200-case agent-adaptation evaluations. Wall time includes model-service latency, agent reasoning, and local command execution. For attempts without a final usage record, token counts are estimated from wall-clock duration using the mean tokens per minute of recorded attempts under the same condition; cached input is a subset of input rather than an additional token category.
Condition
Usage records
Input tokens
Cached-input subset
Output tokens
Wall hours
Timeouts
pxrd-codex , Sol (high)
156/160
1,110.30M
1,089.35M
6.28M
49.4
2
pxrd-codex , Luna (max)
112/160
1,827.73M
1,792.60M
12.02M
79.5
29
adapted-codex , Sol (high)
156/160
621.02M
605.17M
5.56M
44.9
3
Appendix
Table 15: Compute and token accounting for the repeated-sampling campaigns. Each row contains 160 attempted rollouts. Token totals sum attempts with a final usage event and are therefore lower bounds; cached input is included in, rather than additional to, total input.
Stage
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
S0
E
E
M
E
M
M
E
E
E
E
M
E
M
M
E
E
E
E
M
E
J0
C×
C×
C✓
S✓
C✓
C✓
S✓
S✓
S✓
S✓
C✓
S✓
C✓
C✓
C×
S✓
S✓
C×
C✓
C×
C1
E
E
E
–
A
M
–
–
–
–
M
–
E
M
A
–
–
M
E
E
J1
C×
C×
C×
–
S✓
C✓
–
–
–
–
C✓
–
S✓
C✓
S✓
–
–
C✓
C×
C×
S2
E
E
E
–
–
M
–
–
–
–
E
–
–
M
–
–
–
M
M
E
J2
C×
S✓
C×
–
–
C✓
–
–
–
–
C×
–
–
S×
–
–
–
C✓
C✓
S✓
Appendix
Table 16: Case-level execution of verifier-gated inference-time scaling on pXRD-Bench-20. Columns follow the fixed panel order. Solve rows report E when selected.cif is exact, A when only a declared alternative is exact, and M when no submitted CIF is exact; – denotes a case already terminated by the gate. Review rows report S (stop) or C (continue), with ✓ or × indicating retrospective agreement or disagreement, respectively, with candidate-set exactness at that stage. S0 is the initial blind solve, C1 the evidence-conditioned continuation, S2 and S3 fresh solves, and Jt the intervening reviews.
Figure 11: Candidate-level fixed-parameter Rwp distributions under the shared fixedparams-v1 GSAS-II forward model. Blue and red distributions denote candidates accepted and rejected, respectively, by the benchmark structure matcher. Solid lines indicate means and dashed lines indicate medians. Both reference and candidate profiles are clean, independently maximum-normalized simulations; no fitting, alignment, or refinement is performed. The horizontal scale and forward model are shared across all panels.
Figure 12: Candidate-level framework-native Rwp distributions. Colors and vertical-line styles follow Figure 11 . Because the angular or reciprocal-space grids, peak shapes, broadening models, noise, and discretization vary between panels, absolute values should be compared only within a panel and not used to rank frameworks. For matsci-codex , pxrd-codex , and adapted-codex , the displayed range ends at the pooled candidate 95th percentile; the upper 5% tails remain included in the reported means and medians. The RealPXRD-Solver panel records that a fixed-grid native residual is undefined for its released sparse-peak input representation.
Case
Generating phase inventory
Intended difficulty
c01
CaWO4
Single-phase control with an available parent reference.
c02
ZnWO4 , ZnO , WO3
Low counts; a 7.3 wt% residual precursor.
c03
SrWO4 , SrCO3 , WO3
All SrWO4 entries withheld; carbonate-derived mixture.
c04
MnWO4 , Mn3O4 , WO3
All MnWO4 entries withheld; weak residual oxide.
c05
CaWO4 , Ca3WO6 , WO3
Withheld 10.7 wt% intermediate and overlapping reflections.
c06
PbWO4 , Pb3O4 , PbO , WO3
Four phases; weak PbO and specimen broadening.
Appendix
Table 17: Initial synthetic curriculum. Generating phase identities and fractions are visible to the evolver and evaluator, but not supplied to the solver. Withholding removes the specified references from the solver’s library; chemically related structures may remain available.
Component
Setting
Radiation and geometry
Cu K α1/α2 : 1.540598/1.544426A˚ , intensity ratio I2/I1=0.5 ; Bragg–Brentano reflection, radius 200mm .
Angular grid
9.90343322∘ – 100.09455322∘ in 2θ , step 0.0108664∘ .
Count level and noise
Curator-selected signal maximum of 5,000, 15,000, or 40,000 counts, before background; independent Poisson noise on signal plus background.
Table 18: Frozen synthetic-generator settings. U[a,b] and L[a,b] denote uniform and log-uniform draws. Probabilities apply per phase for strain and texture, and per scan for roughness and background. Instrument coefficients use GSAS-II’s native parameter convention. Control recipes can disable effects or specify a background family.
Figure 13: Deposited and agent solutions for the Cu measurement (006). The deposited PG calculated profile is compared with the median-residual attempt in each agent condition (r2 for both baseline and v6). The left panels show the full evaluation window and the right panels expand 22∘ – 40∘ . Observations are black points, calculations are colored lines, and differences are gray with a vertical offset. All rows share the observed maximum as a display normalization and identical vertical limits within each column. Residuals use the original counts over the full evaluation window. PG’s saved calculated curve is rescored; agent curves are independently replayed from their final projects. No additional fitting is performed for this figure.
Case
Run
PG
Sol high
Baseline Sol-max
Evolved v6
Framework access
Framework + v6
Evolved framework v5
Opus
001 (BaCO 3 )
1
8.00
8.73
7.22
7.15
5.85
6.87
6.67
7.30
2
8.73
7.22
7.23
6.08
6.65
5.83
3
8.82
7.15
7.58
5.92
6.79
5.96
002 (BaO 2 )
1
7.20
8.87
7.59
8.73
8.51
7.06
7.12
8.30
2
7.68
12.67
7.65
7.24
5.78
7.21
3
7.80
7.12
7.10
7.21
8.29
7.60
Appendix
Table 19: All initial experimental profile scores, Rwp (%). Bold marks the lowest residual among three attempts within each condition and case, before rounding. Baseline and the four adapted/context conditions use Sol-max. Single PG and Opus scores appear only on the first row of each case. Aggregates are over measurement-level means. Every reaction also includes WO3 . † Documented replacement for a user-cancelled attempt.
Version
Mean change
Required mean gain
Worst case change
v1
-11.81
10.13
-39.66
v2
-3.55
13.94
-32.62
v3
+4.16
11.95
-21.83
v4
-6.68
8.46
-38.34
v5
+2.96
10.06
-37.62
v6
-6.04
8.43
-35.93
Appendix
Table 20: Later framework-access evolution. Each row compares 18 candidate attempts with baseline attempts on the same six cases. Positive changes improve the synthetic score. No version satisfies either the required mean gain or the permitted worst-case regression.
Figure 14: Case-level gains and regressions for the later framework v5. Bars compare three v5 attempts with three framework-baseline attempts. The mean improvement of 2.96 score points combines a 38.50-point gain on a missing- Mn3WO6 case with a 37.62-point loss on the harder Zn problem. The dashed line marks the permitted case-regression limit.
Figure 15: Experimental input panel. Each scan is independently max-normalized for display; solvers receive the original counts. Titles give the precursor pair and firing temperature. Background, peak widths, relative intensities, and phase overlap vary across the measurements. The arrangement does not define a train/test partition.
Condition
Background terms
Nondefault SH/L
Phases
Baseline
15 (6–20)
12/45
2.47
Evolved v6
8 (6–12)
17/45
2.91
Framework access
16 (10–100)
44/45
2.64
Framework + v6
8 (3–33)
45/45
2.80
Evolved framework v5
12 (8–30)
45/45
2.51
Appendix
Table 21: Diagnostics of the 45 submitted projects per condition. Background terms are polynomial or interpolation coefficients, excluding fixed-background curves and auxiliary peaks; entries give the median and range. Nondefault SH/L means a submitted value different from 0.002. Phase counts are means over all attempts.
Powder X-ray diffraction (XRD) is central to materials characterization, yet reliable end-to-end automation remains challenging. An XRD agent must interpret diffraction evidence, operate refinement software, manage coupled parameters in a defensible order, and distinguish numerical improvement from physical validity. In this paper, we propose AutoXRD, an autonomous large language model (LLM) agent framework that organizes powder-XRD analysis as stepwise refinement, grounds actions in observed evidence, and applies deterministic crystallographic and physical checks before accepting results. We further introduce XRDBench with two complementary tracks. XRDBench-QA contains 100 bounded diagnostic tasks that isolate scientific reasoning and decision-making, whereas XRDBench-E2E contains 34 executable workflows that test whether agents can compose these capabilities into complete analyses requiring file inspection, crystallographic-software execution, iterative refinement, evidence preservation, and reporting. We evaluate ten recent LLMs across 1,340 model--task runs. Models average only 57.8 out of 100, falling from 61.9 on XRDBench-QA to 53.7 on XRDBench-E2E. They perform best on refinement-history assessment and result acceptance, but remain substantially weaker on refinement-action selection, phase quantification, indexing, and Rietveld refinement. GPT-5.6 Sol achieves the highest overall score of 81.1, GPT-5.6 Terra the highest XRDBench-E2E point estimate of 81.0, and GPT-5.6 Luna the best score--cost trade-off. Ablations show that all six AutoXRD components consistently improve performance, supporting the framework design. Finally, execution-trace analysis reveals recurring failures in coupled-parameter control, quantitative reasoning, evidence preservation, and workflow termination, motivating stronger scientific constraints, uncertainty-aware decisions, and more efficient planning.
Yuetong Wu, Maojun Sun
The Chinese University of Hong Kong Hong Kong, China · The Hong Kong Polytechnic University Hong Kong, China
Determining crystal structures from powder X-ray diffraction (PXRD) would enable routine characterization of newly synthesized materials, including those absent from reference databases. Generative models offer a promising approach, but existing evaluations remain fragmented and largely assume single-phase samples with known composition. To fill this gap, we introduce PXRD-Bench, a unified benchmark that extends evaluation to unknown per-element atom counts and recovery of both structures in two-phase mixtures. It provides approximately 4.1 million training structures with simulated PXRD, tests generalization to unseen compositions and structural prototypes, and evaluates recovery from experimental PXRD after excluding structurally matching training entries. To provide a strong reference baseline, we also introduce Bragglite, a PXRD-to-structure model developed through a systematic study of PXRD modeling choices. It achieves a 44.2% Top-1 match rate on experimental RRUFF patterns with known composition, 24.6% above the strongest prior baseline. Further analysis yields several key findings: some models fail to benefit from experimental PXRD, while generalization to unseen structural prototypes and recovery from two-phase mixtures remain challenging. Together, our benchmark, systematic studies, and Bragglite clarify the capabilities and limitations of current PXRD-to-structure models.
Dongyeop Woo, Nofit Segal, Nayoung Kim +3
Korea Advanced Institute of Science & Technology (kaist.edu) · Massachusetts Institute of Technology (mit.edu)
Powder X-ray diffraction (PXRD) is the routine probe of crystalline matter, yet its analysis is the rate-limiting step as laboratories automate acquisition. Deep-learning analyzers excel on simulated patterns and degrade on measured ones. This simulation-to-real gap is structural, not additive: synthetic denoising gives no measurable lift on real spectra, whereas correcting a small peak-position drift more than doubles median retrieval correlation. Real-spectrum fine-tuning, peak-aligned reranking, and recalibration narrow what remains and restore the coverage synthetic anchors lose. Xtalyst integrates these in an agent-orchestrated system spanning phase identification, refinement, and calibrated property prediction. On a frozen held-out partition (n=534) each module measured on both splits reproduces its development finding -- including the synthetic-anchor under-coverage, whose magnitude differs between the two pools -- while held-out refinement converges and preserves symmetry without reaching profile-quality fits, and on a diffractometer its wet-dry recommend-rescan-reanalyze loop flips a blinded silicon standard to a gated PASS and changes which minor phase is resolved on a multi-metal alloy.
Shaoguang Wang, Weiyu Guo, Ben Fei +3
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · The Chinese University of Hong Kong, Hong Kong SAR, China. · Suzhou National Laboratory, Suzhou, China. +2