cond-mat.mtrl-sciAug 30, 2026

AutoXRD: Autonomous LLM Agents and Comprehensive Evaluation for Powder Diffraction Analysis

Authors: Yuetong Wu, Maojun Sun

Organizations: The Chinese University of Hong Kong Hong Kong, China · The Hong Kong Polytechnic University Hong Kong, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Powder X-ray diffraction (XRD) is central to materials characterization, yet reliable end-to-end automation remains challenging. An XRD agent must interpret diffraction evidence, operate refinement software, manage coupled parameters in a defensible order, and distinguish numerical improvement from physical validity. In this paper, we propose AutoXRD, an autonomous large language model (LLM) agent framework that organizes powder-XRD analysis as stepwise refinement, grounds actions in observed evidence, and applies deterministic crystallographic and physical checks before accepting results. We further introduce XRDBench with two complementary tracks. XRDBench-QA contains 100 bounded diagnostic tasks that isolate scientific reasoning and decision-making, whereas XRDBench-E2E contains 34 executable workflows that test whether agents can compose these capabilities into complete analyses requiring file inspection, crystallographic-software execution, iterative refinement, evidence preservation, and reporting. We evaluate ten recent LLMs across 1,340 model--task runs. Models average only 57.8 out of 100, falling from 61.9 on XRDBench-QA to 53.7 on XRDBench-E2E. They perform best on refinement-history assessment and result acceptance, but remain substantially weaker on refinement-action selection, phase quantification, indexing, and Rietveld refinement. GPT-5.6 Sol achieves the highest overall score of 81.1, GPT-5.6 Terra the highest XRDBench-E2E point estimate of 81.0, and GPT-5.6 Luna the best score--cost trade-off. Ablations show that all six AutoXRD components consistently improve performance, supporting the framework design. Finally, execution-trace analysis reveals recurring failures in coupled-parameter control, quantitative reasoning, evidence preservation, and workflow termination, motivating stronger scientific constraints, uncertainty-aware decisions, and more efficient planning.

Figures & tables

Figure 1. Overview of AutoXRD and the XRDBench evaluation protocol. AutoXRD combines an autonomous agent runtime, skill-guided powder-diffraction workflows, executable crystallographic backends, and evidence-based scientific validation. XRDBench evaluates diagnostic reasoning through XRDBench-QA and autonomous end-to-end XRD analysis through XRDBench-E2E.
 Supported XRD modulesAgent resourcesAgent evaluation scope
SystemPhase/Struct.RefineFit Diag.Valid.ReportSkillsToolsTaskQAE2EMtls.TasksRunsLLMs
LLM-CSP Agent ( Cao et al., 2025b )✓✗✗✓✗05PXRD structure pred.✗✓∼\sim 6.4K sim.∼\sim 6.4K∼\sim 6.4K1
AI X-ray Scientist ( Chen et al., 2026 )✗✗✗✓✓03SCXRD alignment✗✓2 real1223
Rongzai Agent ( Li et al., 2026 )✗✓✓✓✓–4NPD refinement✗✓5 real5204
AgentBuild-Rietveld ( Shin et al., 2026 )✗✓✓✓✓06PXRD refinement✗✓3 real8341
NeuDiff Agent ( Xiao et al., 2026 )✓✓✓✓✓04SCND structure det.✗✓1 real1102
Table 1. Comparison of language agents for X-ray and neutron diffraction. ✓ / ✗ indicate explicit support or evaluation/absence or no report. The five module columns cover phase or structure analysis, executable refinement, fit diagnosis, scientific validation, and structured reporting. Skills counts named procedures; Tools counts scientific callable interfaces, including skill-linked Python programs but excluding general file and shell tools. E2E requires agent-controlled execution of a material-specific task; Mtls. gives the number and source type of test materials, where sim. and real denote simulated and measured data; – denotes not reported or not applicable. Runs count model–task executions; LLM counts exclude judges and meta-optimizers.
Figure 2. Composition of XRDBench. The inner ring separates the 100 diagnostic tasks from the 34 executable end-to-end workflows; the middle ring groups related capabilities; and the outer ring shows nine XRDBench-QA capability categories and eleven XRDBench-E2E task families. A three-ring sunburst summarizes XRDBench by evaluation track, capability stratum, and task family, with separate difficulty counts.
Figure 3. Overall performance, ordered by the overall score. The wider, hatched bar denotes Overall, followed by XRDBench-QA and XRDBench-E2E. A grouped bar chart compares overall, QA, and end-to-end scores for ten models.
Figure 4. Fine-grained performance across 18 dimensions: nine XRDBench-QA capabilities and nine end-to-end reporting domains. Each model has a separate 0–100 radar; numeric radial labels are omitted. Ten radar charts compare nine bounded XRD capabilities and nine end-to-end reporting domains.
Figure 5. XRDBench-E2E bottlenecks across eleven task families, averaged over ten models. Rows are ordered by final score; gate and completion columns are pass rates, while the remaining columns are effective 0–100 components.
Figure 6. Component ablations on XRDBench-QA . Bars show the full-system score minus component removal.
Figure 7. Full-benchmark efficiency and cost-effectiveness. Resource means pool all 134 task runs per model; the vertical axis is the overall score. Four logo scatter plots compare overall score with duration, model turns, tokens, and logarithmic estimated inference cost.
More in this topic

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 318 papers

Explore similar work

CardsList
Oct 6, 2026

A self-learning scientific agent for X-ray diffraction

A central challenge for scientific agents is to turn analytical experience into reusable expertise grounded in physical evidence. Here we introduce Gan Jiang, a self-learning agent for powder X-ray diffraction built on a diffraction-analysis ecosystem we developed: XMatcher, XQueryer, XDecomposer and WPEM. Together, these engines span phase identification, multiphase decomposition and physics-constrained whole-pattern modelling. Gan Jiang converts analytical experience into executable skills by diagnosing failures, revising skill instructions and code, and validating revisions before reuse, without retraining the language model or changing the underlying physical models. Skills selected using development data and frozen before held-out evaluation achieve higher refinement scores than the original expert-designed skills across FullProf, GSAS-II and PyWPEM. The agent resolves strongly overlapping reflections, quantifies a five-phase ancient Egyptian cosmetic, tracks lattice evolution in an operating battery and compares atomic configurations in a disordered oxide catalyst. On DeltaXRDbench, it leads the evaluated methods in single- and multiphase identification across simulated and experimental data. Without supplied composition, single-phase top-1 accuracies reach 96.30%, 81.78% and 40.83% on MP500, RRUFF and opXRD, respectively, compared with 58.00%, 58.47% and 26.45% for the strongest comparator. These results demonstrate how an integrated scientific tool ecosystem can support agents that extract structural knowledge from measurements while accumulating validated analytical expertise that transfers to new samples.

Bin Cao, Huichi Zhou, Runyu Yang +7

The Hong Kong University of Science and Technology (Guangzhou), China · University College London, United Kingdom · AI Lab, The Yangtze River Delta, China +1

AI Agents for Scientific DiscoveryMaterials ScienceAgent Skill LearningSelf-Improving Agents
AskPDF
Aug 23, 2026

Diagnosing and narrowing the simulation-to-real gap in powder X-ray diffraction with a wet-dry agentic loop

Powder X-ray diffraction (PXRD) is the routine probe of crystalline matter, yet its analysis is the rate-limiting step as laboratories automate acquisition. Deep-learning analyzers excel on simulated patterns and degrade on measured ones. This simulation-to-real gap is structural, not additive: synthetic denoising gives no measurable lift on real spectra, whereas correcting a small peak-position drift more than doubles median retrieval correlation. Real-spectrum fine-tuning, peak-aligned reranking, and recalibration narrow what remains and restore the coverage synthetic anchors lose. Xtalyst integrates these in an agent-orchestrated system spanning phase identification, refinement, and calibrated property prediction. On a frozen held-out partition (n=534) each module measured on both splits reproduces its development finding -- including the synthetic-anchor under-coverage, whose magnitude differs between the two pools -- while held-out refinement converges and preserves symmetry without reaching profile-quality fits, and on a diffractometer its wet-dry recommend-rescan-reanalyze loop flips a blinded silicon standard to a gated PASS and changes which minor phase is resolved on a multi-metal alloy.

Shaoguang Wang, Weiyu Guo, Ben Fei +3

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · The Chinese University of Hong Kong, Hong Kong SAR, China. · Suzhou National Laboratory, Suzhou, China. +2

Synthetic-to-Real Domain AdaptationMaterials ScienceIterative Refinement
AskPDF
May 28, 2026

CrystalXRD-Bench: Benchmarking Vision-Language Models for XRD Peak Indexing Across Diverse Crystalline Materials

Miller-index identification from powder XRD patterns requires capabilities untested by existing multimodal benchmarks: the model must read a narrow peak location from a rendered scientific curve and then connect that observation to multi-step crystallographic reasoning. We introduce CrystalXRD-Bench, a 250-sample benchmark built from 10 public crystallographic databases for a single task: recover the full set of HKLs contributing to the highest-intensity peak in an XRD pattern. Each sample pairs the rendered XRD image with the source CIF text and chemical formula, so visual extraction errors and reasoning errors can be examined side by side. We evaluate seven vision-language models. The best Jaccard score is 0.5888 (GPT-5.4) with an exact-match rate of 37.6%, yet six of seven models remain below Jaccard 0.50; the task is far from solved. Error patterns vary systematically: double-peak cases are especially brittle, recall-heavy models gain coverage by over-predicting HKLs, and access to CIF text does not close the gap in crystallographic calculation. Alongside model rankings, the benchmark identifies the conditions under which current VLMs fail on quantitative scientific figures. All data and evaluation code will be publicly available.

Chengliang Xu, Xiaogang Li, Peiyao Xiao +3

Alibaba Group, Beijing, China

VLM EvaluationMaterials ScienceVLM Reasoning
AskPDF