cond-mat.mtrl-sciAug 30, 2026

AutoXRD: Autonomous LLM Agents and Comprehensive Evaluation for Powder Diffraction Analysis

Authors: Yuetong Wu, Maojun Sun

Organizations: The Chinese University of Hong Kong Hong Kong, China · The Hong Kong Polytechnic University Hong Kong, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Powder X-ray diffraction (XRD) is central to materials characterization, yet reliable end-to-end automation remains challenging. An XRD agent must interpret diffraction evidence, operate refinement software, manage coupled parameters in a defensible order, and distinguish numerical improvement from physical validity. In this paper, we propose AutoXRD, an autonomous large language model (LLM) agent framework that organizes powder-XRD analysis as stepwise refinement, grounds actions in observed evidence, and applies deterministic crystallographic and physical checks before accepting results. We further introduce XRDBench with two complementary tracks. XRDBench-QA contains 100 bounded diagnostic tasks that isolate scientific reasoning and decision-making, whereas XRDBench-E2E contains 34 executable workflows that test whether agents can compose these capabilities into complete analyses requiring file inspection, crystallographic-software execution, iterative refinement, evidence preservation, and reporting. We evaluate ten recent LLMs across 1,340 model--task runs. Models average only 57.8 out of 100, falling from 61.9 on XRDBench-QA to 53.7 on XRDBench-E2E. They perform best on refinement-history assessment and result acceptance, but remain substantially weaker on refinement-action selection, phase quantification, indexing, and Rietveld refinement. GPT-5.6 Sol achieves the highest overall score of 81.1, GPT-5.6 Terra the highest XRDBench-E2E point estimate of 81.0, and GPT-5.6 Luna the best score--cost trade-off. Ablations show that all six AutoXRD components consistently improve performance, supporting the framework design. Finally, execution-trace analysis reveals recurring failures in coupled-parameter control, quantitative reasoning, evidence preservation, and workflow termination, motivating stronger scientific constraints, uncertainty-aware decisions, and more efficient planning.

Figures & tables

Figure 1. Overview of AutoXRD and the XRDBench evaluation protocol. AutoXRD combines an autonomous agent runtime, skill-guided powder-diffraction workflows, executable crystallographic backends, and evidence-based scientific validation. XRDBench evaluates diagnostic reasoning through XRDBench-QA and autonomous end-to-end XRD analysis through XRDBench-E2E.
 Supported XRD modulesAgent resourcesAgent evaluation scope
SystemPhase/Struct.RefineFit Diag.Valid.ReportSkillsToolsTaskQAE2EMtls.TasksRunsLLMs
LLM-CSP Agent ( Cao et al., 2025b )✓✗✗✓✗05PXRD structure pred.✗✓∼\sim 6.4K sim.∼\sim 6.4K∼\sim 6.4K1
AI X-ray Scientist ( Chen et al., 2026 )✗✗✗✓✓03SCXRD alignment✗✓2 real1223
Rongzai Agent ( Li et al., 2026 )✗✓✓✓✓–4NPD refinement✗✓5 real5204
AgentBuild-Rietveld ( Shin et al., 2026 )✗✓✓✓✓06PXRD refinement✗✓3 real8341
NeuDiff Agent ( Xiao et al., 2026 )✓✓✓✓✓04SCND structure det.✗✓1 real1102
Table 1. Comparison of language agents for X-ray and neutron diffraction. ✓ / ✗ indicate explicit support or evaluation/absence or no report. The five module columns cover phase or structure analysis, executable refinement, fit diagnosis, scientific validation, and structured reporting. Skills counts named procedures; Tools counts scientific callable interfaces, including skill-linked Python programs but excluding general file and shell tools. E2E requires agent-controlled execution of a material-specific task; Mtls. gives the number and source type of test materials, where sim. and real denote simulated and measured data; – denotes not reported or not applicable. Runs count model–task executions; LLM counts exclude judges and meta-optimizers.
Figure 2. Composition of XRDBench. The inner ring separates the 100 diagnostic tasks from the 34 executable end-to-end workflows; the middle ring groups related capabilities; and the outer ring shows nine XRDBench-QA capability categories and eleven XRDBench-E2E task families. A three-ring sunburst summarizes XRDBench by evaluation track, capability stratum, and task family, with separate difficulty counts.
Figure 3. Overall performance, ordered by the overall score. The wider, hatched bar denotes Overall, followed by XRDBench-QA and XRDBench-E2E. A grouped bar chart compares overall, QA, and end-to-end scores for ten models.
Figure 4. Fine-grained performance across 18 dimensions: nine XRDBench-QA capabilities and nine end-to-end reporting domains. Each model has a separate 0–100 radar; numeric radial labels are omitted. Ten radar charts compare nine bounded XRD capabilities and nine end-to-end reporting domains.
Figure 5. XRDBench-E2E bottlenecks across eleven task families, averaged over ten models. Rows are ordered by final score; gate and completion columns are pass rates, while the remaining columns are effective 0–100 components.
Figure 6. Component ablations on XRDBench-QA . Bars show the full-system score minus component removal.
Figure 7. Full-benchmark efficiency and cost-effectiveness. Resource means pool all 134 task runs per model; the vertical axis is the overall score. Four logo scatter plots compare overall score with duration, model turns, tokens, and logarithmic estimated inference cost.
More in this topic

Autonomous Research Agents

Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.

See all 318 papers

Explore similar work

CardsList

  1. A self-learning scientific agent for X-ray diffraction

    Oct 6, 2026Bin Cao, Huichi Zhou, Runyu Yang +7AI Agents for Scientific DiscoveryMaterials Science

    The Hong Kong University of Science and Technology (Guangzhou), China · University College London, United Kingdom · AI Lab, The Yangtze River Delta, China +1

  2. Diagnosing and narrowing the simulation-to-real gap in powder X-ray diffraction with a wet-dry agentic loop

    Aug 23, 2026Shaoguang Wang, Weiyu Guo, Ben Fei +3Synthetic-to-Real Domain AdaptationMaterials Science

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · The Chinese University of Hong Kong, Hong Kong SAR, China. · Suzhou National Laboratory, Suzhou, China. +2

  3. CrystalXRD-Bench: Benchmarking Vision-Language Models for XRD Peak Indexing Across Diverse Crystalline Materials

    May 28, 2026Chengliang Xu, Xiaogang Li, Peiyao Xiao +3VLM EvaluationMaterials Science

    Alibaba Group, Beijing, China