DFT-Agent Bench: Scaffold Effects and Execution Provenance in Autonomous DFT
Authors: Shijie Liu, Jin Yang, Weilin Wan, Jingtao Han, Jia LI, Tianqi Deng
Organizations: Shanghai Jiao Tong University · State Key Laboratory of Silicon and Advanced Semiconductor Materials and School of Materials Science and Engineering, Zhejiang University, Hangzhou, Zhejiang 310027, China · Institute of Advanced Semiconductors, ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University, Hangzhou, Zhejiang 311200, China · School of Physics, Zhejiang University, Hangzhou 310027, China · Fudan University · Project Numina & WarpDrive Intelligence
Large language models are increasingly deployed as autonomous agents for first-principles materials simulation. We report empirical findings from DFT-Agent Bench, a benchmark of 200+ Quantum ESPRESSO tasks spanning relaxation, electronic structure, equation-of-state, and elastic-tensor pipelines over multiple material tiers. Each task carries a QE-derived ground truth with public-reference cross-checks where available, and a provenance audit checks whether the required QE execution contract was met. Under one execution harness, we compare a near-bare ReAct baseline with a bundled v5.1 scaffold on two models. The complete scaffold improves bands+DOS by +0.08/+0.11 and d-electron vc-relax by +0.17 on both models, while high-scoring relaxation and equation-of-state cells leave little room for improvement. The elastic result is model-dependent (0.18 to 0.40 for deepseek-v4-pro and 0.62 to 0.59 for claude-opus-4-8). A fixed 100-task analysis with retrieval disabled supports the aggregate pattern: the fully frozen DeepSeek comparison gains +0.17, while the Opus comparison gains +0.10 against a historical pool baseline (p < 0.001). The provenance gate catches incomplete or unmarked executions that can receive high rubric scores; it is an execution-contract check, not a test for intentional fraud. These results characterize the evaluated DFT workflows. The anonymous review artifact contains the frozen diagnostic slice, prompts, scorer, and solver revisions: https://anonymous.4open.science/r/231e31re91287h/README.md.
Figures & tables
raw
gated
Tier & task
n (v1/v5.1)
v1
v5.1
v1
v5.1
M1 vc-relax
116/116
0.96
0.97
0.96
0.91
M1 bands+DOS
116/116
0.78
0.91
0.78
0.91
M1 EOS
104/104
0.97
0.98
0.95
0.97
M1 elastic
104/104
0.45
0.51
0.43
0.51
M2 vc-relax
56/ 0 56
0.94
0.95
0.93
0.94
Table 1: Historical/descriptive pooled summary of per-cell raw and provenance-gated scores, v1 vs. v5.1, pooled over both models : best score per retained historical model–task record. The n values are retained historical pool entries, not unique released task IDs: the six M1+M2 rows sum to 552 entries per arm because repeats and task versions are retained, whereas the released set has 245 unique IDs. Raw and gated nearly coincide except M1 vc-relax, where flagship contract misses are voided ( 0.97 vs 0.91 ; Finding 5). The pooled elastic row averages a bimodal split ( 0.18→0.40 deepseek vs 0.62→0.59 opus; Fig. 1 ); M3 rows are per-model in Fig. 2 . M2 multi-step cells ( n=10 ) omitted.
Figure 1: Model-matched v1 vs. v5.1 with 95% task-clustered bootstrap CIs (M1+M2 common tasks; no cross-model pooling). Three cells separate: bands+DOS on both models and elastic on deepseek-v4-pro only; vc-relax and EOS are flat, and the opus bare baseline already reaches 0.62 on elastic (Finding 3).
Figure 2: Observed package gains in selected lower-scoring cells, per model : Δ= gated(v5.1) − gated(v1) within each model over paired tasks. M1 bands+DOS gains are +0.14 / +0.13 and M3 vc-relax gains are +0.17 / +0.17 ; elastic separates on deepseek only ( +0.20 ; Finding 3). The one negative cell, opus M1 vc-relax ( −0.08 ), is dominated by audit contract misses (Finding 5). Hatched: n<10 , too thin to trust.
Density functional theory (DFT) serves as the basis for computational discovery in materials science and chemistry, yet each calculation demands extensive human effort: adjusting algorithms when convergence stalls, revising plans when unexpected physics emerges, and inserting steps as intermediate results reshape the problem. Existing LLM-based agents automate only the initial planning stage, producing a full execution plan upfront and leaving all subsequent adaptation to hand-crafted rules. As a result, these workflows remain fragile, do not generalize well beyond pre-planned scenarios, and often require expert intervention when failures or unexpected intermediate results require changes to the calculation path. Here, we introduce AutoDFT, a closed-loop multi-agent framework that embeds LLM reasoning into every stage of the DFT lifecycle, where a strategic planner produces a skeletal plan of step objectives; a step planner generates numerical parameters just in time from preceding results; and a monitor-recover-reflect cycle diagnoses failures, repairs them, and revises the plan when the evidence justifies it. We demonstrate both breadth and depth: breadth on VASPBench, a purpose-built benchmark spanning 34 tasks and 9 DFT calculation types, where AutoDFT achieves 94.1% task-level success with GPT-5.2; and depth on established materials databases, where AutoDFT produces quantitatively reliable property predictions across electronic, magnetic, and energetic properties. By closing the loop between planning and execution, AutoDFT enables experimentalists without deep computational expertise to obtain reliable first-principles results.
Penghui Yang, Zhonghan Zhang, Yue Li +6
Nanyang Technological University, Singapore · Singapore Management University
Density Functional Theory (DFT) is a cornerstone of materials science, yet executing DFT in practice requires coordinating a complex, multi-step workflow. Existing tools and LLM-based solutions automate parts of the steps, but lack support for full workflow automation, diverse task adaptation, and accuracy-cost trade-off optimization in DFT configuration. To this end, we present TritonDFT, a multi-agent framework that enables efficient and accurate DFT execution through an expert-curated, extensible workflow design, Pareto-aware parameter inference, and multi-source knowledge augmentation. We further introduce DFTBench, a benchmark for evaluating the agent's multi-dimensional capabilities, spanning science expertise, trade0off optimization, HPC knowledge, and cost efficiency. TritonDFT provides an open user interface for real-world usage. Our website is at https://www.tritondft.com. Our source code and benchmark suite are available at https://github.com/Leo9660/TritonDFT.git.
Zhengding Hu, Kuntal Talit, Zhen Wang +8
University of California, San Diego · University of California, Merced · Los Alamos National Laboratory
Evaluating AI agents on scientific research tasks is constrained by the time and resources required for the underlying experiments or calculations. In computational materials research, repeating the same expensive simulations across agents and trials can make evaluation impractical. We introduce CompMat-Bench, a benchmark of 94 tasks derived from recently published computational materials studies, each asking agents to complete a step toward achieving the study's scientific goal. We reproduce the research steps in advance and assess agents on preparing inputs and analyzing outputs for expensive simulations, so expensive simulations can be avoided during evaluation. The reproduced inputs and results serve as ground truth for grading agents with fixed rules, without an LLM judge. The benchmark supports four evaluation conditions: single tasks and workflows composed of related tasks, each with full or reduced methodological guidance. With full guidance on single tasks, agents based on three LLMs demonstrate the ability to complete individual materials research steps, with pass rates of 66.0-90.4% across 94 tasks. Both longer workflows and reduced guidance can limit agent performance, but in different ways for different agents: they lower the pass rates of the weaker agents, whereas the strongest agent falls only when a long workflow is combined with reduced guidance. Failure analysis attributes most failures to scientific errors rather than to errors in software usage. CompMat-Bench provides a basis for comparing agents on the steps of real materials research and for analyzing agent failure modes.
Chenmu Zhang, Levi Felix, Jun-Jie Zhang +6
Department of Materials Science and NanoEngineering Rice University, Houston, TX 77005, USA