DFT-Agent Bench: Scaffold Effects and Execution Provenance in Autonomous DFT
Authors: Shijie Liu, Jin Yang, Weilin Wan, Jingtao Han, Jia LI, Tianqi Deng
Organizations: Shanghai Jiao Tong University · State Key Laboratory of Silicon and Advanced Semiconductor Materials and School of Materials Science and Engineering, Zhejiang University, Hangzhou, Zhejiang 310027, China · Institute of Advanced Semiconductors, ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University, Hangzhou, Zhejiang 311200, China · School of Physics, Zhejiang University, Hangzhou 310027, China · Fudan University · Project Numina & WarpDrive Intelligence
Large language models are increasingly deployed as autonomous agents for first-principles materials simulation. We report empirical findings from DFT-Agent Bench, a benchmark of 200+ Quantum ESPRESSO tasks spanning relaxation, electronic structure, equation-of-state, and elastic-tensor pipelines over multiple material tiers. Each task carries a QE-derived ground truth with public-reference cross-checks where available, and a provenance audit checks whether the required QE execution contract was met. Under one execution harness, we compare a near-bare ReAct baseline with a bundled v5.1 scaffold on two models. The complete scaffold improves bands+DOS by +0.08/+0.11 and d-electron vc-relax by +0.17 on both models, while high-scoring relaxation and equation-of-state cells leave little room for improvement. The elastic result is model-dependent (0.18 to 0.40 for deepseek-v4-pro and 0.62 to 0.59 for claude-opus-4-8). A fixed 100-task analysis with retrieval disabled supports the aggregate pattern: the fully frozen DeepSeek comparison gains +0.17, while the Opus comparison gains +0.10 against a historical pool baseline (p < 0.001). The provenance gate catches incomplete or unmarked executions that can receive high rubric scores; it is an execution-contract check, not a test for intentional fraud. These results characterize the evaluated DFT workflows. The anonymous review artifact contains the frozen diagnostic slice, prompts, scorer, and solver revisions: https://anonymous.4open.science/r/231e31re91287h/README.md.
Figures & tables
raw
gated
Tier & task
n (v1/v5.1)
v1
v5.1
v1
v5.1
M1 vc-relax
116/116
0.96
0.97
0.96
0.91
M1 bands+DOS
116/116
0.78
0.91
0.78
0.91
M1 EOS
104/104
0.97
0.98
0.95
0.97
M1 elastic
104/104
0.45
0.51
0.43
0.51
M2 vc-relax
56/ 0 56
0.94
0.95
0.93
0.94
Table 1: Historical/descriptive pooled summary of per-cell raw and provenance-gated scores, v1 vs. v5.1, pooled over both models : best score per retained historical model–task record. The n values are retained historical pool entries, not unique released task IDs: the six M1+M2 rows sum to 552 entries per arm because repeats and task versions are retained, whereas the released set has 245 unique IDs. Raw and gated nearly coincide except M1 vc-relax, where flagship contract misses are voided ( 0.97 vs 0.91 ; Finding 5). The pooled elastic row averages a bimodal split ( 0.18→0.40 deepseek vs 0.62→0.59 opus; Fig. 1 ); M3 rows are per-model in Fig. 2 . M2 multi-step cells ( n=10 ) omitted.
Figure 1: Model-matched v1 vs. v5.1 with 95% task-clustered bootstrap CIs (M1+M2 common tasks; no cross-model pooling). Three cells separate: bands+DOS on both models and elastic on deepseek-v4-pro only; vc-relax and EOS are flat, and the opus bare baseline already reaches 0.62 on elastic (Finding 3).
Figure 2: Observed package gains in selected lower-scoring cells, per model : Δ= gated(v5.1) − gated(v1) within each model over paired tasks. M1 bands+DOS gains are +0.14 / +0.13 and M3 vc-relax gains are +0.17 / +0.17 ; elastic separates on deepseek only ( +0.20 ; Finding 3). The one negative cell, opus M1 vc-relax ( −0.08 ), is dominated by audit contract misses (Finding 5). Hatched: n<10 , too thin to trust.