MineralExp: Benchmarking LLM Agents for Experimental Design in Real-World Mineral Exploration
Authors: Bing Wang, Mingyang Wang, Yuanhang Mi, Yue Xu, Kaibo Su, Li'ang Xiao, Chunguo Wu, Changchun Li, Ximing Li, Kun-Feng Qiu, Bo Yang
Organizations: College of Computer Science and Technology, and Key Laboratory of Symbolic Computation and Knowledge Engineering of the MoE, Jilin University, Changchun, China · State Key Laboratory of Geological Processes and Mineral Resources, Frontiers Science Center for Deep-time Digital Earth, School of Earth Science and Resources, China University of Geosciences (Beijing), Beijing, China
In mineral exploration, verifying geological hypotheses relies on designing a variety of geochemical experiments. While large language model (LLM) agents increasingly automate downstream geoscientific data analysis, a crucial upstream capability remains largely unexplored: automated geochemical experimental design, i.e., deciding which experiments to conduct before committing to costly laboratory work. Evaluating this capability of LLM agents is difficult for two reasons. First, geochemical experiments exhibit high methodological diversity and subjectivity: to test the same hypothesis, different geologists may choose different minerals and instruments, so no single protocol can serve as the reference answer. Second, open-ended experimental plans resist representation as structured labels, which makes objective scoring difficult. To address these challenges, we introduce MineralExp, a benchmark that reformulates geochemical experimental design as a hypothesis-based experiment selection problem, i.e., a multi-label classification problem. We abstract concrete instrument names and procedural details into a shared taxonomy of 19 experimental types across five mineral-system processes: Source, Transport, Deposition, Modification, and Preservation. This taxonomy yields deterministic multi-label targets while accommodating diverse experimental protocols. Constructed from published ore-deposit literature and verified by domain experts, MineralExp comprises 277 cases and 1,572 reference experimental plans. Given a geological hypothesis and initial context, an LLM agent selects the relevant tasks and generates concrete experimental plans, which are scored by multi-label classification metrics and an LLM-as-a-Judge metric. The experimental results show that current agents generally achieve high recall but low precision, tending to propose more experiments than are necessary to test the hypothesis.
Figures & tables
Figure 1. A representative example from MineralExp . Given a geological hypothesis and its initial context, an LLM agent selects the relevant experiments and proposes the corresponding experimental plans.
Figure 2. Overall construction pipeline of MineralExp . From 353 candidate studies, we extract (O,E,F) records to separately construct decontaminated inputs Xi=(Hi,Ci,Ai) and mapped reference targets (Oi∗,Yi∗) , followed by expert review and validation to yield 277 cases and 1,572 reference plans.
Process
Task IDs
Tasks
Positive Cases
Source
S1–S4
4
160
Transport
T1–T5
5
170
Deposition
D1–D4
4
146
Modification
M1–M3
3
143
Preservation
P1–P3
3
46
Total
-
19
277
Table 1. Process-level composition of MineralExp .
ID
Experimental Types
Cases
S1
Magma sources and evolution
50
S2
Magmatic metallogenic potential
19
S3
Sources of metals and sulfur
128
S4
Metamorphic fluids and metal release
1
T1
Ore-fluid sources
95
T2
Fluid evolution and episodes
95
Table 2. The 19 experimental types in MineralExp and their corresponding number of positive cases.
Agent
LLM
Source
Transport
Deposition
Modification
Preservation
Overall
R.
F1
R.
F1
R.
F1
R.
F1
R.
F1
R.
F1
OpenCode
GPT-6.1 Sol
72.08
68.77
76.19
67.16
63.86
56.09
62.80
57.87
81.25
72.90
70.76
63.84
GPT-6 Luna
79.19
53.70
84.03
54.01
79.21
46.51
76.83
46.24
79.17
46.91
80.58
50.53
Claude Sonnet 5.5
78.68
68.28
82.91
65.92
71.29
50.70
71.95
51.75
85.42
68.33
77.89
60.42
Claude Haiku 5.5
84.26
54.16
82.91
59.86
67.82
50.37
76.83
46.75
83.33
51.61
79.03
53.87
DeepSeek-V4.1-Flash
86.29
48.57
88.80
50.28
82.18
41.60
88.41
45.74
79.17
41.76
86.36
46.77
Table 3. Experimental results of 4 open-source agent harnesses and 7 foundation LLMs on our benchmark MineralExp .
Figure 3. Detailed selection metrics of MineralExp on OpenCode.
Agent
DeepSeek-V4.1-Flash
GPT-6.1 Sol
#Strict
#Part
#Weig
#Strict
#Part
#Weig
OpenCode
32.44
87.53
59.99
24.05
83.59
53.82
OpenHands
36.83
93.51
65.17
27.48
85.81
56.65
Biomni
35.56
92.49
64.03
22.65
84.99
53.82
ChemToolAgent
32.63
91.28
61.96
24.49
83.78
54.13
Table 4. LLM-as-a-Judge scoring of the experimental plans.
Figure 4. Token consumption of MineralExp .
Figure 5. Statistics of used tools of MineralExp .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10Figure 11Figure 12
Figure 6. Detailed selection metrics of MineralExp on OpenHands, Biomni, and ChemToolAgent.
Component
Case A (ID: G19-0224): Incomplete Plan Coverage
Case B (ID: G19-0175): Full Plan Coverage
Input X(H,C,A)
∙ Hypothesis ( H ): Test whether sulfur-gas and cover indicators developed in arid regions remain informative for detecting concealed gold mineralization beneath semi-humid grassland cover at the Zhaishang deposit. ∙ Context ( C ): Sediment-hosted gold deposit under semi-humid cover; applicability of soil-gas indicators is unresolved. ∙ Conditions ( A ): Exploration lines 143 and 175, a cover profile, and known orebody positions for validation.
∙ Hypothesis ( H ): Determine the formation temperatures, oxygen/sulfur fugacities ( fO2,fS2 ), and pH conditions of arsenopyrite and chlorite across successive mineralization stages at the Jinshan gold deposit. ∙ Context ( C ): Prior work links mineralization to magmatism, deformation, and mixed fluids, but stage-wise physicochemical evolution is unresolved. ∙ Conditions ( A ): Existing ore, arsenopyrite, chlorite, and host-strata samples.
Ground Truth (Y∗,P∗)
∙ Reference Tasks ( Y∗ ): P2, P3 (2 tasks) ∙ Reference Operations ( P∗ , 7 ops): - O03-O05 : ICP-MS assay of Au, AES assay of Ag, and HG-AFS assay of As/Hg across two soil grain-size fractions . - O07-O08 : ICP-MS assay of operationally extractable Au and four operational Au phase pools (water, clay, organic, Fe/Mn-oxide). - O09 : Field soil-gas measurements ( H2S,SO2,CO2 ). - O10 : Total organic carbon (TOC) in two soil fractions.
∙ Reference Tasks ( Y∗ ): M2, T2, T3 (3 tasks) ∙ Reference Operations ( P∗ , 3 ops): - O02 : Transmitted/reflected light microscopy on ore sections to establish mineral assemblages and relationships across three paragenetic stages . - O04 : Quantitative EPMA compositional analysis of stage-resolved arsenopyrite generations (Apy-II1, II2, III). - O05 : Quantitative EPMA compositional analysis of stage-resolved chlorite generations (Chl-I, II, III).
∙ Predicted Tasks ( Y ): M2 ✓ , T2 ✓ , T3 ✓ , D1 × , D2 ×∙ Generated Plans ( P ): - Design ( dt ): First establish a paragenetic scheme from polished thin sections via reflected/transmitted light microscopy and SEM-BSE; then sample coexisting arsenopyrite and chlorite from each recognized stage for quantitative EPMA and fluid-inclusion microthermometry. - Rationale ( rt ) & Evidence ( et ): Prevent mixing records from different stages; obtain stage-specific compositions to constrain temperature, fO2/fS2 , and pH shifts.
Evaluation & Diagnosis
∙ Task Metrics: Recall =100.0% , Precision =33.3% , F1 =50.0%∙ Evidence Coverage ( Cov ): 21.4% ( 1.5/7 operations) ∙ Judge Diagnosis: Full credit ( 1.0 ) on O09 (soil gas) and partial credit ( 0.5 ) on O10 (organic matter without grain-size fractions). Scored 0 on O03-O05 and O07-O08 : generic “pathfinder elements” and sulfur leaching fail to specify grain-size fractionated Au/Ag/As/Hg assays and four-phase extractable-Au extraction.
∙ Task Metrics: Recall =100.0% , Precision =60.0% , F1 =75.0%∙ Evidence Coverage ( Cov ): 100.0% ( 3.0/3 operations) ∙ Judge Diagnosis: Full credit ( 1.0 ) on all three operations ( O02, O04, O05 ). The generated plans explicitly align scientific purpose, analytical method, and target material—linking stage-resolved optical petrography directly to generation-specific EPMA of arsenopyrite and chlorite.
Appendix
Table 5. Case study comparing two instances evaluated with OpenHands and DeepSeek-V4.1-Flash. Both cases achieve 100% task recall on Y , but exhibit a sharp contrast in plan-level evidence coverage ( Cov : 21.4% vs. 100.0% ). Correct task predictions are marked with ✓ , and over-selected false positives with × .
Real-world molecular design remains challenging for large language model (LLM)-based agents. It requires them to interpret design contexts, satisfy multiple constraints, identify infeasible specifications, and reason over multi-step tool outputs. Existing benchmarks do not capture this complexity, focusing instead on explicit and narrow constraints, only feasible problems, and single-path solutions. To address this gap, we propose MolDesignBench, a scenario-grounded benchmark that more closely reflects real-world molecular design for evaluating tool-augmented LLM agents. MolDesignBench comprises 2K generation and optimization instances that combine implicit requirements embedded in design narratives with explicit property and functional-group constraints, including infeasible cases, and require the effective use of 17 specialized chemistry tools. Experiments across diverse frontier LLMs reveal low success rates--with the best achieving only ∼43%--and frequent failures in implicit-constraint reasoning, infeasibility detection, and tool reasoning. The corresponding fine-grained failure-mode analysis identifies implicit constraint interpretation and infeasibility detection as the primary bottlenecks, establishing MolDesignBench as a rigorous testbed to guide future research on chemical agents. The benchmark, tool interface, and evaluation code are publicly available.
Yongjun Jeong, Hanbum Ko, Ye Rin Kim +6
Department of Artificial Intelligence, Korea University · LG AI Research · Department of Statistics, Korea University
While general-purpose Large Language Models (LLMs) applied to Geology often hallucinate when reasoning about subsurface structures and deep-time evolution, current AI in Earth sciences predominantly targets surface remote sensing and GIS. To bridge this gap, we introduce Geo-Expert, a family of parameter-efficient geological LLMs fine-tuned on a custom-curated, high-quality instruction dataset processed using our custom instruction synthesis pipeline. We investigate the impact of model scaling and architecture by fine-tuning three base models: Qwen3-8B, Qwen3-32B, and Gemma-3-27B, with Low-Rank Adaptation (LoRA) method. Our extensive evaluation on a novel domain-specific benchmark, Geo-Eval, reveals that a domain-aligned 8B model can outperform open-weight 70B generalists and proprietary GPT-4o on specialized geological reasoning, while a 32B variant approaches frontier reasoning models. The optimized 8B model further offers a competitive cost-performance ratio for deployment. This work provides a reproducible recipe for democratizing scientific LLMs and establishes a baseline for geological artificial intelligence.
Chenyou Guo, Zongqi Liu, Yizhou Zhang +2
Ocean University of China · Peking University · Monash University
AI for Research (AI4Research) leverages AI to automate and improve scientific workflows. While experimental design is a critical stage of the research process, prior work has focused primarily on code implementation and execution, overlooking the importance of this stage, and no benchmark exists to evaluate AI's ability to conduct systematic experiment design. To bridge this gap, we propose SCOPE, a Scientific COmprehensive Planning Evaluation Benchmark constructed from 300 high-quality latest papers across 19 research domains from top-tier venues (e.g., ICML, NeurIPS, and ICLR),evaluating LLMs on two dimensions: High-Level planning completeness (main, ablation, and analysis experiments) and Low-Level configuration accuracy and rationality (datasets, baselines, and metrics). Benchmarking reveals three findings: (1) most LLMs cannot directly design high-quality experiments; (2) all LLMs exhibit a performance bottleneck in low-level configuration; and (3) search mode does not improve design quality. Furthermore, to address these challenges, we propose OptED, a novel agentic workflow to optimize LLM-based experimental design, that enhances LLM-based experimental planning through stage isolation, tool augmentation, and rule-based constraints, effectively alleviating the configuration bottleneck.
Zejun Liu, Jian Wu, Ru Peng +4
Westlake University · Zhejiang University · Nanjing University of Science and Technology +1